[Thread Prev][Thread Next][Index]

Re: [ferret_users] maximum size of variable (again)



Hi Ansley,

Thank you for your help!

On Tue, Jun 29, 2021 at 3:42 AM Ansley Manke <a.c.manke@xxxxxxxxx> wrote:

I don't think this is a variable-size error on writing to the file; that limitation has long-since been handled with updates to the netCDF libraries.  Maybe this has something to with the grids of the variables as defined by pyferret as it initializes the file. When you open your files and check SHOW GRID var1 etc, does the grid have the dimension of length 36 in the m direction? 

Yes: The listing from "show grid var1[d=1], var2[d=2]" is

    GRID GEX1
 name       axis              # pts   start                end                 subset
 XU        LONGITUDE          500mr   105.1E               155E                full
 YV        LATITUDE           400 r   44.9S                5S                  full
 ZT        DEPTH (m)           67 i-  2.5                  767.22              full
 normal    T
 FREQ_AX1  E (cycles/day)      36 r   0.0027397            0.09863             full
 normal    F
    GRID GEX1
 name       axis              # pts   start                end                 subset
 XU        LONGITUDE          500mr   105.1E               155E                full
 YV        LATITUDE           400 r   44.9S                5S                  full
 ZT        DEPTH (m)           67 i-  2.5                  767.22              full
 normal    T
 FREQ_AX1  E (cycles/day)      36 r   0.0027397            0.09863             full
 normal    F
 
(These are Fourier coefficients and I use the ensemble dimension to represent frequency.)

If the grid does have the shape you expect, then I think that the repeat loop should not have the/mlimits inside it.  Assuming var1 and var2 have an m axis of length 36, I would do this:

  save/clobber/mlimits=36/m=1 var1[d=1], var2[d=1]
  repeat/m=2:36 (save/append var1[d=1], var2[d=2])

which SHOULD automatically write first the data at m=2, then m=3 etc.  Or try,

  save/clobber/mlimits=36/m=1 var1[d=1], var2[d=1]
  repeat/m=2:36 (save/append var1[d=1,m=`_m`], var2[d=2,m=`_m`])

I'm glad that these should work.

Actually I've found that we can forget about the loop.  The following statement gives the same error

let mend = `var2,return=mend` ! I've confirmed that mend == 36 .
save/clobber/file=tmp.nc/mlimits=1:`mend`/m=1 var1[d=1,m=1], var2[d=2,m=1]

The error message is
NetCDF: One or more variable sizes violate format constraints (OPeNDAP/netCDF Error code -62)

In contrast,

save/clobber/file=tmp2.nc/m=1 var1[d=1,m=1], var2[d=2,m=1]

does work, and so does

save/clobber/file=tmp.nc/mlimits=1:`mend`/m=1 var1[d=1,m=1,k=1:37], var2[d=2,m=1,k=1:37]

If you replace "k=1:37" with "k=1:38", you get the same error as the above.

Strangely, if you save only one variable

save/clobber/file=tmp.nc/mlimits=1:`mend`/m=1 var1[d=1,m=1]

it works as it should.

The most consistent summary of these observations may be that

When saving two variables, and when isize * jsize * ksize * msize > 2GiB for each variable,

save/clobber . . . /mlimits=1:msize/m=1 var1[m=1], var2[m=1]

fails.

(As I said in my last message, k=1:37 gives isize * jsize * ksize * msize = 1.98 GiB, whereas it's 2.04 GiB for k=1:38.)

Ryo


[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement