Re: [ferret_users] Appending output data to a netCDF file in i-dimension

["Fiedler, Russell (O&A, Hobart)" <Russell.Fiedler@xxxxxxxx>]

Hi,

I'd also note that striding through the file this way is extremely inefficient. Unless you've done some rather odd chunking of the input files the read/writes will not

not be contiguous. It would be far better to loop over the j index processing all longitudes at once. If this still requires too mush memory then you may require a double loop. With luck you may be able to use non unit stride in the I direction. Also note that you don't need to redifine your variables in the loop.

Something like

let v_ij=var_dta

let v_norm=v_ij[GT=tdaily@mod]

...

save/clob/i=1/j=1/ilimits=../jlimits=../file=.. v_norm...

!Save 277 values at once (1385=5*277)

repeat/j=1:596 (repeat/range=1:1385:277/name=s save/app/i=`s`:`s+276`/app/file=.. v_norm...)

If 277 points requires too much memory You can use smaller strides and replace `s+276` by `min(s+stride-1,1385)` in the above.

Cheers,

Russ

---

From: owner-ferret_users@xxxxxxxx <owner-ferret_users@xxxxxxxx> on behalf of Ansley C. Manke <ansley.b.manke@xxxxxxxx>
Sent: Thursday, 6 February 2020 10:43 AM
To: ferret_users@xxxxxxxx <ferret_users@xxxxxxxx>
Subject: Re: [ferret_users] Appending output data to a netCDF file in i-dimension

 

The general answer for appending to a file in directions other than time, is to first create the file with the full limits that you want to appear in the file, using /ILIMITS, /JLIMITS, /KLIMITS, in all except the time axis.  Then make sure the variable is defined on the same grid, so that when you're writing data at i=1, it's i=1 on the axis, with the same axis names so that it can be appended.

The documentattion for /ILIMITS, etc is here,

https://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/commands-reference/LIST#_VPINDEXENTRY_1409

and follow the links there to examples of appending in I, J, or Z.

-Ansley

On 2/5/2020 12:46 PM, Ge Peng - NOAA Affiliate wrote:

I need to compute daily climate normals (30-yrs averages) from a daily dataset  (variable: var_dta), with dimensions of I:1385; J:596; L:10957.

 

As the machine memory is not large enough for me to compute the normals for the entire field, I had to do so at each data point along the i-dimension (1:1385) and output the results to an intermediate nc file, using the code below. (Potentially, I may have to do so at each of xy data points. Printing out values at each of all xy data points to an ascii file takes too long!)

 

Unfortunately, I am getting an error message (see below) and would appreciate any help you may provide. 

If you have a better way for computing daily climate normals with a large dataset, I'd appreciate it if you could share your method with me.

 

Thanks,

 

--- Peng

 

------

! - Define a daily modulo time axis

def axis/t=0:365.2425/edges/npoints=365/t0=1-JAN-0001/units=days/modulo tdaily

 

! – Compute daily normals along i-dimension

     sp rm -f v_ij.nc

     repeat/range=1:`idm`/name=ii (;\

     let v_ij=var_dta[i=`ii`] ;\

     let v_norm=v_ij[GT=tdaily@mod] ;\

     let v_std=(v_ij[GT=tdaily@modvar])^0.5 ;\

     let v_min=v_ij[GT=tdaily@modmin] ;\

     let v_max=v_ij[GT=tdaily@modmax] ;\

     let v_npts=v_ij[GT=tdaily@modngd] ;\

     save/file=v_ij.nc/app v_norm,v_std,v_min,v_max,v_npts ;\

     can var v_* ;\

     )

 

**TMAP ERR: Requested data range is outside of data set limits

                        Expected: V_NORM[I=           2 :           2 ] Found: V_NORM[I=           1 :           1 ]

LIST/FORMAT=CDF/file=v_ij.nc/app v_norm,v_std,v_min,v_max,v_npts

 

The same error message if I use either one of the following two:

save/file=v_ij.nc/app/I=1 v_norm,v_std,v_min,v_max,v_npts ;\

save/file=v_ij.nc/app/I=`ii` v_norm,v_std,v_min,v_max,v_npts ;\

 

--

Ge Peng, PhD
Research Scholar
NOAA's Cooperative Institute for Satellite Earth System Studies (CISESS) at 

NOAA’s National Centers for Environmental Information (NCEI)

Center for Weather and Climate (CWC)

151 Patton Ave, Asheville, NC 28801 USA
+1 828 257 3009; ge.peng@xxxxxxxx

ORCID: http://orcid.org/0000-0002-1986-9115

Following CICS-NC on Facebook