Re: [ferret_users] memory problem with large file when regredding

["Fiedler, Russell (O&A, Hobart)" <Russell.Fiedler@xxxxxxxx>]

Hi,

I normally do this by writing out just one point/row and then set up a loop to do the appending. You can either  do single rows or batches depending on how much memory you are willing to use. If using chunked data, double loops can be used to minimize I/O

e.g.

let jpts=`mask,return=jend`

let block=5

save/clobber/file="($03)"/jlimits=1:`jpts`/j=1 mask

repeat/range=2:`jpts`:`block`/name=s save/file="($03)"/app mask[j=`s`:`s+block-1`]

Russ

---

From: owner-ferret_users@xxxxxxxx <owner-ferret_users@xxxxxxxx> on behalf of Ansley C. Manke <ansley.b.manke@xxxxxxxx>
Sent: Tuesday, 21 May 2019 2:50:01 AM
To: ferret_users@xxxxxxxx
Subject: Re: [ferret_users] memory problem with large file when regredding

 

Hi Patrick

The /JLIMITS qualifier refers to the indices of the grid you  are writing out.  It is used to will create the full-size output file to begin with, so it refers to the grid you're writing.  The output grid in your example is J=1:180.

So, something like this should work -- with the /JLIMITS and the [j= ] referring to the output grid not the source grid.  You'll need to find the range of mask[j=j1:j2] to use for your input data.

save/clobber/file="($03)"/JLIMITS=1:180 mask[j=1:60]
save/append/file="($03)" mask[j=61:120]
save/append/file="($03)" mask[j=121:180]

...

The first command computes mask for a subset of the result grid, but writes it to a file with the full range; then the rest of the commands fill in more data.

-Ansley

On 5/20/2019 7:13 AM, Patrick Brockmann wrote:

Re,

Thank you Satyesh for your answer.

Unfortunatelly, my variable does not have a time dimension.

So spread the work into parts on L dimension won't help.

My variable is defined as follows:

dimensions:
        lon = 36000 ;
        lat = 18000 ;

If I  use JLIMITS then I get this error message:

save/clobber/file="($03)"/JLIMITS=1:300 mask
 !-> LIST/FORMAT=CDF/clobber/file="tyty.nc"/JLIMITS=1:300 mask
 LISTing to file tyty.nc
 **ERROR: invalid command: Does not encompass data to be written: JLIMITS=1:300
          Variable: MASK

The complete test is:

wget https://www.ngdc.noaa.gov/mgg/shorelines/data/gshhg/latest/gshhg-shp-2.3.7.zip

unzip gshhg-shp-2.3.7.zip

gdal_rasterize -a area -te -180 -90 180 90 -tr 0.01 0.01 i/GSHHS_i_L1.shp mask_i_L1_0.01.nc

---> Rasterize the vectors shorelines into a 0.01° gridded land/water variable (4.9Go).

Now my purpose is to create coarser resolutions (0.05, 0.25, 0.5) of this variable with percent of land/water occupation.

It can be nicelly done with ferret regridding transformations such as: [G=MYGRID@AVE]

The script runs with a region but not for the entire globe.

use mask_i_L1_0.01.nc

let mask1 = if band gt 1 then 1 else 0

def axis/x=-180:180:0.50/edges/units=degrees_east myxaxis
def axis/y=-90:90:0.50/edges/units=degrees_north myyaxis
def grid/x=myxaxis/y=myyaxis mygrid
let mask = mask1[G=MYGRID@AVE]
save/clobber/file=LandWater_0.50.nc mask

Patrick

--
Data Analysis and Visualization Engineer / Project Manager
LSCE/IPSL, CEA-CNRS-UVSQ laboratory
LSCE - Climate and Environment Sciences Laboratory
IPSL - Institut Pierre Simon Laplace
--

---

De: "Satyesh Ghetiya" <satyeshghetiya@xxxxxxxxx>
À: "Patrick Brockmann" <patrick.brockmann@xxxxxxxxxxxx>
Cc: "ferret users" <ferret_users@xxxxxxxx>
Envoyé: Lundi 20 Mai 2019 15:33:42
Objet: Re: [ferret_users] memory problem with large file when regredding

Dividing data with many indeces (e.g. L or I or J or K) is good method to handle saving of large files.

I don't know how many L values are there, but suppose you have total values of L = Lmax (i.e. L=1:Lmax), then you may try following:

save/clobber/file=file_0.1.nc/llimit=Lmax/l=1 mask   ! put Lmax as a number say 365 or something .....

repeat/range=2:Lmax/name=aa (save/append/file=file_0.1.nc mask[=`aa`])

Hope it works. (LLIMITS can be used for I also, for e.g. ILIMIT)

On Mon, May 20, 2019 at 6:35 PM Patrick Brockmann <patrick.brockmann@xxxxxxxxxxxx> wrote:

Hi all,

I have a 5Go variable at 0.01x0.01 resolution and I would like to save it at a coarser resolution.

My regridding script cannot be run because it requires too much memory.

Is there a strategy to run it ?

I have tried to save it with different /append calls but cannot figure out

because anunlimited dimension seems to be possible only for time dimension (L).

Here are the simplified lines from my script for a regredding from 0.01 to 0.1°

!============================

use file_0.01.nc     ! mask1 is the variable to regrid

def axis/x=-180:180:0.1/edges/units=degrees_east myxaxis
def axis/y=-90:90:0.1/edges/units=degrees_north myyaxis 
def grid/x=myxaxis/y=myyaxis mygrid
let mask = mask1[G=MYGRID@AVE] 

save/clobber/file=file_0.1.nc mask

!============================

Any help would be appreciated.

Regards

Patrick

--
Data Analysis and Visualization Engineer / Project Manager
LSCE/IPSL, CEA-CNRS-UVSQ laboratory
LSCE - Climate and Environment Sciences Laboratory
IPSL - Institut Pierre Simon Laplace
--

--

G Satyesh

--
Ansley Manke
NOAA/PMEL Science Data Integration Group
7600 Sand Point Way NE
206-526-6246