Re: [ferret_users] memory problem with large file when regredding

["Ansley C. Manke" <ansley.b.manke@xxxxxxxx>]

Hi Patrick

The /JLIMITS qualifier refers to the indices of the grid you  are writing out.  It is used to will create the full-size output file to begin with, so it refers to the grid you're writing.  The output grid in your example is J=1:180.

So, something like this should work -- with the /JLIMITS and the [j= ] referring to the output grid not the source grid.  You'll need to find the range of mask[j=j1:j2] to use for your input data.

save/clobber/file="($03)"/JLIMITS=1:180 mask[j=1:60]
save/append/file="($03)" mask[j=61:120]
save/append/file="($03)" mask[j=121:180]

...

The first command computes mask for a subset of the result grid, but writes it to a file with the full range; then the rest of the commands fill in more data.

-Ansley

On 5/20/2019 7:13 AM, Patrick Brockmann wrote:

Re,

Thank you Satyesh for your answer.

Unfortunatelly, my variable does not have a time dimension.

So spread the work into parts on L dimension won't help.

My variable is defined as follows:

dimensions:
        lon = 36000 ;
        lat = 18000 ;

If I  use JLIMITS then I get this error message:

save/clobber/file="($03)"/JLIMITS=1:300 mask
 !-> LIST/FORMAT=CDF/clobber/file="tyty.nc"/JLIMITS=1:300 mask
 LISTing to file tyty.nc
 **ERROR: invalid command: Does not encompass data to be written: JLIMITS=1:300
          Variable: MASK

The complete test is:

wget https://www.ngdc.noaa.gov/mgg/shorelines/data/gshhg/latest/gshhg-shp-2.3.7.zip

unzip gshhg-shp-2.3.7.zip

gdal_rasterize -a area -te -180 -90 180 90 -tr 0.01 0.01 i/GSHHS_i_L1.shp mask_i_L1_0.01.nc

---> Rasterize the vectors shorelines into a 0.01° gridded land/water variable (4.9Go).

Now my purpose is to create coarser resolutions (0.05, 0.25, 0.5) of this variable with percent of land/water occupation.

It can be nicelly done with ferret regridding transformations such as: [G=MYGRID@AVE]

The script runs with a region but not for the entire globe.

use mask_i_L1_0.01.nc

let mask1 = if band gt 1 then 1 else 0

def axis/x=-180:180:0.50/edges/units=degrees_east myxaxis
def axis/y=-90:90:0.50/edges/units=degrees_north myyaxis
def grid/x=myxaxis/y=myyaxis mygrid
let mask = mask1[G=MYGRID@AVE]
save/clobber/file=LandWater_0.50.nc mask

Patrick

--
Data Analysis and Visualization Engineer / Project Manager
LSCE/IPSL, CEA-CNRS-UVSQ laboratory
LSCE - Climate and Environment Sciences Laboratory
IPSL - Institut Pierre Simon Laplace
--

---

De: "Satyesh Ghetiya" <satyeshghetiya@xxxxxxxxx>
À: "Patrick Brockmann" <patrick.brockmann@xxxxxxxxxxxx>
Cc: "ferret users" <ferret_users@xxxxxxxx>
Envoyé: Lundi 20 Mai 2019 15:33:42
Objet: Re: [ferret_users] memory problem with large file when regredding

Dividing data with many indeces (e.g. L or I or J or K) is good method to handle saving of large files.

I don't know how many L values are there, but suppose you have total values of L = Lmax (i.e. L=1:Lmax), then you may try following:

save/clobber/file=file_0.1.nc/llimit=Lmax/l=1 mask   ! put Lmax as a number say 365 or something .....

repeat/range=2:Lmax/name=aa (save/append/file=file_0.1.nc mask[=`aa`])

Hope it works. (LLIMITS can be used for I also, for e.g. ILIMIT)

On Mon, May 20, 2019 at 6:35 PM Patrick Brockmann <patrick.brockmann@xxxxxxxxxxxx> wrote:

Hi all,

I have a 5Go variable at 0.01x0.01 resolution and I would like to save it at a coarser resolution.

My regridding script cannot be run because it requires too much memory.

Is there a strategy to run it ?

I have tried to save it with different /append calls but cannot figure out

because anunlimited dimension seems to be possible only for time dimension (L).

Here are the simplified lines from my script for a regredding from 0.01 to 0.1°

!============================

use file_0.01.nc     ! mask1 is the variable to regrid

def axis/x=-180:180:0.1/edges/units=degrees_east myxaxis
def axis/y=-90:90:0.1/edges/units=degrees_north myyaxis 
def grid/x=myxaxis/y=myyaxis mygrid
let mask = mask1[G=MYGRID@AVE] 

save/clobber/file=file_0.1.nc mask

!============================

Any help would be appreciated.

Regards

Patrick

--
Data Analysis and Visualization Engineer / Project Manager
LSCE/IPSL, CEA-CNRS-UVSQ laboratory
LSCE - Climate and Environment Sciences Laboratory
IPSL - Institut Pierre Simon Laplace
--

--

G Satyesh

--
Ansley Manke
NOAA/PMEL Science Data Integration Group
7600 Sand Point Way NE
206-526-6246