Re: [ferret_users] memory problem with large file when regredding

[Martin Schmidt <martin.schmidt@xxxxxxxxxxxxxxxxx>]

Hi Patrick,

if using time slices does not solve the probelm, you may also use a domain decomposition like in parallel ocean models. One method would be to write the result in separate files for each domain and use the methods from MOM or NEMO to glue them into a single file.

You may also stay in the "ferret space". The following script works for me. wind_2008 contains 1 year daily wind with .25 deg resolution in that region also covered by the coarse grid. So not really huge.

I am aware, this is not the ultimate check that this method saves memory. You need to load one time slice once completely, just to write the x/y axes not allowing for extension with "append". I also left ferret between two slices and started from the beginning, for sure without writing the axes again. So I would expect that ferret loads only data for the subregions.

The script needs streamlining to estimate the indices etc. But it shows the method.

Best,

Martin

use wind_2008.nc

define axis/x=-60:30:1/edges/unit=degrees_east xax
define axis/y=-60:20:1/edges/unit=degrees_north yax

can axis/modulo xax

define grid /x=xax/y=yax gc
let wc = wind_speed[g=gc@ave]

can region
save/l=1 /clobber/file=test.nc/outtype=float wc

set region/x=-60:-30 /y=-60:-40
save/append/file=test.nc/outtype=float wc[i=1:30]
set region/x=-30:0 /y=-60:-40
save/append/file=test.nc/outtype=float wc[i=31:60]
set region/x=0:30 /y=-60:-40
save/append/file=test.nc/outtype=float wc[i=61:90]

! you may leave ferret here and may restart
set region/x=-60:-30 /y=-40:-20
save/append/file=test.nc/outtype=float wc[i=1:30,j=21:40]
set region/x=-30:0 /y=-40:-20
save/append/file=test.nc/outtype=float wc[i=31:60,j=21:40]
set region/x=0:30 /y=-40:-20
save/append/file=test.nc/outtype=float wc[i=61:90,j=21:40]

! you may leave ferret here and may restart
set region/x=-60:-30 /y=-20:20
save/append/file=test.nc/outtype=float wc[i=1:30,j=41:80]
set region/x=-30:0 /y=-20:20
save/append/file=test.nc/outtype=float wc[i=31:60,j=41:80]
set region/x=0:30 /y=-20:20
save/append/file=test.nc/outtype=float wc[i=61:90,j=41:80]

Am 20.05.2019 um 15:03 schrieb Patrick Brockmann:

Hi all,

I have a 5Go variable at 0.01x0.01 resolution and I would like to save it at a coarser resolution.

My regridding script can be run because it requires too much memory.

Is there a strategy to run i ?

I have tried to save it with different /append calls but cannot figure out

because anunlimited dimension seems to be possible only for time dimension (L).

Here are the simplified lines from my script for a regredding from 0.01 to 0.1°

!============================

use file_0.01.nc     ! mask1 is the variable to regrid

def axis/x=-180:180:0.1/edges/units=degrees_east myxaxis
def axis/y=-90:90:0.1/edges/units=degrees_north myyaxis 
def grid/x=myxaxis/y=myyaxis mygrid
let mask = mask1[G=MYGRID@AVE] 

save/clobber/file=file_0.1.nc mask

!============================

Any help would be appreciated.

Regards

Patrick

--
Data Analysis and Visualization Engineer / Project Manager
LSCE/IPSL, CEA-CNRS-UVSQ laboratory
LSCE - Climate and Environment Sciences Laboratory
IPSL - Institut Pierre Simon Laplace
--