Re: [ferret_users] appending ensemble slabs

["Ansley C. Manke" <ansley.b.manke@xxxxxxxx>]

Hi all,

This turns out to have been an issue when the data we're appending has an "abstract" axis in the appending directions. It's not particular to the E or F axes, but happens for any axis except for time. This seems to be a long-standing behavior that nobody had run into before.

It's best to define fixed axes for use with appending - I've added a note on this to the section of the Users Guide "Appending a slab to NetCDF files". 

However what Ryo tried to do should work, and there is a fix for this which will be in the next Ferret release - or which can be picked up now if you are getting code from GitHub and building Ferret yourself.

Ansley

On 4/30/2018 10:20 AM, Ansley C. Manke wrote:

Hi Ryo, Russ -

Thank you for the discussion  and the report.  It looks like Ferret has a bug in SAVE/APPEND in the E direction, but only when the axis in the E direction is an abstract axis.  The variable "evals"  is on an abstract axis, of undetermined length.

yes? let evals = esequence({1,2,3,4,5})
yes? show grid evals
    GRID EABSTRACT
 name       axis              # pts   start                end                 subset
 normal    X
 normal    Y
 normal    Z
 normal    T
 ABSTRACT  E             99999999 r   1                    1.E+08              full
 normal    F

These abstract axes are handy -- they define the grids of pseudo-variables, and can combine with other axes in expressions -- but they also cause us some headaches.  As you found Ryo, your script does work in the T direction, so there's definitely something wrong, and we'll find out what that is.  But there's also a good workaround.  Going back to your original script, just define the variable "evals" on a fixed axis.

yes? define axis/e=1:5:1 e5ax
yes? let evals = _e[ge=e5ax]
yes? show grid evals
    GRID (G002)
 name       axis              # pts   start                end                 subset
 normal    X
 normal    Y
 normal    Z
 normal    T
 E5AX      E                    5 r   1                    5                   full
 normal    F

With this change your script should work fine.

--Ansley

On 4/29/2018 8:31 PM, Ryo Furue wrote:

Thank you, Russ.

On Mon, Apr 30, 2018 at 9:52 AM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:

 Why don't you load the files in as an ensemble directly rather than individually? There's no real need to merge the files but the following should work.
[ . . . ]
let myfiles = spawn("ls -1 tmp?.nc")
ensemble tmp_all = myfiles

Thank you for the example.  Your code, of course, does work.  I didn't know this is the standard way to define an ensemble.

But, in fact, I'm looking for a way to load ascii data files. So,

    ensemble tmp_all = {"data01.nc", "data02.nc", . . . }

isn't the case for me.  Because I didn't know the ENSEMBLE command, I thought using a set of netCDF files would simplify my example.

So, how does the method of appending slabs in the Ferret manual work in the ensemble dimension?

As an workaround, I currently use the L dimension (time) as a fake ensemble dimension, because my original data doesn't include a time dimension.  It's easy to add slabs in the L dimension.  (A new workaround I've just learned is to convert all my ascii files to netCDF and then combine them as an ensemble.)

By the way, when I looked at the netCDF files generated with Russ's method, I found that the ensemble dimension isn't open. I thought it would be always better to open the last dimension, so that you can easily extend the data at least in that dimension.

Cheers,

Ryo