Re: [ferret_users] pyferret fails to read a very long directory

[Karl Smith - NOAA Affiliate <karl.smith@xxxxxxxx>]

Hi Martin,

I was starting to write you about how Ferret/PyFerret reads the results of a command (the C function fopen) when I noticed that there is a hard limit of 2000 strings per string variable still being imposed in the code without proper error handling.  I am fairly sure this limitation should no longer be imposed, but will need to be investigated.  And, obviously, if it does still have to be imposed, the limit needs to be increased and given proper error handling.

Best,

Karl

On Mon, Jun 3, 2019 at 4:06 AM Martin Schmidt <martin.schmidt@xxxxxxxxxxxxxxxxx> wrote:

Dear all,

I am trying to glue a large amount of files using the aggregate
capability of ferret. The problem is the file list itself. The files are
ordered by a 4-digit number. So

ls -l wind_0*.nc

gives me 1000 file names,

ls -l wind_[0-2]*.nc

results in 3000 file names. All works well with up to 2000 file names.
In ferret:

yes? let file_list = SPAWN("ls -1 wind_0*nc")

yes? list file_list

gives the correct list and the aggregation works great as it should.

yes? let file_list = SPAWN("ls -1 wind_[0-2]*nc")

yes? list file_list

results in huge traceback and ferret quits.

yes? list file_list
*** Error in `/sw/data/python/miniconda/OS_42.3/2.7.15/bin/python':
free(): invalid pointer: 0x00007fcfbc746698 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x740ef)[0x7fcfbc41b0ef]
/lib64/libc.so.6(+0x79646)[0x7fcfbc420646]
/lib64/libc.so.6(+0x7a393)[0x7fcfbc421393]
/sw/viz/ferret/py_ferret_v750_os423_py_2715_netcdf463/lib/python2.7/site-packages/pyferret/libpyferret.so(FerMem_Free+0x1a)[0x7fcfb622ee83]


...

/sw/data/python/miniconda/OS_42.3/2.7.15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7d14)[0x7fcfbd1b2ea4]


... several 100 similar lines from many other libraries in the miniconda
based python including a memory map  and finally

Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File
"/sw/viz/ferret/py_ferret_v750_os423_py_2715_netcdf463/lib/python2.7/site-packages/pyferret/__init__.py",
line 482, in init
     result = run()
   File
"/sw/viz/ferret/py_ferret_v750_os423_py_2715_netcdf463/lib/python2.7/site-packages/pyferret/__init__.py",
line 887, in run
     retval = libpyferret._run(str_command)
RuntimeError: etc

I have no idea, which limit or ulimit, (but it is not the stacksize)
could be violated.  I have no idea how to produce this list in python
directly, to check if this a python or ferret issue.

in _init_.py it crashs here:

         readline.set_completer_delims(FERRET_READLINE_DELIMS)
     # the actual Ferret function call
     retval = libpyferret._run(str_command)

Is there eventually a limit in ferret itself?

Any hints would be very welcome.

Best,

Martin

--

Karl M. Smith, Ph.D.
JISAO Univ. Wash. and PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."