|
Hi Martin,
Some of the hard limits have been eliminated, and others raised
in recent versions of Ferret/PyFerret. There still are some fixed
limits, however, on the number of open files, grids, and axes that
can be defined. Here's a table with many of those limits:
https://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/Grids-Regions/FERRET-PROGRAM-LIMITS
You'll see that the upper limit for axes is 1500. I assume you
are doing a time aggregation. Each time axis in the member files
will use one of these axis definitions. Here is a case where you
might do better with a descriptor file, if that is a possibility
for the data you are working with. Descriptor files define just
one time axis for the multi-file set. Or, as you are doing,
working with subsets of the data, and finding other means to put
it all together.
Ansley
On 6/3/2019 10:49 AM, Martin Schmidt
wrote:
Hi Karl,
thank you very much. So it is not a simple internal python
issue or something like this. I also did not notice another
similar limitation. Shortly before the limit in teh string
number shows up, an internal limit is met:
**TMAP ERR: limit on number of axes has been reached
Meanwhile I am working "piecewise".
Best,
Martin
Am 03.06.2019 um 18:41 schrieb Karl
Smith - NOAA Affiliate:
Hi Martin,
I was starting to write you about how Ferret/PyFerret
reads the results of a command (the C function fopen) when I
noticed that there is a hard limit of 2000 strings per
string variable still being imposed in the code without
proper error handling. I am fairly sure this limitation
should no longer be imposed, but will need to be
investigated. And, obviously, if it does still have to be
imposed, the limit needs to be increased and given proper
error handling.
Best,
Karl
Dear all,
I am trying to glue a large amount of files using the
aggregate
capability of ferret. The problem is the file list itself.
The files are
ordered by a 4-digit number. So
ls -l wind_0*.nc
gives me 1000 file names,
ls -l wind_[0-2]*.nc
results in 3000 file names. All works well with up to 2000
file names.
In ferret:
yes? let file_list = SPAWN("ls -1 wind_0*nc")
yes? list file_list
gives the correct list and the aggregation works great as it
should.
yes? let file_list = SPAWN("ls -1 wind_[0-2]*nc")
yes? list file_list
results in huge traceback and ferret quits.
yes? list file_list
*** Error in
`/sw/data/python/miniconda/OS_42.3/2.7.15/bin/python':
free(): invalid pointer: 0x00007fcfbc746698 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x740ef)[0x7fcfbc41b0ef]
/lib64/libc.so.6(+0x79646)[0x7fcfbc420646]
/lib64/libc.so.6(+0x7a393)[0x7fcfbc421393]
/sw/viz/ferret/py_ferret_v750_os423_py_2715_netcdf463/lib/python2.7/site-packages/pyferret/libpyferret.so(FerMem_Free+0x1a)[0x7fcfb622ee83]
...
/sw/data/python/miniconda/OS_42.3/2.7.15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7d14)[0x7fcfbd1b2ea4]
... several 100 similar lines from many other libraries in
the miniconda
based python including a memory map and finally
Traceback (most recent call last):
File "<string>", line 1, in <module>
File
"/sw/viz/ferret/py_ferret_v750_os423_py_2715_netcdf463/lib/python2.7/site-packages/pyferret/__init__.py",
line 482, in init
result = run()
File
"/sw/viz/ferret/py_ferret_v750_os423_py_2715_netcdf463/lib/python2.7/site-packages/pyferret/__init__.py",
line 887, in run
retval = libpyferret._run(str_command)
RuntimeError: etc
I have no idea, which limit or ulimit, (but it is not the
stacksize)
could be violated. I have no idea how to produce this list
in python
directly, to check if this a python or ferret issue.
in _init_.py it crashs here:
readline.set_completer_delims(FERRET_READLINE_DELIMS)
# the actual Ferret function call
retval = libpyferret._run(str_command)
Is there eventually a limit in ferret itself?
Any hints would be very welcome.
Best,
Martin
--
Karl M. Smith, Ph.D.
JISAO Univ. Wash. and PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."
--
Ansley Manke
NOAA/PMEL Science Data Integration Group
7600 Sand Point Way NE
206-526-6246
|