Re: [ferret_users] reading a small portion of stream binary file

To: "Ansley C. Manke" <ansley.b.manke@xxxxxxxx>

Subject: Re: [ferret_users] reading a small portion of stream binary file

From: Ryo Furue <furue@xxxxxxxxxx>

Date: Tue, 29 Sep 2020 12:36:21 +0900

Arc-authentication-results: i=4; mx.google.com; dkim=pass header.i=@hawaii-edu.20150623.gappssmtp.com header.s=20150623 header.b=PobrYFC2; arc=pass (i=2 dkim=pass dkdomain=hawaii-edu.20150623.gappssmtp.com); spf=pass (google.com: domain of furue@xxxxxxxxxx designates 209.85.219.70 as permitted sender) smtp.mailfrom=furue@xxxxxxxxxx

Arc-authentication-results: i=3; mx.google.com; dkim=pass header.i=@hawaii-edu.20150623.gappssmtp.com header.s=20150623 header.b=PobrYFC2; arc=pass (i=2 dkim=pass dkdomain=hawaii-edu.20150623.gappssmtp.com); spf=pass (google.com: domain of furue@xxxxxxxxxx designates 209.85.219.70 as permitted sender) smtp.mailfrom=furue@xxxxxxxxxx

Arc-authentication-results: i=2; mx.google.com; dkim=pass header.i=@hawaii-edu.20150623.gappssmtp.com header.s=20150623 header.b=PobrYFC2; arc=pass (i=1 spf=pass spfdomain=hawaii.edu dkim=pass dkdomain=hawaii-edu.20150623.gappssmtp.com); spf=softfail (google.com: domain of transitioning furue@xxxxxxxxxx does not designate 65.153.203.176 as permitted sender) smtp.mailfrom=furue@xxxxxxxxxx

Arc-authentication-results: i=1; mx.google.com; dkim=pass header.i=@hawaii-edu.20150623.gappssmtp.com header.s=20150623 header.b=PobrYFC2; spf=pass (google.com: domain of furue@xxxxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=furue@xxxxxxxxxx

Arc-message-signature: i=4; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-archive:list-help:list-post:list-id:mailing-list:precedence :message-id:cc:to:subject:date:from:in-reply-to:references :mime-version:sender:dkim-signature; bh=bZdyA8iAxmjovD6udt2/UXgdIQeUGrIj+mE1BUnsKuM=; b=edbLvNI5zd/3ZIxIpPVZ6WWlIN+mKO6l/29FYX3V8OFaYAKNBU7G+GI7UgBFPMUITt MCYkyUHhq7ri/NPb1AMuxxLQ3WyYeYq/atlxafJfdue8DZSs3gZCtcxUqkaWZmxzagwy UlVtth06R4jBssngJvvLGoQULRv97j9137pqB97qz9RYkva6ei8EBYm7THFI5Oom3VWC 50zZBMSBdJv2z1grCb3S3krEJA5zuHTYpAavArpBfQNxv0TC2AwzntDvMdUEVQ4LoW8A UoufLzdzePxvljhSh6f5Sf8qjNFMuq+Ms5wR/XS1f8EiPWHQbkWO+nOvrT11Z828kFIr QZTg==

Arc-message-signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:cc:to:subject:date:from:in-reply-to:references :mime-version:dkim-signature; bh=bZdyA8iAxmjovD6udt2/UXgdIQeUGrIj+mE1BUnsKuM=; b=Ca0geKqBCGNFatNnw/0JcPzjamUW+Q+JWjgcmBtnUUvYJDq66wgWXRNh1BzEKZwKFW rDKPsKj+T+CRKtX+i/G/hckNNmVakAPa5TwL0Dwg6ruWgsZidURY6Bd8oZn5+slJd2Sb nS5NSK88wfZ7Btt/xINjLzvrGgKRhbMSOaTV91H4cAgjqa9JNFVY/ZfJ6mWOKr1VqMj1 8750YSh7s0kaWNoV8LPYKhWxGJtNn7+pOkPO3DxydDlmW2a5oCSqJ+nwV9aTw3YC/u27 8TbQmvlBZZzQJnTEI6yXSHgezRwCGK7bk/Tza3Ae5whDsa5bBdYFSDr/mBAgrH4HCA71 4H6g==

Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=bZdyA8iAxmjovD6udt2/UXgdIQeUGrIj+mE1BUnsKuM=; b=qqM+B7D+qR89pPthbChacI+gAhU1nNscyuUBxNJFbr5e8sqT4DyMEmWxkCEIL/uQyd Bv81ElUigcQGRKK3Pg1OUduyhHrke7rHaG7L2o4DHgBtSpY59dJzbYJqwm1dryv5YNcW l6Wa6XwiAVnfbTJMuf1iOuDNEj6I9p4UdWpxTIhq8Ea0csyBhWq8mqd/6cuH53XGk3ln UYqyho29aRlNT4d+5CTXZLMpaVv34+4mUi0PTLFnb95iEUXwJSIcr2uphycPot33Ng2H hztqioRiqVZLAsJxSud1pOU7XT8uLEPrUb0hPq10S77JnmkXEP0wj0Datt2o7GW0BITI B/Dg==

Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=bZdyA8iAxmjovD6udt2/UXgdIQeUGrIj+mE1BUnsKuM=; b=GJptOY+M8TVej5u4RxYqofwY34cTkz9qfSDt6N+ryf0BhbvJKLqK+FNQFEqcIMSWyh Xq2WrkP9TBxXjL21PxcM09W0vrC9ms8ZA4SWgHpSK0uH9XRt44jOhqqpiPawiR5yu0DE yPlf1Cfi/C8XiD8YzBHrM6vBjkQb5Ua9eO5B02FiPYGtKmWDTzdhMYgLdjdvOzG/grXF LyRaj89giUP2m1oazJnee+wyx8lkgH9QtYmITs/sQMfe1wNQfX4FMBzQcI9dwnQGZUB2 5yfyFptT81X7Ok/Gaxp11HGDQ7aDey/0iDdno30IP4Pfva5sKyjFjJ8ieYWl19+/Wd3V jS2g==

Arc-seal: i=4; a=rsa-sha256; t=1601350596; cv=pass; d=google.com; s=arc-20160816; b=qKu8OJKhOJQOAra7pR17WOngwlIvt0Sh314JlQu7VOKh1wMpKWlZjD7P10GSJWTDd6 EA0R4DjHxMJ6Lz82lQFh8AkIwRUfKqnMQmZY8qtUL7N0AknVX66Qn3IA5e5eOl9pEEPu YnaeW6txNRBF0iz+pczgjAkJixshql1LvDg5HlusfEmx55UgaKTfUR3H9ls6Qg64Lvs0 /QgU50iMF2q/nMt/TdcPuc/jNBAkykWXVR0KEuFqm33GHHVE7w6v5r0Dk1rVjcBLUaQU 8Um2GdmB8bVL4eTw29sDvWxjCWnDJDwJOY/1r6JlPj0uSnMizhC79Q4Al6V+RFlh2FBT 1iDQ==

Arc-seal: i=3; a=rsa-sha256; t=1601350595; cv=pass; d=google.com; s=arc-20160816; b=wzz8Ph43NUFtJOuHVO8pSYV9C6q46u0Qi3F+TOHOE4k6lYFazZyWCZ403AMIqOApvw dG/eja7oBGemrjt4GJVrF5wpMrjxX4xTNkB9PV7F7G4IUTl9LMksFhtX6ttQNNKhDwAE PC9ecIoaK8KxPqmGm5glKcFq23nDvXHL2BOMKNcObfkgKIlKLYJRhyKZkD4xxmD/DgNJ I9hviQenbpm35FZ8+p7P9kVCBGCMnvbsrsa1DFdphgXMA2Rl8xa+wR7o+737S8gx7EhW x5BjBqAR8vkGjuwn8/pGlpQBvlzMlUbWqX6jzCbPcbHwHBQuKwl5CTIeWcPH1Ea/hClM XYPg==

Arc-seal: i=2; a=rsa-sha256; t=1601350593; cv=pass; d=google.com; s=arc-20160816; b=Te19AzlslbdlxTpygG1g9JW7a2JhM8ZEtuRRnmBNVBg8sSFDn12tfoYzZdNpmwJiLp AMjYv5JUv0Y2az9G8+6+5XFPJDRQ7qfydyAA3uI3u6XYzSth659yWyxwnZl/IRYIthfo GPjGIHhqfCH1aNORXnW9dYF7DgnjQjr2I2/RdJCNKR5HMLjroZprVOKRpqTP2om7avFY wBwTMvrM+FV3qWGDItcUSBvfp2GIMmZBa2zu2Wm55VRWgnhmgYnOei2mI13BPv3kOdzQ UtwUqNMgwV/jsEPda5VyY6iWJRo5mqKmnAukI2lRFHuMdWyfhs6/j9ibhF3BzY4ghB5X udmg==

Arc-seal: i=1; a=rsa-sha256; t=1601350592; cv=none; d=google.com; s=arc-20160816; b=twRoHVPxOA+tLBaX7NzWSAMu8xW/ECrCCwuh48eRtrrfiDY+3Q+x8WBAM0xlkG/p9M xyMHXsFfAyoi0dLujOAihVCxi9yfGkM8VZ/9esQmhKdXGUyvsk2jL4pEBD9o1q/LRoTO ooK9qkqI0z0R0oM5WFKmts4Z7v7w5V7f+VAPXXHZiMQobfB2cDTmA9LOVyVeEsK6qQjj qg54dnz+/lLu1B4nPGlvL9Qu7jBp7abVQcUoyXEz6qeNXZxO7qs9qqoDIJ50ScCkNlya 5mqJ3SGofJ3fICkVKp0AWiox80iy8KZHt3mumNoKwpgxi7PzPtmF3U0pTw0/SqeIhZ/v PNmg==

Cc: ferret users <ferret_users@xxxxxxxx>

Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=noaa.gov; s=google; h=sender:mime-version:references:in-reply-to:from:date:subject:to:cc :message-id:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive; bh=bZdyA8iAxmjovD6udt2/UXgdIQeUGrIj+mE1BUnsKuM=; b=MxtT48DGYuBsfTPkOQ0+1KKRORRlh1Tu06gVODTo+umneKQsS3a/SSQTKlFrHVFOKn 5xdLzs58JyNk0mD9BDxe64+L9s01jT13t+EfwpVA+lmOriV7VKdXC1/MiryrSbxsZmHv iggI4gjK2VchqnOcpxoN8ImZ8P55CzJY5BpTlsPJmXDwLn07kXHuH4L1Frroy40oElAk gU1KAjgFYFyVIANNVOOxyKXkkFs9BGARzx/XoZ6ArTh2CxKi4aooJzDPpiEmLKQgeMnu 54woOVTeFl3lTCQgvb3PSYjMksIt9yNH02wvb81Ra0/a7cxGoRTPtos+jIKm+I2SZOxY lQdQ==

In-reply-to: <90b08c55-3be3-5d00-2081-b2cc04b85620@noaa.gov>

List-archive: <https://groups.google.com/a/noaa.gov/group/ferret_users/>

List-help: <https://support.google.com/a/noaa.gov/bin/topic.py?topic=25838>, <mailto:ferret_users+help@noaa.gov>

List-id: <ferret_users.noaa.gov>

List-post: <https://groups.google.com/a/noaa.gov/group/ferret_users/post>, <mailto:ferret_users@noaa.gov>

Mailing-list: list ferret_users@xxxxxxxx; contact ferret_users+owners@xxxxxxxx

References: <CAEwQpaKoS5wXZbvUZdgUTC2eh_2zP6qVgL1ofFKQRABBryv-nw@mail.gmail.com> <CAFXLzPh+=ux+KzmMOyU-LSRuNkhPG1L4UmF2dQL-d3acrrTzmA@mail.gmail.com> <CAEwQpaJQp0HGvfvyKz66iaoyRtmYpu8X_Fw+MnxyH9sdiy-9cA@mail.gmail.com> <90b08c55-3be3-5d00-2081-b2cc04b85620@noaa.gov>

Sender: owner-ferret_users@xxxxxxxx

Hi Ansley,

On Tue, Sep 29, 2020 at 1:18 AM Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:

I agree that using other tools is the best solution.

I'm not sure about that. I haven't found a better tool than Ferret.

Actually I think "ncgen" would be the RIGHT tool. That is, I'm thinking of pushing the netCDF developers to extend "ncgen" to allow for binary files. Currently, ncgen accepts only ASCII data:

data:
VAR = 0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, . . .

It would make perfect sense to extend the format to allow for an external binary file in such a way as

data:
VAR: file("path/to/mydata.bin", stream, big_endian)

just for an example.

Until that happens, Ferret is still the best tool, isn't it? For my problem at hand, I chopped the original binary file into smaller pieces and was able to combine them into a single netCDF file with just ~5 lines of additional Ferret code (plus a five-line shell program to split the binary file).

Ryo