Re: [ferret_users] ferret and ERDDAP

To: Ryo Furue <furue@xxxxxxxxxx>

Subject: Re: [ferret_users] ferret and ERDDAP

From: Roland Schweitzer - NOAA Affiliate <roland.schweitzer@xxxxxxxx>

Date: Mon, 2 Nov 2020 05:25:22 -0800

Arc-authentication-results: i=2; mx.google.com; dkim=pass header.i=@noaa.gov header.s=google header.b=fCY5Hc8r; spf=pass (google.com: domain of roland.schweitzer@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=roland.schweitzer@xxxxxxxx; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=noaa.gov

Arc-authentication-results: i=1; mx.google.com; dkim=pass header.i=@noaa.gov header.s=google header.b=fCY5Hc8r; spf=pass (google.com: domain of roland.schweitzer@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=roland.schweitzer@xxxxxxxx; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=noaa.gov

Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-archive:list-help:list-post:list-id:mailing-list:precedence:cc :to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=Th0zsMYBkAoqtFt9v1tna62pK2NjzSMnpluRiWVm++c=; b=sHLNdyGNPuCGgYO3TgC/hgaZ7Z9V3cKXgegfFaCwMw8vRn0cvfXNpGcn6+hTLdZwY5 C1Tmi3q5TsTRZRB22v3rq3d7Ow90C0WXIXwq9r48/nr1iLLSyhl6osEzkwqLdAF59KUM E+60MSdvSDkwIBmxYpPteAMT5N7kI1/JMkjzo9j9fykEXaUVwbpH0mofZs6LqZYO9IT7 YyEI1Vk2neS+bSCbWOp8FWkiwl4MgJ9uSNUjovD6qI6o0LqnSDihi6CoZWG3yu6K/8dT mr+36KrgwaGIrpkhVJK1NKVs7L7I1xdMD1yaRGCglkgSFwHXumVwwhKTamzdVcSiE7h7 WGrw==

Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=Th0zsMYBkAoqtFt9v1tna62pK2NjzSMnpluRiWVm++c=; b=zQDDI+8L3Gt+LBUeOFjpUhB4I3allZtBv4a7Uupcd7ZYKTBwZ4sHDK7I1YvVYDiw9p nsdY6THTy6mnma5V4l4z1IRriwqTEw6FBnhPc+NJDN6ISokyx8tNX+2H9/Uhy4wXKqvF xtkL0S/ugxjF7ZKbaj9WUwOQCyBkMMUG3hWz2gVgbGzsp5Bszs7+BVafXnnpJHhtKAZA 9CQ4wZqPtBi8rwcKLr2ZCyGrn8Gq1CDRVJ6hlzB6ZdnFO8/4YS/z/0ZM1xd0KP7VW06M OvT0CLI4wwEFZ76rOjiBOfJ5oQ3EkB06k4z8REmDdqEsnV4OWH2YkaSRam4b/XwsJMHL o/aA==

Arc-seal: i=2; a=rsa-sha256; t=1604323534; cv=pass; d=google.com; s=arc-20160816; b=gPa7fNipxkUyvy5ECqHpiOx2Eg6/e2u+803+xORpt5Zc5Gy6o1HfsQxfYyh0j/oMZc 5j3q82H7k7guvL7o7ZPNSPqLv5765ZeMbZ6mFr/XBcXFud+rTrxTfEvQv1e/d33tq05U +iAjpjyYiU2X8/N3qeKrg9QLHINhUxmqHGFtPSAtGixGYZvLNgcYBGwnvZONk8sk1ANS oVeCDF3qQsDREmBG8qwJr62g2bzgRDswf0nzE62SjRhMTGc/A1By9ralCx6BkES+LN65 IwPKDZ5FSCcPt3anSOX9yNd1gJF5T2leRvEVNo9WN+RNxJJUTr409XngDdXOVl3Q93cQ ksxA==

Arc-seal: i=1; a=rsa-sha256; t=1604323533; cv=none; d=google.com; s=arc-20160816; b=S6QRpMBzS4YIagAKc7QrTnZ0AgMPpaSEMK0bORCc+LibKqNsoSJJ1s22PTq1eRt2B2 8La0eFmCIunjg/xtrVRJ1391ZC9slEDKqfwFmUFigEG96aBZw4/LgObQEHIdI5zx3vtw mWgC/kqvFAEtRR8MJ+9KldOP0LrN09ywhy+2EDr84GBx1CWhJ1QyeqlQzppoLfISbQCR YBl+wTR3U8Llgd1Tt1RgPsqceYQanEjSbI4wtQdRTs04h9D2NAFSQB2arVU2iXJdRjIZ 3G0hGoMZ+vztLmJxsB/jQu+yBdAnl0bl9NVv7wfqLIvSCnqHwmh4MyiMJWXsCc8p98bv 5gRg==

Cc: Joshua Osborne - NOAA Affiliate <joshua.osborne@xxxxxxxx>, Patrick Brockmann <patrick.brockmann@xxxxxxxxxxxx>, ferret users <ferret_users@xxxxxxxx>

Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=noaa.gov; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive; bh=Th0zsMYBkAoqtFt9v1tna62pK2NjzSMnpluRiWVm++c=; b=PkdLx7W9m93cN0DmJ82G2JkAprfhsfXX5zX9P98C6xHs+C2BQyi3k1dtOQM6Acj4zj DDr4pgx0Izw/gAZh7UFts8qnqPAezlOE0Xuq1o+8qhq1onmasTDtiVhyIt6zhYmxZCbR QXpcY9AM8WTjhhozNtRAnMBsFAx6n40sAOYbj/8fPgbNOFanwnogQHE3kAIPRkfJRYHy jx4WLochY0b8CKmR61ptzYtkzQYq9CbQqebazl9HD9z6MQ2Wesacf+q5nLfYxcws1VBL yvdn94PoSwQTx6HUGVwiWBGXQtzXwrcc1YYw1mNOGfVSYbKHLl9M5RGHI1frJ5Wu64Dk jwMg==

In-reply-to: <CAEwQpaJ4gzovu9ycACFVzbtMou9LjmD+F0-aUaJmusgtpw5kgw@mail.gmail.com>

List-archive: <https://groups.google.com/a/noaa.gov/group/ferret_users/>

List-help: <https://support.google.com/a/noaa.gov/bin/topic.py?topic=25838>, <mailto:ferret_users+help@noaa.gov>

List-id: <ferret_users.noaa.gov>

List-post: <https://groups.google.com/a/noaa.gov/group/ferret_users/post>, <mailto:ferret_users@noaa.gov>

Mailing-list: list ferret_users@xxxxxxxx; contact ferret_users+owners@xxxxxxxx

References: <1364465171.1725595.1603905692525.JavaMail.zimbra@lsce.ipsl.fr> <CAHFMwDWu7Qw19qtH8hGEm_Xz6ux+f2CQbn8wMznXq2Ojc_ESRw@mail.gmail.com> <CAEwQpaJ4gzovu9ycACFVzbtMou9LjmD+F0-aUaJmusgtpw5kgw@mail.gmail.com>

Sender: owner-ferret_users@xxxxxxxx

HI,

I'll take a stab at elaborating. An ERDDAP server is an OPeNDAP server and takes the later part of its name from the Data Access Protocol (DAP) of OPeNDAP.

Broadly speaking, data in ERDDAP servers are organized into two subcategories which the ERDDAP documentation refers to as "TABLEDAP" and "GRIDDAP" data sets. GRIDDAP datasets are rectilinear grids in up to 4 dimensions (XYZT) which follow the CF conventions for grids, and TABLEDAP data sets are served as OPeNDAP sequences (more in a moment). However, both data types can also be retrieved from the server in a couple of dozen different data formats. Given an ERDDAP URL (https://coastwatch.pfeg.noaa.gov/erddap/) you can get a nice list of the GRIDDAP data sets using this URL (https://coastwatch.pfeg.noaa.gov/erddap/griddap/index.html) and the TABLEDAP data sets using this URL (https://coastwatch.pfeg.noaa.gov/erddap/tabledap/index.html)

So to use a grid data set in PyFerret, you operate just as you did in the example you shared (clicking on the "data" link on one of the data sets the "GRIDDAP" URL above) you land on an interactive page for that data set. The PyFerret accessible URL is the same minus the ".html".

[rhs@dunkel ~]$ pyferret
NOAA/PMEL TMAP
PyFerret v7.63 (optimized)
Linux 2.6.32-754.33.1.el6.x86_64 - 09/28/20
2-Nov-20 04:48

yes? use "https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplAvisoSshMon"
yes? show data
currently SET data sets:
1> https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplAvisoSshMon (default)
name title I J K L
SSHAG Sea Surface Height Above Geoid 1:360 1:180 ... 1:219
NOBS Number of Observations 1:360 1:180 ... 1:219
STDERR Standard Error 1:360 1:180 ... 1:219

yes?

Using PyFerret with the TABLEDATA data is a bit different. OPeNDAP sequences are indeterminant in length and it's difficult for PyFerret to negotiate with the server efficiently for the information in needs via the OPeNDAP sequences DAP protocol. So, when we use TABLEDAP data sets in our own software (like LAS, and the older version running against the data Patrick first mentioned) we generally use ERDDAP constraint expressions to determine the subset of the data we find interesting and then download a netCDF CF DSG data file to use with Ferret.

Suppose you wanted just the November data from this experiment, you could get is with a URL like this: https://ferret.pmel.noaa.gov/pmel/erddap/tabledap/ThirtyMinute_fd49_7acb_ec76.ncCF?S_41&latitude>=-90&latitude<=90&time>=1995-11-01T00:00:00.000Z&time<=1995-11-30T00:00:00.000Z&orderBy("id,time"). You can download it with wget (using the encoded version which you can construct and copy from the page I linked above) or from you browser which will do the encoding for you.

Then you can plot the result in PyFerret using the new DSG capabilities.

[rhs@dunkel ~]$ pyferret
NOAA/PMEL TMAP
PyFerret v7.63 (optimized)
Linux 2.6.32-754.33.1.el6.x86_64 - 09/28/20
2-Nov-20 05:17

yes? use ThirtyMinute_fd49_7acb_ec76.ncCF
yes? plot S_41
yes? go land_detail

I hope this helps.

Roland

On Sun, Nov 1, 2020 at 2:04 AM Ryo Furue <furue@xxxxxxxxxx> wrote:

Dear Patrick and Joshua,

On Sat, Oct 31, 2020 at 1:37 AM Joshua Osborne - NOAA Affiliate <joshua.osborne@xxxxxxxx> wrote:
[ . . . ]
You can, therefore, use ERDDAP as a data source from within PyFerret just as you would any other data source.

I wonder if you can elaborate on this? I'm curious and interested.

I'm familiar with OPeNDAP servers. If you visit a data server such as apdrc.soest.hawaii.edu , you find this kind of page:

http://apdrc.soest.hawaii.edu/dods/public_data/Argo_Products/annual_mean/Gridded_annual_mean.info

where you find a URL of the dataset you want to use, such as

http://apdrc.soest.hawaii.edu:80/dods/public_data/Argo_Products/annual_mean/Gridded_annual_mean

You open this URL on PyFerret, like so

yes? set data "http://apdrc.soest.hawaii.edu:80/dods/public_data/Argo_Products/annual_mean/Gridded_annual_mean"
yes? show data

and then you can use the dataset as if it were a local netCDF file.

What about ERDDAP ? The page (https://ferret.pmel.noaa.gov/pmel/erddap/index.html ) seems to be saying that OPeNDAP is somehow included or covered or something.

Regards,
Ryo

The policy of the DOC and NOAA requires me to inform you that the opinions in this email are mine and do not necessarily represent the opinion or policy of the Department of Commerce or the National Oceanic and Atmospheric Administration.