Re: [ferret_users] precision issue ?

[Ryo Furue <furue@xxxxxxxxxx>]

Hi Alexa,

| yes? list UTEDDYC[k=1,j=500,i=205]*UTEDDYA[k=1,j=500,i=205]
|             
|          -1.458E+09
| yes? list VTEDDYC[k=1,j=500,i=205]*VTEDDYA[k=1,j=500,i=205]
|             
|           1.458E+09
| yes? list 
|  UTEDDYC[k=1,j=500,i=205]*UTEDDYA[k=1,j=500,i=205]
| +VTEDDYC[k=1,j=500,i=205]*VTEDDYA[k=1,j=500,i=205]
|             
|          -128.0
| 
| so listing the terms individually, they should add up to zero.
| Is this a consequence of ferret operating in single precision ?

I think so.  But, what is your goal?  Do you think that this
precision is not enough for your purpose?

Let's look at the *relative* error:

      128/1.458E9 ~= 8.8E-8 < 1E-7

A single precision variable has 7 significant digits or so.
So, your result is excellent considering how many siginificant
digits you have. 

| This makes me not trust my ferret calculations any more..

It shouldn't.  If your calculation requires more precision,
you need to use another programming language to process your
data before bringing it to Ferret.  But, if you are only
concerned that there might be something wrong with Ferret,
you don't need to worry.

Ryo
---------------------------
P.S.  It's interesting that the error was -128.
I guess it was because the difference between the
two big numbers was in the least signinicant
binary digit (bit).  That would give you
1.000...0 x 10^n = 2^n.  (The left-hand side
is in binary and the right hand-side is in decimal.)