[Thread Prev][Thread Next][Index]

[ferret_users] Wrong result in division



Dear all,

I am sorry to send another question but I am puzzled as to why this simple division is giving the wrong result. The same calculations in R give the correct result.

I have 2 variables calculated with the following

let myvar1 = var1[k=@sum] * 30 / .12
let nw = var2 * var3 * 30 / 1000
let myvar2 = nw[k=@sum]

and my result
let result = myvar1/myvar2

              MYVAR1  MYVAR2  RESULT
 ---- J:30
25E   / 103:  0.0054  0.0049   1.09
27E   / 104:  0.0356  0.0344   1.03
 ---- J:31
25E   / 103:  0.0490  0.0390   5.54
27E   / 104:  0.0458  0.0338   1.36
 ---- J:35
25E   / 103:  0.1618  0.1680  47.26
27E   / 104:  0.0028  0.1906   0.20

But clearly RESULT is wrong, why is this happening?

Thanks in advance

[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement