Showing posts with label ruby. Show all posts
Showing posts with label ruby. Show all posts

Monday, March 3, 2008

Improbable Truths in Ruby

The following statements are all true in Ruby 1.8.6: 0.6 - 0.5 < 0.1 (0.29 * 100).to_i == 28 0.28 * 100 != 28 (0.28 * 100).to_i == 28 (0.27 * 100) == 27 Well, what can I say? Floating point errors in Ruby can really bite you. I had always thought higher precision numbers were necessary only when dealing with very small numbers or when computing compound interest, but look out! Ruby floating point can really bite you with some strange, unexpected results. It makes me wonder why Matz included a to_i method at all!

Saturday, March 1, 2008

Weird Floating Point Problem in Ruby

I'm working on a simple postage stamp calculator, just one of several projects I have for some Ruby on Rails practice. So I set up my rspec tests and find that one of my tests is failing. After stepping through the test, I found something that I find very surprising: 0.5 - 0.4 < 0.1: irb(main):001:0> 0.5-0.4 => 0.1 irb(main):002:0> 0.5-0.4<0.1 => true irb(main):003:0> ((0.5-0.4)*10) => 1.0 irb(main):004:0> ((0.5-0.4)*10).to_i => 0 irb(main):005:0> ((0.5-0.4)*20).to_i => 1 irb(main):006:0> ((0.5-0.4)*20) => 2.0 ...well, this certainly violates the principle of least astonishment, but I guess the lesson is that I should be using values in terms of pennies (ie, integers not floats).