Saturday, March 1, 2008

Weird Floating Point Problem in Ruby

I'm working on a simple postage stamp calculator, just one of several projects I have for some Ruby on Rails practice. So I set up my rspec tests and find that one of my tests is failing. After stepping through the test, I found something that I find very surprising: 0.5 - 0.4 < 0.1: irb(main):001:0> 0.5-0.4 => 0.1 irb(main):002:0> 0.5-0.4<0.1 => true irb(main):003:0> ((0.5-0.4)*10) => 1.0 irb(main):004:0> ((0.5-0.4)*10).to_i => 0 irb(main):005:0> ((0.5-0.4)*20).to_i => 1 irb(main):006:0> ((0.5-0.4)*20) => 2.0 ...well, this certainly violates the principle of least astonishment, but I guess the lesson is that I should be using values in terms of pennies (ie, integers not floats).

No comments:

Post a Comment