irb(main):001:0> 0.5-0.4
=> 0.1
irb(main):002:0> 0.5-0.4<0.1
=> true
irb(main):003:0> ((0.5-0.4)*10)
=> 1.0
irb(main):004:0> ((0.5-0.4)*10).to_i
=> 0
irb(main):005:0> ((0.5-0.4)*20).to_i
=> 1
irb(main):006:0> ((0.5-0.4)*20)
=> 2.0
...well, this certainly violates the principle of least astonishment, but I guess the lesson is that I should be using values in terms of pennies (ie, integers not floats).
Saturday, March 1, 2008
Weird Floating Point Problem in Ruby
I'm working on a simple postage stamp calculator, just one of several projects I have for some Ruby on Rails practice. So I set up my rspec tests and find that one of my tests is failing. After stepping through the test, I found something that I find very surprising: 0.5 - 0.4 < 0.1:
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment