Tags
Daily
Created
Nov 10, 2020 1:27 AM
Published
@November 9, 2020 ¤ Journal
TIL that 27 digits of significance are possible in Ruby...why not? Here's something that should just be 0.009
:
>> 19120.01.to_d / 2124445
=> 0.9000002353555869886017289e-2
Say you were estimating how many significant digits this way:
>> (19120.01.to_d / 2124445).to_s.split(".")[1].size
=> 27
This digit estimation works great for small, cleaner divisions. But it falls apart, since floating point math never truly escapes us. As usual, #round
to the rescue:
>> (19120.01.to_d / 2124445).round(5)
=> 0.9e-2
>> (19120.01.to_d / 2124445).round(5).to_s
=> "0.009"
Just a friendly reminder that you'll probably, always, want to #round
your BigDecimal
's when dividing. Thank me later.
¤ @qrush