Math Error caused by Floor Function?

EatingMyHatEatingMyHat Member Posts: 1,246
edited May 2012 in Tech Support

These 4 boxes have the same math expression. The boxes on the left use a Real attribute, the boxes on the right use Integer. The ones on the top do not use FLOOR, the ones in the bottom use FLOOR (again, all have the same math expression).

The expression is simple: 11111040*1.3/4.5/26 (don't ask why...). As you can see, there is no decimal part to the results (validated by the Real attribute and by Excel), however FLOOR seem to produce a different result.

Here is the project for you to look at:

Anyone encounter something like that?



  • MotherHooseMotherHoose Member Posts: 2,456
    well integerAttributes store numbers without decimalValues …
    when you ask for a floor() function … in this calculating instance … it rounds the value down
    though the ceil() function does not round it up

    go figure??? … guess if you want all the same values … eliminate the floor()

    just putting the equation itself in the Display Text …
    … works well without interveningAttribute
    or changing a text type attribute To: the expression
    …stores nicely … and can be used in mathematical operations
    and, when we get writable tables … you will probably store there

    okay … don't know where you are going with this one @EatingMyHat
    but, love how you show the first 6 numbers in numerical order!

    image MH
  • EatingMyHatEatingMyHat Member Posts: 1,246
    @MotherHoose The issue is that I need the Floor since when there is a decimal I want it to round it down. but given that for these numbers there is no decimal I needed it to have the same result as it should.

    I put the equation in the Display Text to simplify the test, in my code these are all attributes (REAL). But when I started to get errors, I broke it down and placed the numbers there. Not sure how Writeable tables will help with the math errors.

    I checked up to 45 figures after the decimal point... there is no hidden decimal there. Checked it with ROUNDDOWN and TRUNC functions and still, Excel insists that there is no decimal to round here.

  • EatingMyHatEatingMyHat Member Posts: 1,246
    Bug opened...
Sign In or Register to comment.