Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Math Error caused by Floor Function?

EatingMyHatEatingMyHat Posts: 1,246Member
edited May 2012 in Tech Support

These 4 boxes have the same math expression. The boxes on the left use a Real attribute, the boxes on the right use Integer. The ones on the top do not use FLOOR, the ones in the bottom use FLOOR (again, all have the same math expression).

The expression is simple: 11111040*1.3/4.5/26 (don't ask why...). As you can see, there is no decimal part to the results (validated by the Real attribute and by Excel), however FLOOR seem to produce a different result.

Here is the project for you to look at:

Anyone encounter something like that?



  • MotherHooseMotherHoose Posts: 2,456Member
    well integerAttributes store numbers without decimalValues …
    when you ask for a floor() function … in this calculating instance … it rounds the value down
    though the ceil() function does not round it up

    go figure??? … guess if you want all the same values … eliminate the floor()

    just putting the equation itself in the Display Text …
    … works well without interveningAttribute
    or changing a text type attribute To: the expression
    …stores nicely … and can be used in mathematical operations
    and, when we get writable tables … you will probably store there

    okay … don't know where you are going with this one @EatingMyHat
    but, love how you show the first 6 numbers in numerical order!

    image MH
  • EatingMyHatEatingMyHat Posts: 1,246Member
    @MotherHoose The issue is that I need the Floor since when there is a decimal I want it to round it down. but given that for these numbers there is no decimal I needed it to have the same result as it should.

    I put the equation in the Display Text to simplify the test, in my code these are all attributes (REAL). But when I started to get errors, I broke it down and placed the numbers there. Not sure how Writeable tables will help with the math errors.

    I checked up to 45 figures after the decimal point... there is no hidden decimal there. Checked it with ROUNDDOWN and TRUNC functions and still, Excel insists that there is no decimal to round here.

  • EatingMyHatEatingMyHat Posts: 1,246Member
    Bug opened...
This discussion has been closed.