02 DSD-NL 2016 - Simona Gebruikersmiddag - Floating point onnauwkeurigheid en gevoeligheid van rekenmodellen - Werner Kramer, VORtech

1. VORtech
floating point arithmetic
The consistency, accuracy and performance dilemma
Werner Kramer
17-05-2016

2. 17-05-2016
VORtech 20 jaar (en trakteerde kennis)
http://www.meetup.com/VORtech-Scientific-Software-Engineering/

3. 17-05-2016
floating point arithmetic
• IEEE 754 Binary floating-point standard
• Pick two:
• Reproducibility
• Accuracy
• Performance
• Testing results with finite
accuracy

4. 17-05-2016
SIMONA-4315
Results depend on the location of the grid file
SIMONA-4328
Unexpected differences when cleaning roughcombination-
files
PS. The RMM-model is known to be slightly unstable.

5. 17-05-2016
SIMONA-4256
test-models give different results for release2014 and trunk
Change in revision r5784:
waquaref.tab
#
14 RESTART NCHAR = 7 OPT
#
IREP = 1 NAME = EXP_RESTART NCHAR = 3 JREP = 1 TYPE = 3 OPT
IREP = 1 NAME = SDS_RESTART NCHAR = 3 JREP = 1 TYPE = 3 MAND
+IREP = 1 NAME = TIME_EPS NCHAR = 8 JREP = 1 TYPE = 2 DEF = 1e-4
#
Triggered a difference in a matrix factorization (with duplicate eigen values),
which resulted in up to 10 cm difference in the water level.
Differences disappear when compiling with debug flags.

6. 17-05-2016 6
IEEE Floating point number
fraction between 1 and 2 (significand)
most significant bit (MSB) not stored
0.15625 * 2 = 0.31250 0
0.31250 * 2 = 0.62500 0
0.62500 * 2 = 1.25000 1
0.25000 * 2 = 0.50000 0
0.50000 * 2 = 1.00000 1
0. 00101
1. 01 2
biased exponent = exponent + 127
sign bit: 0 = positive, 1 = negative
0. 1562532 bit memory value for a decimal number

7. 17-05-2016 7
Some examples
decimal S exponent fraction
1.000000 0 01111111 00000000000000000000000
-1.000000 1 01111111 00000000000000000000000
0.5000000 0 01111110 00000000000000000000000
0.1000000 0 01111011 10011001100110011001101
0.7578125 0 01111110 10000100000000000000000
0.0000E+00 0 00000000 00000000000000000000000
0.0000E+00 1 00000000 00000000000000000000000
Infinity 0 11111111 00000000000000000000000
-Infinity 1 11111111 00000000000000000000000
(s)NaN * 11111111 (0)1**********************
• 1.0/0.0 =
• 1.0/0.0 =
• 0.0/0.0 = 0.0 × = NaN
• sqrt(-1.) = NaN

8. 17-05-2016
Distribution of values
• Spacing is dependent on the value of the exponent
• Without denormals there would be a gap around zero

9. 17-05-2016 9
Rounding errors
Addition
3.0 +
6.0 1.10 × 2
1.10 × 2 +
1.10 × 2
0.11 × 2 +
10.01 × 2
1.001 × 2 ?
a) round to nearest (roundTiesToEven ) 1.00 × 2 = 8
b) round down (towards ) 1.00 × 2 = 8
c) round up (towards ) 1.01 × 2 = 10
d) round towards zero 1.00 × 2 = 8

10. 17-05-2016
Factors that affect reproducibility
• Floating-point semantics
• Use of higher-precision intermediate results
fused multiply add instruction (fma) A*x + y
• Differences in math libraries (e.g. sin function)
-fimf-arch-precision=(high, medium, low)
• Data alignment changing vectorization
• Parallelism changing operation order
• Implementation differences between processors
-fimf-arch-consistency=true math library gives same results
across processors

11. 17-05-2016
Reassociation
• Addition and multiplication are mathematically associative,
but not computationally associative
• (a+b)+c = a+(b+c)
• (a*b)*c = a*(b*c)
• Divide using multiply by reciprocal x*y => x*(1/y)
• C and C++ disallow reassociation, specify left-to-right order
• Fortran allows reordering as long as parentheses are honored
(–assume protect_parens)
• Compiler may not obey these by default

12. 17-05-2016
Reassociation
integer ::i, n
real, dimension(n) :: A = 1.0
real :: C = -1.0, tiny = 1e-20
do i = 1, n
A(i) = A(i) + C + tiny
end do
original code optimized code
integer ::i, n
real, dimension(n) :: A = 1.0
real :: C = -1.0, tiny = 1e-20
C + tiny
do i = 1, n
A(i) = A(i) + C
end do
-fp-model keyword
• fast : value-unsafe optimizations (default)
• precise(source): value-safe optimizations only
• strict : precise + diable fma

13. 17-05-2016
Vectorization
• Vector operation works on multiple data at once (e.g. 16 byte
block = 4 reals)
• Vectorized math functions are very slightly less accurate but
faster than the scalar versions
• Unaligned data -> both scalar and vector versions are called
• Can change results run-to-run!
• OS stack alignment
• Address Space Layout Randomization

14. 17-05-2016
Vectorization
https://software.intel.com/en-us/articles/what-are-peel-and-remainder-loops-fortran-
vectorization-support
0x00
0x04
0x08
0x0c
0x10
0x14
0x18
0x1c
0x20
0x24
0x28
real(kind=4),
dimension(DIM_A) :: x
16 byte
16 byte
vectorized loop:
vector operation works on 16
byte block at once (SSE2)
do i=1,DIM_A
a(i) = sin(a(i))
end do
remainder:
scalar operation on remaining
array elements
peel loop:
loop iterations in scalar mode
until it reaches a 16 byte
boundary

15. 17-05-2016
SIMONA testing & improvements
Field max(dif) time 99%(dif) rms(dif) mean(dif)
solution_flow.sep.sep 0.001162 1395.00 0.000525 0.000170 0.000119
solution_flow.up.up 0.006302 975.00 0.001386 0.000284 0.000106
solution_flow.vp.vp 0.011412 1185.00 0.002075 0.000657 0.000127
test suite containing a large number of models
test suite has quantify random option -qr
inverts loop when solving matrix system
use -fp-model source to validate code
modifications
change sensitive parts to double precision
release with -fp-model source?

16. 17-05-2016
Technical/legacy
Software correctness is determined by comparison to previous
(baseline) results.
Debugging/porting
When developing and debugging, a higher degree of run-to-run stability
is required to find potential problems.
Legal
Accreditation or approval of software might require exact reproduction of
previously defined results.
Customer perception
Developers may understand the technical issues with reproducibility but
still require reproducible results since end users or customers will be
disconcerted by the inconsistencies.
Why Reproducibility