We examine the behavior of a multi-threaded genomics code on physical and Amazon EC2 hardware - and reveal changes that improve performance by 3x.

1. Optimizing Thread Performance for a
Genomics Variant Caller

2. This talk
• Introduce two tools that can help improve the performance of
multithreaded code
• Apply the tools to a real world Genomics code

3. caption
Tool 1: Allinea Performance Reports – benchmarking and
characterization

4. Tool 2: Allinea Forge - Debugging and Profiling
• Debug and profile from
one interface,
configuration
• Secure native remote and
local access
• Rapidly switch between
the tasks
• Edit, build, commit,
debug, profile, optimize..

5. Small data files
<5% slowdown
No instrumentation
No recompilation
Our profiler finds the performance bottlenecks

6. Our debugger helps bugs and performance
• Observe why
workload is
imbalanced
• Observe why
particular code paths
are followed
• .. And fix any bugs
that optimization
creates!

7. Above all…
• The tools are aimed at any performance problem that matters
– Focus on time: the ultimate judge of performance
• Do not prejudge the problem
– Don’t assume it’s MPI messages, threads or I/O before profiling!
• If there’s a problem..
– Allinea Performance Reports shows it, and advises you on solutions
– Allinea Forge’s profiler shows it, next to your code

8. 6 steps to improve performance
Get a realistic test
case
• Performance on real data
matters
• Keep the test case for
reference and re-use
Profile your code
• Add “-g” flag to your
compilation
• Run with a profiler
Look for the significant
• Which part/phase of the
code dominates time?
• Is there any unexpected
significant time use?
What is the nature of
the problem?
• Compute? I/O? MPI?
Thread synchronization?
• Display the metrics that
show the problem best
Apply brain to solve
• MPI – can you balance the
work better?
• Compute – is memory time
dominant – can you improve
layout?
Think of the future
• Try larger process or thread
counts to watch for
scalability problems
• Keep the profile (.map file)
for future comparison

9. Example: Improving Thread Usage in Genomics
• DISCOVAR
– Variant caller and small genome assembler
– Sub-mammalian sized genomes
– Newer DISCOVAR de novo for larger genomes
• C++ and OpenMP
• Developed by Broad Institute at MIT

10. A first look – on real hardware
• It’s not I/O intensive
• Good quantity of
OpenMP time
• No vectorization

11. OpenMP in detail
• Physical cores are
200% loaded:
hyperthreading is on
• 17% of parallel region
time is synchronization
• .. That’s quite high

12. Investigating the OpenMP synchronization
• Horizontal time axis:
colour coded
– Dark green – single core
– Light green – OpenMP work
– Light blue – pthread
synchronization
– Gray – idle
• Vertical axis
– #cores doing something
• Something’s very wrong
towards the end – with
all the gray

13. Zoom in on the region
• Stacks, code, regions,
time are all focused on
zoom area
• Key observation:
– OpenMP region with
“omp critical” is where
the time is being wasted

14. Fixing
• #pragma omp critical
– Execute exactly one
thread at a time to
ensure safety
• Is costing too much
– Passing “token” from
thread to thread to do
small pieces of work.
• Run whole section on
one thread instead
– Has same semantics

15. Impact of change
• Runtime down by 7%

16. As a performance report
• Improvements in
– Runtime
– Synchronization
overhead

17. Let’s try something bigger – into Amazon cloud!
• C4.8xlarge
– 36 hyperthreaded cores
– 60GB RAM
– Xeon E5-2666 v3 Haswell
– 25MB Cache
– 2.6GHZ
vs
• Our physical server
– 24 hyperthreaded cores
– 24 GB RAM
– Xeon E5-2407 v2
– 10MB Cache
– 2.4GHz
$ ./runme.sh
discovar version: Discovar r52488
loadaverage: 0.05 0.98 1.36 1/790 16317
2015-07-27 07:57 PERF: REAL 835.857 USER 36.188
SYSTEM 5.441 PERC 4.71
835 seconds to run on EC2
… vs …
~448 seconds on our physical server
Why?

18. Profile with Allinea Forge to find where the problem is
• Focus on initial 300
seconds: something
must be wrong here
• Serious lack of good
“green” compute

19. In detail…
• 36 threads, waiting… but who is using madvise?!

20. Why is glibc so bad?
• madvise system call in
_int_free()
– At least two context
switches each call ..
– This glibc version has
issues…?
• What other options are
there?

21. Maybe Google TCMalloc?
• Optimized for multi-
threaded applications
• No-win
– Same run time
– Issue is use of sys_futex
not madvise
• .. Not optimized for this
multithreaded
application!

22. Jemalloc?
• As recommended by
the Broad Institute
• … same runtime

23. Jemalloc – same problem
• Source proves the issue
again…

24. Can Intel libraries help?
• We try the Intel TBB
multithreaded allocator
• 14 minutes down to 10
minutes!
• .. But still this code has
scope for more…

25. Real optimization of OpenMP regions
• NB – still profiling for
first 300 seconds only
• Significant inactivity in
final 60 seconds
• OpenMP region
– #pragma omp parallel for
• Is it working?
– No – the threads are idle
• Let’s remove

26. After the first fix…
• Now able to run to
completion
– 358 seconds
• Still inactivity at end of
run

27. Zoomed to the inactivity…
• Another OpenMP region
• Quick edit: comment out
the OpenMP, again!

28. … and the impact
• Down to 304 seconds

29. Finally… something to sort out
• Recursive, in-place
multithreaded sorter
• Is not scaling well in
thread counts
• Options?
– Re-engineer
– Replace
– Tune

30. Let’s tune
• Try limiting the thread pool to 8 workers
– Better than 36 clashing threads?

31. Result…
• Runtime 4.7 minutes
• 3x improvement on
original
• #1 position on the
Broad Benchmark list
for a sub-$2 / hour
system!

32. Lessons learned
• Real codes exhibit many different performance patterns
– Profiling real data sets at real scales is vital to target the effort
– Small test cases do not expose all the problems
– Small thread counts can be too small to find real problems
• Changing code can be simple
– Use threads wisely – it will not always be faster
– Changing libraries – someone else might have fixed your problem
• Re-engineering is sometimes necessary
– Take advantage of vector units
– Take advantage of threads

33. Increase the performance of your software
Analyze and tune
with Allinea
Performance Reports
Develop, profile and
debug applications
with Allinea Forge
With professional
support when you
need it most
Read more!