Building high-performance data processing and storage systems that can both scale up and scale out is hard. In this talk, we'll examine some lessons my team and I learned while building record-setting sorting systems at UCSD, and how re-examining your assumptions, understanding your problem, and designing for experimentation can make building high-performance systems easier.

1. UNORTHODOX PATHS
TO HIGH PERFORMANCE

2. @ALEXRAS

3. TRITONSORT (NSDI 2011)
THEMIS (SOCC 2012)
MapReduce Really Fast
Sort Really Fast

4. 2011
!!!!!
2010
!!
2014
!!!

5. THIS TALK:
HOW WE DID IT

6. LANGUAGE
FRAMEWORK
TOOLS
API

7. 1. SOFTWARE-HARDWARE CO-DESIGN
2. BUILDING FOR EXPERIMENTATION
3. CAREFULLY MANAGING MEMORY

8. MOTIVATION

9. [SORT] IS AN EXCELLENT TEST
OF THE INPUT-OUTPUT
ARCHITECTURE OF A COMPUTER
AND ITS OPERATING SYSTEM.
"A measure of transaction processing power"
Datamation 1985

10. THE SORT BENCHMARK
‣ SORTING K/V PAIRS (RECORDS)
‣ MANY CATEGORIES, VARIANTS
‣ TODAY: GRAYSORT (100TB)

11. 2009: YAHOO! SORTS 100TB
IN 173 MINUTES
ON 3452 HADOOP NODES

12. 2.79 MBPS PER NODE
9.6 GB PER SECOND
578 GB PER MINUTE
3452 NODES

13. THE GAP IN THEORETICAL PER
NODE PERFORMANCE AND
WHAT IS ACTUALLY ACHIEVED
HAS BECOME
GLARINGLY LARGE.
Anderson and Tucek
“Efficiency Matters!” SIGOPS OSR 2010

14. HOW CAN
WE DO
BETTER?

15. HARDWARE & SOFTWARE
CO-DESIGNED
FOR WORKLOAD

16. UNDERSTANDING THE PROBLEM

17. HOW TO MAXIMIZE
PER-NODE SPEED?

18. CPU
RAM
NETWORK
DISK ~15 disks/NIC
10Gbps

19. ‣8 CORES
‣16 DISKS
‣24 GB RAM
‣10 GBPS NIC

20. WHAT ARE THE
EXPENSIVE
OPERATIONS?

21. SEEKING
ROTATION

22. WRITING"
☠ SEEKING

23. KEEP OFF THE DISK
TWO READS +
TWO WRITES
PER RECORD

24. SEEK INFREQUENTLY
BIG READS
BIG WRITES

25. WHICH DISTRIBUTED
SORTING ALGORITHM?

26. MERGESORT
1 2 8 7 12 103 5 4 6 9 11
31 2 4 5 6 7 98 12 10 11
1 2 3 4 5 6 7 8 9 1210 11

27. THE TROUBLE WITH MERGESORT
AT SCALE, MANY SORTED CHUNKS
FETCHING RANDOMLY CAUSES SEEKS

28. DISTRIBUTION SORT
3 1 24 56
1 2 8 7 12 103 5 4 6 9 11
8 7 9 12 10 11
31 2 4 65 7 8 9 1210 11

29. DISTRIBUTION SORT
‣ BIG WRITES IN FIRST PASS
‣ SEQUENTIAL I/O IN SECOND PASS

30. NOW, TO BUILDING!

31. IT WON'T JUST
MAGICALLY
BE FAST

32. EXPERIMENTATION!

33. FLEXIBLE
MODULAR
GRAPHS

34. STAGE

35. READ HASH SEND
RECEIVE PARTITION WRITE
NETWORK
READ SORT WRITE

36. SYNTHETIC HASH SEND
RECEIVE PARTITION SINK
NETWORK
REALLY FAST DISKS?

37. READ HASH SINK
SYNTHETIC PARTITION WRITE
NETWORK
REALLY FAST NETWORK?

38. DIFFERENT SORTS?
READ RADIX SORT WRITE
READ QUICKSORT WRITE
READ TIMSORT WRITE

39. MEASUREMENT

40. HOW LARGE ARE WRITES?
WHERE IS THE BOTTLENECK?
ARE STAGES BLOCKED? IDLE?

41. TONS AND TONS
OF LOGS

43. AGGREGATES

44. TIME-SERIES

45. RUNTIME INFO

46. ORGANIZING LOGS
/2016/06/14/frob_widgets_1.tar.bz2

47. ORGANIZING LOGS
/2016/06/14/frob_widgets_1.tar.bz2
/cluster_nodes.txt
/node.conf

48. ORGANIZING LOGS
/2016/06/14/frob_widgets_1.tar.bz2
/cluster_nodes.txt
/node.conf
/notes.md

49. SMARTLY CONTROLLING MEMORY

51. BUFFER POOLS ARE
FAST AND SIMPLE
AND INFLEXIBLE

52. MALLOC IS
SIMPLE AND FLEXIBLE
AND DANGEROUS

53. By default, Linux follows an
optimistic memory allocation
strategy. This means that when
malloc() returns non-NULL there is
no guarantee that the memory really
is available.
In case it turns out that the
system is out of memory, one or
more processes will be killed by
the OOM killer.

55. ARE WE SOLVING
THE RIGHT PROBLEM?

56. WHAT IF MALLOC WAITED?
WAITING PROVIDES BACKPRESSURE
CALLERS CAN BE SCHEDULED
INTERFACE STAYS SIMPLE
DECISIONS CAN BE GLOBAL

57. POOLS
QUOTAS
CONSTRAINTS

58. WRAPPING UP

59. 300 MBPS PER NODE
15.6 GB PER SECOND
938 GB PER MINUTE
52 NODES

60. 632 MBPS PER NODE
112.6 GB PER SECOND
6757 GB PER MINUTE
178 NODES

61. LESSONS
LEARNED:

62. BOTTLENECKS
SHAPE YOUR
ARCHITECTURE

63. STRUCTURE SOFTWARE
FOR EXPERIMENTATION
AND MEASUREMENT

64. SAVE YOUR LOGS
SAVE YOUR CONFIG
SAVE YOUR NOTES

65. SOMETIMES YOU NEED
MORE CONTROL THAN
THE OS WILL GIVE YOU

66. GOING FAST
IS HARD

67. THANKS