Understanding the performance of distributed dataflow systems like Apache Spark, Apache Flink, and Tensorflow is hard. Parallel computation is interleaved with data and control communication, and execution dependencies typically span multiple system components. In such environments, bottleneck detection is cumbersome and currently relies heavily on humans. After decades of systems research, state-of-the-art performance analysis techniques are commonly based on offline trace processing and thus are only suitable for batch computations and postmortem reports. Vasia Kalavri offers an overview of Strymon, a system for predictive data center analytics, and its online critical path analysis module. Strymon analyzes live traces from distributed dataflow systems to predict bottlenecks and provide insights on streaming application performance—leveraging logging and monitoring pipelines of modern production data centers to ingest cross-layer events in a streaming fashion and predict possible effects of such events in what-if sc

1. ONLINE
PERFORMANCE
ANALYSIS
OF DISTRIBUTED DATAFLOW SYSTEMS
Vasia Kalavri
kalavriv@inf.ethz.ch
Support:
O’Reilly Velocity London
20 October 2017

2. ABOUT ME
▸ Postdoc at ETH Zürich
▸ Systems Group: https://www.systems.ethz.ch/
▸ PMC member of Apache Flink
▸ Research interests
▸ Large-scale graph processing
▸ Streaming dataflow engines
▸ Current project
▸ Predictive datacenter analytics and
management
2
@vkalavri

3. 33
traces, configuration,
topology updates, …
Datacenter
Strymon
queries, complex analytics,
simulations, …
policy enforcement,
what-if scenarios, …
STRYMON: ONLINE DATACENTER ANALYTICS AND MANAGEMENT
Datacenter
model
event streams
strymon.systems.ethz.ch

4. STRYMON IS BUILT ON TIMELY
4
▸ A steaming framework for data-parallel computations
▸ Arbitrary cyclic dataflows
▸ Logical timestamps (epochs)
▸ Asynchronous execution
▸ Low latency
D. Murray, F. McSherry, M. Isard, R. Isaacs, P. Barham, M. Abadi.
Naiad: A Timely Dataflow System. In SOSP, 2013.
4
https://github.com/frankmcsherry/timely-dataflow

5. What-if scenarios
Real-time
datacenter analytics
Incremental
network routing
Online critical
path analysis
Explaining
simulations
Strymon
5

6. Apache Flink
SnailTrail: Strymon’s
component for online
performance analysis of
distributed dataflows
6
What-if scenarios
Real-time
datacenter analytics
Incremental
network routing
Online critical
path analysis
Explaining
simulations
Strymon

7. DATAFLOW PROGRAMMING
7
sources
sinks
operators
data exchange

8. UNDERSTANDING THE PERFORMANCE OF DISTRIBUTED DATAFLOWS
Hard to troubleshoot
▸ long-running, dynamic
workloads
▸ many tasks, activities,
operators, dependencies
▸ bottleneck causes are
usually not isolated but
span multiple processes
8
client
W1
W1
scheduler

9. PERFORMANCE METRICS
9

10. PERFORMANCE METRICS
9
Dataflow graph

11. PERFORMANCE METRICS
9
Duration
Dataflow graph

12. PERFORMANCE METRICS
9
Duration
Aggregate data exchange
Dataflow graph

13. PERFORMANCE METRICS
9
Duration
Aggregate data exchange
Custom
aggregate metrics
Dataflow graph

14. W1
W2
W3
serialization processing
waiting
deserialization
10
processing
message
A PARALLEL EXECUTION

15. W1
W2
W3
serialization processing
waiting
deserialization
10
processing
message
A PARALLEL EXECUTION

16. W1
W2
W3
serialization processing
waiting
deserialization
10
processing
messageProcessing is the most time-consuming activity
A PARALLEL EXECUTION

17. W1
W2
W3
serialization processing
waiting
deserialization
11
processing
What if we optimize it?
A PARALLEL EXECUTION

18. W1
W2
W3
serialization
waiting
deserialization
12
A PARALLEL EXECUTION

19. 13
W1
W2
W3
serialization
waiting
deserialization
A PARALLEL EXECUTION

20. 13
W1
W2
W3
serialization
waiting
deserialization
A PARALLEL EXECUTION

21. 13
W1
W2
W3
serialization
waiting
deserialization
No performance benefit
for the parallel execution!
A PARALLEL EXECUTION

22. CONVENTIONAL
PROFILING CAN
BE MISLEADING

23. CRITICAL PATH ANALYSIS

24. THE PROGRAM ACTIVITY GRAPH (PAG)
16
W1
W2
W3
a b
c d
t=k t=k+1
x u v z

25. THE PROGRAM ACTIVITY GRAPH (PAG)
16
W1
W2
W3
a b
c d
t=k t=k+1
x u v z
Nodes are timestamped events:
start or end of a worker activity

26. THE PROGRAM ACTIVITY GRAPH (PAG)
16
W1
W2
W3
a b
c d
t=k t=k+1
x u v z
Nodes are timestamped events:
start or end of a worker activity
u = {
timestamp: k+1,
worker: 2
}

27. THE PROGRAM ACTIVITY GRAPH (PAG)
17
W1
W2
W3
a b
c d
t=k t=k+1
x u v z
Edges represent activities
annotated with a type and duration

28. THE PROGRAM ACTIVITY GRAPH (PAG)
17
W1
W2
W3
a b
c d
t=k t=k+1
x u v z
Edges represent activities
annotated with a type and duration
(u, v) = {
type: serialization
duration: 1
}

29. W1
W2
W3
The Program Activity
Graph captures
computational
dependencies among
parallel workers
▸ Which activities delay the overall execution?
▸ i.e. which activities lie on the critical path of execution?
18

30. CRITICAL PATH
19
W1
W2
W3
a b
c d
The longest path in the execution history
(not considering waiting activities)

31. CRITICAL PATH
19
W1
W2
W3
a b
c d
The longest path in the execution history
(not considering waiting activities)

32. CRITICAL PATH
19
W1
W2
W3
a b
c d
The longest path in the execution history
(not considering waiting activities)

33. CRITICAL PATH
19
W1
W2
W3
a b
c d
The longest path in the execution history
(not considering waiting activities)

34. CRITICAL PATH
19
W1
W2
W3
a b
c d
The longest path in the execution history
(not considering waiting activities)

35. CRITICAL PATH
19
W1
W2
W3
a b
c d
The longest path in the execution history
(not considering waiting activities)

36. CRITICAL PATH
19
W1
W2
W3
a b
c d
The longest path in the execution history
(not considering waiting activities)

37. 20
W1
W2
W3
a b
c d
CRITICAL PATH

38. CRITICAL PATH
21
W1
W2
W3
a b
c d

39. CRITICAL PATH
21
W1
W2
W3
a b
c d
Reduced execution time

40. POST-MORTEM CRITICAL PATH ANALYSIS IS EASY
1. Collect traces during execution
job start job end
profiler
22

41. POST-MORTEM CRITICAL PATH ANALYSIS IS EASY
1. Collect traces during execution
job start job end
profiler
22
2. Analyze traces offline
analyzer
performance
summaries

42. How to compute the critical path for
continuously running,
distributed streaming applications,
with potentially unbounded input?
23

43. How to compute the critical path for
continuously running,
distributed streaming applications,
with potentially unbounded input?
▸ There might be no “job end”
▸ The PAG and critical path are continuously evolving
▸ Stale profiling information is not useful
23

44. ONLINE CRITICAL PATH ANALYSIS

45. ONLINE ANALYSIS OF TRACE SNAPSHOTS
25
input stream output stream

46. ONLINE ANALYSIS OF TRACE SNAPSHOTS
25
input stream output stream
periodic
snapshot

47. ONLINE ANALYSIS OF TRACE SNAPSHOTS
25
input stream output stream
periodic
snapshot
trace snapshot
stream
analyzer
performance
summaries
stream

48. PROGRAM ACTIVITY GRAPH SNAPSHOT
26
W1
W2
W3
a b
c d
t=k t=k+1
x u v z
ts te

49. W1
W2
W3
a b
c d
x u v z
ts te
27

50. ▸ All paths have the same length: te - ts
W1
W2
W3
a b
c d
x u v z
ts te
27

51. W1
W2
W3
a b
c d
x u v z
ts te
▸ All paths have the same length: te - ts
28

52. W1
W2
W3
a b
c d
x u v z
ts te
▸ All paths have the same length: te - ts
29

53. W1
W2
W3
a b
c d
x u v z
ts te
▸ All paths have the same length: te - ts
▸ Choosing a random path might miss critical activities
30

54. W1
W2
W3
a b
c d
x u v z
ts te
▸ All paths have the same length: te - ts
▸ Choosing a random path might miss critical activities
31

55. W1
W2
W3
a b
c d
x u v z
ts te
▸ All paths have the same length: te - ts
▸ Choosing a random path might miss critical activities
31

56. ▸ All paths have the same length: te - ts
▸ Choosing a random path might miss critical activities
▸ Enumerating all paths is impractical
W1
W2
W3
a b
c d
x u v z
ts te
32

57. W1
W2
W3
a b
c d
x u v z
ts te
How to rank activities with regard to criticality?
All paths are
potentially part
of the evolving
critical path
33

58. W1
W2
W3
a b
c d
x u v z
ts te
How to rank activities with regard to criticality?
All paths are
potentially part
of the evolving
critical path
Intuition: the more paths an activity appears on
the more probable it is that this activity is critical
33

59. 1
2
3
4
5
6
7
8
9
W1
W2
W3
a b
c d
x u v z
ts te
34

60. 1
2
3
4
5
6
7
8
9
W1
W2
W3
a b
c d
x u v z
ts te
9
0
0
6 6
35

61. 36
CRITICAL PARTICIPATION (CP METRIC)
An estimation of the activity’s participation in the critical path
total number of paths
in the snapshot
activity duration: edge weight
centrality: the number of
paths this activity appears on
tion 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN}
set of N transient paths of snapshot G[ts,te]. The tran-
path centrality of an edge e 2 G[ts,te] is defined as
c(e) =
NX
i=1
ci(e), where ci(e) =
8
>><
>>:
0 : e < ~pi
1 : e 2 ~pi
e following holds:
CPa =
TPC(a) · aw
N(te ts)
(3)
Sp
di↵er
ysis:
graph
whos
ers (t
graph
all wo
tions
1 We p
4

62. 36
CRITICAL PARTICIPATION (CP METRIC)
An estimation of the activity’s participation in the critical path
total number of paths
in the snapshot
activity duration: edge weight
centrality: the number of
paths this activity appears on
Can be computed
without path
enumeration!
tion 8. Transient Path Centrality: Let P = {~p1, ~p2, ...~pN}
set of N transient paths of snapshot G[ts,te]. The tran-
path centrality of an edge e 2 G[ts,te] is defined as
c(e) =
NX
i=1
ci(e), where ci(e) =
8
>><
>>:
0 : e < ~pi
1 : e 2 ~pi
e following holds:
CPa =
TPC(a) · aw
N(te ts)
(3)
Sp
di↵er
ysis:
graph
whos
ers (t
graph
all wo
tions
1 We p
4

63. ONLINE PERFORMANCE ANALYSIS WITH
SNAILTRAIL

64. 38
reference application SnailTrail
Timely
Trace ingestion
CP-based
performance
summaries
PAG construction
CP computation and
activity ranking
trace streams
Profiling
Trace generation
Apache Flink,
Apache Spark,
TensorFlow,
Heron,
Timely Dataflow, ...
SNAILTRAIL IN ACTION

65. EXAMPLE: TASK SCHEDULING IN APACHE SPARK
39
DRIVER
W1
W2
W3
Venkataraman, Shivaram, et al. "Drizzle: Fast and adaptable stream processing at scale." Spark Summit (2016).

66. SCHEDULING BOTTLENECK IN APACHE SPARK
40
0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
CP
0 5 10 15
Snapshot
%weight
Processing Scheduling
Conventional ProfilingSnailTrail Profiling
Apache Spark: Yahoo! Streaming Benchmark, 16 workers, 8s snapshots

67. SNAILTRAIL CP-BASED SUMMARIES
▸ Activity Summary
▸ which activity type is a bottleneck?
41

68. 0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
1.0
CP
DataMessage
Unknown
Buffer
Deserialization
Serialization
Processing
Activity Summary
Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots
42

69. 0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
1.0
CP
DataMessage
Unknown
Buffer
Deserialization
Serialization
Processing
Activity Summary
Optimize
serialization!
Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots
42

70. SNAILTRAIL CP-BASED SUMMARIES
▸ Activity Summary
▸ which activity type is a bottleneck?
▸ Straggler Summary
▸ which worker is a bottleneck?
43

71. 0 5 10 15
Snapshot
0.00
0.05
0.10
0.15
CP
W1
W2
W3
W4
Straggler Summary
Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots
44

72. 0 5 10 15
Snapshot
0.00
0.05
0.10
0.15
CP
W1
W2
W3
W4
Straggler Summary
Steal work from W1
Apache Flink: Dhalion WordCount Benchmark, 4 workers, 1s snapshots
44

73. SNAILTRAIL CP-BASED SUMMARIES
▸ Activity Summary
▸ which activity type is a bottleneck?
▸ Straggler Summary
▸ which worker is a bottleneck?
▸ Operator Summary
▸ which operator is a bottleneck?
45

74. Operator Summary
Apache Flink: Dhalion WordCount Benchmark, 10 workers, 1s snapshots
46

75. Operator Summary
Increase
flatmap’s parallelism!
Apache Flink: Dhalion WordCount Benchmark, 10 workers, 1s snapshots
46

76. SNAILTRAIL CP-BASED SUMMARIES
▸ Activity Summary
▸ which activity type is a bottleneck?
▸ Straggler Summary
▸ which worker is a bottleneck?
▸ Operator Summary
▸ which operator is a bottleneck?
▸ Communication Summary
▸ which communication channels are bottlenecks?
47

77. Communication Criticality
Communication Summary
48
0 1 2 3 4 5 6 7 8 9 10 11 12
Worker
0
1
2
3
4
5
6
7
8
9
10
11
12
Worker
1 32 54 86 7 109 1211
Worker
Worker
1
2
3
4
5
6
7
8
9
10
11
12

78. Communication Criticality
Communication Summary
48
0 1 2 3 4 5 6 7 8 9 10 11 12
Worker
0
1
2
3
4
5
6
7
8
9
10
11
12
Worker
1 32 54 86 7 109 1211
Worker
Worker
1
2
3
4
5
6
7
8
9
10
11
12
Collocate worker 5
with 2, 9, 10, 11

79. SNAILTRAIL PERFORMANCE
▸ Low instrumentation overhead
▸ < 10% for all reference systems
▸ High throughput
▸ 1.2 million events per second
▸ Always online
▸ 1s of traces in 6ms, 256s of traces in < 25s
49
Traces from Apache Flink Sessionization, 48 workers, 1s-256s snapshots
SnailTrail on Intel Xeon E5-4640, 2.40GHz, 32 cores, 512GB RAM

80. RECAP
50

81. RECAP
50
Strymon: online datacenter analytics
and management

82. RECAP
50
Strymon: online datacenter analytics
and management
0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
CP
0 5 10 15
Snapshot
%weight
Processing Scheduling
Conventional profiling is misleading

83. RECAP
50
Strymon: online datacenter analytics
and management
0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
CP
0 5 10 15
Snapshot
%weight
Processing Scheduling
Conventional profiling is misleading
CP-metric: online critical path analysis

84. RECAP
50
Strymon: online datacenter analytics
and management
0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
CP
0 5 10 15
Snapshot
%weight
Processing Scheduling
Conventional profiling is misleading
CP-metric: online critical path analysis SnailTrail: online CP-based summaries

85. THE STRYMON TEAM & FRIENDS
51
Vasiliki Kalavri
Zaheer Chothia
Andrea Lattuada
Prof. Timothy Roscoe
Sebastian Wicki
Moritz Hoffmann
Desislava Dimitrova
John Liagouris
Frank McSherry

86. THE STRYMON TEAM & FRIENDS
51
Vasiliki Kalavri
Zaheer Chothia
Andrea Lattuada
Prof. Timothy Roscoe
Sebastian Wicki
Moritz Hoffmann
Desislava Dimitrova
John Liagouris
? ?
Frank McSherry

87. THE STRYMON TEAM & FRIENDS
51
Vasiliki Kalavri
Zaheer Chothia
Andrea Lattuada
Prof. Timothy Roscoe
Sebastian Wicki
Moritz Hoffmann
Desislava Dimitrova
John Liagouris
? ?
Frank McSherry
IT COULD BE YOU!
strymon.systems.ethz.ch
www.systems.ethz.ch/positions

88. ONLINE
PERFORMANCE
ANALYSIS
OF DISTRIBUTED DATAFLOW SYSTEMS
Vasia Kalavri
kalavriv@inf.ethz.ch
Support:
O’Reilly Velocity London
20 October 2017