A modern enterprise datacenter is a complex, multi-layered system whose components often interact in unpredictable ways. Yet, to keep operational costs low and maximize efficiency, we would like to foresee the impact of changing workloads, updating configurations, modifying policies, or deploying new services. In this talk, I will share our research group’s ongoing work on Strymon: a system for predicting datacenter behavior in hypothetical scenarios using queryable online simulation. Strymon leverages existing logging and monitoring pipelines of modern production datacenters to ingest cross-layer events in a streaming fashion and predict possible effects of such events in what-if scenarios. Predictions are made online by simulating the hypothetical datacenter state alongside the real one. Driven by a real-use case from our industrial partners, I will highlight the challenges we are facing in building Strymon to support a diverse set of data representations, input sources, query languages, and execution models. Finally, I will share our initial design decisions and give an overview of Timely Dataflow; a high-performance distributed streaming engine and our platform of choice for Strymon’s core implementation.

1. PREDICTIVE
DATACENTER ANALYTICS
WITH STRYMON
Vasia Kalavri
kalavriv@inf.ethz.ch
Support:
QCon San Francisco
14 November 2017

2. ABOUT ME
▸ Postdoc at ETH Zürich
▸ Systems Group: https://www.systems.ethz.ch/
▸ PMC member of Apache Flink
▸ Research interests
▸ Large-scale graph processing
▸ Streaming dataflow engines
▸ Current project
▸ Predictive datacenter analytics and
management
2
@vkalavri

3. DATACENTER
MANAGEMENT

4. https://code.facebook.com/posts/1499322996995183/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/
4

5. 5
https://www.theregister.co.uk/2017/09/28/amadeus_booking_software_outages_lead_to_global_delayed_flights/

6. DATACENTER MANAGEMENT
66
Workloadfluctuations
Network failures
Configuration
updates
Software updatesResource scaling
Service
deployment

7. DATACENTER MANAGEMENT
6
Can we predict the effect of changes?
6
Workloadfluctuations
Network failures
Configuration
updates
Software updatesResource scaling
Service
deployment

8. DATACENTER MANAGEMENT
6
Can we predict the effect of changes?
Can we prevent catastrophic faults?
6
Workloadfluctuations
Network failures
Configuration
updates
Software updatesResource scaling
Service
deployment

9. Predicting outcomes under hypothetical conditions
What-if Analysis:
7

10. Predicting outcomes under hypothetical conditions
‣ How will response time change if we migrate a large service?
‣ What will happen to link utilization if we change the routing protocol costs?
‣ Will SLOs will be violated if a certain switch fails?
‣ Which services will be affected if we change load balancing strategy?
What-if Analysis:
7

11. Test deployment?
Data Center Test Cluster
▸ expensive to operate and maintain
▸ some errors only occur in a large scale!
a physical small-scale cluster
to try out configuration changes and what-ifs
8

12. Analytical model?
Data Center
infer workload distribution from samples and analytically
model system components (e.g. disk failure rate)
▸ hard to develop, large design space to explore
▸ often inaccurate
9

13. MODERN ENTERPRISE
DATACENTERS ARE ALREADY
HEAVILY INSTRUMENTED
10

14. Trace-driven online simulation
Use existing instrumentation to build a datacenter model
▸ construct DC state from real events
▸ simulate the state we cannot directly observe
11
Data Center
current DC
state
forked what-if
DC state
forked what-if
DC state
forked what-if
DC state

15. 1212
traces, configuration,
topology updates, …
Datacenter
Strymon
queries, complex analytics,
simulations, …
policy enforcement,
what-if scenarios, …
STRYMON: ONLINE DATACENTER ANALYTICS AND MANAGEMENT
Datacenter
state
event streams
strymon.systems.ethz.ch

16. TIMELY DATAFLOW
ingress egress
feedback

17. STRYMON’S OPERATIONAL REQUIREMENTS
‣ Low latency: react quickly to network failures
‣ High throughput: keep up with high stream rates
‣ Iterative computation: complex graph analytics on the
network topology
‣ Incremental computation: reuse already computed results
when possible
‣ e.g. do not recompute forwarding rules after a link update
14

18. TIMELY DATAFLOW: STREAM PROCESSING IN RUST
15
▸ Data-parallel computations
▸ Arbitrary cyclic dataflows
▸ Logical timestamps (epochs)
▸ Asynchronous execution
▸ Low latency
D. Murray, F. McSherry, M. Isard, R. Isaacs, P. Barham, M. Abadi.
Naiad: A Timely Dataflow System. In SOSP, 2013.
15
https://github.com/frankmcsherry/timely-dataflow

19. WORDCOUNT IN TIMELY
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = InputHandle::new();
let mut probe = ProbeHandle::new();
let index = worker.index();
worker.dataflow(|scope| {
input.to_stream(scope)
.flat_map(|text: String|
text.split_whitespace()
.map(move |word| (word.to_owned(), 1))
.collect::<Vec<_>>()
)
.aggregate(
|_key, val, agg| { *agg += val; },
|key, agg: i64| (key, agg),
|key| hash_str(key)
)
.inspect(|data| println!("seen {:?}", data))
.probe_with(&mut probe);
});
//feed data
…
}).unwrap();
}
16

20. WORDCOUNT IN TIMELY
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = InputHandle::new();
let mut probe = ProbeHandle::new();
let index = worker.index();
worker.dataflow(|scope| {
input.to_stream(scope)
.flat_map(|text: String|
text.split_whitespace()
.map(move |word| (word.to_owned(), 1))
.collect::<Vec<_>>()
)
.aggregate(
|_key, val, agg| { *agg += val; },
|key, agg: i64| (key, agg),
|key| hash_str(key)
)
.inspect(|data| println!("seen {:?}", data))
.probe_with(&mut probe);
});
//feed data
…
}).unwrap();
}
16
initialize and run
a timely job

21. WORDCOUNT IN TIMELY
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = InputHandle::new();
let mut probe = ProbeHandle::new();
let index = worker.index();
worker.dataflow(|scope| {
input.to_stream(scope)
.flat_map(|text: String|
text.split_whitespace()
.map(move |word| (word.to_owned(), 1))
.collect::<Vec<_>>()
)
.aggregate(
|_key, val, agg| { *agg += val; },
|key, agg: i64| (key, agg),
|key| hash_str(key)
)
.inspect(|data| println!("seen {:?}", data))
.probe_with(&mut probe);
});
//feed data
…
}).unwrap();
}
16
create input and
progress handles

22. WORDCOUNT IN TIMELY
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = InputHandle::new();
let mut probe = ProbeHandle::new();
let index = worker.index();
worker.dataflow(|scope| {
input.to_stream(scope)
.flat_map(|text: String|
text.split_whitespace()
.map(move |word| (word.to_owned(), 1))
.collect::<Vec<_>>()
)
.aggregate(
|_key, val, agg| { *agg += val; },
|key, agg: i64| (key, agg),
|key| hash_str(key)
)
.inspect(|data| println!("seen {:?}", data))
.probe_with(&mut probe);
});
//feed data
…
}).unwrap();
}
16
define the dataflow
and its operators

23. WORDCOUNT IN TIMELY
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = InputHandle::new();
let mut probe = ProbeHandle::new();
let index = worker.index();
worker.dataflow(|scope| {
input.to_stream(scope)
.flat_map(|text: String|
text.split_whitespace()
.map(move |word| (word.to_owned(), 1))
.collect::<Vec<_>>()
)
.aggregate(
|_key, val, agg| { *agg += val; },
|key, agg: i64| (key, agg),
|key| hash_str(key)
)
.inspect(|data| println!("seen {:?}", data))
.probe_with(&mut probe);
});
//feed data
…
}).unwrap();
}
16
watch for
progress

24. WORDCOUNT IN TIMELY
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = InputHandle::new();
let mut probe = ProbeHandle::new();
let index = worker.index();
worker.dataflow(|scope| {
input.to_stream(scope)
.flat_map(|text: String|
text.split_whitespace()
.map(move |word| (word.to_owned(), 1))
.collect::<Vec<_>>()
)
.aggregate(
|_key, val, agg| { *agg += val; },
|key, agg: i64| (key, agg),
|key| hash_str(key)
)
.inspect(|data| println!("seen {:?}", data))
.probe_with(&mut probe);
});
//feed data
…
}).unwrap();
}
16
a few Rust
peculiarities to
get used to :-)

25. PROGRESS TRACKING
▸ All tuples bear a logical timestamp (think event time)
▸ To send a timestamped tuple, an operator must hold a
capability for it
▸ Workers broadcast progress changes to other workers
▸ Each worker independently determines progress
17
A distributed protocol that allows operators reason
about the possibility of receiving data

26. MAKING PROGRESS
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
…
…
…
for round in 0..10 {
input.send(("round".to_owned(), 1));
input.advance_to(round + 1);
while probe.less_than(input.time()) {
worker.step();
}
}
}).unwrap();
}
18

27. MAKING PROGRESS
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
…
…
…
for round in 0..10 {
input.send(("round".to_owned(), 1));
input.advance_to(round + 1);
while probe.less_than(input.time()) {
worker.step();
}
}
}).unwrap();
}
18
push data to the
input stream

28. MAKING PROGRESS
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
…
…
…
for round in 0..10 {
input.send(("round".to_owned(), 1));
input.advance_to(round + 1);
while probe.less_than(input.time()) {
worker.step();
}
}
}).unwrap();
}
18
advance the
input epoch

29. MAKING PROGRESS
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
…
…
…
for round in 0..10 {
input.send(("round".to_owned(), 1));
input.advance_to(round + 1);
while probe.less_than(input.time()) {
worker.step();
}
}
}).unwrap();
}
18
do work while there’s
still data for this epoch

30. TIMELY ITERATIONS
timely::example(|scope| {
let (handle, stream) = scope.loop_variable(100, 1);
(0..10).to_stream(scope)
.concat(&stream)
.inspect(|x| println!("seen: {:?}", x))
.connect_loop(handle);
});
19

31. TIMELY ITERATIONS
timely::example(|scope| {
let (handle, stream) = scope.loop_variable(100, 1);
(0..10).to_stream(scope)
.concat(&stream)
.inspect(|x| println!("seen: {:?}", x))
.connect_loop(handle);
});
19
loop 100 times
at most
advance
timestamps by 1
in each iteration
create the
feedback loop

32. TIMELY ITERATIONS
timely::example(|scope| {
let (handle, stream) = scope.loop_variable(100, 1);
(0..10).to_stream(scope)
.concat(&stream)
.inspect(|x| println!("seen: {:?}", x))
.connect_loop(handle);
});
19
loop 100 times
at most
advance
timestamps by 1
in each iteration
create the
feedback loop
t
(t, l1)
(t, (l1, l2))

33. TIMELY & I
20

34. TIMELY & I
20
Relationship status:
it’s complicated

35. TIMELY & I
20
Relationship status:
it’s complicated
APIs &
libraries
performance
ecosystem
fault-tolerance
deployment
debugging
expressiveness
incremental
computation
performance

36. STRYMON
USE-CASES

37. What-if analysis
Real-time
datacenter analytics
Incremental
network routing
Online critical
path analysis
Strymon
22
‣ Query and analyze state online
‣ Control and enforce configuration
‣ Simulate what-if scenarios
‣ Understand performance

38. What-if analysis
Real-time
datacenter analytics
Incremental
network routing
Online critical
path analysis
Strymon
23
‣ Query and analyze state online
‣ Control and enforce configuration
‣ Simulate what-if scenarios
‣ Understand performance

39. RECONSTRUCTING USER SESSIONS
24
Application A
Application B
A.1
A.2
A.3
B.1
B.2
B.3
B.4
Time: 2015/09/01 10:03:38.599859
Session ID: XKSHSKCBA53U088FXGE7LD8
Transaction ID: 26-3-11-5-1

40. PERFORMANCE RESULTS
25
Log event
A
B
1 2
1-1 1-2
1
1-3
1-1
Client Time
1
1-1 1-2
2
Inactivity
Transaction IDn-2-8Transaction
‣ Logs from 1263
streams and 42 servers
‣ 1.3 million events/s at
424.3 MB/s
‣ 26ms per epoch vs. 2.1s
per epoch with Flink
Zaheer Chothia, John Liagouris, Desislava Dimitrova, and Timothy Roscoe.
Online Reconstruction of Structural Information from Datacenter Logs. (EuroSys '17).
More Results:

41. What-if analysis
Real-time
datacenter analytics
Incremental
network routing
Online critical
path analysis
Strymon
26
‣ Query and analyze state online
‣ Control and enforce configuration
‣ Simulate what-if scenarios
‣ Understand performance

42. ROUTING AS A STREAMING COMPUTATION
Network
changes
G R
delta-join
iteration
Updated rules
Forwarding
rules
27
‣ Compute APSP and keep forwarding rules as operator state
‣ Flow requests translate to lookups on that state
‣ Network updates cause re-computation of affected rules only

43. REACTION TO LINK FAILURES
▸ Fail random link
▸ 500 individual runs
▸ 32 threads
28
Desislava C. Dimitrova et. al.
Quick Incremental Routing Logic for Dynamic Network Graphs. SIGCOMM Posters and Demos '17
More Results:

44. What-if analysis
Real-time
datacenter analytics
Incremental
network routing
Online critical
path analysis
Strymon
29
‣ Query and analyze state online
‣ Control and enforce configuration
‣ Simulate what-if scenarios
‣ Understand performance

45. ONLINE CRITICAL PATH ANALYSIS
30
input stream output stream
periodic
snapshot
trace snapshot
stream
analyzer
performance
summaries
Apache Flink,
Apache Spark,
TensorFlow,
Heron,
Timely Dataflow

46. TASK SCHEDULING IN APACHE SPARK
31
DRIVER
W1
W2
W3
Venkataraman, Shivaram, et al. "Drizzle: Fast and adaptable stream processing at scale." Spark Summit (2016).

47. SCHEDULING BOTTLENECK IN APACHE SPARK
32
0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
CP
0 5 10 15
Snapshot
%weight
Processing Scheduling
Conventional ProfilingStrymon Profiling
Apache Spark: Yahoo! Streaming Benchmark, 16 workers, 8s snapshots

48. SCHEDULING BOTTLENECK IN APACHE SPARK
32
0 5 10 15
Snapshot
0.0
0.2
0.4
0.6
0.8
CP
0 5 10 15
Snapshot
%weight
Processing Scheduling
Conventional ProfilingStrymon Profiling
Apache Spark: Yahoo! Streaming Benchmark, 16 workers, 8s snapshots

49. What-if analysis
Real-time
datacenter analytics
Incremental
network routing
Online critical
path analysis
Strymon
33
‣ Query and analyze state online
‣ Control and enforce configuration
‣ Simulate what-if scenarios
‣ Understand performance

50. EVALUATING LOAD BALANCING STRATEGIES
▸ 13k services
▸ ~100K user requests/s
▸ OSPF routing
▸ Weighted Round-Robin
load balancing
34
Traffic matrix simulation
under topology and load
balancing changes

51. WHAT-IF DATAFLOW
35
Logging Servers
Topology Discovery
Configuration DBs
RPC logs
(text)
Topology changes
(graph)
Device specs,
application placement
(relational)
ETL Load Balancing Routing
LB View
Construction
Routing View
Construction
Traffic Matrix
Construction
DC Model
construction

52. STRYMON DEMO

53. Strymon 0.1.0 has been released!
https://strymon-system.github.io
https://github.com/strymon-system
strymon-users@lists.inf.ethz.ch
Try it out:
Send us feedback:
37

54. THE STRYMON TEAM & FRIENDS
38
Vasiliki Kalavri
Zaheer Chothia
Andrea Lattuada
Prof. Timothy Roscoe
Sebastian Wicki
Moritz HoffmannDesislava Dimitrova
John Liagouris
Frank McSherry

55. THE STRYMON TEAM & FRIENDS
38
Vasiliki Kalavri
Zaheer Chothia
Andrea Lattuada
Prof. Timothy Roscoe
Sebastian Wicki
Moritz HoffmannDesislava Dimitrova
John Liagouris
? ?
Frank McSherry

56. THE STRYMON TEAM & FRIENDS
38
Vasiliki Kalavri
Zaheer Chothia
Andrea Lattuada
Prof. Timothy Roscoe
Sebastian Wicki
Moritz HoffmannDesislava Dimitrova
John Liagouris
? ?
Frank McSherry
IT COULD BE YOU!
strymon.systems.ethz.ch

57. PREDICTIVE
DATACENTER ANALYTICS
WITH STRYMON
Vasia Kalavri
kalavriv@inf.ethz.ch
Support:
QCon San Francisco
14 November 2017