Presentation of Timely Dataflow at Flink meetup

1. Timely Dataflow
Frank McSherry
mcsherry@gmail.com

2. Data-parallel dataflow
Simple programming model
Performant implementations
Too
Sub

3. Some background
from [Gonzalez et al, OSDI2014]

4. 20xPR cores twitter_rv uk_2007_05
Spark 128 857s 1759s
Giraph 128 596s 1235s
GraphLab 128 249s 833s
GraphX 128 419s 462s

5. 20xPR cores twitter_rv uk_2007_05
Spark 128 857s 1759s
Giraph 128 596s 1235s
GraphLab 128 249s 833s
GraphX 128 419s 462s
Laptop 1

6. 20xPR cores twitter_rv uk_2007_05
Spark 128 857s 1759s
Giraph 128 596s 1235s
GraphLab 128 249s 833s
GraphX 128 419s 462s
Laptop 1 300s 651s

7. 20xPR cores twitter_rv uk_2007_05
Spark 128 857s 1759s
Giraph 128 596s 1235s
GraphLab 128 249s 833s
GraphX 128 419s 462s
Laptop 1 300s 651s
110s 256s

8. Connectivity cores twitter_rv uk_2007_05
Spark 128 1784s 8000s+
Giraph 128 200s 8000s+
GraphLab 128 242s 714s
GraphX 128 251s 800s
Laptop 1

9. Connectivity cores twitter_rv uk_2007_05
Spark 128 1784s 8000s+
Giraph 128 200s 8000s+
GraphLab 128 242s 714s
GraphX 128 251s 800s
Laptop 1 153s 417s

10. Connectivity cores twitter_rv uk_2007_05
Spark 128 1784s 8000s+
Giraph 128 200s 8000s+
GraphLab 128 242s 714s
GraphX 128 251s 800s
Laptop 1 153s 417s
15s 30s

11. What went so wrong?
Simple programming model
Performant implementations
Too
Sub

12. Back to basics
(of data-parallel computation)

13. Batch Dataflow
fn main () {
input.map_reduce(|record| keys_vals(record),
|key, vals| reducer(key, vals))
}
Parallelism!

14. Batch Dataflow
fn my_func(input) {
input.map_reduce(|record| keys_vals(record),
|key, vals| reducer(key, vals))
}
Parallelism!

15. Batch Dataflow
fn my_func(input) {
input.map_reduce(|record| keys_vals(record),
|key, vals| reducer(key, vals))
}
fn main() {
my_func(my_func(input));
}
Parallelism!

16. fn my_func(input) {
input.map_reduce(|record| keys_vals(record),
|key, vals| reducer(key, vals))
}
fn main() {
my_func(my_func(input));
for record in source {
input.push(record);
}
}
State!
Batch DataflowStreaming Datoflow

17. fn my_func(input) {
input.map_reduce(|record| keys_vals(record),
|key, vals| reducer(key, vals))
}
fn main() {
let output = my_func(my_func(input));
for record in source {
input.push(record);
}
}
Streaming Datoflow

18. fn my_func(input) {
input.map_reduce(|record| keys_vals(record),
|key, vals| reducer(key, vals))
}
fn main() {
let output = my_func(my_func(input));
for record in output {
input.push(record);
}
}
Iteration!
Streaming DatoflowIterative Dataflow

19. Where is this going?
Iteration!
State!
Parallelism!
Data Parallel
Computation

20. What is missing?
Structured Programming Languages
Zero-overhead implementations
Iterative dataflow is like one while loop with if/then/else.
Abstractions makes a software stack feel valuable.
Need modular abstractions which still compose.
Important to expose a performant low-level interface.

21. Structured Programming Languages
a worked example: graph connectivity

25. Connected
Components
Connected
Components
Strong
Connectivity

26. Good news everyone!
RecvAt(time, data): // you write
SendBy(time, data): // you call
NotifyAt(time): // you call
OnNotify(time): // you write
You can do this today with “Timely Dataflow”
Operators need time-generic implementations

28. Zero-Overhead Implementations
Sales pitch for Rust (www.rust-lang.org)

29. What is this code doing?

30. What is this code doing?
laptop (17.5s)

31. What is this code doing?

34. What is this code doing?
dataflow (18.5s)
dataflow (12.3s)

35. Simple programming model
Performant implementations
enriched to reflect program structure
which provide zero-cost abstractions