Today's end-to-end data pipelines need to combine many diverse workloads such as machine learning, relational operations, stream dataflows, tensor transformations, and graphs. For each of these workload types exist several frontends (e.g., DataFrames/SQL, Beam, Keras) based on different programming languages as well as different runtimes (e.g., Spark, Flink, Tensorflow) that target a particular frontend and possibly a hardware architecture (e.g., GPUs). Putting all the pieces of a data pipeline together simply leads to excessive data materialisation, type conversions and hardware utilisation as well as miss-matches of processing guarantees. Our research group at RISE and KTH in Sweden has founded Arc, an intermediate language that bridges the gap between any frontend and a dataflow runtime (e.g., Flink) through a set of fundamental building blocks for expressing data pipelines. Arc incorporates Flink and Beam-inspired stream semantics such as windows, state and out of order processing as well as concepts found in batch computation models. With Arc, we can cross- compile and optimise diverse tasks written in any programming language into a unified dataflow program. Arc programs can run on various hardware backends efficiently as well as allowing seamless, distributed execution on dataflow runtimes. To that end, we showcase Arcon a concept runtime built in Rust that can execute Arc programs natively as well as presenting a minimal set of extensions to make Flink an Arc-ready runtime.

1. Introducing Arc
Klas Segeljakt Max Meldrum
@FlinkForward
A Common Intermediate Language for
Unified Batch and Stream Analytics

2. Outline
• Project Introduction
• The Arc Intermediate Representation (IR)
• Arc Examples
• Arc + Flink Integration?
• Conclusions
2

3. The Big Picture
3
• Flink plays an important
role in the data science
landscape
?
?
?
?
? ?

4. The Big Picture
3
• Flink plays an important
role in the data science
landscape
• Combining Flink with other
frameworks can lead to
interesting applications
?
?
?
?
? ?

5. The Big Picture
3
• Flink plays an important
role in the data science
landscape
• Combining Flink with other
frameworks can lead to
interesting applications
• However, there is a
language barrier
?
?
?
?
? ?

6. Intuition
4
f1 f2 f3

7. Intuition
4
f1 f2 f3
• No cross-optimisation
optimisation is possible,
e.g. resource sharing

8. Intuition
4
f1 f2 f3
• No cross-optimisation
optimisation is possible,
e.g. resource sharing
• Data movement
costs ( )

9. Intuition
4
#Frameworks
Performance
f1 f2 f3
• No cross-optimisation
optimisation is possible,
e.g. resource sharing
• Data movement
costs ( )

10. Intuition
4
#Frameworks
Performance
f1 f2 f3
• No cross-optimisation
optimisation is possible,
e.g. resource sharing
• Data movement
costs ( )

11. Intuition
4
#Frameworks
Performance
f1 f2 f3
• No cross-optimisation
optimisation is possible,
e.g. resource sharing
• Data movement
costs ( )
f1 f2 f3IR IR IR

12. Intuition
4
#Frameworks
Performance
f1 f2 f3
• No cross-optimisation
optimisation is possible,
e.g. resource sharing
• Data movement
costs ( )
f1 f2 f3IR IR IR
f1 + f2 + f3IR

13. The Arc IR
5
• Streams
• Tables
• Linear algebra
High-level
• Runners
• Hardware
Low-level

14. Arc
The Arc IR
5
• Streams
• Tables
• Linear algebra
High-level
• Runners
• Hardware
Low-level

15. Arc
The Arc IR
5
• Streams
• Tables
• Linear algebra
High-level
• Runners
• Hardware
Low-level
Abstractions
• Pipelines (Operators/Sources/Sinks)
• User-defined Windows
• Out-of-Order Processing, ...

16. Arc
The Arc IR
5
• Streams
• Tables
• Linear algebra
High-level
• Runners
• Hardware
Low-level
Optimisations
• Compiler: Partial evaluation, ...
• Dataflow: Operator fusion,
fission, reordering, ...
Abstractions
• Pipelines (Operators/Sources/Sinks)
• User-defined Windows
• Out-of-Order Processing, ...

17. Compiler Pipeline
6
Arc (High Level IR)
Frontends
Logical Dataflow IR
Binaries
Physical Dataflow IR

18. Compiler Pipeline
6
Arc (High Level IR)
Frontends
Logical Dataflow IR
Binaries
Physical Dataflow IR
Flink Backend
Flink Frontend

19. 7
What is Arc?

20. 7
What is Arc?
A restrictive language for describing batch and stream transformations

21. 7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:

22. 7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)

23. 7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)
• Builders: Write-only data types (e.g. Appender[T])

24. 7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)
• Builders: Write-only data types (e.g. Appender[T])
• Values are written to builders, and builders are lazily materialised back into values

25. 7
What is Arc?
A restrictive language for describing batch and stream transformations
Transformations are modelled through:
• Values: Read-only data types (e.g. Vec[T], Stream[T], i8..i64)
• Builders: Write-only data types (e.g. Appender[T])
• Values are written to builders, and builders are lazily materialised back into values
➡ Dependencies between values and builders form a dataflow graph

26. 8
source
evenSink
oddSink
map(v+5)
filter(v%2==0)
filter(v%2!=0)
Arc Example

27. 8
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
let mapped = result(for(source,
StreamAppender[i32],
|out, v| merge(out, v + 5)));
for(mapped, evenSink, |out, v|
if (v % 2 == 0, merge(out, v), out));
for(mapped, oddSink, |out, v|
if (v % 2 != 0, merge(out, v), out))
Arc
source
evenSink
oddSink
map(v+5)
filter(v%2==0)
filter(v%2!=0)
Arc Example

28. 9
source
evenSink
oddSink
map(v+5)
filter(v%2==0)
filter(v%2!=0)
Arc Example
source
evenSink
oddSink
map(v+5) then if(x%2==0)

29. 10
source
evenSink
oddSink
map(v+5) then if(x%2==0)
Arc Example (Fused)
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
let mapped = result(for(source,
StreamAppender[i32],
|out, v| merge(out, v + 5)));
for(mapped, evenSink, |out, v|
if (v % 2 == 0, merge(out, v), out));
for(mapped, oddSink, |out, v|
if (v % 2 != 0, merge(out, v), out))
Unfused

30. 10
source
evenSink
oddSink
map(v+5) then if(x%2==0)
Arc Example (Fused)
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
let mapped = result(for(source,
StreamAppender[i32],
|out, v| merge(out, v + 5)));
for(mapped, evenSink, |out, v|
if (v % 2 == 0, merge(out, v), out));
for(mapped, oddSink, |out, v|
if (v % 2 != 0, merge(out, v), out))
Unfused
|source:Stream[i32],
evenSink:StreamAppender[i32],
oddSink:StreamAppender[i32]|
for(source,
{evenSink,oddSink},
|out, v|
let x = v + 5;
if (x % 2 == 0,
{merge(out.$1, x), out.$2},
{out.$1, merge(out.$2, x)}))
Fused

31. 11
Arc + Flink?

32. • Benefits:
11
Arc + Flink?

33. • Benefits:
• Enable stronger optimisations
11
Arc + Flink?

34. • Benefits:
• Enable stronger optimisations
• Use your other favourite libraries together with Flink
11
Arc + Flink?

35. • Benefits:
• Enable stronger optimisations
• Use your other favourite libraries together with Flink
• Make life easier for data scientists
11
Arc + Flink?

36. The black box problem
UDFs are black boxes
12
stream.map( )
.filter( )
.reduce( )

37. The black box problem
UDFs are black boxes
➡ Flink is unaware of what is being executed
inside of each black box
12
stream.map( )
.filter( )
.reduce( )

38. Fusion Levels
13
= Flink Task ~ Thread

39. Fusion Levels
13
= Flink Task ~ Thread
x + 1 x + 1 x + 1 x + 11. No Fusion

40. Fusion Levels
13
= Flink Task ~ Thread
x + 1 x + 1 x + 1 x + 11. No Fusion
x + 1 x + 1 x + 1 x + 12.Task Fusion

41. Fusion Levels
13
= Flink Task ~ Thread
x + 1 x + 1 x + 1 x + 11. No Fusion
x + 1 x + 1 x + 1 x + 12.Task Fusion
3. Invocation-level
Fusion
x + 1
for-loop
4X

42. Fusion Levels
13
= Flink Task ~ Thread
x + 1 x + 1 x + 1 x + 11. No Fusion
x + 1 x + 1 x + 1 x + 12.Task Fusion
4. Instruction-level
Fusion
x + 4
3. Invocation-level
Fusion
x + 1
for-loop
4X

43. 14
Experiment Results
100
101
102
103
ExecutionTime(seconds)
None
Task(Flink)
Invocation
Instruction
50 maps on 10M elements
N
one
Task(Flink)
Invocation
Instruction
Optimisation level
(Lower is better)

44. Example Frontend Code
(Pandas + Beam)
15

45. Example Frontend Code
(Pandas + Beam)
15

46. Example Frontend Code
(Pandas + Beam)
15
import arc.beam as beam
import arc.beam.transforms.window as window
import arc.beam.transforms.combiners as combiners
import arc.pandas as pandas

47. Example Frontend Code
(Pandas + Beam)
15
import arc.beam as beam
import arc.beam.transforms.window as window
import arc.beam.transforms.combiners as combiners
import arc.pandas as pandas
def normalise(elements):
series = pandas.Series(elements)
avg = series.sum() / series.count()
return series / avg

48. Example Frontend Code
(Pandas + Beam)
15
import arc.beam as beam
import arc.beam.transforms.window as window
import arc.beam.transforms.combiners as combiners
import arc.pandas as pandas
def normalise(elements):
series = pandas.Series(elements)
avg = series.sum() / series.count()
return series / avg
p = beam.Pipeline()
(p
| beam.io.ReadFromText(path='input.txt').with_output_types(int)
| beam.WindowInto(window.FixedWindows(size=5))
| beam.CombineGlobally(normalise)
| combiners.ToList()
| beam.io.WriteToText(path='output.txt'))
p.run()

49. Example Frontend Code
(Pandas + Beam)
15
import arc.beam as beam
import arc.beam.transforms.window as window
import arc.beam.transforms.combiners as combiners
import arc.pandas as pandas
def normalise(elements):
series = pandas.Series(elements)
avg = series.sum() / series.count()
return series / avg
p = beam.Pipeline()
(p
| beam.io.ReadFromText(path='input.txt').with_output_types(int)
| beam.WindowInto(window.FixedWindows(size=5))
| beam.CombineGlobally(normalise)
| combiners.ToList()
| beam.io.WriteToText(path='output.txt'))
p.run()
?

50. 16
One more thing...

51. • Flink inspired dataflow engine built in Rust
• Goals:
• Common runtime for Arc applications
• Support dynamic task execution
• First-class support for hardware acceleration
17
Arcon: Native Arc Runner

52. • Arc is an IR for batch and stream programming.
• By raising the level of abstraction, Arc is able to both
optimise the dataflow and the code within it.
18
Conclusions
Arc and experiments can be found at https://github.com/cda-group
Contact info: klasseg@kth.se & mmeldrum@kth.se

53. References
Publications:
• Kroll, L., Segeljakt, K., Carbone, P., Schulte, C. and Haridi, S., 2019, June.
Arc: an IR for batch and stream programming. In Proceedings of the 17th
ACM SIGPLAN International Symposium on Database Programming
Languages (pp. 53-58). ACM.
• Meldrum, M., Segeljakt, K., Kroll, L., Carbone, P., Schulte, C. and Haridi,
S., 2019, August. Arcon: Continuous and Deep Data Stream Analytics.
In Proceedings of Real-Time Business Intelligence and Analytics (p. 3).
ACM.
19