In this talk, we describe the design and implementation of the Python Streaming API support that has been submitted for inclusion in mainline Flink. Python is one of the most popular programming languages for data analysis. Its readability emphasizes development productivity and as a scripting language, it does not require a compilation nor complex development environment setup. Flink already has support for Python APIs for batch programming and unfortunately, the mechanism used to support batch programs (i.e., DataSet APIs) do does not work for Streaming API. We describe the limitations with the batch implementation and provide insights into how we solved this using Jython. We will walk through some of the examples programs using the new Python API and compare programmability and performance with the Java and Scala streaming APIs.

1. www.parallelm.com
Python Streaming API
1
Zohar Mizrahi
Senior Software Architect
ParallelM
Flink Forward 2017,
Berlin, Germany

2. www.parallelm.com
What Will Be Covered
● ParallelM
● Python in Machines Learning
● Python Batch API
● Python Streaming API
● Live Demo
2

3. www.parallelm.com
ParallelM
ParallelM accelerates time to value of AI initiatives
by helping ML Ops and Data Science teams
deploy and manage Machine Learning (ML) in Production
We have put much effort in Flink, because of its
exceptional design and ability to handle high speed
real time and true stream processing
3

4. www.parallelm.com
Python In Machine Learning
● Popular in data analysis (NumPy, SciPy, Matplotlib,
Panda, etc.)
● Very easy to learn
● Very easy to read
● Does not require compilation
● Awesome online community
4

5. www.parallelm.com
Python Batch Processing Overview
5
Client (JVM)
Script
(Python)
HDFS
JobManager
(JVM)
TaskManager
(JVM)
Script
(Python)
MMAP
Files

6. www.parallelm.com
… but Flink is a Stream Processing framework ...
What about Python for Streaming Data?
6

7. www.parallelm.com
Proposed Python Streaming API Architecture
7
Jython,
Python Script
Java Streaming API
Client (JVM)
HDFS
JobManager
(JVM)
Jython,
Python
Script
UDFs
TaskManager (JVM)
Python Streaming API
Python Streaming
API
- Python naming
- Serialization/Deseri
alization
- Thin layer

8. www.parallelm.com
Jython
● Python engine in java (http://www.jython.org/)
● Possible to use CPython extensions like NumPy or SciPy
with JyNI (http://jyni.org/)
● Glitches
○ The latest supported Python version is 2.7
○ No official statement for coming support in Python 3.x
8

9. www.parallelm.com
What Jython Challenges I had to solve?
● Java class serial ID mismatch
○ Execution of the same script multiple times
● Different namespaces for python classes with same name
but different code
○ Execution of different scripts having the same class name
● Python paths and imports issues
○ A python script may import additional files and folder
9

10. www.parallelm.com
Performance Considerations
● Initialisation of the Jython framework impose a fixed
overhead of 2 ~ 5 seconds:
○ Client - whenever submitted
○ TaskManager - only once, on the first submitted job
● Java/Scala vs. Python
○ No high-scale tests were conducted yet
○ There’s a room for optimizations
10

11. www.parallelm.com
Python Script - main
def main():
env = PythonStreamExecutionEnvironment.get_execution_environment()
env.read_text_file(“/tmp/book.txt”)
.flat_map(Tokenizer())
.key_by(Selector())
.time_window(milliseconds(50))
.reduce(Sum())
.print()
env.execute(True)
11

12. www.parallelm.com
Python Script - UDF
class Tokenizer(FlatMapFunction):
def flatMap(self, value, collector):
for word in value.lower().split():
collector.collect((1, word))
class Selector(KeySelector):
def getKey(self, input):
return input[1]
class Sum(ReduceFunction):
def reduce(self, input1, input2):
count1, word1 = input1
count2, word2 = input2
return (count1 + count2, word1)
12

13. www.parallelm.com
Status / API Coverage
● Pending pull request (#3838)
● New project under:
flink-libraries/flink-streaming-python
● Partial coverage of the whole streaming API (Beta)
13

14. www.parallelm.com
Tests / Examples
● Internal tests are under:
flink-libraries/flink-streaming-python/src/test/python/org/a
pache/flink/streaming/python/api
● One complete example:
flink-examples/flink-examples-streaming/src/main/python/fibo
nacci.py
14

15. www.parallelm.com
> ./bin/pyflink-stream.sh /tmp/fibonacci.py - --local
Notes:
● New command line tool: pyflink-stream.sh
● Command line arguments: after the dash(`-`)
● For local execution: env.execute(True)
● Cluster mode requires HDFS
How to execute
15

16. www.parallelm.com 16
● Fibonacci python example
● Functionality
○ Calculates fibonacci series up to an upper bound
● Input
○ “<x>, <y>” - stream of pairs of numbers
● Output
○ ((<x>, <y>), <#iters>) - the original pair along with the iterations
number
Demo Time