The capabilities of machine learning are now pretty well understood and there are great tools to do data science and construct models that answer nontrivial questions about your data. These tools are often used from Python. The key new challenge is making the trained prediction model usable in real time, while the user is interacting with your software. Getting answers from an ML model takes a lot of CPU/GPU and must be done at serious scale. The ML tools are optimized mainly for batch-processing a lot of data at once, and often the implementations aren't parallelized. In this talk I will show one approach which allows you to write a low-latency, auto-parallelized and distributed stream processing pipeline in Java that seamlessly integrates with a data scientist's work taken in almost unchanged form from their Python development environment. The talk includes a live demo using the command line and going through some Python and Java code snippets.

1. Real-Time Streaming with
Python ML Inference
Marko Topolnik
1

2. About Us
Hazelcast started as an In-Memory Data Grid project
(distributed caching with data-local computation)
Java codebase, polyglot clients
we focus on Simplicity
in 2017 we added Hazelcast Jet: In-Memory Distributed Stream
Processing
My role: core engineer at Hazelcast Jet
2

3. Example: Salary Prediction
Python, SciKit Learn
3

4. Sample Input and Output
Input: {
"age": 25,
"workclass": "Self-emp",
"fnlwgt": 176756,
"education": "HS-grad",
"education-num": 9,
"marital-status": "Never-married",
"occupation": "Farming-fishing",
"relationship": "Own-child",
"capital-gain": 0,
"capital-loss": 0,
"hours-per-week": 35,
"native-country": "United-States"
}
Output: {
"probability": 0.85
"income": "<=50K"
}
4

5. (Showing Project Directory)
5

6. Data Scientist's Perspective
Do Data Science Profit!
6

7. Data Engineer's Perspective
Do Data Science
Monitor
Profit
(fingers crossed)
7
Package
Load-Balance
Scale Up
Scale Out
Deploy
Re-Train

8. Productionizing the REST service
8
Request #1
Request #2
Request #3
R
E
S
T
Batching?

9. Productionizing the REST service
9
Request #1
Request #2
Request #3
R
E
S
T
Request
Queue
Batch
Response
Queue
Parallelism?

10. Productionizing the REST service
10
Request #1
Request #2
Request #3
R
E
S
T
R
E
S
T
Load
Balancer
Request
Queue
Response
Queue
Batch
Batch

11. Replace REST with Distributed Streaming
11
Request #1
Request #2
Request #3
Kafka
Hazelcast
Jet Cluster
$ jet submit ML

12. Hazelcast Jet Code
Pipeline p = Pipeline.create();
p.readFrom(Kafka.source())
.apply(mapUsingPython(new PythonServiceConfig()
.setBaseDir("/Users/mtopol/dev/python/sklearn")
.setHandlerModule("example_1_inference_jet")))
.writeTo(Kafka.sink());
jet.newJob(p);
$ mvn package
$ jet submit target/my-job.jar
12

13. Jet Node 1
Jet Job's Execution Plan
Source
mapUsing
Python
mapUsing
Python
Sink
13
Kafka Topic A
Python
process
Kafka Topic B
Batch
Python
process
Batch

14. Cluster Elasticity and Resilience
● Jet jobs are fault-tolerant
● nodes can join and leave the cluster, jobs go on
● automatically rescale to available hardware
14

15. Let's Start a Jet Cluster!
15

16. Cluster Self-Formation
16
Hazelcast Jet natively supports:
● Amazon AWS
● Google GCP
● Kubernetes
With simple configuration, the nodes self-discover in these
environments

17. Source and Sink Connectors
● Kafka
● Hadoop HDFS
● S3 bucket
● JDBC
● JMS queue and topic
● custom connector
● test connectors: convenient to test the code before submitting
17

18. Stream Operators
● windowed aggregation using Event Time
○ sliding, session window
○ count, sum, average, linear regression, ...
○ custom aggregate function
● rolling aggregation
● streaming join (co-grouping)
● hash join (enrichment)
● contact arbitrary external services
○ mapUsingPython uses this
18

19. Thanks for attending!
Q&A
marko@hazelcast.com
@mtopolnik
19