1
Stream Processing with
Hazelcast Jet
Rafał Leszko
@RafalLeszko

© 2018 Hazelcast Inc. Confidential & Proprietary
Agenda
● Introduction to Jet
○ What is Hazelcast?
○ What is Stream Processing and Hazelcast Jet?
○ Example 1: Word Count
● Jet Under the Hood
○ How does it work?
○ Infinite Streams
○ Example 2: Twitter Cryptocurrency Analysis
● Jet Features & Use Cases
○ Jet Features
○ Why would I need it?
○ Example 3: Web Crawler

© 2018 Hazelcast Inc. Confidential & Proprietary
Introduction to Jet

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?
Products:

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?
Products:

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?
Products:
My Role:
● Cloud Software Engineer

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?
DAG - Direct Acyclic Graph

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?

© 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Problem:
Count the number of occurrences of each word in the given text.
Sample Input:
Lorem ipsum dolor, dolor.
Sample Output:
lorem=1
ipsum=1
dolor=2

© 2018 Hazelcast Inc. Confidential & Proprietary
Pure Java
Pattern delimiter = Pattern.compile("W+");
return lines.entrySet().stream()
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> Arrays.stream(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.collect(
groupingBy(
identity(),
counting()));
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Hazelcast Jet
Pattern delimiter = Pattern.compile("W+");
Pipeline pipeline = Pipeline.create();
pipeline.drawFrom(Sources.<Long, String>map(LINES))
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> traverseArray(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.groupingKey(wholeItem())
.aggregate(counting())
.drainTo(Sinks.map(COUNTS));
return pipeline;

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Pure Java
Pattern delimiter = Pattern.compile("W+");
return lines.entrySet().stream()
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> Arrays.stream(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.collect(
groupingBy(
identity(),
counting()));

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Hazelcast Jet
Pattern delimiter = Pattern.compile("W+");
Pipeline pipeline = Pipeline.create();
pipeline.drawFrom(Sources.<Long, String>map(LINES))
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> traverseArray(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.groupingKey(wholeItem())
.aggregate(counting())
.drainTo(Sinks.map(COUNTS));
return pipeline;

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Demo:
https://github.com/hazelcast/hazelcast-jet-code-samples

© 2018 Hazelcast Inc. Confidential & Proprietary
Jet Under the Hood

© 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?

© 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?

© 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?

© 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?
Under the Hood:
● Generate DAG representation from Pipeline
● Serialize DAG
● Send DAG to every Node
● Deserialize DAG
● Executes DAG on each Node

© 2018 Hazelcast Inc. Confidential & Proprietary
Infinite Streams

© 2018 Hazelcast Inc. Confidential & Proprietary
Infinite Streams
Examples:
● Currency Exchange Rates
● Tweets from Twitter
● Events in some Event-Based system
● ...

© 2018 Hazelcast Inc. Confidential & Proprietary
Infinite Streams
Windowing
pipeline.drawFrom(...)
.addTimestamps()
.window(sliding(30_000, 10_000))

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 2: Twitter Cryptocurrency Analysis
Problem:
Present in real-time the sentiments about cryptocurrencies
Input:
Tweets are streamed from Twitter and categorized by coin type
(BTC, ETC, XRP, etc)
Output:
Tweets sentiments (last 30 sec, last minute, last 5 minutes)

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 2: Twitter Cryptocurrency Analysis
Demo:
https://jet.hazelcast.org/demos/

© 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features & Use Cases

© 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features
Categories of Features
● Easy to Use
● Performance

© 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: Performance

© 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: Performance

© 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: other features

© 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: discovery

© 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?

© 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?
● Big Data Projects

© 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?
● Big Data Projects
● Speed up Everything

© 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?
● Big Data Projects
● Speed up Everything

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 3: Web Crawler
Problem:
Parse all blog posts from the webpage
Input:
URL of Blog Trips
Output:
All the content from the Blog

© 2018 Hazelcast Inc. Confidential & Proprietary
Example 3: Web Crawler
Demo:
https://github.com/leszko/geodump

© 2018 Hazelcast Inc. Confidential & Proprietary
Thank You!