SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
Stream Processing in the Cloud - Athens Kubernetes Meetup 16.07.2019
3.
Hands Up
Raise your hand if…
● ...you know what Stream Processing is?
4.
Hands Up
Raise your hand if…
● ...you know what Stream Processing is?
● ...you have ever used Stream Processing?
5.
Hands Up
Raise your hand if…
● ...you know what Stream Processing is?
● ...you have ever used Stream Processing?
● ...you have ever used Hazelcast Jet?
6.
Agenda
● Part 1: Stream Processing Basics
○ What is Stream Processing and Hazelcast Jet?
○ Example: Word Count
● Part 2: Jet Under the Hood
○ How does it work?
○ Infinite Streams
○ Example: Twitter Cryptocurrency Analysis
● Part 3: Jet in the Cloud
○ Cloud (Kubernetes) integration
○ Example: Stock Trade Aggregator
● Part 4: Jet Features & Use Cases
○ Why would I need it?
○ Example: Web Crawler
15.
Example 1: Word Count
Problem:
Count the number of occurrences of each word in the given text.
Sample Input:
Lorem ipsum dolor, dolor.
Sample Output:
lorem=1
ipsum=1
dolor=2
16.
Example 1: Word Count
Pure Java
Pattern delimiter = Pattern.compile("W+");
return lines.entrySet().stream()
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> Arrays.stream(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.collect(
groupingBy(
identity(),
counting()));
33.
How does it work?
Under the Hood:
● Generate DAG representation from Pipeline
● Serialize DAG
● Send DAG to every Node
● Deserialize DAG
● Executes DAG on each Node
37.
Example 2: Twitter Cryptocurrency Analysis
Problem:
Present in real-time the sentiments about cryptocurrencies
Input:
Tweets are streamed from Twitter and categorized by coin type
(BTC, ETC, XRP, etc)
Output:
Tweets sentiments (last 30 sec, last minute, last 5 minutes)
38.
Example 2: Twitter Cryptocurrency Analysis
Demo:
https://jet.hazelcast.org/demos/
47.
Jet in the Cloud: deploying on k8
$ helm install stable/hazelcast-jet
48.
Jet in the Cloud: deploying on k8
$ kubectl scale <name> --replicas=6
49.
Example 3: Stock Trade Aggregator
Problem:
Present in real-time the aggregated trade price of stocks
Input:
Stock trades with name and price
Output:
Sum of prices per stock name
50.
Example 3: Stock Trade Aggregator
Demo:
https://github.com/hazelcast/hazelcast-jet-code-samples/t
ree/master/integration/kubernetes