Successfully reported this slideshow.

Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018

0

Share

1 of 47
1 of 47

More Related Content

More from Rafał Leszko

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018

  1. 1. 1 Stream Processing with Hazelcast Jet Rafał Leszko @RafalLeszko
  2. 2. © 2018 Hazelcast Inc. Confidential & Proprietary Agenda ● Introduction to Jet ○ What is Hazelcast? ○ What is Stream Processing and Hazelcast Jet? ○ Example 1: Word Count ● Jet Under the Hood ○ How does it work? ○ Infinite Streams ○ Example 2: Twitter Cryptocurrency Analysis ● Jet Features & Use Cases ○ Jet Features ○ Why would I need it? ○ Example 3: Web Crawler
  3. 3. © 2018 Hazelcast Inc. Confidential & Proprietary Introduction to Jet
  4. 4. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast?
  5. 5. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast? Products:
  6. 6. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast? Products:
  7. 7. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast? Products: My Role: ● Cloud Software Engineer
  8. 8. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet?
  9. 9. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet? DAG - Direct Acyclic Graph
  10. 10. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet?
  11. 11. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet?
  12. 12. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Problem: Count the number of occurrences of each word in the given text. Sample Input: Lorem ipsum dolor, dolor. Sample Output: lorem=1 ipsum=1 dolor=2
  13. 13. © 2018 Hazelcast Inc. Confidential & Proprietary Pure Java Pattern delimiter = Pattern.compile("W+"); return lines.entrySet().stream() .map(e -> e.getValue().toLowerCase()) .flatMap(t -> Arrays.stream(delimiter.split(t))) .filter(word -> !word.isEmpty()) .collect( groupingBy( identity(), counting())); Example 1: Word Count
  14. 14. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  15. 15. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  16. 16. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  17. 17. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  18. 18. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  19. 19. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Hazelcast Jet Pattern delimiter = Pattern.compile("W+"); Pipeline pipeline = Pipeline.create(); pipeline.drawFrom(Sources.<Long, String>map(LINES)) .map(e -> e.getValue().toLowerCase()) .flatMap(t -> traverseArray(delimiter.split(t))) .filter(word -> !word.isEmpty()) .groupingKey(wholeItem()) .aggregate(counting()) .drainTo(Sinks.map(COUNTS)); return pipeline;
  20. 20. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Pure Java Pattern delimiter = Pattern.compile("W+"); return lines.entrySet().stream() .map(e -> e.getValue().toLowerCase()) .flatMap(t -> Arrays.stream(delimiter.split(t))) .filter(word -> !word.isEmpty()) .collect( groupingBy( identity(), counting()));
  21. 21. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Hazelcast Jet Pattern delimiter = Pattern.compile("W+"); Pipeline pipeline = Pipeline.create(); pipeline.drawFrom(Sources.<Long, String>map(LINES)) .map(e -> e.getValue().toLowerCase()) .flatMap(t -> traverseArray(delimiter.split(t))) .filter(word -> !word.isEmpty()) .groupingKey(wholeItem()) .aggregate(counting()) .drainTo(Sinks.map(COUNTS)); return pipeline;
  22. 22. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  23. 23. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  24. 24. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Demo: https://github.com/hazelcast/hazelcast-jet-code-samples
  25. 25. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Under the Hood
  26. 26. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work?
  27. 27. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work?
  28. 28. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work?
  29. 29. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work? Under the Hood: ● Generate DAG representation from Pipeline ● Serialize DAG ● Send DAG to every Node ● Deserialize DAG ● Executes DAG on each Node
  30. 30. © 2018 Hazelcast Inc. Confidential & Proprietary Infinite Streams
  31. 31. © 2018 Hazelcast Inc. Confidential & Proprietary Infinite Streams Examples: ● Currency Exchange Rates ● Tweets from Twitter ● Events in some Event-Based system ● ...
  32. 32. © 2018 Hazelcast Inc. Confidential & Proprietary Infinite Streams Windowing pipeline.drawFrom(...) .addTimestamps() .window(sliding(30_000, 10_000))
  33. 33. © 2018 Hazelcast Inc. Confidential & Proprietary Example 2: Twitter Cryptocurrency Analysis Problem: Present in real-time the sentiments about cryptocurrencies Input: Tweets are streamed from Twitter and categorized by coin type (BTC, ETC, XRP, etc) Output: Tweets sentiments (last 30 sec, last minute, last 5 minutes)
  34. 34. © 2018 Hazelcast Inc. Confidential & Proprietary Example 2: Twitter Cryptocurrency Analysis Demo: https://jet.hazelcast.org/demos/
  35. 35. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features & Use Cases
  36. 36. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features Categories of Features ● Easy to Use ● Performance
  37. 37. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: Performance
  38. 38. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: Performance
  39. 39. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: other features
  40. 40. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: discovery
  41. 41. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it?
  42. 42. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it? ● Big Data Projects
  43. 43. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it? ● Big Data Projects ● Speed up Everything
  44. 44. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it? ● Big Data Projects ● Speed up Everything
  45. 45. © 2018 Hazelcast Inc. Confidential & Proprietary Example 3: Web Crawler Problem: Parse all blog posts from the webpage Input: URL of Blog Trips Output: All the content from the Blog
  46. 46. © 2018 Hazelcast Inc. Confidential & Proprietary Example 3: Web Crawler Demo: https://github.com/leszko/geodump
  47. 47. © 2018 Hazelcast Inc. Confidential & Proprietary Thank You!

×