Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018

868 views

Published on

Talk I gave at the Voxxed Days Thessaloniki 2018 conference

Published in: Technology
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018

  1. 1. 1 Stream Processing with Hazelcast Jet Rafał Leszko @RafalLeszko
  2. 2. © 2018 Hazelcast Inc. Confidential & Proprietary Agenda ● Introduction to Jet ○ What is Hazelcast? ○ What is Stream Processing and Hazelcast Jet? ○ Example 1: Word Count ● Jet Under the Hood ○ How does it work? ○ Infinite Streams ○ Example 2: Twitter Cryptocurrency Analysis ● Jet Features & Use Cases ○ Jet Features ○ Why would I need it? ○ Example 3: Web Crawler
  3. 3. © 2018 Hazelcast Inc. Confidential & Proprietary Introduction to Jet
  4. 4. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast?
  5. 5. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast? Products:
  6. 6. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast? Products:
  7. 7. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast? Products: My Role: ● Cloud Software Engineer
  8. 8. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet?
  9. 9. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet? DAG - Direct Acyclic Graph
  10. 10. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet?
  11. 11. © 2018 Hazelcast Inc. Confidential & Proprietary What is Hazelcast Jet?
  12. 12. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Problem: Count the number of occurrences of each word in the given text. Sample Input: Lorem ipsum dolor, dolor. Sample Output: lorem=1 ipsum=1 dolor=2
  13. 13. © 2018 Hazelcast Inc. Confidential & Proprietary Pure Java Pattern delimiter = Pattern.compile("W+"); return lines.entrySet().stream() .map(e -> e.getValue().toLowerCase()) .flatMap(t -> Arrays.stream(delimiter.split(t))) .filter(word -> !word.isEmpty()) .collect( groupingBy( identity(), counting())); Example 1: Word Count
  14. 14. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  15. 15. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  16. 16. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  17. 17. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  18. 18. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  19. 19. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Hazelcast Jet Pattern delimiter = Pattern.compile("W+"); Pipeline pipeline = Pipeline.create(); pipeline.drawFrom(Sources.<Long, String>map(LINES)) .map(e -> e.getValue().toLowerCase()) .flatMap(t -> traverseArray(delimiter.split(t))) .filter(word -> !word.isEmpty()) .groupingKey(wholeItem()) .aggregate(counting()) .drainTo(Sinks.map(COUNTS)); return pipeline;
  20. 20. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Pure Java Pattern delimiter = Pattern.compile("W+"); return lines.entrySet().stream() .map(e -> e.getValue().toLowerCase()) .flatMap(t -> Arrays.stream(delimiter.split(t))) .filter(word -> !word.isEmpty()) .collect( groupingBy( identity(), counting()));
  21. 21. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Hazelcast Jet Pattern delimiter = Pattern.compile("W+"); Pipeline pipeline = Pipeline.create(); pipeline.drawFrom(Sources.<Long, String>map(LINES)) .map(e -> e.getValue().toLowerCase()) .flatMap(t -> traverseArray(delimiter.split(t))) .filter(word -> !word.isEmpty()) .groupingKey(wholeItem()) .aggregate(counting()) .drainTo(Sinks.map(COUNTS)); return pipeline;
  22. 22. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  23. 23. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count
  24. 24. © 2018 Hazelcast Inc. Confidential & Proprietary Example 1: Word Count Demo: https://github.com/hazelcast/hazelcast-jet-code-samples
  25. 25. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Under the Hood
  26. 26. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work?
  27. 27. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work?
  28. 28. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work?
  29. 29. © 2018 Hazelcast Inc. Confidential & Proprietary How does it work? Under the Hood: ● Generate DAG representation from Pipeline ● Serialize DAG ● Send DAG to every Node ● Deserialize DAG ● Executes DAG on each Node
  30. 30. © 2018 Hazelcast Inc. Confidential & Proprietary Infinite Streams
  31. 31. © 2018 Hazelcast Inc. Confidential & Proprietary Infinite Streams Examples: ● Currency Exchange Rates ● Tweets from Twitter ● Events in some Event-Based system ● ...
  32. 32. © 2018 Hazelcast Inc. Confidential & Proprietary Infinite Streams Windowing pipeline.drawFrom(...) .addTimestamps() .window(sliding(30_000, 10_000))
  33. 33. © 2018 Hazelcast Inc. Confidential & Proprietary Example 2: Twitter Cryptocurrency Analysis Problem: Present in real-time the sentiments about cryptocurrencies Input: Tweets are streamed from Twitter and categorized by coin type (BTC, ETC, XRP, etc) Output: Tweets sentiments (last 30 sec, last minute, last 5 minutes)
  34. 34. © 2018 Hazelcast Inc. Confidential & Proprietary Example 2: Twitter Cryptocurrency Analysis Demo: https://jet.hazelcast.org/demos/
  35. 35. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features & Use Cases
  36. 36. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features Categories of Features ● Easy to Use ● Performance
  37. 37. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: Performance
  38. 38. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: Performance
  39. 39. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: other features
  40. 40. © 2018 Hazelcast Inc. Confidential & Proprietary Jet Features: discovery
  41. 41. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it?
  42. 42. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it? ● Big Data Projects
  43. 43. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it? ● Big Data Projects ● Speed up Everything
  44. 44. © 2018 Hazelcast Inc. Confidential & Proprietary Why would I need it? ● Big Data Projects ● Speed up Everything
  45. 45. © 2018 Hazelcast Inc. Confidential & Proprietary Example 3: Web Crawler Problem: Parse all blog posts from the webpage Input: URL of Blog Trips Output: All the content from the Blog
  46. 46. © 2018 Hazelcast Inc. Confidential & Proprietary Example 3: Web Crawler Demo: https://github.com/leszko/geodump
  47. 47. © 2018 Hazelcast Inc. Confidential & Proprietary Thank You!

×