Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to In-Memory Stream Processing with Hazelcast Jet @MorningAtLohika
Similar to In-Memory Stream Processing with Hazelcast Jet @MorningAtLohika (20)
Recently uploaded
Recently uploaded (20)
In-Memory Stream Processing with Hazelcast Jet @MorningAtLohika
- 1. IN-MEMORY STREAM PROCESSING WITH Nazarii Cherkas | Hazelcast nazarii@hazelcast.com https://twitter.com/n_cherkas
- 2. Brief Agenda • Why Stream Processing? • What‘s special about Streaming Data • Challenges when processing the Infinite Stream • Hazelcast Jet: The modern Stream Processing Engine • Overview and Key Concepts • Infinite Stream Processing • Fault Tolerance • Jet Performance • Summary 2© 2018 Hazelcast Inc.
- 3. About me 3© 2018 Hazelcast Inc.
- 4. About me • 7+ years of experience of on different positions from Java Engineer to Team Lead 4© 2018 Hazelcast Inc.
- 5. About me • 7+ years of experience of on different positions from Java Engineer to Team Lead • Solutions Architect at Hazelcast, I solve problems of our users and interact with community 5© 2018 Hazelcast Inc.
- 6. Why Stream Processing? 6© 2018 Hazelcast Inc.
- 7. Streaming Data is everywhere 7© 2018 Hazelcast Inc.
- 8. What's special about Streaming Data 8© 2018 Hazelcast Inc.
- 9. What's special about Streaming Data • Infinite data sets 9© 2018 Hazelcast Inc.
- 10. What's special about Streaming Data • Infinite data sets • Small size of data record 10© 2018 Hazelcast Inc.
- 11. What's special about Streaming Data • Infinite data sets • Small size of data record • Near real-time insights 11© 2018 Hazelcast Inc.
- 12. What's special about Streaming Data • Infinite data sets • Small size of data record • Near real-time insights • Variance in throughput and variance in disorder 12© 2018 Hazelcast Inc.
- 13. Definitions of Stream Processing 13© 2018 Hazelcast Inc.
- 14. Definitions of Stream Processing “...a type of data processing that is designed with infinite data sets in mind...” https://jet.hazelcast.org/use-cases/real-time-stream-processing/ https://data-artisans.com/what-is-stream-processing https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 14© 2018 Hazelcast Inc.
- 15. Definitions of Stream Processing “...a type of data processing that is designed with infinite data sets in mind...” “...processing of data in motion, or in other words, computing on data directly as it is produced or received…” https://jet.hazelcast.org/use-cases/real-time-stream-processing/ https://data-artisans.com/what-is-stream-processing https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 15© 2018 Hazelcast Inc.
- 16. Definitions of Stream Processing “...a type of data processing that is designed with infinite data sets in mind...” “...processing of data in motion, or in other words, computing on data directly as it is produced or received…” “...a technique to process the data on-the-fly, prior to it’s storage...” https://jet.hazelcast.org/use-cases/real-time-stream-processing/ https://data-artisans.com/what-is-stream-processing https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 16© 2018 Hazelcast Inc.
- 17. Stream vs Batch Processing 17© 2018 Hazelcast Inc.
- 18. Stream vs Batch Processing https://aws.amazon.com/streaming-data/ 18 Batch processing Stream processing Data scope Queries or processing over all or most of the data in the dataset Queries or processing over data within a rolling time window, or on just the most recent data record © 2018 Hazelcast Inc.
- 19. Stream vs Batch Processing https://aws.amazon.com/streaming-data/ 19 Batch processing Stream processing Data scope Queries or processing over all or most of the data in the dataset Queries or processing over data within a rolling time window, or on just the most recent data record Data size Large batches of data Individual records or micro batches consisting of a few records © 2018 Hazelcast Inc.
- 20. Stream vs Batch Processing https://aws.amazon.com/streaming-data/ 20 Batch processing Stream processing Data scope Queries or processing over all or most of the data in the dataset Queries or processing over data within a rolling time window, or on just the most recent data record Data size Large batches of data Individual records or micro batches consisting of a few records Responsiveness Latencies in minutes to hours Requires latency in the order of seconds or milliseconds © 2018 Hazelcast Inc.
- 21. Stream vs Batch Processing https://aws.amazon.com/streaming-data/ 21 Batch processing Stream processing Data scope Queries or processing over all or most of the data in the dataset Queries or processing over data within a rolling time window, or on just the most recent data record Data size Large batches of data Individual records or micro batches consisting of a few records Responsiveness Latencies in minutes to hours Requires latency in the order of seconds or milliseconds Analyses Complex analytics Aggregates, approximation algorithms and simple response functions © 2018 Hazelcast Inc.
- 22. Layers of Stream Processing 22© 2018 Hazelcast Inc.
- 23. Challenges of Stream Processing 23© 2018 Hazelcast Inc.
- 24. Challenges of Stream Processing • Distributed system coordination 24© 2018 Hazelcast Inc.
- 25. Challenges of Stream Processing • Distributed system coordination • Notion of time 25© 2018 Hazelcast Inc.
- 26. Challenges of Stream Processing • Distributed system coordination • Notion of time • Memory management 26© 2018 Hazelcast Inc.
- 27. Challenges of Stream Processing • Distributed system coordination • Notion of time • Memory management • Fault-tolerance 27© 2018 Hazelcast Inc.
- 28. Hazelcast Jet: In-Memory Streaming and Fast Batch Processing 28© 2018 Hazelcast Inc.
- 29. What is Hazelcast Jet 29© 2018 Hazelcast Inc. Source Sink
- 30. What is Hazelcast Jet https://github.com/hazelcast/hazelcast-jet/ Apache License 2.0 30© 2018 Hazelcast Inc. Source Sink
- 31. Hazelcast Jet use cases 31© 2018 Hazelcast Inc.
- 32. Hazelcast Jet use cases • Low-latency Stream processing and analytics 32© 2018 Hazelcast Inc.
- 33. Hazelcast Jet use cases • Low-latency Stream processing and analytics • Fast Batch processing and ETL 33© 2018 Hazelcast Inc.
- 34. Hazelcast Jet use cases • Low-latency Stream processing and analytics • Fast Batch processing and ETL • Distributed java.util.stream 34© 2018 Hazelcast Inc.
- 35. Hazelcast Jet use cases • Low-latency Stream processing and analytics • Fast Batch processing and ETL • Distributed java.util.stream • Implementing event sourcing and CQRS 35© 2018 Hazelcast Inc.
- 36. Hazelcast Jet use cases • Low-latency Stream processing and analytics • Fast Batch processing and ETL • Distributed java.util.stream • Implementing event sourcing and CQRS • Data processing microservice architectures 36© 2018 Hazelcast Inc.
- 37. Hazelcast Jet: Architecture Overview 37 Core API java.util.stream Batch Readers and Writers Batch Processing Pipeline API Streaming Readers and Writers Stream Processing Networking Deployment Data Structures and Partition Management Execution Engine Cluster Management with Cloud Discovery SPI Java Client Fault-Tolerance Connectors High-Level APIs Processing Core © 2018 Hazelcast Inc. 31
- 38. Hazelcast Jet: Architecture Overview 38 Core API java.util.stream Batch Readers and Writers Batch Processing Pipeline API Streaming Readers and Writers Stream Processing Networking Deployment Data Structures and Partition Management Execution Engine Cluster Management with Cloud Discovery SPI Java Client Fault-Tolerance Connectors High-Level APIs Processing Core © 2018 Hazelcast Inc. 32
- 39. Hazelcast Jet: Architecture Overview 39 Core API java.util.stream Batch Readers and Writers Batch Processing Pipeline API Streaming Readers and Writers Stream Processing Networking Deployment Data Structures and Partition Management Execution Engine Cluster Management with Cloud Discovery SPI Java Client Fault-Tolerance Connectors High-Level APIs Processing Core © 2018 Hazelcast Inc. 33
- 40. Talk is cheap, show me the Word Count Demo Word Count problem is the “Hello, World” in the Land of Stream Processing • Input • Text book in the single file • Stop-list of words to ignore i.e. ”this”, “that”, “of” etc. • Output • Top N word occurrences in the book, saved as key -> value pairs 40© 2018 Hazelcast Inc. https://es.wikiquote.org/wiki/Linus_Torvalds https://github.com/ncherkas/hazelcast-jet-demos
- 41. Key Concepts 41© 2018 Hazelcast Inc.
- 42. Key concepts Distributed Acyclic Graph (DAG) 42© 2018 Hazelcast Inc.
- 43. Key concepts Distributed Acyclic Graph (DAG) 43© 2018 Hazelcast Inc.
- 44. Key concepts Distributed Acyclic Graph (DAG) 44© 2018 Hazelcast Inc.
- 45. Key concepts Distributed Acyclic Graph (DAG) 45© 2018 Hazelcast Inc.
- 46. Key concepts Distributed Acyclic Graph (DAG) 46© 2018 Hazelcast Inc.
- 47. Key concepts Distributed Acyclic Graph (DAG) 47© 2018 Hazelcast Inc.
- 48. Key concepts Jet Cluster 48© 2018 Hazelcast Inc.
- 49. Key concepts Jet Cluster 49© 2018 Hazelcast Inc.
- 50. Key concepts Job Execution 50© 2018 Hazelcast Inc.
- 51. Key concepts Job Execution 51© 2018 Hazelcast Inc.
- 52. Key concepts Job Execution 52© 2018 Hazelcast Inc. Hazelcast Member Hazelcast Member Hazelcast Member
- 53. Jet APIs 53© 2018 Hazelcast Inc.
- 54. Jet APIs • Pipeline API • First choice to use Jet. Build rich data pipelines on a variety of sources and sinks 54© 2018 Hazelcast Inc.
- 55. Jet APIs • Pipeline API • First choice to use Jet. Build rich data pipelines on a variety of sources and sinks • Distributed java.util.stream • Entry-level usage, simple transform-aggregate operations on IMap, JCache and IList 55© 2018 Hazelcast Inc.
- 56. Jet APIs • Pipeline API • First choice to use Jet. Build rich data pipelines on a variety of sources and sinks • Distributed java.util.stream • Entry-level usage, simple transform-aggregate operations on IMap, JCache and IList • Core DAG API • Low-level API for fine-grained tuning and integration 56© 2018 Hazelcast Inc.
- 57. Sources and Sinks 57© 2018 Hazelcast Inc. Resource Infinite? IList ❌ IMap, ICache ❌ Remote IMap, ICache ❌ Event Journal ✅ Remote Event Journal ✅ HDFS ❌ Kafka ✅ Files ❌ File Watcher ✅ TCP Socket ✅ Application Log N/A
- 58. Sources and Sinks 58© 2018 Hazelcast Inc. Resource Infinite? Replyable? Checkpointing? IList ❌ ✅ ❌ IMap, ICache ❌ ✅ ❌ Remote IMap, ICache ❌ ✅ ❌ Event Journal ✅ ✅ ✅ Remote Event Journal ✅ ✅ ✅ HDFS ❌ ✅ ❌ Kafka ✅ ✅ ✅ Files ❌ ✅ ❌ File Watcher ✅ ❌ ❌ TCP Socket ✅ ❌ ❌ Application Log N/A N/A ❌
- 59. Sources and Sinks 59© 2018 Hazelcast Inc. Resource Infinite? Replyable? Checkpointing? Distributed? IList ❌ ✅ ❌ ❌ IMap, ICache ❌ ✅ ❌ ✅ Remote IMap, ICache ❌ ✅ ❌ ✅ Event Journal ✅ ✅ ✅ ✅ Remote Event Journal ✅ ✅ ✅ ✅ HDFS ❌ ✅ ❌ ✅ Kafka ✅ ✅ ✅ ✅ Files ❌ ✅ ❌ ❌ File Watcher ✅ ❌ ❌ ❌ TCP Socket ✅ ❌ ❌ ❌ Application Log N/A N/A ❌ ❌
- 60. Sources and Sinks 60© 2018 Hazelcast Inc. Resource Infinite? Replyable? Checkpointing? Distributed? Data Locality IList ❌ ✅ ❌ ❌ ❌ IMap, ICache ❌ ✅ ❌ ✅ Src ✅ Sink ❌ Remote IMap, ICache ❌ ✅ ❌ ✅ ❌ Event Journal ✅ ✅ ✅ ✅ ❌ Remote Event Journal ✅ ✅ ✅ ✅ ❌ HDFS ❌ ✅ ❌ ✅ ✅ Kafka ✅ ✅ ✅ ✅ ❌ Files ❌ ✅ ❌ ❌ ✅ File Watcher ✅ ❌ ❌ ❌ ✅ TCP Socket ✅ ❌ ❌ ❌ ❌ Application Log N/A N/A ❌ ❌ ✅
- 61. Infinite Stream Processing with Jet 61© 2018 Hazelcast Inc.
- 62. Jet Streaming Demo Flight Telemetry Processing a near real-time Flight Telemetry Stream from ADS-B Exchange - https://www.adsbexchange.com/ 62© 2018 Hazelcast Inc.
- 63. Jet Streaming Demo Flight Telemetry Processing a near real-time Flight Telemetry Stream from ADS-B Exchange - https://www.adsbexchange.com/ • Filter out planes outside of defined airports 63© 2018 Hazelcast Inc.
- 64. Jet Streaming Demo Flight Telemetry Processing a near real-time Flight Telemetry Stream from ADS-B Exchange - https://www.adsbexchange.com/ • Filter out planes outside of defined airports • Detect whether the plane is ascending, descending or staying in the same level 64© 2018 Hazelcast Inc.
- 65. Jet Streaming Demo Flight Telemetry Processing a near real-time Flight Telemetry Stream from ADS-B Exchange - https://www.adsbexchange.com/ • Filter out planes outside of defined airports • Detect whether the plane is ascending, descending or staying in the same level • Based on the plane type and phase of the flight calculate the maximum noise levels nearby to an airport and estimate C02 emissions for a region 65© 2018 Hazelcast Inc. https://github.com/ncherkas/hazelcast-jet-demos
- 66. Jet Streaming Demo Dashboard Pipeline 66© 2018 Hazelcast Inc.
- 67. Jet Streaming Demo Dashboard Pipeline 67© 2018 Hazelcast Inc.
- 68. Jet Streaming Demo Dashboard Pipeline 68© 2018 Hazelcast Inc.
- 69. Jet Streaming Demo Dashboard Pipeline 69© 2018 Hazelcast Inc.
- 70. Jet Streaming Demo Dashboard Pipeline 70© 2018 Hazelcast Inc.
- 71. Jet Streaming Demo Dashboard Pipeline 71© 2018 Hazelcast Inc.
- 72. Jet Streaming Demo Dashboard Pipeline 72© 2018 Hazelcast Inc.
- 73. Jet Streaming Demo Dashboard Pipeline 73© 2018 Hazelcast Inc.
- 74. Jet Streaming Demo Dashboard Pipeline 74© 2018 Hazelcast Inc.
- 75. Pipeline transformations 75© 2018 Hazelcast Inc.
- 76. Pipeline transformations • Time-agnostic transformations • Filter • Map • Flatmap 76© 2018 Hazelcast Inc.
- 77. Pipeline transformations • Time-agnostic transformations • Filter • Map • Flatmap • Aggregation and Grouping • Build-in count, different kind averages, min/max, linear trends and many more 77© 2018 Hazelcast Inc.
- 78. Pipeline transformations • Time-agnostic transformations • Filter • Map • Flatmap • Aggregation and Grouping • Build-in count, different kind averages, min/max, linear trends and many more • Co-Aggregation 78© 2018 Hazelcast Inc.
- 79. Pipeline transformations • Time-agnostic transformations • Filter • Map • Flatmap • Aggregation and Grouping • Build-in count, different kind averages, min/max, linear trends and many more • Co-Aggregation • Hash-Join 79© 2018 Hazelcast Inc.
- 80. Windowing 80© 2018 Hazelcast Inc.
- 81. Windowing 81© 2018 Hazelcast Inc.
- 82. Windowing 82© 2018 Hazelcast Inc.
- 83. Windowing 83© 2018 Hazelcast Inc.
- 84. Windowing Example: 30-second Window Sliding by 10 Seconds 84© 2018 Hazelcast Inc.
- 85. Windowing Example: 30-second Window Sliding by 10 Seconds 85© 2018 Hazelcast Inc.
- 86. Windowing Example: 30-second Window Sliding by 10 Seconds 86© 2018 Hazelcast Inc.
- 87. Windowing Example: 30-second Window Sliding by 10 Seconds 87© 2018 Hazelcast Inc.
- 88. Watermarks to handle Late Events Makes an educated guess that “from this point on there will be no more items with timestamp less than this” 88© 2018 Hazelcast Inc.
- 89. Watermarks to handle Late Events Makes an educated guess that “from this point on there will be no more items with timestamp less than this” 89© 2018 Hazelcast Inc.
- 90. Watermarks to handle Late Events Makes an educated guess that “from this point on there will be no more items with timestamp less than this” 90© 2018 Hazelcast Inc.
- 91. Watermarks in Jet Predefined Watermark Policies • With Fixed Lag • Limiting Lag and Delay • Limiting Lag and Lull • Limiting Timestamp and Wall-Clock Lag 91© 2018 Hazelcast Inc.
- 92. Fault Tolerance 92© 2018 Hazelcast Inc.
- 93. Jet Processing Fault Tolerance Cluster elects a Coordinator Member who takes care of the Job Coordination among the Cluster Members 93© 2018 Hazelcast Inc.
- 94. Jet Processing Fault Tolerance Jet achieves fault tolerance in streaming jobs by making a snapshot of the internal processing state 94© 2018 Hazelcast Inc.
- 95. Jet Processing Fault Tolerance Coordinator Member detects the other Member failure and restarts the Job using new topology 95© 2018 Hazelcast Inc.
- 96. Jet Processing Fault Tolerance When the Coordinator Member crashes the new one is elected by the Cluster 96© 2018 Hazelcast Inc.
- 97. Distributed Snapshots Technique 1st described in a paper by Chandy and Lamport in 1989 97© 2018 Hazelcast Inc.
- 98. Distributed Snapshots Technique 1st described in a paper by Chandy and Lamport in 1989 98© 2018 Hazelcast Inc.
- 99. Distributed Snapshots Technique 1st described in a paper by Chandy and Lamport in 1989 99© 2018 Hazelcast Inc.
- 100. Distributed Snapshots Technique 1st described in a paper by Chandy and Lamport in 1989 100© 2018 Hazelcast Inc.
- 101. Distributed Snapshots Technique 1st described in a paper by Chandy and Lamport in 1989 101© 2018 Hazelcast Inc.
- 102. Jet Processing Guarantees 102© 2018 Hazelcast Inc.
- 103. Jet Processing Guarantees • At-Least Once 103© 2018 Hazelcast Inc.
- 104. Jet Processing Guarantees • At-Least Once • Exactly Once 104© 2018 Hazelcast Inc.
- 105. Jet Processing Guarantees • At-Least Once • Exactly Once • At-Most Once (meaning that the Fault Tolerance is turned off) 105© 2018 Hazelcast Inc.
- 106. Performance 106© 2018 Hazelcast Inc.
- 107. Hazelcast Jet Performance Key Design Decisions 107© 2018 Hazelcast Inc.
- 108. Hazelcast Jet Performance Key Design Decisions • DAG to Model Computations 108© 2018 Hazelcast Inc.
- 109. Hazelcast Jet Performance Key Design Decisions • DAG to Model Computations • In-Memory Data Locality 109© 2018 Hazelcast Inc.
- 110. Hazelcast Jet Performance Key Design Decisions • DAG to Model Computations • In-Memory Data Locality • Partition Mapping Affinity 110© 2018 Hazelcast Inc.
- 111. Hazelcast Jet Performance Key Design Decisions • DAG to Model Computations • In-Memory Data Locality • Partition Mapping Affinity • SP/SC Queues 111© 2018 Hazelcast Inc.
- 112. Hazelcast Jet Performance Key Design Decisions • DAG to Model Computations • In-Memory Data Locality • Partition Mapping Affinity • SP/SC Queues • Cooperative Multithreading (Green Threads) 112© 2018 Hazelcast Inc.
- 113. Jet Streaming Performance 113© 2018 Hazelcast Inc. https://jet.hazelcast.org/performance/
- 114. Jet Throughput 114© 2018 Hazelcast Inc. https://jet.hazelcast.org/performance/
- 115. Running in Production 115© 2018 Hazelcast Inc.
- 116. Running Jet in Production • Docker images - https://github.com/hazelcast/hazelcast-jet-docker 116© 2018 Hazelcast Inc.
- 117. Running Jet in Production • Docker images - https://github.com/hazelcast/hazelcast-jet-docker • Cluster Management: Mesos, Yarn 117© 2018 Hazelcast Inc.
- 118. Running Jet in Production • Docker images - https://github.com/hazelcast/hazelcast-jet-docker • Cluster Management: Mesos, Yarn • Cluster Discovery • Cloud Providers: AWS, Windows Azure, GCP, PCF, Heroku • Kubernetes • Consul, Eureka, Zookeeper 118© 2018 Hazelcast Inc.
- 119. Summary Why you should consider to use the Hazelcast Jet 119© 2018 Hazelcast Inc.
- 120. Summary Why you should consider to use the Hazelcast Jet • High Performance | Industry Leading 120© 2018 Hazelcast Inc.
- 121. Summary Why you should consider to use the Hazelcast Jet • High Performance | Industry Leading • Out-of-box integration with Hazelcast IMDG | Source, Sink, Enrichment 121© 2018 Hazelcast Inc.
- 122. Summary Why you should consider to use the Hazelcast Jet • High Performance | Industry Leading • Out-of-box integration with Hazelcast IMDG | Source, Sink, Enrichment • Easy to start with and integrate | Zero dependencies, developer friendly 122© 2018 Hazelcast Inc.
- 123. Summary Why you should consider to use the Hazelcast Jet • High Performance | Industry Leading • Out-of-box integration with Hazelcast IMDG | Source, Sink, Enrichment • Easy to start with and integrate | Zero dependencies, developer friendly • Simple to deploy | Embedded 10MB jar or Client-Server 123© 2018 Hazelcast Inc.
- 124. Summary Why you should consider to use the Hazelcast Jet • High Performance | Industry Leading • Out-of-box integration with Hazelcast IMDG | Source, Sink, Enrichment • Easy to start with and integrate | Zero dependencies, developer friendly • Simple to deploy | Embedded 10MB jar or Client-Server • Works in every Cloud | Same as Hazelcast IMDG 124© 2018 Hazelcast Inc.
- 125. Summary Why you should consider to use the Hazelcast Jet • High Performance | Industry Leading • Out-of-box integration with Hazelcast IMDG | Source, Sink, Enrichment • Easy to start with and integrate | Zero dependencies, developer friendly • Simple to deploy | Embedded 10MB jar or Client-Server • Works in every Cloud | Same as Hazelcast IMDG • For Developers by Developers | Code it 125© 2018 Hazelcast Inc.
- 126. Questions? Version 0.6 is the current release with 0.7 coming Q3 2018 aiming for 1.0 this year http://jet.hazelcast.org https://groups.google.com/forum/#!forum/hazelcast-jet https://gitter.im/hazelcast/hazelcast 126© 2018 Hazelcast Inc.
Editor's Notes
- TODO: add contacts !!! TODO: what’s written? :)
- TODO: add contacts !!! TODO: what’s written? :)
- !!! Mention about conferences TODO: add contacts !!! TODO: what’s written? :)
- - the answer is that the streaming data [definition of term] is everywhere and it’s usually about … - all these examples of data are generated all the time and usually come with some important real-time insights that require the processing here and now TODO: too much, remove gaming activities
- - fraud detection - alerts generation - variance in throughput -> auto-scaling - disorder -> e.g., a plane full of people taking their phones out of airplane mode after having used them offline for the entire flight - disorder -> producer parallelism and retries – specific to the tools that are used, due to the internals, especially when using batching
- - fraud detection - alerts generation - variance in throughput -> auto-scaling - disorder -> e.g., a plane full of people taking their phones out of airplane mode after having used them offline for the entire flight - disorder -> producer parallelism and retries – specific to the tools that are used, due to the internals, especially when using batching
- - fraud detection - alerts generation - variance in throughput -> auto-scaling - disorder -> e.g., a plane full of people taking their phones out of airplane mode after having used them offline for the entire flight - disorder -> producer parallelism and retries – specific to the tools that are used, due to the internals, especially when using batching
- - fraud detection - alerts generation - variance in throughput -> auto-scaling - disorder -> e.g., a plane full of people taking their phones out of airplane mode after having used them offline for the entire flight - disorder -> producer parallelism and retries – specific to the tools that are used, due to the internals, especially when using batching
- - fraud detection - alerts generation - variance in throughput -> auto-scaling - disorder -> e.g., a plane full of people taking their phones out of airplane mode after having used them offline for the entire flight - disorder -> producer parallelism and retries – specific to the tools that are used, due to the internals, especially when using batching
- - let’s try to understand what is Stream Processing - the key things: on the fly priour to it’s storage, infinite data set in mind, data in motion
- - let’s try to understand what is Stream Processing - the key things: on the fly priour to it’s storage, infinite data set in mind, data in motion
- - the key things: on the fly priour to it’s storage, infinite data set in mind, data in motion
- - the key things: on the fly priour to it’s storage, infinite data set in mind, data in motion
- K-means, Hyper Log Log… How it’s different from classical Batch Processing, when we run periodical jobs to handle our data? TODO: review and maybe come up with own points TODO: combine 1 & 2
- K-means, Hyper Log Log… How it’s different from classical Batch Processing, when we run periodical jobs to handle our data? TODO: review and maybe come up with own points TODO: combine 1 & 2
- TODO: review and maybe come up with own points
- TODO: review and maybe come up with own points
- TODO: review and maybe come up with own points
- 1. Architecturally, stream processing system usually consists of the following 2 layers. 2. Now let’s see how the typical Stream Processing system looks in practice. … 3. Hence, all this doesn’t come for free, there are multiple challenges to solve when you are Processing the Infinite Stream [moving to the next slide].
- - problems: how to form the cluster how to coordinate and how to control the required level of consistency
- - problems: how to form the cluster how to coordinate and how to control the required level of consistency
- - problems: how to form the cluster how to coordinate and how to control the required level of consistency
- - problems: how to form the cluster how to coordinate and how to control the required level of consistency
- - how to solve these problems? - next slide -
- - Hazelcast Jet is one of the products which aim to solve such problem
- Architecturally, Jet consists of the following layers
- TODO: where is DAG API here? Jet Member is also a fully functional Hazelcast IMDG Member and a Jet Cluster is also a Hazelcast IMDG Cluster Hazelcast IMDG provides Layer of cluster management, deployment, data partitioning and networking In-Memory store for Jet Processing state Shared state to connect multiple Jet Jobs Remote data caching Enrichment data source
- TODO: where is DAG API here? Jet Member is also a fully functional Hazelcast IMDG Member and a Jet Cluster is also a Hazelcast IMDG Cluster Hazelcast IMDG provides Layer of cluster management, deployment, data partitioning and networking In-Memory store for Jet Processing state Shared state to connect multiple Jet Jobs Remote data caching Enrichment data source
- TODO: Add a client App and make animations.
- - Hazelcast Jet is one of the products which aim to solve such problem
- TODO: unify orange color among slides! TODO: animation
- TODO: unify orange color among slides! TODO: animation
- TODO: unify orange color among slides! TODO: animation
- TODO: unify orange color among slides! TODO: animation
- TODO: unify orange color among slides! TODO: animation
- TODO: unify orange color among slides! TODO: animation
- Uses Hazelcast IMDG Clustering under the hood Peer-To-Peer communication Members can be either set statically or automatically discovered Elastically scales up or down Topologies Embedded Client-Server
- Uses Hazelcast IMDG Clustering under the hood Peer-To-Peer communication Members can be either set statically or automatically discovered Elastically scales up or down Topologies Embedded Client-Server
- Unit of work described by DAG which is submitted to the cluster for execution Asynchronous, Distributed Submitted to each running member *Scales up/down when adding removing members Embeds JAR with the source code, if needed
- Unit of work described by DAG which is submitted to the cluster for execution Asynchronous, Distributed Submitted to each running member *Scales up/down when adding removing members Embeds JAR with the source code, if needed
- Unit of work described by DAG which is submitted to the cluster for execution Asynchronous, Distributed Submitted to each running member *Scales up/down when adding removing members Embeds JAR with the source code, if needed
- [DON’T MAKE MUCH INTRO???] Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- [DON’T MAKE MUCH INTRO???] Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Automatic dependent surveillance — broadcast (ADS–B) is a surveillance technology in which an aircraft determines its position via satellite navigation and periodically broadcasts it, enabling it to be tracked. The information can be received by air traffic control ground stations as a replacement for secondary surveillance radar, as no interrogation signal is needed from the ground. It can also be received by other aircraft to provide situational awareness and allow self-separation. ADS–B is "automatic" in that it requires no pilot or external input. It is "dependent" in that it depends on data from the aircraft's navigation system.[1]
- Automatic dependent surveillance — broadcast (ADS–B) is a surveillance technology in which an aircraft determines its position via satellite navigation and periodically broadcasts it, enabling it to be tracked. The information can be received by air traffic control ground stations as a replacement for secondary surveillance radar, as no interrogation signal is needed from the ground. It can also be received by other aircraft to provide situational awareness and allow self-separation. ADS–B is "automatic" in that it requires no pilot or external input. It is "dependent" in that it depends on data from the aircraft's navigation system.[1]
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- TODO: more info plus diagram
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- Co-Aggregation – join page visits, user data and payments Context propagation for map, flatMap and filter 2) Aggregation and Grouping Transformation of a set of input values sharing the same distinct key into a single output value Build-in Aggregate Operations for count, different kind avagares, min/max, linear trends and many more Easy to implement own aggregations 3) Co-Aggregation groupBy over the items from more than one contributing stream Like JOIN with the Group By in SQL Typical use case - collecting stats over the user activity coming from the several streams 4) Hash-Join Join of one finite stream with another, possibly infinite stream Optimized for data enrichment - when each item of the primary stream gets enriched with the data resolved by a hashtable lookup To optimize the performance, the entire enriching stream is replicated on each Jet member
- TODO: must be “Event time” on axis - commutative and associative operations
- TODO: must be “Event time” on axis
- TODO: must be “Event time” on axis
- TODO: must be “Event time” on axis
- TODO: must be “Event time” on axis
- TODO: must be “Event time” on axis
- TODO: must be “Event time” on axis
- TODO: Add a client App and make animations.
- TODO: ANIMATIONS!!! TODO: add a final step – when the snapshot completed - due to parallelism, in most cases a processor receives data from more than one upstream processor -
- TODO: ANIMATIONS!!! TODO: add a final step – when the snapshot completed - due to parallelism, in most cases a processor receives data from more than one upstream processor -
- TODO: ANIMATIONS!!! TODO: add a final step – when the snapshot completed - due to parallelism, in most cases a processor receives data from more than one upstream processor -
- TODO: ANIMATIONS!!! TODO: add a final step – when the snapshot completed - due to parallelism, in most cases a processor receives data from more than one upstream processor -
- TODO: ANIMATIONS!!! TODO: add a final step – when the snapshot completed - due to parallelism, in most cases a processor receives data from more than one upstream processor -
- TODO: animations
- TODO: animations
- TODO: animations
- TODO: animations
- Why it’s worth considering Jet for your next stream processing task
- Why it’s worth considering Jet for your next stream processing task
- TODO: Key Competitive Differentiators?
- TODO: Key Competitive Differentiators?
- TODO: Key Competitive Differentiators?
- TODO: Key Competitive Differentiators?
- TODO: Key Competitive Differentiators? Mention that this is an open product, e.g. it’s easy to implement a connector
- TODO: add resources