Event Streaming in Academia With John Desjardins | Current 2022 A systematic review and meta-analysis of event streaming and real-time data applications using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement were performed from an original inquiry of 540 articles from 12 digital libraries. An independent review of selected studies assessed the risk of bias and obtained data from research and academic studies. The talk will cover the systematic review workflow and obtained results from the academic literature. It will demonstrate best practices of event streaming and real-time applications in academia and research communities using Google Scholar for scholarly literature search. The findings will be made available publicly on Github and arXiv.org

1. © HAZELCAST | CONFIDENTIAL | 1
© HAZELCAST | CONFIDENTIAL | 1
A Systematic Literature
Review and Meta-Analysis of
Event Streaming in Academia
John DesJardins @johmdesjardins
(study by Fawaz Ghali)

2. © HAZELCAST | CONFIDENTIAL | 2
© HAZELCAST | CONFIDENTIAL | 2
Study by
✦ Fawaz Ghali @fawazghali
✦ Developer Advocate at Hazelcast
✦ PhD in Computer Science
✦ +46 peered-review publications
✦ Google Scholar i10-index: 19
✦ Basically, I know this field well.

3. © HAZELCAST | CONFIDENTIAL | 3
© HAZELCAST | CONFIDENTIAL | 3
This picture is derived from Greek
mythology: the blind giant Orion
carried his servant Cedalion on his
shoulders to act as the giant's eyes

4. © HAZELCAST | CONFIDENTIAL | 4
© HAZELCAST | CONFIDENTIAL | 4
Where do you stand?
Standing on the shoulders of giants (Google Scholar)
90% of the Research Literature is useless (said someone)

5. © HAZELCAST | CONFIDENTIAL | 5
© HAZELCAST | CONFIDENTIAL | 5
Literature Research (Framework + Results)
+
Best Practices
=
Success

6. © HAZELCAST | CONFIDENTIAL | 6
© HAZELCAST | CONFIDENTIAL | 6
Part 1: Framework: Preferred Reporting Items for
Systematic Reviews and Meta-Analyses (PRISMA)

7. © HAZELCAST | CONFIDENTIAL | 7
© HAZELCAST | CONFIDENTIAL | 7
Included
Eligibility
Screening
Identification
(stream-
processing)
560 articles
12 Digital
libraries
duplicates
removed
Assessed Qualitative
Further
identified
with ML
Quantitative
Removed if
no real-time
Removed if
before 2021

8. © HAZELCAST | CONFIDENTIAL | 8
© HAZELCAST | CONFIDENTIAL | 8
Part 2: Results
✦ DOI10.5455/jjcit.71-1646838830: A Novel True-real-time Spatiotemporal Data Stream Processing
Framework

9. © HAZELCAST | CONFIDENTIAL | 9
© HAZELCAST | CONFIDENTIAL | 9
Part 2: Results

10. © HAZELCAST | CONFIDENTIAL | 10
© HAZELCAST | CONFIDENTIAL | 10

11. © HAZELCAST | CONFIDENTIAL | 11
© HAZELCAST | CONFIDENTIAL | 11

12. © HAZELCAST | CONFIDENTIAL | 12
© HAZELCAST | CONFIDENTIAL | 12
✦ scalability (Sc), data analytics (DA), multiple event types (MET), prediction tools (PT), data storage
(DS), realtime (Rt), performance evaluation (PE) and stream processing (SP)

13. © HAZELCAST | CONFIDENTIAL | 13
© HAZELCAST | CONFIDENTIAL | 13
A Survey of Distributed Data Stream Processing
Frameworks
✦ DOI: 10.1109/ACCESS.2019.2946884

14. © HAZELCAST | CONFIDENTIAL | 14
© HAZELCAST | CONFIDENTIAL | 14
Quantitative Analysis for SQL over Streaming
0
500
1000
1500
2000
2500
3000
3500
2010-2012 2012-2014 2014-2016 2016-2018 2018-2020 2020-2022
# of publications including citations

15. © HAZELCAST | CONFIDENTIAL | 15
© HAZELCAST | CONFIDENTIAL | 15
Quantitative Analysis for ML over Streaming
To Be Added / Coming Soon!

16. © HAZELCAST | CONFIDENTIAL | 16
© HAZELCAST | CONFIDENTIAL | 16
Part 3: Best Practices
John DesJardins - Background in Streaming Analytics
✦ Software AG - Worked in Advanced Technologies Team -
involved in onboarding/enablement for acquired technologies:
• 2010 - acquired RTM Group - Java based streaming
engine, later launched as webMethods Business Events
• 2013 - acquired Apama Complex Event Processing
✦ Cloudera - 2015 - 2018
• Apache Spark and Kafka Streams
✦ Hazelcast - 2018 to present
• Hazelcast Jet, unified into Hazelcast Platform in 2021
Gigascale Stream Processing Engine
https://hazelcast.com/blog/billion-events-per-second-with-
millisecond-latency-streaming-analytics-at-giga-scale/

17. © HAZELCAST | CONFIDENTIAL | 17
© HAZELCAST | CONFIDENTIAL | 17
Part 3: Best Practices - Streaming Architectures
✦ Do Your Homework - Research on Technologies/Papers, not just Vendors &
Analysts, Hacker News, etc. - but also look for emerging technologies/projects
✦ Favor Modern, Distributed Architectures - Best Suited to Cloud/Cloud-Native
✦ Favor Simpler Architectures - Easier to Operate, More Resilient
✦ Ensure Completeness of Capabilities in Your Architecture, such as:
➢ Stream Processing with Windowing, Joins, etc.
➢ Data Pipelines
➢ Connectors & CDC
➢ SQL for stored and streaming data
➢ Fast, Integrated Storage for Stateful Streaming
➢ Ability to Integrate/Execute ML
✦ Run Your Own Benchmarks & POC - Always “Trust but Verify”

18. Anti-Pattern for Real-time Stream Processing
Data Lake Pattern
✦ Complex Architecture
✦ Long Lag from Data Birth to Value Creation
✦ Often There Are Changes to Data as It Is Consolidated
✦ Doesn’t Support Zero Downtime
✦ Costly to Implement and Operate
Ingest
“Raw”
Refine &
Enrich Cleansed
Real-time
Analytics &
Machine
Learning
Data Lakes
Batch
Loads
Data Is
Born
Action
Taken,
Value
Created

19. Source
Source
Source
Data Processing
Live Events
Analytics
Live Events
Analytics
Live Events
Analytics
Live Events
Analytics
Sink or Client
App
Unified Architecture
Scale Ingest/Data/Compute/ML - Together
Streaming Ingest Queries, Logic &
Machine Learning
Streaming Ingest
Queries, Logic &
Machine Learning
Streaming Ingest
Queries, Logic &
Machine Learning
Streaming Ingest Queries, Logic &
Machine Learning
Event Stream Input
MLOps
Runtimes
Both Compute & Data Are Partitioned. Compute Is Partition Aware.

20. Hazelcast Advantages
✦ Partitioning of Both Compute & Data = Maximizing Parallel & Distributed Architecture
✦ Data-Aware Processing = Data Locality – Less Movement of Data
✦ Collaborative Worksharing = Simplifies Scalability
✦ In-Memory Optimized = Further Drive Down Latency
✦ Simple Peer to Peer Architecture, Cloud-Native
✦ Multi-Region DR, Live Job Upgrades – Do Not Require Restarts – ZERO DOWNTIME
Additional Unique Features:
✦ Easier to Operate across Dev & Ops
✦ Great Developer Experience
✦ Cooperative Multi-threading - Efficiently Uses Multi-Core Processors

21. © HAZELCAST | CONFIDENTIAL | 21
© HAZELCAST | CONFIDENTIAL | 21
Future work
✦ Blog/Paper - with details on this Systemic Literature Review & additional quantitative findings -
Dr. Fawaz Ghali, Hazelcast
✦ Work to get existing paper peer reviewed
• Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile
https://arxiv.org/abs/2103.10169
✦ Additional research and papers - focused on the value of a Unified approach to real-time
stream processing, along with machine learning highlighting:
• Benefits of combining compute with data, as well as partitioned, data aware processing.
⁃ Reducing latency
⁃ Increasing throughput
⁃ Improving resilience
• Work with researchers on applying these technologies to problems in areas such as
healthcare imaging
• Papers around our work on next generation architecture - another order of magnitude

22. © HAZELCAST | CONFIDENTIAL | 22
© HAZELCAST | CONFIDENTIAL | 22
Hazelcast Viridian Serverless
▪ Try Us TODAY!
▪ Create a Cluster in Seconds
▪ Free 2GiB of in-memory storage
▪ Declarative API
▪ Support for streaming SQL
▪ Cross-Region replication capabilities
▪ Maximum throughput and low latency

23. © HAZELCAST | CONFIDENTIAL | 23
© HAZELCAST | CONFIDENTIAL | 23
Join the Hazelcast Developer Community
✦ Slack: Join the Hazelcast Community on slack to chat with users and developers of Hazelcast.
hazelcastcommunity.slack.com
✦ Local events: Speak at or meet the Hazelcast Local community
✦ Twitch live streams: Live sessions, interviews, office hours
https://www.twitch.tv/thehazelcast
✦ Community Blog: Write about your experience or read how other developers use Hazelcast
✦ Hazelcast Heroes: Become a Hazelcast Hero
✦ Email: developer-relations@hazelcast.com
✦ Contribute:
https://github.com/hazelcast

24. © HAZELCAST | CONFIDENTIAL | 24
© HAZELCAST | CONFIDENTIAL | 24
Thank you!
Q&A
Special thanks to the work of Fawaz Ghali –
Developer Advocate
@fawazghali