Apache Ignite is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time. But, did you know it provides streaming and complex event processing (CEP)? In this hands-on demonstration we will take Apache Ignite’s Streaming and CEP features for a test drive. We will start with an example streaming use case then demonstrate how to implement each component in Apache Ignite. Finally we will show how to connect a dashboard application to Apache Ignite to display the results.

1. TEST DRIVING STREAMING AND CEP
ON APACHE IGNITE
MATT COVENTON
See all the presentations from the In-Memory Computing
Summit at http://imcsummit.org

2. ABOUT ME
 Big Data Services Lead at Innovative Software Engineering
 http://www.iseinc.biz
 mattcoventon@iseinc.biz

3. WHAT ARE WE GOING TO DO TODAY?
 An Overview of Apache Ignite Streaming and CEP
 Dive into some code!
 A simple streaming/CEP use case

4. APACHE IGNITE
STREAMING AND
CEP
OVERVIEW

5. WHAT IS STREAMING?
 Most commonly, streaming refers to processing unbounded data sets as they arrive to achieve lower
latency and therefore more timely results.
 If you haven’t already, read these helpful posts that clarify the terms, techniques, and design patterns:
 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102

6. WHAT IS CEP?
 Complex event processing, or CEP, is event processing that combines data from multiple sources to
infer events or patterns that suggest more complicated circumstances. The goal of complex event
processing is to identify meaningful events (such as opportunities or threats) and respond to them as
quickly as possible (https://en.wikipedia.org/wiki/Complex_event_processing)

7. IN THE APACHE IGNITE CONTEXT
 Apache Ignite In-Memory Data Fabric is a high-performance, integrated and distributed in-memory
platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster
than possible with traditional disk-based or flash technologies.

8. APACHE IGNITE STREAMING
 Primarily a high performance means of
inserting unbounded data sets into the Ignite
Data Grid (cache) using IgniteDataStreamer
API
 StreamReceiver API offers custom pre-
processing
 Other data processing through queries
(including continuous queries) and cache
policies
 Backed by all kinds of Ignite goodness:
 Scalable
 Fault-tolerant
 High throughput
 Streaming functionality atop a convergent
data platform – the future is bright!

9. IGNITE DATA STREAMER API
IgniteData
Streamer
MQTT
Streamer
Kafka
Streamer
Camel
Streamer
Stream
Receiver
Stream
Transformer
Stream
Visitor
JMS Data
Streamer
Other...

10. IGNITE DATA STREAMER API
 IgniteDataStreamer API is the basic building block to writing unbounded data to Ignite
 Scalable
 Fault-tolerant
 At-least-once-guarantee (watch out for duplicate data)
 Buffers data and writes in batches (may introduce unwanted latency, set perNodeBufferSize() and
autoFlushFrequency() accordingly)

11. STREAM RECEIVER API
 StreamReceiver API allows you to add custom, collocated pre-processing of the streaming data prior to
putting it into the cache.
 Does not put data into the cache automatically, you need to handle that during processing
 Single receiver per IgniteDataStreamer
 Two out of the box implementation of StreamReceiver
 StreamTransformer updates data in the stream cache based on its previous value
 StreamVisitor visits every key-value tuple in the stream
 Might be possible to implement watermark, trigger, accumulation patterns (depending on use case, see
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102)

12. WINDOWING
 Achieved through cache eviction and expiry policies
 Use eviction policies for size/batch based
 Consider SortedEvictionPolicy with custom comparator for “x most recent events”
 Use expiry policies for time based
 Consider notion of event time, ingestion time, and processing time
 CreatedExpiryPolicy is ingestion time based
 What if data is delayed?
 Consider a custom expiry policy based on event time

13. QUERYING
 All Ignite data indexing capabilities as well as Ignite SQL, TEXT, and Predicate based cache queries are
available (it’s just another cache after all)
 Leverage continuous queries to filter events on the node and receive real-time notifications that match
your criteria
 Another option to implement watermark, trigger, and accumulation patterns
 This is where the complex event processing (CEP) magic happens leveraging distributed joins and
cross-cache joins

14. DIVE INTO SOME
CODE
A SIMPLE IOT USE CASE

15. A SIMPLE IOT USE CASE
 Monitor productivity on manufacturing lines
 Sensors stream number of items per second through IgniteDataStreamer
 Data is retained in the cache for 60 seconds (windowing)
 Dashboard shows number of items per minute for each active line and the total items
per minute for the entire factory
1
2
n
Ignite Data
Streamer
Ignite Cache
3
Dashboard

16. IGNITE POM DEPENDENCIES

17. PRODUCTION LINE EVENT

18. CACHE CONFIG

19. MONITORING APPLICATION MAIN CLASS

20. MONITORING APPLICATION REST CONTROLLER

21. STARTING THE IGNITE NODE

22. H2 DEBUG CONSOLE WITH ONE LINE REPORTING

23. DASHBOARD

24. THANK YOU!