"Getting actionable insights from events and addressing common business scenarios usually require bridging together several technologies and programming models to achieve the expected business outcome and bring the flexibility modern applications need. This talks is about how we can concretely wire together Apache Kafka to collect and distribute real-time events, Apache Flink and stream processing to reduce the noise and transform raw events into business events and a rule engine to analyse complex patterns of events, detect business-meaningful situations in context and leveraging business rules to implement the detection logic. Many Apache Flink users in various industries are already combining these technologies to make their event solutions more powerful and more flexible. Built on a fictitious but realistic airline scenario, this talk shows how to technically integrate a rule engine technology, like Redhat Drools, with Kafka and Apache Flink and pinpoint on how stream processing and complex event processing are complementary and different at the same time, and where one need to pay attention to scale the solution to production environment."

1. Stream Processing and Complex Event
Processing together with Kafka, Flink and a
Rule Engine
Stephane Mery
IBM Distinguished Engineer
CTO, Event Integration
IBM Automation
Dale Lane
Senior Technical Staff Member
Chief Architect, Event Integration
IBM Automation

2. © 2024 IBM Corporation
2
If you knew what is going on in
your business right now, what
could you do?
• Deliver an always-on, always up-to-date
and responsive experience to your
Customers
• Provide hyper-personalized services to
your Customers
• Address threats as they occur
• Seize opportunities in the moment
One of the top 5 priorities
In 2023, rapid response is the new
baseline. Uncertainty is expected and
complexity is compounding. As threats
materialize on multiple fronts,
organizations must reduce the time from
insight to action.

3. 3
Perishable insigths have exponentially more value than after-the-fact
analytics
Analytics is done much later
Data is born fast
You have less time to decide
3
© 2023 IBM Corporation
Calls for a new architecture for a new generation of event-driven solutions:
an Event Driven Architecture
It’s a challenge today
3
© 2024 IBM Corporation

4. It often starts with a threat to address or an opportunity to seize… before it’s too late
4
What are the use-cases?
4
Operational
Efficiency
Customer
Service
Fraud
Detection
Increase
Sales
How can you protect customers from malicious actors
right now?
How do you adjust inventory levels to the demand
right now?
What offer should you make to your customer to
trigger a sales right now?
How can you prevent your passengers from missing
their flight right now?
© 2024 IBM Corporation

5. 5
IBM Hursley Americas Summit, October 2023 / © 2023 IBM Corporation
Event Processing is Key
“Event-stream processing is a critical capability of event-driven
architecture. It actively tracks and analyzes continuous streams
of data between producers and consumers of events within and
across enterprises so that change, risk and opportunities can be
proactively identified and interpreted.”
The Event-Driven Enterprise – 451 Research - Jan 2023
• To shape the data the way the consumer
expects it
• To have an always up-to-date view of the data
• To extract insights from events
• To correlate events over time & space to
uncover business situations
© 2024 IBM Corporation

6. Multiple types of Event Processing
Depending on the use-case, choose one paradigm or combine
• Stream Processing Framework
• Direct Acyclic Graph programming model
• Live SQL queries
• Streaming analytics algorithms
• General Event Processing Framework
• Actor-based programming model
• CEP/Rules
• State Management
• External services (predictive/scoring models,
decision services…)
Continuous ETL
Anomaly Detection
Event Transformation
Stream Processing Complex Event Processing
USE-CASES
TECHNOLOGY
Real-time monitoring/Alerting
Event-driven Processes (state machine)
Event pattern detection
Situation detection/correlation
Often piped: SP to prepare, CEP to detect
Raw to Relevant Detect
6
• Few sources
High rate of low value raw events
• Aggregated on short time windows
• Multiple sources
• Lower rate of business events
• Aggregated on long time windows
© 2024 IBM Corporation

7. Typology of use-cases
7
Stream processing
User might want to
transform his stream
of events to generate
more useful events.
Transformation
User might want to
derive insights from
his stream of events
to remove some
noise and generate
more synthetic
events.
Derive insights
Complex Event Processing
User might want to
detect a specific
situation.
Detect situations
User might want to
drive a workflow
based on a state where
transitions are
triggered by an event
Event driven
processes
© 2024 IBM Corporation

8. 8
Programming Models
Stream Processing Complex Event Processing
Stream of events
Virtual Table
Live
Query
Stream of
derived events
SQL
Direct Acyclic Graph
Compiled into
State
Query-based Programming Model Actor-based programming model
Processing logic
Stream of events
Stream of events
Router
State
Actor
Entity
(Customer,
Account…)
Context
(calculated)
External
Sources
of Data
Subscribe
Subscribe
(business) Events
(business) Events
Processing
logic
Timer
Ø Fit well transformation scenarios
Ø Limited state management
Ø Can’t model complex processing context
Ø Flexibility to model processing context
Ø State Management
Ø Support complex logic
Ø Handling of missing events
(derived) Events
(derived) Events
(derived) Events
(raw) Events
Ø Events build-up a situational context
used to detect situations
Stream of events
(business) Events
Subscribe
© 2024 IBM Corporation

9. 9
Business Situation
360 View
of Customer | Passenger | Patient…
(Stateful Context)
Conditions
Event
Pattern
Trends,
anomalies…
Clock
Business
Event
Predictive
Models
Event
Streams
Stream Analytics
Complex Event Processing
(SQL) Filter, transform, enrich…
ML
Rules
Situation Detection Logic (rules)
When a transaction occurs
If
there is at least 3 debit transactions
during the last period of 7 days
and
the account holder risk score is greater than 60%
and
the transaction amount is 50% more than the
average amount spent over the past 3 months
Then
-> potential fraudulent transaction
What makes a Situation?
Reason to Act
© 2024 IBM Corporation

10. Why a Rule Engine for CEP?
10
• A Rule Engine allows to isolate the Business Logic from the application
• A Rule Engine can infer new facts (events) from existing facts (context and events) by applying logical reasoning
• Rule Engines can be used in stateless mode (request-reply) or in stateful mode (efficient update of a context makes
new rules eligible and remember which rules have been triggered already) – RETE algorithm
• Rule Engines are good at “Pattern Matching” (detection of sequence) and are strong on “time-based reasoning”
Complex correlation logic can quickly become a nightmare when implemented in Java or other programming languages
Several options on the market
Operational Decision Manager
RedHat Decision
Manager Open Edition
© 2024 IBM Corporation

11. Integrating a Rule Engine in Apache Flink
11
• 2 technologies, 2 missions
• Flink ingests events, builds a resilient context and maintains the system clock
• The rule engine processes the events and apply the correlation/detection logic
• 2 integration patterns
• Remote call to a Decision Service from Flink
• The rule engine is “embedded” – in-memory invocation of the rule engine from Flink
• 2 Flink integration patterns
• Inside a DataStream Operator
• Using Stateful Functions
© 2024 IBM Corporation

12. Challenges
12
• Performance – need to handle 1000’ of events/sec => roll-out the remote invocation
• Change of the event processing logic – how to update the rules?
• Out-of-order events, missing and delayed events -> often need compensation logic
• Stateful Function? Is it still active?
© 2024 IBM Corporation

13. Integration Patterns
13
Remote call to a rule-based Decision Service
Decision
Service
Decision
Service
Flink Event Processing DAG
Enrich events
with a decision
Apply
correlation
logic
Business
rules
Business
rules
• Business logic is isolated
• Easy to manage Rule logic lifecycle independent from
the event processing flow
• Only work with stateless rules (context has to be
maintained in Flink and sent at every call)
• Performance cost if event rate is high
Embedding a rule engine in Flink
RichFlatMapFunction
Rule Engine
(jar)
Flink Event Processing DAG
Business
rules
(binary)
• Business logic is isolated
• Rule logic lifecycle independent from the event processing flow
• Embedded in the Flink Application
• Or sent/updated with a Broadcast Stream
• State shared between Flink & the Rule engine
• No performance impact
State management
Rule invocation
Update State
Send outbound Events
Flink
State
© 2024 IBM Corporation

14. Integration Patterns
14
Using Stateful Function
Fligh #777 Agent
(Stateful Function)
Rule Engine
(jar)
Flink Event Processing DAG
Business
rules
(binary)
State management
Rule invocation
Update State
Send outbound Events
Router
Passenger “John Li”
Agent
(Stateful Function)
Rule Engine
(jar)
Business
rules
(binary)
State management
Rule invocation
Update State
Send outbound Events
Stateful Functions (embedded mode) Flink Event
Processing DAG
• Business logic is isolated
• Agent-based Programming Model
• Per entity subscription
• Allow more complex scenarios
• Inter-SF communication
• Loop
• Manage Rule logic lifecycle independent
from the event processing flow
• Update rules by deploying new SF
• Send rules within an Event
• Shared state between the SF and the rule
engine
• No performance impact
• What’s happening to Stateful Functions?
DataStream -> SF
bridge
SF -> DataStream
bridge
Flink
State
Flink
State
Event
Transformation,
Enrichment,
Aggregation
Event
Correlation
Event
Transformation
© 2024 IBM Corporation

15. Integration Patterns
Code – Rule Engine embedded in a DataStream Operator
RichFlatMapFunction to
process events in a stream
State to keep context (events
and other information derived
from the rules)
Get the rule engine (IBM ODM)
Prepare context to send to the
rule engine
Invoke the rule engine
Get back an updated context
and outbound events to send
Store the new context
Note: if you want to use Timers (to
detect missing events for instance) you
need to use a KeyedProcessFunction
© 2024 IBM Corporation

16. 16
What IBM is doing in this space?
Put business event to work by enabling users to detect situations, act in real-
time, automate decisions, and maximize their revenue potential
Event Endpoint Management
Build a self-service catalog of event
sources for users to securely browse
and utilize
Event Stream
Collect streams of real-time business
events with enterprise-grade Apache
Kafka
Event Processing
Define business situations in an
intuitive, easy-to-use authoring canvas
in order to act in real-time and
automate decisions
Composable set of capabilities
IBM Event Automation
16
© 2024 IBM Corporation

17. Kafka Connect
Data
Source
Event Distribution
Event
Gateway
SQL Job SQL Job
Stream Processing Engine
(Apache Flink Cluster)
Event Management Event Processing
Low Code Editor
Endpoint Catalog
Deploy
17
IBM Event Automation Architecture
© 2024 IBM Corporation

18. With decision automation:
− Business logic is accessible, consistent, auditable
− Interactions can be personalized to provide richer,
more seamless experiences
− Decisions can be changed with no down time in
seconds, not months
− Fewer errors and faster time to market as decision
services are independent of apps
− Isolates business logic to allow quick
updates when conditions change without
impact to surrounding systems
− Empowers business users to create and
update policies with business-friendly
language, models and tables
− Combines analytical insights with business
rules using machine learning to anticipate
changing conditions
− Surfaces decision history, to allow for
straightforward, easy tracking
IBM Automation © 2023 IBM Corporation
What IBM is doing in this space?
IBM Decision Automation
IBM Operational Decision Manager
IBM Decision Manager Open Edition (Drools Open Source)
© 2024 IBM Corporation

19. 19
Wrap-up
ü Becoming an Event-Driven Enterprise makes the
enterprise more agile and responsive
ü Event Processing is key. Not all processing are the same
ü Complex Event Processing is better achieved with a Rule
Engine
ü Flink + Rule Engines can seamlessly integrate
“Virtually all key performance indicators can be
improved when change, risk and opportunity are
quickly sensed (and predicted) and effectively
responded to. Enhanced business outcomes can be
achieved by strengthening the sense-and-respond
reflexes of an organization and investing in
technologies that enable the event-driven enterprise.”
The Event-Driven Enterprise
451 Research – Jan 2023
19
© 2024 IBM Corporation

20. Thank you
© Copyright IBM Corporation 2023. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any
kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and
[insert other IBM trademarks listed on the IBM Trademarks List—and use serial commas], are trademarks or registered trademarks of International Business Machines Corporation, in the
United States and/or other countries. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on ibm.com/trademark.
20
Stephane Mery
IBM Distinguished Engineer
CTO, Event Integration
—
stephane.mery@fr.ibm.com
ibm.com
Dale Lane
STSM
Chief Architect, Event Integration
Dale.Lane@uk.ibm.com
© 2024 IBM Corporation