Kcell, a large telecommunications company in Kazakhstan, implemented a complex event processing platform using Apache Flink to enable real-time processing of millions of user events per second. The system handles SMS, call, data usage, location and other events to trigger personalized notifications, detect fraud and optimize business operations and analytics. Key to the solution was using a single Flink job to run hundreds of dynamic rules via control topics to update triggers, while keeping state local for high performance.

1. Complex Event Processing platform handling
millions of users
Krzysztof Zarzycki - CTO @ Getindata

2. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
About us
Founded in 2014 by
ex-Spotify engineers.
Focus only on Big Data and
Cloud (from day 1)
Community builders (Big
Data Tech Warsaw
organizers)
50+ Big Data engineers

3. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
What is it?
The application logic, analytics, and queries exist continuously, and data flows through them continuously.
Stream Processing

4. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Why is it important for business?
Stream Processing

5. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Actionable insights
Stream Processing

6. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Stream Processing
Why is it important for engineering?

7. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
It’s NOT only about real-time
It’s just natural - data comes continuously.
Stream Processing

8. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Stream Processing
User sessions spanning minutes, hours, or days
Batch boundaries are often artificial.
♪ ♪
♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪
♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪
♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪
[9:00 - 10:00) [10:00-11:00)

9. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Complex Event Processing
● Analyze patterns,relations, cause-and-effect
○ If A & B then C
● Infer business-relevant events from raw technical stream
● .. and cascade extraction of even higher-level events
● Alerts, triggers, workflow automation

10. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Complex Event Processing
● behavioral marketing
● product analytics
● business activity monitoring
● technical monitoring and anomaly detection
● IoT
● fraud detection

11. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
ESP vs CEP
● Difference is blurry and diminishing
● Traditional CEP
○ Complex proc, low latency, single-machine
○ high-level language like SQL
● Traditional ESP
○ straightforward, high-throughput, distributed
○ Broader, more generic and low-level
● NOW: Best of both!
○ Often called “Streaming Analytics”

12. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
The need
● Streaming model and real-time
● On par with batch
○ Enrichment, Joins
○ Aggregation
○ Reprocessing of historical data
○ Machine Learning scoring, inference
○ Complex Event Processing
● large scale, high-throughput
● correctness and fault tolerance

13. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
The solution
Apache Flink
open-source stateful processor over massive data streams

14. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Who uses Flink

15. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
We use Flink!
Banks Telcos Automotive Adtech
Commiters to Flink

16. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Late & Out-of-sequence
events
Breaks correctness.
Often handled with very tedious user code.
Or solved in batch by “waiting enough” and “processing twice”.

17. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Late & Out-of-sequence
events
Handled by Flink in the framework
Based on watermarks heuristics, that marks the progress of event time in the stream.
Asserts that all earlier events have probably arrived.

18. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Local State
Operational state obligatory for analytics
Used for accumulators, windows, source offsets, tracking patterns, ...
6
sum
1
3
2
4
1
1

19. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Local State
Persistent operational state local to computation
Maximize performance with millions of updates per second & core.
Enable out-of-core (more than RAM) processing, with RocksDb
State
Task 1
Logic State
Task N
Logic

20. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Fault tolerance
(Checkpointing)
State survives abrupt crashes or just maintenance
Checkpointed regularly to resilient external storage.
Accurate - keeps stream offsets, accumulators or windows in perfect sync, consistency.
Efficient - almost no impact on the processing.

21. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Flink Cluster

22. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
APIs
Java/Scala
high-level Table API
mid-level dataflow
low-level advanced for tricky cases
Developer Data Scientist Analyst
SQL
Incl. analytical functions
MATCH_RECOGNIZE
and UDF extensibility
Python
Based on Table API

23. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Big picture

24. Assisting Millions of User in Real-Time
Kcell case

25. About Kcell
Kcell has a strong software
development team and lots
of experience in building
services and products
We like innovations
> 10 000 000 subscribers
Largest GSM operator
in Kazakhstan
4G (40%), 3G (73%), 2G
(96%) population
Great network
coverage
There is the ongoing
process of company digital
transformation
Not only telco

26. Business needs
Assisting Millions of Users in Real-Time
SMS events
Voice usage events
Data usage events
Roaming events
Location events
Input Process Actions

27. Use Cases
Use case scenarios. Just few of many.
Case
If subscriber top-ups her balance too often in
short period of time. We can offer her a less
expensive tariff or auto-payment services.
Balance Top Up Case
Trigger UI

28. Roaming
Fraud
Trigger to Marketing Platform if subscriber
visited X country OR/AND registered in Y
visited mobile network and his device's type
is Z
Roaming case
Send an email to the anti-fraud unit if
subscriber registered in roaming but his
balance at the moment is equal to 0.
This situation is impossible in standard case.
Fraud case in roaming

29. Personalized Notifications
Business Automation
Regulatory

30. Future Work
We have already done a lot. But more great things are coming.
2020 Q3 2020 Q4 2021 Q1 Bright Future
More Data Sources
More Triggers
Geolocation data
Equipment logs
Commoditize Machine Learning
Extract value from ML company-wide!
Enable easy ML training and productization
of models in real-time
Real-time BI
Intraday view on business and
operations
Monetize valuable insights from
our combined rich data sources.
Data Monetization
Predictive maintenance
Network Optimization
To lower operational costs
And make better investments
And many more...
Create behavioral profile of the
customers for better
personalised serving
Customer 360 view

31. Old System
Why did we start to look for the new solution?
External Vendor
Solution
Blackbox Solution
Scalability issues
Not reliable
1
2
3
Kcell Developers can’t fix, tweak or optimize it
Limited to ~2000 events / sec
Can’t support all needed data sources
Multiple accidents which took too much time to resolve

32. Scale
Required system throughput
500K
Events / second
10M
Subscribers
40
TB / month

33. New Solution
Real-time Stream Processing
ingestion outgestion
events
hub
events
processing
HTTP
push/pull
FTP
NFS
MQ
HTTP
push/pull
FTP
MQ

34. New Solution
Real-time Stream Processing
flink
ingestion outgestion
events
hub
events
processing
HTTP
push/pull
FTP
NFS
MQ
HTTP
push/pull
FTP
MQ
flink flink

35. New Solution (Operations)
Web UI, Monitoring, Security
flink
ingestion outgestion
events
hub
events
processing
HTTP
push/pull
FTP
NFS
MQ
HTTP
push/pull
FTP
MQ
Admin UI
(Triggers workbench)
Monitoring
Loki - logs
Prometheus/Grafana -
metrics
Security
FreeIPA
Kerberos
LDAP/AD
API (kafka based)
flink flink

36. New Solution (Data Lake)
Data Lake and Sub-second OLAP Analytics
flink
ingestion outgestion
events
hub
events
processing
HTTP
push/pull
FTP
NFS
MQ
HTTP
push/pull
FTP
MQ
Data Lake
Historical Storage (HDFS)
Batch (Spark) SQL (Hive)
Keep history, Report, Explore
Column-oriented
Data store
OLAP (Druid)
Interactive BI
flink flink

37. Processing Flow
Real-time Stream Processing
raw call events
data usage events
transform
transformed events
transform
transformed events
local state
RocksDB
control topic
Admin UI
HTTP
calls
notification
events
outgestion
ingestion
ingestion
submit/stop
triggers

38. Dynamic Rules
Design
Some treats for Squirrels

39. Dynamic Rules Design
Key Points
● We want to run 100s of triggers/business rules
● A typical approach: job per rule
● Won’t work in our case:
○ Run 100s of topologies/jobs = multiplied resources cost
○ Pull data from Kafka 100s of times
○ State (user features) replicated 100s times
○ Starting rule requires deployment of the job

40. Dynamic Rules Design
Key Points
● Our approach: One job to run all triggers/rules
○ And to consume all the sources
● Trigger “templates” still coded with java
● adding/removing rules without restarting application
● 100s of rules running efficiently

41. Dynamic Rules Design
The Overview
billing events
roaming
Sort by time
control topic
notification
events
Deduplicate Router
Late events
Trigger 1
Trigger 2
State
Updater
Apply Triggers (CoProcess Function)
Keyed by User

42. Dynamic Rules Design
Pros and Cons
Shared resources and costs
● CPU, RAM, state, shuffle
● Pulling data from Kafka
One bad rule affects whole system
● Watermarks are shared
● Failures are shared
No job restart on start of new rules
● Rules started by business, no IT
involved
Still need to code rule template in
the job
● No way to use SQL, Table API, CEP
Sharing of state
● Build customer features, that can
be seen by all rules
Can be tricky to debug
● Code is shared
● Code paths enabled externally

43. Dynamic Rules Design
Issue: lagging sources slow down all rules
Source A:
highly unordered, late
Source B:
Ordered, low latency
Late notifications
Low latency
notifications
Triggers
Triggers
Group 1
Triggers
Group 2
Source A:
highly unordered, late
Source B:
Ordered, low latency
Triggers
Group
Late notifications
Problem Solution

44. Flink Changes Wishlist
What could be even better?
attach new branch to existing topology
that receives the same data
Dynamic Topologies
Cheaper topologies
● Graph of topologies that pass
data locally in Flink
● Other words: Local
Proxying/fan-out of Kafka traffic
Share inputs between topologies
Dynamic SQL
SQL
{ }

45. Decisions made
Some decisions our team made before or during project implementation
Streaming-first approach
Apache Kafka for event hub
Apache Flink
Powerful Real-Time Analytics

46. Apache Avro
Keep state local to the process
Ingest reference data for local joins and
enrichment
● No need to query external systems
while processing
● Data time correlation correctness
Performance
transformed
events
transformed
events
Subscriber profile data
(events)
Local State
Not at >100K
events / sec

47. Nifi for data ingestion (no coding)
● but not for CEP
Web UI for configuring triggers
Ease of Use

48. Flink on YARN, with HDFS
HA for redundancy and running ~24/7
Prometheus & Grafana for monitoring &
alerting
Loki for logs collection and aggregation
Reliability and battle-tested techniques
Kerberos and AD thanks to FreeIPA
Apache Ranger for authorization
Security

49. One platform for the whole Enterprise
Batch (adhoc) queries too
● Spark, Hive/Presto
Online analytics
● OLAP
Extensiveness
HDP
Open-source technologies
HDP as a licence-free distribution
Just start with a bunch of servers
Cost-Efficiency

50. Testing
def "should notify when user's balance drops below threshold"() {
given:
BalanceDropTrigger trigger =balanceDropTrigger()
.threshold(
50.0)
.outgestionSystem(
'campaignSystem')
.build()
admin.createsTrigger(trigger)
and:
user.withBalance(
60.0)
when:
user.makesCall(
phoneCall().amountSpent(
20.0))
then:
wait(allowedEventLateness)
and:
List<Notification> actualNotifications =
campaignSystem.getNotifications(
user, trigger)
and:
actualNotifications.size() ==1
assertThat(actualNotifications.first())
.hasMsisdn(
user.msisdn)
.hasBalanceAfter(
40.0)
cleanup:
admin.deletesTrigger(trigger)
}
flink
ingestion outgestion
events
hub
events
processing
Fake
Campaign
System
HTTP
push/pull
FTP
NFS
MQ
Test event generators
Preprod environment

51. Our Collaboration
Two heads are better than one
Joint development team
Not a vendor solution
Development as one team
Code quality
Code review and
automated tools for
code quality control
Agile Practices
Distant geographic
locations, but
everyday standups
Go live quickly!
<4 months to first
production case
running 24/7!
Deliver
DevOps/Automation
Knowledge sharing
Constant knowledge
exchange in areas of
expertise
Testing
Separate testing
environment
Automated Unit/E2E tests

52. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Q&A