This talk introduces RBEA (Rule-Based Event Aggregator), the scalable real-time analytics platform developed by King’s Streaming Platform team. We have built RBEA to make real-time analytics easily accessible to game teams across King without having to worry about operational details. RBEA is built on top of Apache Flink and uses the framework’s capabilities to it’s full potential in order to provide highly scalable stateful and windowed processing logic for the analytics applications. We will talk about how we have built a high-level DSL on the abstractions provided by Flink and how we tackled different technical challenges that have come up while developing the system.

2. © King.com Ltd 2016 – Commercially confidential
RBea: Scalable Real-Time Analytics
at King
Gyula Fóra
Data Warehouse Engineer
Apache Flink PMC
Page 2

3. © King.com Ltd 2016 – Commercially confidential
We make awesome mobile games
463 million monthly active users
30 billion events per day
And a lot of data…
Page 3
About King

4. © King.com Ltd 2016 – Commercially confidential
From streaming perspective…
Page 4
DB
30 billion events / day
Complex processing logic
Terabytes of state
DB
DB

5. © King.com Ltd 2016 – Commercially confidential
How do we use Flink
Page 5
Standalone deployment
Few heavy streaming jobs
RocksDB state backend with caching
Lot of custom tooling

6. © King.com Ltd 2016 – Commercially confidential
The RBea platform
Page 6
Scripting on the live streams
Deploy from browser/python
Window aggregates
Stateful computations
Scalable + fault tolerant

7. © King.com Ltd 2016 – Commercially confidential
RBea architecture
Page 7
Events Output
REST API
RBEA web frontend
Libraries
http://hpc-asia.com/wp-content/uploads/2015/09/World-Class-Consultancy-Seeking-Data-Scientist-CA-Hobson-Associates-Matthew-Abel-Recruiter.jpg
Data Scientists

8. © King.com Ltd 2016 – Commercially confidential
RBea backend implementation
Page 8
One stateful Flink job / game
Stream events and scripts
Events are partitioned by user id
Scripts are broadcasted
Output/Aggregation happens downstream
S1 S2
S3
S4
S5
Add/Remove scripts
Event stream Loop over deployed
scripts and process
CoFlatMap
Output based
on API calls

9. © King.com Ltd 2016 – Commercially confidential
Dissecting the DSL
Page 9
@ProcessEvent(semanticClass=SCPurchase.class)
def process(SCPurchase purchase,
Output out,
Aggregators agg) {
long amount = purchase.getAmount()
String curr = purchase.getCurrency()
out.writeToKafka("purchases", curr + "t" + amount)
Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10)
numPurchases.increment()
}

10. © King.com Ltd 2016 – Commercially confidential
Dissecting the DSL
Page 10
@ProcessEvent(semanticClass=SCPurchase.class)
def process(SCPurchase purchase,
Output out,
Aggregators agg) {
long amount = purchase.getAmount()
String curr = purchase.getCurrency()
out.writeToKafka("purchases", curr + "t" + amount)
Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10)
numPurchases.increment()
}
Processing methods by annotation
Event filter conditions
Flexible argument list
Code-generate Java classes
=> void processEvent(Event e, Context ctx);

11. © King.com Ltd 2016 – Commercially confidential
Dissecting the DSL
Page 11
@ProcessEvent(semanticClass=SCPurchase.class)
def process(SCPurchase purchase,
Output out,
Aggregators agg) {
long amount = purchase.getAmount()
String curr = purchase.getCurrency()
out.writeToKafka("purchases", curr + "t" + amount)
Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10)
numPurchases.increment()
}
Output calls create Output events
Output(KAFKA, “purchases”, “…” )
These events are filtered downstream and
sent to a Kafka sink

12. © King.com Ltd 2016 – Commercially confidential
Dissecting the DSL
Page 12
@ProcessEvent(semanticClass=SCPurchase.class)
def process(SCPurchase purchase,
Output out,
Aggregators agg) {
long amount = purchase.getAmount()
String curr = purchase.getCurrency()
out.writeToKafka("purchases", curr + "t" + amount)
Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10)
numPurchases.increment()
}
Aggregator calls create Aggregate events
Aggr (MYSQL, 60000, “PurchaseCount”, 1)
Flink window operators do the aggregation

13. © King.com Ltd 2016 – Commercially confidential
Aggregators
Page 13
long size = aggregate.getWindowSize();
long start = timestamp - (timestamp % size);
long end = start + size;
TimeWindow tw = new TimeWindow(start, end);
Event time windows
Window size / aggregator
Script1
Script2
Window 1Window 2 NumGames
Revenue
W1: 8999
W2: 9001
W1: 200
W2: 300
MyAggregator
W1: 10
W2: 5
Dynamic window assignment

14. © King.com Ltd 2016 – Commercially confidential
User states
Page 14
RBea
Script
LAST_GAMESTART
LAST_PURCHASE
LAST_…
+ User defined states
Fields
byte[]
User State
byte[]byte[]
LRU Cache
Scripts can read all state
But write only their own

15. © King.com Ltd 2016 – Commercially confidential
Things you might wonder…
Page 15
Can slow scripts affect other scripts?
Yes, but we are working on it
Separate test/live environments
What does the backend know about the scripts?
Outputs produced
Failures (causes) => these are propagated
Runtime stats in the future
Is RBea useful compared to custom Flink jobs?
Writing + maintaining streaming jobs is hard
Especially with state, windowing, MySQL etc.

16. © King.com Ltd 2016 – Commercially confidential Page 16
RBea physical plan

17. © King.com Ltd 2016 – Commercially confidential
Wrap up
Page 17
RBea makes streaming accessible to every
data scientist at King
We leverage Flink’s stateful and windowed
processing capabilities
People love it because it’s simple and
powerful

18. Thank you!