Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RBea: Scalable Real-Time Analytics at King

507 views

Published on

This talk introduces RBEA (Rule-Based Event Aggregator), the scalable real-time analytics platform developed by King’s Streaming Platform team. We have built RBEA to make real-time analytics easily accessible to game teams across King without having to worry about operational details. RBEA is built on top of Apache Flink and uses the framework’s capabilities to it’s full potential in order to provide highly scalable stateful and windowed processing logic for the analytics applications. We will talk about how we have built a high-level DSL on the abstractions provided by Flink and how we tackled different technical challenges that have come up while developing the system.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

RBea: Scalable Real-Time Analytics at King

  1. 1. © King.com Ltd 2016 – Commercially confidential RBea: Scalable Real-Time Analytics at King Gyula Fóra Data Warehouse Engineer Apache Flink PMC Page 2
  2. 2. © King.com Ltd 2016 – Commercially confidential We make awesome mobile games 463 million monthly active users 30 billion events per day And a lot of data… Page 3 About King
  3. 3. © King.com Ltd 2016 – Commercially confidential From streaming perspective… Page 4 DB 30 billion events / day Complex processing logic Terabytes of state DB DB
  4. 4. © King.com Ltd 2016 – Commercially confidential How do we use Flink Page 5 Standalone deployment Few heavy streaming jobs RocksDB state backend with caching Lot of custom tooling
  5. 5. © King.com Ltd 2016 – Commercially confidential The RBea platform Page 6 Scripting on the live streams Deploy from browser/python Window aggregates Stateful computations Scalable + fault tolerant
  6. 6. © King.com Ltd 2016 – Commercially confidential RBea architecture Page 7 Events Output REST API RBEA web frontend Libraries http://hpc-asia.com/wp-content/uploads/2015/09/World-Class-Consultancy-Seeking-Data-Scientist-CA-Hobson-Associates-Matthew-Abel-Recruiter.jpg Data Scientists
  7. 7. © King.com Ltd 2016 – Commercially confidential RBea backend implementation Page 8 One stateful Flink job / game Stream events and scripts Events are partitioned by user id Scripts are broadcasted Output/Aggregation happens downstream S1 S2 S3 S4 S5 Add/Remove scripts Event stream Loop over deployed scripts and process CoFlatMap Output based on API calls
  8. 8. © King.com Ltd 2016 – Commercially confidential Dissecting the DSL Page 9 @ProcessEvent(semanticClass=SCPurchase.class) def process(SCPurchase purchase, Output out, Aggregators agg) { long amount = purchase.getAmount() String curr = purchase.getCurrency() out.writeToKafka("purchases", curr + "t" + amount) Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10) numPurchases.increment() }
  9. 9. © King.com Ltd 2016 – Commercially confidential Dissecting the DSL Page 10 @ProcessEvent(semanticClass=SCPurchase.class) def process(SCPurchase purchase, Output out, Aggregators agg) { long amount = purchase.getAmount() String curr = purchase.getCurrency() out.writeToKafka("purchases", curr + "t" + amount) Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10) numPurchases.increment() } Processing methods by annotation Event filter conditions Flexible argument list Code-generate Java classes => void processEvent(Event e, Context ctx);
  10. 10. © King.com Ltd 2016 – Commercially confidential Dissecting the DSL Page 11 @ProcessEvent(semanticClass=SCPurchase.class) def process(SCPurchase purchase, Output out, Aggregators agg) { long amount = purchase.getAmount() String curr = purchase.getCurrency() out.writeToKafka("purchases", curr + "t" + amount) Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10) numPurchases.increment() } Output calls create Output events Output(KAFKA, “purchases”, “…” ) These events are filtered downstream and sent to a Kafka sink
  11. 11. © King.com Ltd 2016 – Commercially confidential Dissecting the DSL Page 12 @ProcessEvent(semanticClass=SCPurchase.class) def process(SCPurchase purchase, Output out, Aggregators agg) { long amount = purchase.getAmount() String curr = purchase.getCurrency() out.writeToKafka("purchases", curr + "t" + amount) Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10) numPurchases.increment() } Aggregator calls create Aggregate events Aggr (MYSQL, 60000, “PurchaseCount”, 1) Flink window operators do the aggregation
  12. 12. © King.com Ltd 2016 – Commercially confidential Aggregators Page 13 long size = aggregate.getWindowSize(); long start = timestamp - (timestamp % size); long end = start + size; TimeWindow tw = new TimeWindow(start, end); Event time windows Window size / aggregator Script1 Script2 Window 1Window 2 NumGames Revenue W1: 8999 W2: 9001 W1: 200 W2: 300 MyAggregator W1: 10 W2: 5 Dynamic window assignment
  13. 13. © King.com Ltd 2016 – Commercially confidential User states Page 14 RBea Script LAST_GAMESTART LAST_PURCHASE LAST_… + User defined states Fields byte[] User State byte[]byte[] LRU Cache Scripts can read all state But write only their own
  14. 14. © King.com Ltd 2016 – Commercially confidential Things you might wonder… Page 15 Can slow scripts affect other scripts? Yes, but we are working on it Separate test/live environments What does the backend know about the scripts? Outputs produced Failures (causes) => these are propagated Runtime stats in the future Is RBea useful compared to custom Flink jobs? Writing + maintaining streaming jobs is hard Especially with state, windowing, MySQL etc.
  15. 15. © King.com Ltd 2016 – Commercially confidential Page 16 RBea physical plan
  16. 16. © King.com Ltd 2016 – Commercially confidential Wrap up Page 17 RBea makes streaming accessible to every data scientist at King We leverage Flink’s stateful and windowed processing capabilities People love it because it’s simple and powerful
  17. 17. Thank you!

×