Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scalable Real-Time Complex Event
Processing @Uber
Shuyi Chen
Uber Technology Inc.
● 6 continents, 70 countries and 400+ cities
● Transportation as reliable as running water, everywhere, for
everyone
Uber
Outline
• Motivation
• Architecture
• Limitations
• Challenges
Outline
• Motivation
• Architecture
• Limitations
• Challenges
Uber is a data-driven company
Thousands of Kafka topics from different services
We can extract a lot of useful information from this rich
set of logs in real-time!
Multiple logins from the same IP in the last 10 minutes
Partner accepted a trip
→ partner calls rider through the Uber APP
→ rider cancels the trip
Partners reject the second pickup of a UberPOOL trip
Multiple logins from the same IP in the last 10 minutes
Window Aggregation
Partner accepted a trip
→ partner calls rider through the Uber APP
→ rider cancels the trip
Pattern detection
Partners reject the second pickup of a UberPOOL trip
Filter
Can we use declarative semantics to specify these stream
processing logics?
Complex event processing
• Combines data from multiple sources to infer events or patterns that
suggest more complicated c...
WSO2/Siddhi: Complex event processing engine
• Lightweight, extensible, open source, released as a Java library
• Features...
How Siddhi works
• Specify processing logic declaratively with SiddhiQL
How Siddhi works
• Query is parsed at runtime into an execution plan runtime
• As events flow in, the execution plan runti...
How can we make it scalable at Uber scale?
Apache Samza
• A distributed stream processing framework
– Distributed and Scalable
– Built-in State management
– Built-in...
How can we make the stream processing output useful?
Actions
• Generalize a set of common action templates to make it easy for
services and human to harness the power of realt...
Actions
Real-time Scalable Complex Event Processing
Outline
• Motivation
• Architecture
• Limitations
• Challenges
Partitioner
• Re-partition events based on key
• Support predicate pushdown through query analysis
• Support column prunin...
Query processor
• Parse Siddhi queries into execution plan runtime
• Process events in Siddhi execution plan runtime
• Che...
Action processor
• Execute actions upon the complex event processing output
• Support various kinds of actions for easy in...
How do we translate a query into psychical plan that
runs?
DAG (Directed Acyclic Graph) generation
• Analyze Siddhi query to automatically generate the stream
processing DAG in Samz...
Join, window, pattern
More complicated
No stream processing logic is hard-coded in any of the
processors
REST API backend
• All queries, actions are stored externally in database.
• RESTFUL API for CRUD operations
• If query/ac...
Unified management and monitoring
• Every use case
– share the same set of processors
– Use queries and actions to describ...
Production status
• 100+ production use cases
• 30+ billion messages processed per day
Applications
• Real-time fraud detection
• Real-time anomaly detection
• Real-time marketing campaign
• Real-time promotio...
Outline
• Motivation
• Architecture
• Limitations
• Challenges
Out-of-order event handling
• Not a big concern
– Events of the same rider/partner are usually seconds aparts
• K-slack ex...
Auto-scaling
• Manually re-partition kafka topics to increase parallelism
• Manually tune container memory if needed
• Fut...
Outline
• Motivation
• Architecture
• Limitations
• Challenges
Large checkpointing state
• Samza use Kafka to log state changes
• Siddhi engine snapshot can be large
• Kafka message siz...
Synchronous checkpointing
• If state is large, time to checkpoint can be long
• Samza uses single-threaded model, unsafe t...
Exactly once state processing?
• Can not commit state and offset atomically
• No exactly once state processing
Custom business logic
• Common logic implemented as Siddhi extensions
• Ad-hoc logic implemented as UDF in javascript or s...
Intermediate Kafka messages
• Samza uses Kafka as message queue for intermediate processing
output
– This can create large...
Multi-tenancy
• Older Siddhi version process events using a thread pool
– Bad for multi-tenancy in YARN
– Consume more CPU...
Upgrading Samza jobs
• Upgrade Samza jobs require a full restart, and can take minutes due
to
– Offset checkpointing topic...
Our solution: non-interrupted handoff
• For critical jobs, we use replication during upgrade
– Start a shadow job
– Upgrad...
Thank You!
WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber
WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber
Upcoming SlideShare
Loading in …5
×

WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber

2,713 views

Published on

The Marketplace data team at Uber has built a scalable complex event processing platform to solve many challenging real-time data needs for various Uber products. This platform has been in production for more than a year and supports over 100 real-time data use cases with a team of 3. In this talk, we will share the detail of the design and our experience, and how we employ Siddhi, Kafka and Samza at scale.

Published in: Technology

WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber

  1. 1. Scalable Real-Time Complex Event Processing @Uber Shuyi Chen Uber Technology Inc.
  2. 2. ● 6 continents, 70 countries and 400+ cities ● Transportation as reliable as running water, everywhere, for everyone Uber
  3. 3. Outline • Motivation • Architecture • Limitations • Challenges
  4. 4. Outline • Motivation • Architecture • Limitations • Challenges
  5. 5. Uber is a data-driven company
  6. 6. Thousands of Kafka topics from different services
  7. 7. We can extract a lot of useful information from this rich set of logs in real-time!
  8. 8. Multiple logins from the same IP in the last 10 minutes
  9. 9. Partner accepted a trip → partner calls rider through the Uber APP → rider cancels the trip
  10. 10. Partners reject the second pickup of a UberPOOL trip
  11. 11. Multiple logins from the same IP in the last 10 minutes Window Aggregation
  12. 12. Partner accepted a trip → partner calls rider through the Uber APP → rider cancels the trip Pattern detection
  13. 13. Partners reject the second pickup of a UberPOOL trip Filter
  14. 14. Can we use declarative semantics to specify these stream processing logics?
  15. 15. Complex event processing • Combines data from multiple sources to infer events or patterns that suggest more complicated circumstances • CEP is used across many industries for various use cases, including: – Finance: Trade analysis, fraud detection – Airlines: Operations monitoring – Healthcare: Claims processing, patient monitoring – Energy and Telecommunications: Outage detection • CEP uses declarative rule/query language to specify event processing logic
  16. 16. WSO2/Siddhi: Complex event processing engine • Lightweight, extensible, open source, released as a Java library • Features supported – Filter – Join – Aggregation – Group by – Window – Pattern processing – Sequence processing – Event tables – Event-time processing – UDF – Extensions – Declarative query language: SiddhiQL
  17. 17. How Siddhi works • Specify processing logic declaratively with SiddhiQL
  18. 18. How Siddhi works • Query is parsed at runtime into an execution plan runtime • As events flow in, the execution plan runtime process events inside the CEP engine according the query logic
  19. 19. How can we make it scalable at Uber scale?
  20. 20. Apache Samza • A distributed stream processing framework – Distributed and Scalable – Built-in State management – Built-in fault tolerant – At-least-once message processing
  21. 21. How can we make the stream processing output useful?
  22. 22. Actions • Generalize a set of common action templates to make it easy for services and human to harness the power of realtime stream processing • Currently we support – Make an RPC call – Invoke a Webhook endpoint – Index to ElasticSearch – Index to Cassandra – Kafka – Statsd – Chat service – Email – Push notification
  23. 23. Actions Real-time Scalable Complex Event Processing
  24. 24. Outline • Motivation • Architecture • Limitations • Challenges
  25. 25. Partitioner • Re-partition events based on key • Support predicate pushdown through query analysis • Support column pruning through query analysis (WIP)
  26. 26. Query processor • Parse Siddhi queries into execution plan runtime • Process events in Siddhi execution plan runtime • Checkpoint state regularly to ensure recovery upon crash/restart using RocksDB
  27. 27. Action processor • Execute actions upon the complex event processing output • Support various kinds of actions for easy integration • Implement action retry mechanism using RocksDB to provide at-least-once delivery
  28. 28. How do we translate a query into psychical plan that runs?
  29. 29. DAG (Directed Acyclic Graph) generation • Analyze Siddhi query to automatically generate the stream processing DAG in Samza using the processors Filter, transformation
  30. 30. Join, window, pattern
  31. 31. More complicated
  32. 32. No stream processing logic is hard-coded in any of the processors
  33. 33. REST API backend • All queries, actions are stored externally in database. • RESTFUL API for CRUD operations • If query/action logic changed – Redeploy the Samza DAG if needed – Otherwise, the updated queries/actions will be loaded at runtime w/o interruption
  34. 34. Unified management and monitoring • Every use case – share the same set of processors – Use queries and actions to describe its processing logic • A single monitoring template can be reused across different use cases
  35. 35. Production status • 100+ production use cases • 30+ billion messages processed per day
  36. 36. Applications • Real-time fraud detection • Real-time anomaly detection • Real-time marketing campaign • Real-time promotion • Real-time monitoring • Real-time feedback system • Real-time analytics • Real-time visualizations • And etc.
  37. 37. Outline • Motivation • Architecture • Limitations • Challenges
  38. 38. Out-of-order event handling • Not a big concern – Events of the same rider/partner are usually seconds aparts • K-slack extension in Siddhi for out-of-order event processing
  39. 39. Auto-scaling • Manually re-partition kafka topics to increase parallelism • Manually tune container memory if needed • Future – Use CPU/memory/IO stats to auto-scale the data pipelines
  40. 40. Outline • Motivation • Architecture • Limitations • Challenges
  41. 41. Large checkpointing state • Samza use Kafka to log state changes • Siddhi engine snapshot can be large • Kafka message size limit to 1MB by default • Solution: we build logics to slice state into smaller pieces and checkpoint them.
  42. 42. Synchronous checkpointing • If state is large, time to checkpoint can be long • Samza uses single-threaded model, unsafe to do it asynchronously (SAMZA-863)
  43. 43. Exactly once state processing? • Can not commit state and offset atomically • No exactly once state processing
  44. 44. Custom business logic • Common logic implemented as Siddhi extensions • Ad-hoc logic implemented as UDF in javascript or scala
  45. 45. Intermediate Kafka messages • Samza uses Kafka as message queue for intermediate processing output – This can create large load on Kafka if a heave topic is partitioned multiple times – Encode the intermediate messages to reduce footprint
  46. 46. Multi-tenancy • Older Siddhi version process events using a thread pool – Bad for multi-tenancy in YARN – Consume more CPU resource than claimed • Newer version still use thread pool for scheduled task, but main processing in single thread – Good: CPU consumption per YARN container is bounded
  47. 47. Upgrading Samza jobs • Upgrade Samza jobs require a full restart, and can take minutes due to – Offset checkpointing topic too large → set retention to hours – Changelog topic too large → set retention or enable compaction in Kafka or host affinity (SAMZA-617) • To minimize the interruption during upgrade, it would be nice to have – Rolling restart – Per container restart
  48. 48. Our solution: non-interrupted handoff • For critical jobs, we use replication during upgrade – Start a shadow job – Upgrade shadow – Switch primary and shadow – Upgrade primary – Switch back • Downside: require 2x capacity during upgrade
  49. 49. Thank You!

×