Uber Engineering discusses the challenges and solutions for near real-time analytics and event processing, emphasizing the use of Apache Kafka and Flink for data management and analytics. The presentation outlines various use cases such as growth metrics, fraud detection, and online analytical processing, all powered by sophisticated SQL and streaming technologies. Looking ahead, Uber aims to explore further advancements in real-time machine learning and auto-scaling within their data ecosystem.

1. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Near Real-Time Analytics
Challenges & Lessons at Uber Engineering

2. U B E R | Data
Chinmay Soman @ChinmaySoman
● Staff Software Engineer @ Uber
● Tech Lead on Streaming Platform
● Background in distributed storage and filesystems
● Apache Samza Committer, PMC
Quick Introduction

3. LAS VEGAS SUMMIT 2015
Billion to Trillions ~ PB
Messages/day bytes/day
Apache Kafka at Uber

4. LAS VEGAS SUMMIT 2015
Billions 100s of TB - PB
Messages Processed / day Bytes Processed / day
Near Real-Time Analytics at Uber

5. U B E R | Data
What is near real-time ?
Decision
Time
Seconds to
5 mins

6. U B E R | Data
Agenda
● Evolution of Business Needs
● The case for SQL as building block
● New ecosystem using Flink
● The road ahead

7. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Evolution of Business Needs

8. U B E R | Data
Case I - Growth Metrics
“How many cars are active
right now ?”
“What % of trips have been
delayed in the last 5 mins ?”
“What is the % of Uber X trips
taken by Android users ?”

9. U B E R | Data
Events logged to Kafka
Rider eyeballs
Trip updates

10. U B E R | Data
Artemis
DatabaseKAFKA
Categorize by time
Aggregate value
5 mins

11. U B E R | Data
KAFKA
Artemis
Database
Delayed
events
Corruption
Backpressure
Duplicate
events

12. U B E R | Data
Artemis
Database
Pre-aggregation
KAFKA
Backfill pipeline
Data Lake

13. U B E R | Data
Artemis
Database
Pre-aggregation
Per dimension
KAFKA Data Explosion

14. U B E R | Data
Apollo
Intelligent Caching
KAFKA
Query Language
Fast Accurate Scalable

15. U B E R | Data
Case II - Event processing
FRAUD
“If # Signups per device look
suspicious -> Ban the driver/rider”

16. U B E R | Data
Case II - Event processing
INTELLIGENT ALERTS
“Send me an alert if a leased
vehicle leaves a geo-fence”

17. U B E R | Data
Athena platform using Apache Samza
Robust
Ease of operation
No backpressure issues
Built in state management

18. U B E R | Data
Fraud Rule
Engine
Track count # of sign_ups
categorized by device_imei
Ban
fraudsters in
real-time
Event processing - Apache Samza
KAFKA

19. U B E R | Data
Case III - OLAP (OnLine Analytical Processing)
A / B Tests
See progress of
tests in real-time

20. U B E R | Data
Case III - OLAP use case
FORECASTING
“How many first time
riders will be dropped off
in a given geofence ?”

21. U B E R | Data
Our integrated platform
● Filter events
● Merge streams
● Decorate with external data

22. U B E R | Data
Are we there yet ?
Artemis
Apollo
Event
Processing
OLAP
?
2014 2015 2016 2017
System
Complexity

23. U B E R | Data
What’s missing ?
● Cumbersome for data scientists / Ops people
● Redundant code
● Custom backfill pipelines

24. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
SQL as the building block

25. U B E R | Data
SQL + Stream Processing
70-80% of jobs can be
implemented via SQL

26. Intelligent Promotions
SQL + Stream Processing: Powerful abstraction
Rule Threshold Action
“All trips worth > 10$
in San Francisco
between Friday 5 pm
and Sunday 9 pm
> 100 “Give bonus of $500”

27. U B E R | Data
Complicated rules
● “If number of hours online > 10 …”
● “If amount earned > 700 in a given week, then …”
● “If # uberPOOL rides >10, then …”
● “If trip happens over some geo-fence 10 times in a given weekend, then …”
SQL + Stream Processing: Powerful abstraction

28. U B E R | Data
Intelligent Promotions
SQL + Stream Processing: Powerful abstraction
> 100 trigger_payment()
select count(*) from hp_api_created_trips
WHERE city_id = 1
AND fare > 10
AND request_at > 1491105600
AND request_at <= 1491177600
Rule Threshold Action

29. U B E R | Data
Complicated rules
● “If number of hours online > 10 …”
● “If amount earned > 700 in a given week, then …”
● “If # Uber Pool rides >10, then …”
● “If trip happens over some geo-fence 10 times in a given weekend, then …”
What if we created specific rules for specific driver partners ?
SQL + Stream Processing: Powerful abstraction

30. U B E R | Data
Can be used for alerts as well:
“If a driver X is outside a geofence, then …”
SQL + Stream Processing: Powerful abstraction

31. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
New eco-system: Athena X

32. U B E R | Data
Enter Flink
Apache Calcite (SQL) Integration
Easy to manage and scale
No backpressure problem
Built in state management support
HDFS integration
Not dependent on Kafka

33. U B E R | Data
Rule Store
KAFKA Database
select count(*) from hp_api_created_trips
WHERE city_id = 1
AND fare > 10
AND request_at > 1491105600
AND request_at <= 1491177600
Promotions using Flink

34. U B E R | Data
Promotions using Flink
Rule Store
DatabaseKAFKA
select count(*) from hp_api_created_trips
WHERE city_id = 1
AND fare > 10
AND request_at > 1491105600
AND request_at <= 1491177600

35. U B E R | Data
New Eco-system: Athena X
HDFS
Kafka
Alerts
Kafka
Streams
Other data
destinations
Database
Streams
HDFS
Cassandra
Rule Store
HTTP

36. U B E R | Data
Are we there yet ?
Artemis
Apollo
Event
Processing
OLAP
?
2014 2015 2016 2017
System
Complexity Flink

37. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
The road ahead ...

38. U B E R | Data
Future Discussions
● To (Apache) Beam or not to Beam?
● Real-time Machine Learning
● Auto scaling

39. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
AthenaX - Flink deep dive
Haohui Mai
Bill Liu
(11:45 am)

40. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Thank you
For more: eng.uber.com
Twitter: @UberEng