- The document discusses Uber's use of stream processing to enable real-time analytics and complex event processing over streaming data from its global ridesharing marketplace. - Key applications include real-time OLAP, detecting patterns in event streams, and supply positioning to monitor marketplace health. - The architecture uses Apache Kafka for event collection, Apache Samza for event processing, and storage and visualization applications. It addresses challenges of processing large-scale, real-time geo-temporal data streams.

1. STREAM
PROCESSINGIN
UBERMARKETPLACE

2. ~ 68 countries / 350+ cities
Transportation as reliable as running
water, everywhere, for everyone
2

3. Agenda
What’s on the menu?
•Use Cases
•Problem Space
•Overall Architecture
•Choices & Tradeoffs
•Q & A

4. Use Case: Realtime OLAP

5. There is always need for quick exploration

6. How many open cars in the world, NOW?

8. How many UberXs were driving clients in SF in the past 10
minutes by hexagons?

9. How many UberXs were driving clients in SF in the past 10 minutes by hexagons?

10. Driving time and other metrics over time by hexagonal area

12. Use Case: Complex Event Processing

13. There are patterns in event streams

14. How many drivers cancel requests
more than 3 times in a row within a 10-
minute window?

15. Report riders requesting a pickup 100 miles
apart within a half hour window?

16. IF
This —>
Then that —>
● Sigma is similar - but for offline/batch applications
Complex Event Processing

17. Use Case: Supply Positioning

18. Clusters Of Supply & Demand

19. Predicted Health Metrics
Actual Health Metrics
Monitor Marketplace Health

20. Challenges

21. OLAP of Geo-spatial Temporal Data
Reasonably Large Scale
Near Real Time

22. • Indexing, Lookup, Rendering
• Symmetric Neighbors
• Convex & Compact Regions
• Equal Areas
• Equal Shape
Hexagons

23. Scale
Geo Space Vehicle Types Time Status
X X X

24. Granular Geo Areas

25. Granular Geo Areas
Over 10,000 hexagons in a city

26. Multiple Vehicle Types
7 vehicle types

27. Minute-level Time Buckets
1440 minutes in a day

28. Many Driver States
13 driver states

29. Many Cities
300 cities

30. Granular Data
1 day of data: 300 x 10,000 x 7 x 1440 x 13 = 393 billion
possible combinations

31. Unknown Query Patterns
Any combination of dimensions

32. Variety of Aggregations
- Heatmap
- Top N
- Histogram
- count(), avg(), sum(), percent(), geo

33. Large Data Volume
• Hundreds of thousands of
events per second
• At least dozens of fields in
each event

34. Multiple Topics
Rider States Driver States

35. Let’s build a stream processing pipeline

36. Pipeline Template

37. Event Collection

38. Multiple Event Types with Different Volume

39. Hundreds of Thousands of Events Per Second

40. Events Should Be Available Under a Second

41. Events Should Rarely Get Lost

42. Multiple Consumers

44. Natural Choice: Apache Kafka
- Low latency and high throughput
- Persistent events
- Distributes a topic by partitions
- Groups consumers by consumer groups

46. Event Processing

47. Transformation

48. Event Transformation Example
(Lat, Long) -> (zipcode, hexagon, S2)

49. Pre-aggregation

50. Joining Multiple Streams

51. Sessionization

52. Multi-Staged Processing

53. Minimum Requirements
- Statement Management
- Checkpointing
- Automatic Resource Management
- Multi-staged processing

54. Apache Samza

55. Why Apache Samza?
- DAG on Kafka
- Excellent integration with Kafka
- Built-in checkpointing
- Built-in state management
- Excellent support from our data team

56. Samza Is Conceptually Simple

57. IF
This —>
Then that —>
● Sigma is similar - but for offline/batch applications
Complex Event Processing

58. ● Sigma is similar - but for offline/batch applications
Complex Event Processing

59. ● Sigma is similar - but for offline/batch applications
Complex Event Processing

60. ● Sigma is similar - but for offline/batch applications
Complex Event Processing

61. ● Sigma is similar - but for offline/batch applications
Complex Event Processing

62. ● Sigma is similar - but for offline/batch applications
Slightly Expanded Version

63. ● Sigma is similar - but for offline/batch applications
Slightly Expanded Version

64. ● Sigma is similar - but for offline/batch applications
Slightly Expanded Version

65. ● Sigma is similar - but for offline/batch applications
Slightly Expanded Version

67. Applications

68. Dashboard of Realtime Business Metrics

69. Ad-Hoc Queries

70. Visualization with Streaming

71. Visualization with Streaming
LocationUpdate where city = X
LocationUpdate
where city = Y
and vehicle = ‘UberX’
100%
100%
100%
10%
5%

72. Visualization with Streaming
LocationUpdate where city = X
LocationUpdate
where city = Y
and vehicle = ‘UberX’
100%
100%
100%
10%
5%

73. Visualization with Streaming
LocationUpdate where city = X
LocationUpdate
where city = Y
and vehicle = ‘UberX’
100%
100%
100%
10%
5%

74. Visualization with Streaming
LocationUpdate where city = X
LocationUpdate
where city = Y
and vehicle = ‘UberX’
100%
100%
100%
10%
5%

75. Visualization with Streaming
LocationUpdate where city = X
LocationUpdate
where city = Y
and vehicle = ‘UberX’
100%
100%
100%
10%
5%

76. Visualization with Streaming
LocationUpdate
where city = ‘SF’
LocationUpdate
where city = ‘LA’
and vehicle
10%
5%
100% 100%

77. Ad-hoc Exploration

78. A Few Trade-Offs

79. Lambda vs Kappa

80. We Use Lambda
- Spark + HDFS/S3 for batch processing
- Yes, it is painful, but
- We may need to go way back due to change of business
requirements
- Batch process can run faster — they scale differently
- It was not easy to start a new stream processing instance

81. Processing by Event Time Is Not Always Easy

82. Leverage The Storage Layer

83. Dealing with Limitation of Samza
-No broadcasting. We have to override
SystemStreamPartitionGrouper
-No dynamic topology. Can’t have arbitrary number of
nested CEP queries
-Tedious configuration and deployment of jobs. In house
code-gem and deployment solution

84. Thank You