Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Uber Real Time Data Analytics

1,680 views

Published on

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder.
Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Published in: Engineering

Uber Real Time Data Analytics

  1. 1. Real Time Data Analytics @ UberAnkur Bansal Apache Big Data Europe November 14, 2016
  2. 2. About Me ● Sr. Software Engineer, Streaming Team @ Uber ○ Streaming team supports platform for real time data analytics: Kafka, Samza, Flink, Pinot.. and plenty more ○ Focused on scaling Kafka at Uber’s pace ● Staff software Engineer @ Ebay ○ Build & scale Ebay’s cloud using openstack ● Apache Kylin: Committer, Emeritus PMC
  3. 3. Agenda ● Real time Use Cases ● Kafka Infrastructure Deep Dive ● Our own Development: ○ Rest Proxy & Clients ○ Local Agent ○ uReplicator (Mirrormaker) ○ Chaperone (Auditing) ● Operations/Tooling
  4. 4. Important Use Cases
  5. 5. Stream Processing Real-time Price Surging SURGE MULTIPLIERS Rider eyeballs Open car information KAFKA
  6. 6. Real-time Machine Learning - UberEats ETD
  7. 7. ● Fraud detection ● Share my ETA And many more ...
  8. 8. Apache Kafka is Uber’s Lifeline
  9. 9. Kafka ecosystem @ Uber
  10. 10. 100s of billion 100s TB Messages/day bytes/day Kafka cluster stats Multiple data centers
  11. 11. Kafka Infrastructure Deep Dive
  12. 12. Requirements ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
  13. 13. Kafka Pipeline Local Agent uReplicator
  14. 14. Kafka Pipeline: Data Flow 1 2 3 5 7 64 8
  15. 15. Kafka Clusters Local Agent uReplicator
  16. 16. Kafka Clusters ● Use case based clusters ○ Data (async, reliable) ○ Logging (High throughput) ○ Time Sensitive (Low Latency e.g. Surge, Push notifications) ○ High Value Data (At-least once, Sync e.g. Payments) ● Secondary cluster as fallback ● Aggregate clusters for all data topics.
  17. 17. Kafka Clusters ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
  18. 18. Kafka Rest Proxy Local Agent uReplicator
  19. 19. Why Kafka Rest Proxy ? ● Simplified Client API ● Multi-lang support (Java, NodeJs, Python, Golang) ● Decouple client from Kafka broker ○ Thin clients = operational ease ○ Less connections to Kafka brokers ○ Future kafka upgrade ● Enhanced Reliability ○ Primary & Secondary Kafka Clusters
  20. 20. Kafka Rest Proxy: Internals
  21. 21. Kafka Rest Proxy: Internals
  22. 22. Kafka Rest Proxy: Internals ● Based on Confluent’s open sourced Rest Proxy ● Performance enhancements ○ Simple http servlets on jetty instead of Jersey ○ Optimized for binary payloads. ○ Performance increase from 7K* to 45-50K QPS/box ● Caching of topic metadata. ● Reliability improvements* ○ Support for Fallback cluster ○ Support for multiple Producers (SLA based segregation) ● Plan to contribute back to community *Based on benchmarking & analysis done in Jun ’2015
  23. 23. Rest Proxy: performance (1 box) Message rate (K/second) at single node End-endLatency(ms)
  24. 24. Kafka Clusters + Rest Proxy ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
  25. 25. Kafka Clients Local Agent uReplicator
  26. 26. Client Libraries ● Support for multiple clusters. ● High Throughput ○ Non-blocking, async, batching ○ <1ms produce latency for clients ○ Handles Throttling/BackOff signals from Rest Proxy ● Topic Discovery ○ Discovers the kafka cluster a topic belongs ○ Able to multiplex to different kafka clusters ● Integration with Local Agent for critical data
  27. 27. Client Libraries Add Figure What if there is network glitch / outage?
  28. 28. Client Libraries Add Figure
  29. 29. Kafka Clusters + Rest Proxy + Clients ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
  30. 30. Local Agent Local Agent uReplicator
  31. 31. Local Agent ● Local spooling in case of downstream outage/backpressure ● Backfills at the controlled rate to avoid hammering infrastructure recovering from outage ● Implementation: ○ Reuses code from rest-proxy and kafka’s log module. ○ Appends all topics to same file for high throughput.
  32. 32. Local Agent Architecture Add Figure
  33. 33. Local Agent in Action Add Figure
  34. 34. Kafka Clusters + Rest Proxy + Clients + Local Agent ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
  35. 35. uReplicator Local Agent uReplicator
  36. 36. Multi-DC data flow
  37. 37. CONFIDENTIAL >> INSERT SCREENSHOT HERE << Mirrormaker : existing problems ● New Topic added ● New partitions added ● Mirrormaker bounced ● New mirrormaker added
  38. 38. uReplicator: In-house solution Zookeeper Helix MM Controller Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition MM worker1 MM worker2 MM worker3
  39. 39. uReplicator Zookeeper Helix MM Controller Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition MM worker1 MM worker2 MM worker3
  40. 40. Kafka Clusters + Rest Proxy + Clients + Local Agent ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
  41. 41. uReplicator ● Running in production for 1+ year ● Open sourced: https://github.com/uber/uReplicator ● Blog: https://eng.uber.com/ureplicator/
  42. 42. Chaperone - E2E Auditing
  43. 43. Chaperone Architecture
  44. 44. CONFIDENTIAL >> INSERT SCREENSHOT HERE << Chaperone : Track counts
  45. 45. CONFIDENTIAL >> INSERT SCREENSHOT HERE << Chaperone : Track Latency
  46. 46. Chaperone ● Running in production for 1+ year ● Planning to open source in ~2 Weeks
  47. 47. At-least Once Kafka
  48. 48. Why do we need it? 1 2 3 5 7 64 8 ● Most of infrastructure tuned for high throughput ○ Batching at each stage ○ Ack before produce (ack’ed != committed) ● Single node failure in any stage leads to data loss ● Need a reliable pipeline for High Value Data e.g. Payments
  49. 49. How did we achieve it? ● Brokers: ○ min.insync.replicas=2, can only torrent one node failure ○ unclean.leader.election= false, need to wait until the old leader comes back ● Rest Proxy: ○ Partition Failover ● Improved Operations: ○ Replication throttling, to reduce impact of node bootstrap ○ Prevent catching up nodes to become ISR
  50. 50. Operations/Tooling
  51. 51. Partition Rebalancing Add Figure
  52. 52. Partition Rebalancing ● Calculates partition imbalance and inter-broker dependency. ● Generates & Executes Rebalance Plan. ● Rebalance plans are incremental, can be stopped and resumed. ● Currently on-demand, Automated in the future.
  53. 53. XFS vs EXT4 Add Figure
  54. 54. Summary: Scale ● Kafka Brokers: ○ Multiple Clusters per DC ○ Use case based tuning ● Rest Proxy to reduce connections and better batching ● Rest Proxy & Clients ○ Batch everywhere, Async produce ○ Replace Jersey with Jetty ● XFS
  55. 55. Summary: Reliability ● Local Agent ● Secondary Clusters ● Multi Producer support in Rest Proxy ● uReplicator ● Auditing via Chaperone
  56. 56. Future Work ● Open source contribution ○ Chaperone ○ Toolkit ● Data Lineage ● Active Active Kafka ● Chargeback ● Exactly once mirroring via uReplicator
  57. 57. Questions ? ankur@uber.com
  58. 58. Extra Slides
  59. 59. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 Broker 3 100 101 Leader Committed Producer Acked
  60. 60. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 Broker 3 100 101 Leader Committed Producer Failed Acked
  61. 61. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 Broker 3 100 101 Leader Committed Producer
  62. 62. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 104 105 106 Broker 3 100 101 104 105 Leader Committed Producer Old HW
  63. 63. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 104 105 106 Broker 3 100 101 104 105 Leader Committed Producer X Old HW X
  64. 64. Kafka Durability (acks=1) Broker 1 100 101 104 105 106 Broker 2 100 101 104 105 106 Broker 3 100 101 105 106 Leader Committed Producer data loss!!
  65. 65. Distributed Messaging system * Supported in Kafka 0.8+ ● High throughput ● Low latency ● Scalable ● Centralized ● Real-time
  66. 66. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Broker 2 Broker 3 ZooKeeper
  67. 67. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Partition 0 Broker 2 Partition 1 Broker 3 Partition 2 ZooKeeper
  68. 68. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Partition 0 Partition 2 Broker 2 Partition 1 Partition 0 Broker 3 Partition 2 Partition 1 ZooKeeper
  69. 69. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Partition 0 0 1 2 3 Partition 2 0 1 2 3 Broker 2 Partition 1 0 1 2 3 Partition 0 0 1 2 3 Broker 3 Partition 2 0 1 2 3 Partition 1 0 1 2 3 ZooKeeper
  70. 70. Kafka Concepts

×