Kafka Summit SF 2017 - Billions of Messages a Day – Yelp’s Real-time Data Pipeline
•
1 like•1,013 views
Kafka Summit SF 2017 by Justin Cunningham
Recommended
Recommended
More Related Content
Similar to Kafka Summit SF 2017 - Billions of Messages a Day – Yelp’s Real-time Data Pipeline
Similar to Kafka Summit SF 2017 - Billions of Messages a Day – Yelp’s Real-time Data Pipeline (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Kafka Summit SF 2017 - Billions of Messages a Day – Yelp’s Real-time Data Pipeline
- 1. Justin Cunningham justinc@yelp.com Billions of Messages a Day Yelp's Real-time Data Pipeline
- 2. Yelp’s Mission Connecting people with great local businesses.
- 3. Why build a data pipeline? Start with a monolith 2011: ~1,000,000 lines
- 4. Services Solve Everything! 2014: ~150 services Service Service Service Service Service
- 5. Almost Everything Metcalfe's Law 22,350Omni-Directional Communication Paths 11,175Bi-Directional Communication Paths 150Services
- 6. What about the data?
- 7. Reasonable Becomes Unreasonable 86 Million is a Magic Number I want to process all reviews every day. I want to make 1,000 requests per second to your service, every second, forever.
- 8. Potential Solutions? What if we implement a raw bulk-data API? We could pass it arbitrary SQL to generalize it. What if we take DB snapshots and pass them around? Flags Prefs Category 33939 533248 37
- 9. Failing at Failure session.begin() business = Business() session.add(business) session.commit() my_service_client.notify_business_changed(business.id)
- 10. Failing at Failure session.begin() business = Business() session.add(business) my_service_client.notify_business_changed(business.id) session.commit()
- 11. How do we start solving these problems? Service Service Service Service Service Service Message Bus n2 -> n
- 12. + Offset Key Data 0 47 The 1 21 Quick 2 18 Brown 3 47 Fox 4 21 Jumps Key Offset 18 2 47 3 21 4 Offset Key Data 2 18 Brown 3 47 Fox 4 21 Jumps ...
- 14. What is the Data Pipeline? COMMUNICATION
- 15. Guaranteed Format and Compatibility Schema 1 Schema 1 Schema 2 Schema 2 Schema 3Schema 2 Consumer loads schema dynamically as it receives messages SCHEMATIZER Load Schema 1 Load Schema 3 Load Schema 2 PIPE (topic w/registered schema) PRODUCER CONSUMER CONSUMER SCHEMATIZER
- 18. Core Data Pipeline All About the Data CORE SCHEMA STORE
- 19. How We Use It CORE SCHEMA STORE Application Event Logs MySQL Code / Stream Processor Amazon Redshift S3 Application Event Logs MySQL Code / Stream Processor
- 20. How We Use It Processing Business Changes Amazon Redshift S3 Elasticsearch Code MySQL Stream Processor
- 21. Envelopes are Powerful What we learned? Envelope Contains: uuid, timestamp, schema_id, payload, previous_payload and meta attributes (tuples of schema_id & payload) Message = magic byte + encoded bytes
- 23. What we learned? Distributed Systems are Hard
- 24. Glorious Future? Event-First Architecture Event-Log as System of Record CORE SCHEMA STORE Web Workers Everything Else
- 25. Overall Data Infra Datapipe Producer Bunsen Scribe Replication Handler MySQL Other Data Stores Yelp-main Services MONK DP DP JSON SCHEMATIZER KAFKAKAFKA • Paastorm • Python • Flatmap • Flink* • Java/Scala • Advanced Primitives & Stream SQL Recursive MySQL Services Yelp-main Redshift S3 Flink Kafka Connect Cassandra ES