An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.
•
10 likes•2,160 views
Jay Kreps, Open Source Visionary and Co Founder of Confluent and several open source projects will be visiting LA. I have asked him to come present at our group. He will present his vision and will answer questions regarding Kafka and other projects Bio:- Jay is the co-founder and CEO at Confluent a company built around realtime data streams and the open source messaging system Apache Kafka. He is the original author of several of open source projects including Apache Kafka, Apache Samza, Voldemort, and Azkaban.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.
Similar to An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban. (20)
More from Data Con LA
More from Data Con LA (20)
Recently uploaded
Recently uploaded (20)
An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.
- 1. Apache Kafka and the Rise of the Stream Data Platform Jay Kreps 1
- 3. 2009: We want all our data in Hadoop!
- 4. Initial approach: “gut it out”
- 5. • Data coverage • Many source systems • Relational DBs • Log files • Metrics • Messaging systems • Many data formats • Constant change • New schemas • New data sources Problems
- 6. How does everything else work? ?
- 8. User events
- 9. Application Logs & Operational Metrics
- 10. Messaging
- 11. This is a giant mess
- 13. Idea: Stream Data Platform
- 14. First Attempt: Messaging systems!
- 15. Problems • Throughput • Batch systems • Persistence • Stream Processing • Ordering guarantees • Partitioning
- 16. Second Attempt: Build Kafka!
- 17. What does it do?
- 19. Logs & Publish-Subscribe Messaging
- 20. A Kafka Topic
- 21. Kafka: A Modern Distributed System for Streams Scalability of a filesystem ◦ Hundreds of MB/sec/server throughput ◦ Many TB per server Guarantees of a database ◦ Messages strictly ordered ◦ All data persistent Distributed by default ◦ Replication ◦ Partitioning model Producers, Consumers, and Brokers all fault tolerant and horizontally scalable
- 23. Batch Data => Batch Processing
- 24. Stream processing is a generalization of batch processing and request/response processing
- 25. Request/Response processing: One input => One output
- 26. Batch processing: All inputs => All outputs
- 27. Stream Processing: Some inputs => some outputs (you choose how much “some” is)
- 28. Stream Processing a la carte cat input | grep “foo” | wc -l
- 29. Stream Processing with Frameworks + = Stream Processing
- 31. Usage at LinkedIn • Everything in the company is a real-time stream • > 1.2 trillion messages written per day • > 3.4 trillion messages read per day • ~ 1 PB of stream data • Tens of thousands of producer processes • Backbone for data stores • Search • Social Graph • Newsfeed • Primary storage (in progress) • Basis for stream processing
- 32. Elsewhere
- 33. • Mission: Make this a practical reality everywhere • Product: Confluent Platform • Apache Kafka • Schemas and metadata management • Connectors for common systems • Monitor data flow end-to-end • Stream processing integration
- 34. Sneak Preview • 0.9 Release • Security • Quotas • New consumer client • Connector framework • Confluent Platform 2.0 • C Client • Connectors • etc
- 35. Resources • Confluent • @confluentinc • http://confluent.io • http://blog.confluent.io/2015/02/25/stream-data- platform-1 • Apache Kafka • @apachekafka • http://kafka.apache.org • http://linkd.in/199iMwY • Me • @jaykreps
Editor's Notes
- Year = 2009: This story takes place at LinkedIn. I was on the infra team. Previously had built a distributed key-value store Was leading Hadoop adoption effort.
- Had done initial prototypes, had some valuable things Want Data Lake/Hub
- Various ad hoc load jobs Manual parsing of each new data we got into target schema 14% of data Not the kind of team I wanted
- Data grandfathering 14% of data in Hadoop N new non-hadoop engineers imply O(N) hadoop engineers
- Got interested in things like schemas, metadata, dataflow, etc
- Real-time, mostly lossless—couldn’t go back in time Batch, mostly lossless–high latency Why CSV dumps?
- Batch aggregation Lossy, high-latency, only went to data warehouse/Hadoop
- Splunk
- Low-throughput, lossy, no real scalability story No central system No integration with batch wold
- Reality about 100x more complex 300 services ~100 databases Multi-datacenter Trolling: load into Oracle, search, etc Publish data from Hadoop to a search index Run a SQL query to find the biggest latency bottleneck Run a SQL query to find common error patterns Low latency monitoring of database changes or user activity Incorporate popularity in real-time display and relevance algorithms Products that incorporate user activity
- Not tenable—new systems, data being created faster than we integrate them Not all problems solvable with infrastructure Extract the common pattern…
- Had Hadoop—great that can be our warehouse/archive. But what about real-time data and real-time processing? All the messed up stuff? Files are really just a stream of updates stored together
- This problem is solved…don’t reinvent the wheel!
- Not suitable for ETL Not suitable for high throughput
- Reinvent the wheel! Initial time estimate: 3 months 30 second intro to Kafka
- Producers, consumers, topic Like a messaging system
- Different: Commit log Like a file that goes on forever Stolen from distributed database internals Key abstraction for systems, real-time processing, data integration Formalization of a stream
- Very fast publish subscribe mechanism Unifies batch and real-time processing
- Built like a modern dist system
- Let’s dive into the elements of this platform One of the big things this enables is stream processing…let’s dive into that. Truviso
- First thing you want when you have real-time data streams is real-time transformations 1790 Collected data by Networks=>stream processing 3,929,214 people $44k Horses and wagons are a high latency, batch channel
- Service: One input = one output Batch job: All inputs = all outputs Stream computing: any window = output for that window
- REST, RPC, etc
- REST, RPC, etc
- Batch processing => stream processing with window of 1 day
- Like unix pipe, streams data between programs Distributed, fault-tolerant, elastically scalable Modern version of unix pipes
- Streams + Processing = Stream Processing HDFS analogy Most stream processing system use Kafka Some require it—rollback recovery
- Stream Data Platform Trace data flow
- Everything that happens in the company is a real-time stream
- Describe adoption cycle
- Only one thing you can do if you think the world needs to change, you live in Silicon Valley—quit your job and do it.