In this talk I describe the work we're doing at Lyft to build a world-class streaming platform.

1. Jamie Grier | @jamiegrier
1
Streaming

2. Agenda
• Goals of Lyft’s Streaming Platform
• Streaming Platform Overview
• Why Flink
• Why Kafka
• Open problems
2

3. Goals of Lyft’s Streaming Platform
• Make it easy to build real-time, event-driven, stateful, microservices
• Solve the hard parts of stream processing ONCE for the entire
company
• Be a force multiplier for other teams within Lyft
• Three components: Pub/Sub, Streaming Compute, Stream Registry
3

4. Streaming Platform Overview
4
Streaming
Service One
Streaming
Service Two
Streaming
Service Three
Stream / Schema
Registry
Deployment
Tooling
Metrics &
Dashboards
Alerts Logging
Amazon
EC2
Amazon S3 Wavefront
Salt
(Conifg / Orca)
Docker
Pub/Sub Pub/Sub
Stream Compute

5. Lyft Streaming Platform - Streaming Compute Criteria
Operational Considerations
● Stateful Computation and Exactly-once
Processing Semantics
● Robust State Management
● Data Reprocessing (backfill)
● Asynchronous Checkpoints
● Back-pressure
● High throughput and low-latency
● Deployment Architecture
5
API Considerations:
● Functional / Fluent API
● Flexible Windowing API
● Event Time Support
● Apache Beam Support
● Stream SQL
● Powerful Direct API
● Late Data Handling
The contenders: Apache Flink, Apache Spark Streaming, Apache Kafka Streams

6. Why Flink? API Considerations
• Functional / Fluent API
• Flexible Windowing API
• Event Time Support
• Apache Beam Support
• Stream SQL
• Powerful Direct API
• Late Data Handling
6

7. Why Flink? Operational Considerations
• Stateful Computation and Exactly-once Processing Semantics
• Robust State Management
• Stateful Data Reprocessing (backfill)
• Asynchronous Checkpoints
• Back-pressure
• High throughput and low-latency
• Deployment Architecture
7

8. Lyft Streaming Platform - Pub/Sub Criteria
Operational Considerations
● Write Latency
● Read Latency
● Project Maturity
● Vendor Support
8
Semantics / Features
● Durability
● Consumer Fanout
● Transactions / Idempotent Writes
● Per-Key Ordering Guarantees
● Long-Term Data Storage
● Auto-Scaling
The contenders: Apache Kafka, Amazon Kinesis, Pravega

9. Why Kafka?
Pros
• Durability & Write Latency
• Read Latency & Consumer Fanout
• Transactions & Idempotent Writes
• Operational Concerns & Vendor Support
Cons
• No ordering by key, only partition
• Long term data storage still an issue
• Auto-Scaling still an issue
9

10. Open Problems
• Rescaling Kafka while preserving per-key ordering
• Efficient Dynamic Computations over streams
• Long term storage for events: real-time and historical reads
• Zero Downtime deployments for streaming services
10

11. Rescaling Kafka
• Rescaling Kafka while preserving per-key ordering
• Kafka only provides partition ordering guarantees!
• We want per-key ordering guarantees
• Guarantees should hold across re-partitioning events
• Basic approach: Read old partitions completely before reading new
• Achieve this using something akin to Flink’s checkpoint barriers to mark
re-partitioning events 11

12. Rescaling Kafka while preserving per-key ordering
12

13. Rescaling Kafka while preserving per-key ordering
13

14. Efficient Dynamic Computation Over Streams
• Enable many users to dynamically submit small streaming computations
• Share bandwidth amongst multiple computations
• Share computed sub-results amongst multiple computations
• Correctly handle bootstrapping of computations which depend on
historical data
• Basic approach: Map any computation into a fixed/general data flow
“shape”
14

15. Efficient Dynamic Computations over streams
15

16. Efficient Dynamic Computations over streams
16

17. Long term storage for events: Real-time and historical reads
17

18. Zero Downtime deployments for streaming services
18

19. Summary
• Lyft is building a next generation streaming platform based on Apache
Flink and Apache Kafka
• Stateful stream processing is not a “solved problem”
• There are many hard / open problems left to solve
• If these sort of problems interest you please come join us!
We’re Hiring!
19

20. Thank you!
20
Jamie Grier