"Lyft is a ride-sharing company which is a two-sided marketplace; balancing supply and demand using various levers (passenger pricing, driver incentive etc.) to maintain an efficient system. Lyft has built a real-time optimization platform that helps to build the product faster. This complex system makes real-time decisions using various data sources; machine learning models; and a streaming infrastructure for low latency, reliability and scalability. This infrastructure consumes a massive number of events from different sources to make real-time product decisions. Rakesh discusses how Lyft organically evolved and scaled the streaming platform that provides a consistent view of the marketplace to aid an individual team independently run their optimization. The platform offers online and offline feature access that helps teams to back test their model in the future. Each iteration provided better scale, unlocking different capabilities of the marketplace and new avenues for growth. To accomplish these, the team has set the roadmap for next-generation streaming platform and developing smarter tools. Topics include: * Describing Lyft’s streaming platform for dynamic decision-making * Where Kafka fits in streaming tech stack and how it helped us to scale better * Productionizing the very first pipeline * Tools that simplified pipeline creation development environment that is friendly to Data Science people * Lesson learned"

1. Evolution of
Pipelines @ Lyft
By
Rakesh Kumar
Senior Staff Engineer @ Lyft

2. ● Context
● Streaming Platform Tech Stack
● Pipeline Architecture Evolution
● Tools
● Lesson Learned
● Q/A
Agenda

3. Context

4. Dynamic Pricing
Supply/Demand curve
ETA
Pricing
Notifications
Detect Delays
Coupons
User Delight
Fraud
Behavior Fingerprinting
Monetary Impact
Imperative to act fast
Top Destinations
Core Experience

5. Use Cases

6. Feature Generation Pipeline
TBD
• Enriches raw data into high-quality tagged data
• Shared between aggregation pipelines
• Aggregation pipelines generate real time ML features
• Features are indexed in feature store and published to Kafka topics

7. Model Training Pipelines
TBD
• Models training is triggered by smart trigger
• Model training frequency >= 30 seconds
• Trained and serialized model is stored in artifactory for model serving

8. Model Inference Pipelines
• Consists of Model DAG
• Triggered by smart trigger
• Stores computed value in a service or Kafka topic

9. Streaming Platform
Tech Stack

10. Streaming Tech Stack

11. Streaming Tech Stack

12. Scale
● Powering Business critical products
• Dynamic Pricing, driver incentive, real time supply
management, etc.
• Computing ETA, determining driver routing, & making
dispatch decision for all sessions in real time
● Computing dynamic pricing for 10s million geohashes
● Forecasting marketplace parameters for 644+ cities in the
USA & 12+ cities in Canada
● Computing driver incentive in real time for 1+ million
hotspot per minute
● And many more

13. Evolution

14. MVP

15. Scaling Challenges
Separation of Concerns

16. Scaling for Operation

17. Tools

18. Validation Pipelines
18
● A safe environment to test out changes with production traffic
● Identify and avoid bad changes during the development phase
● Identify scaling issue early & easily
● Higher production pipeline reliability

19. Safe Deploy
19
● Build on top of validation pipelines & K8s operator
● Offers guarantees for healthy rollout
● Seamless rollbacks
● Automate deployment
● Increase developer productivity

20. Auto-Optimizing Capacities
20

21. Dev Experience
Yaml Based Pipelines
21

22. Lesson Learned

23. Lesson Learned
23
● Standardization
● Better abstraction for usability
● Guidelines
● Observability
● Blue-Green operational complexity

24. Thank you
Sept 26, 2023
Q / A