Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Hybrid Apache Spark Architecture with YARN and Kubernetes

Download to read offline

Lyft is on the mission to improve people’s lives with the world’s best transportation. Starting 2019, Lyft has been running both Batch ETL and ML spark workloads primarily on Kubernetes with the Apache Spark on k8s operator. However, with the increasing scale of workloads in frequency and resource requirements, we started hitting numerous reliability issues related to IP allocation, container images, IAM role assignment, and Kubernetes Control Plane.

To continue supporting growing Spark usage with Lyft, the team came up with a hybrid architecture optimized for containerized and non-containerized workload based on Kubernetes and YARN. In this talk, we will also cover a dynamic runtime controller that helps with per environment config overrides and easy switchover between resource managers.

  • Be the first to like this

Hybrid Apache Spark Architecture with YARN and Kubernetes

  1. 1. Hybrid Spark Architecture with Yarn and Kubernetes Catalin Toda (Sr Engineer @ Lyft) Rohit Menon (Staff Engineer @ Lyft)
  2. 2. Agenda ▪ Spark @ Lyft ▪ Challenges with K8s ▪ Hybrid Model ▪ Spark Operator ▪ Image Hierarchy ▪ Spark Wrapper ▪ Progress & Future Plans
  3. 3. What is Spark Used For @ Lyft • Primarily Python Shop with some Scala • Running in AWS with S3 as permanent storage • Interactive Development & ML with Jupyter and Spark • ML Batch Use Cases • Pricing • ETA/Routing • Mapping • ETL Use Cases • Event Ingestion • GraphQL Offline Model • Financial Datasets / SOX Complaint Datasets / Experimentation Offline Batch Analysis and many more • HiveQL to Spark-SQL Migration
  4. 4. 2018 Spark on Yarn • Every major use case had its own ephemeral yarn cluster • Management overhead for infra team • Custom Dependency management per cluster to pull in python dependencies • Tough to test/maintain cluster bootstrap scripts • Custom IAM role/permission overhead
  5. 5. 2019 Spark On Kubernetes • Lyft Infra supported Kubernetes deployment • Google OSS spark-on-k8s operator availability • Flyte (Container native scheduler) took off in Lyft for ML use case • Containerized workloads with easier python dependency management • Simpler support for per job IAM roles
  6. 6. Spark On Kubernetes Architecture
  7. 7. 2020 Spark on Kubernetes • Maturing support for Spark on K8s • Lyft Hadoop/Hive infrastructure as K8s deployment • Auto-scaling handled for YARN cluster based on RM load • Spark ETL workloads move over from Yarn to Spark on k8s • Start hitting limits with Lyft k8s Infra setup • Custom solutions required to support growing scale • Group jobs to reduce spiky requests to k8s and AWS control plane • Add new k8s clusters to support stronger isolation model
  8. 8. Current Spark Scale on K8s - 1000 concurrent jobs - 5 Kubernetes Clusters - 20k executors at peak - 1 AWS region - 5 AWS Availability Zones
  9. 9. Challenges with K8s Model • IPv4 Address Shortage • Shortage across all 5 AZs • Leads to driver and executors startup delays - IAM Wait Delays - AWS IAM assignment could be throttled - IAM wait to assure assignment increasing delays - Infrastructure issues - Etcd size tuning - Impact of bad k8s node
  10. 10. Challenges with K8s Model • Image Overheads • Every project has their own image • Registration of images for different environments • Startup delays caused by uncached image • New nodes • New image releases • Release model • Infra prepares a base image with Spark latest changes • Customers manage final release when the image is tested • Leads to maximum 1 month rollout time due to several images to be updated
  11. 11. Challenges with K8s Model - K8s scheduling • Fixed per namespace quota • Containers not admitted if over quota • No priority between jobs in namespaces - Control plane limits - Maximum number of containers - Short running containers are not typical K8s workload • Hive Deprecation • Today hive scale = 5k jobs at peak • Expected spark load to increase by 6x • For interactive workload pod startup time latency is high on k8s
  12. 12. 2021 Hybrid Model (YARN + K8s) Separation by Workload Type • Containerized (K8s) • Pyspark + Custom Dependencies • ML interactive • Non-containerized (YARN) • SQL Interactive • SQL Batch • Scala Workloads • Simple Pyspark with no dependencies
  13. 13. Hybrid Model Architecture
  14. 14. Advantages of Hybrid Model • YARN executors have low startup latency and can handle spikes • Easier Queue and Resource management • Workloads without custom dependencies do not get penalized with k8s infra overheads • Mature support for dynamic allocation and external shuffle service
  15. 15. Single Entry Point • Spark-on-k8s-operator as single entry point for both YARN and K8s • Integrates easily open source • Compatibility with Flyte • No Lyft specific code • Multi version branch by default • Driver runs on k8s in client mode • We plan to contribute this to OSS • Current OSS design add overhead of spark submit pod • Sets us infra team to move workloads seamlessly between resource managers
  16. 16. Spark Wrapper Design Stage 1 - Part of base image - Downloads and runs stage2 Stage 2 - Manipulate configs - Run spark driver - Capture job logs and results - Push application metrics
  17. 17. Spark Wrapper • Custom image entry point • Allows config management based on environments • Allows switching between resource managers • Metrics • Push to events - queryable/dashboard using lyft stack • Push to statsd for real time monitoring/alerting • Integrates well with Lyft Infra • Spark on k8s operator remains in sync with upstream • Lyft specific logic that integrates with in-house tools • Adds runtime controller to images
  18. 18. Image Hierarchy/Distribution • One base image for spark per spark version • Containerized spark extends base image • Users can further extend containerized base image to add custom dependencies • ML base image • Users maintain their own image • Non-containerized use the base image directly • Infra updates the image • Consistent experience across use cases
  19. 19. Progress so far - Best of Both Worlds - Spark driver startup < 1s - Resource allocation managed in YARN - K8s scale reduced by 20x - IP addresses requirement reduced by 20x - Per job IAM Roles using Web Identity provider
  20. 20. Progress so far - Best of Both Worlds - No migration needed for containerized customers - Python dependency management using an utility library - The latest version is synced in all environments (adhoc, k8s, YARN)
  21. 21. Future Plans - Consolidate Batch Compute on Spark (Hive -> Spark) - Evaluate Data Lake technologies - Continue to scale k8s and Spark Infrastructure
  22. 22. Conclusion - YARN vs K8s - Workload analysis is required before identifying the best solution - For Lyft - existing YARN infrastructure helped choosing a hybrid model - Fixing K8s model requires: - K8s supports to IPv6 in the latest versions - Scaling - Number of k8s clusters and a gateway to perform the routing between them - Image - Design considerations/overheads with high number of images - Quota - Investing in projects trying to solve this aspect - Web Identity Provider - Custom Roles in K8s
  23. 23. Q & A Contact Info: ctoda@lyft.com rmenon@lyft.com

Lyft is on the mission to improve people’s lives with the world’s best transportation. Starting 2019, Lyft has been running both Batch ETL and ML spark workloads primarily on Kubernetes with the Apache Spark on k8s operator. However, with the increasing scale of workloads in frequency and resource requirements, we started hitting numerous reliability issues related to IP allocation, container images, IAM role assignment, and Kubernetes Control Plane. To continue supporting growing Spark usage with Lyft, the team came up with a hybrid architecture optimized for containerized and non-containerized workload based on Kubernetes and YARN. In this talk, we will also cover a dynamic runtime controller that helps with per environment config overrides and easy switchover between resource managers.

Views

Total views

110

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

13

Shares

0

Comments

0

Likes

0

×