Downloaded 229 times
Uploaded byDatabricks
Apache Spark on K8S Best Practice and Performance in the Cloud
The document presents an overview of Tencent Cloud's implementation of Spark on Kubernetes, detailing architecture, best practices, and performance benchmarks. It discusses the motivations behind using Dockerized Spark, the advantages of Kubernetes, and the challenges faced when transitioning from a Yarn setup. Additionally, it provides insights into benchmarking results, root cause analysis of performance issues, and future work aimed at improving Spark's integration with Kubernetes.
More Related Content
Similar to Apache Spark on K8S Best Practice and Performance in the Cloud
![[Spark Summit 2017 NA] Apache Spark on Kubernetes](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkonkubernetespublic1-170929072840-thumbnail.jpg?width=640&height=640&fit=bounds)
Apache Spark on K8S Best Practice and Performance in the Cloud
- 1. Jimmy Chen, JunpingDu Tencent Cloud
- 2. Agenda ⢠Spark servicein the cloud â our practice ⢠Spark on Kubernetes â architecture, best practice and performance
- 3. About us Jimmy Chen Sr.Software Engineer at Tencent Cloud Apache Hadoop, Spark, Parquet Contributor Previously, worked at Intel Junping Du Principal Engineer & Director at Tencent Cloud Apache Hadoop PMC & ASF Member Previously, worked at Hortonwork and VMware
- 4. Top 100 MostValuable Global Brands 2018 ranked 1st in China, ranked 5th global Worldâs 50 Most Innovative Companies 2015 ranked 1st in China The 331th largest company in the Fortune in 2018 About Tencent and Tencent Cloud
- 5. Case study forSpark in the Cloud - Sparkling , , , , , ,
- 6. Sparkling Architecture TencentCloudInfrastructure RDBMS NoSQL KAFKA COS ES ⌠Data LakeData Catalog and Management Atlas Meta Store RDBMS Data Catalog Sparkling Data Warehouse Manager Data Application NotebookSQL IDE Data Sync Livy-Thrift SQL Optimizer Columnar Storage JDBC Execution Engine - Spark QueryEngine Work Flow Query Monitor Analysis BI ML ⌠Work Space
- 7. Motivations towards dockerizedSpark ⢠Serverless â Better isolation ⢠On-premises deployment â Bare metal infrastructure, control protocol not work â DevOps scope â Cost
- 8. Our Practice ⢠Sparklingon Kubernetes â Cloud API server -> Kubernetes API server â CVM resource -> container resource â Cloud load balancer -> Kubernetes load balancer â Cloud DNS -> Kubernetes ingress/coredns â CVM image -> docker image â etc.
- 9. Kubernetes New open-source clustermanager. - github.com/kubernetes/kubernetes libs app kernel libs app libs app libs app Runs programs in Linux containers. 2,000+ contributors and 75,000+ commits.
- 10. Kubernetes Architecture node A Pod1 172.17.0.5 Pod, a unit of scheduling and isolation. â runs a user program in a primary container â holds isolation layers like a virtual IP in an infra container Pod 2 node B Pod 1 Pod 2 node C Pod 1 Pod 2 172.17.0.6 172.17.0.7
- 11. Kubernetes Advantages ⢠.. . . . . . ⢠, . . ,,. , . . , ⢠, â˘
- 12. Spark On KubernetesWorkflow spark-submit Kubernetes spark operator
- 13. Spark on KubernetesStatus ⢠Spark side â Cluster/Client mode â Volume mounts â Multiple-language support, such as: Python, R â Static executor allocation ⢠On-going Work â Dynamic Executor Allocation â Job Queues and Resource Management â Spark Application Management ⢠Kubernetes side â Support spark-ctl CLI â Native cron support for scheduled app â Customized Spark pod, e.g. mounting configMap and volume, setting affinity/anti-affinity â Monitoring through Prometheus ⢠On-going Work â Support GPU resource scheduler â Spark operator HA â Multiple spark version support â Priority queues
- 14. Sparkling on KubernetesArchitecture Livy Server Thrift Server Metastore Airflow Service History Service Kubernetes cluster Grafana Service Notebook Service Zeppelin Service Ingress pod Datasync pod pod pod pod pod pod pod pod Node pod pod Node pod pod Node pod pod Node Persistent volume pod pod Node pod pod DaemonSet
- 15. Submit via spark-submit Step1: check cluster master #kubectl cluster-info Kubernetes master is running at https://169.254.128.15:60002 Step 2: submit application via spark-submit #bin/spark-submit --master k8s://https://169.254.128.7:60002 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.kubernetes.container.image=sherrytima/spark2.4.1:k8s local:///opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar
- 16. Submit via kubectl/sparkctl â˘Install spark operator on your kubernetes cluster # helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator # helm install incubator/sparkoperator --namespace spark-operator ⢠Declare application yaml ⢠Run the application â kubectl create -f spark-pi.yaml â spark-ctl create spark-pi.yaml
- 17. Benchmark Environment ⢠Software Kubernetes:1.10.1 Hadoop/Yarn: 2.7.3 Spark: 2.4.1 OS: Centos-7.2 ⢠Hardware Master â CPU/mem: 8 core + 64G mem â Disk: 200G SSD â NUM: 1 Slaves â CPU/mem:32 core + 128G mem â Disk: 4 HDD â NUM: 6
- 18. Terasort ⢠Data Size100G spark.executor.instance 5 spark.executor.cores 2 spark.executor.memory 10g spark.executor.memoryoverhead 2g spark.driver memory 8g ⢠Data Size 500G â Spark on kubernetes failed â Spark on Yarn (~35 minutes) 0 100 200 300 400 500 600 700 800 900 spark on yarn spark on k8s TimeinSeconds terasort 100g Lower is better
- 19. Root Cause Analysis â˘Pod always be evicted due to disk pressure, the spark.local.dir is mounted as emptyDir. ⢠emptyDir is ephemeral storage on node
- 20. Possible Solutions ⢠mountemptyDir with RAM backed volumes. spark.kubernetes.local.dirs.tmpfs= true ⢠Extend disk capacity of kubelet workspace
- 21. Result 0 100 200 300 400 500 600 700 800 900 spark on yarnspark on k8s (default) spark on k8s with tmpfs TimeinSeconds Terasort 100G lower is better
- 22. Spark-sql-perf ⢠Use spark-sql-perfrepo from Databricks â 10 shuffle bound queries â spark application base on spark-sql-perf ⢠Run in cluster mode for Kubernetes bin/spark-submit --name tpcds-1000 --jars hdfs://172.17.0.8:32001/run-lib/* --class perf.tpcds.TpcDsPerf --conf spark.kubernetes.container.image=sherrytima/spark-2.4.1:k8sv8 --conf spark.driver.args="hdfs://172.17.0.8:32001/run-tpcds 1000 orc true" hdfs://172.17.0.8:32001/tpcds/tpcds_2.11-1.1.jar
- 23. Benchmark Result ⢠Configuration spark.executor.instance24 spark.executor.cores 6 spark.executor.memory 24g spark.executor.memoryoverhead 4g spark.driver memory 8g ⢠Result Spark on kubernetes is much slower than spark on yarn!!! 0.0 200000.0 400000.0 600000.0 800000.0 1000000.0 1200000.0 q4 q11 q17 q25 q29 q64 q74 q78 q80 q93 timeinms tpcds@1000 lower is better Spark on yarn Spark on k8s
- 24. Root Cause ⢠Thekubelet work directory can only be mounted on one disk, so that the spark scratch space only use ONE disk. ⢠While running in yarn mode, the spark scratch space use as many disk as yarn.local.dir configures.
- 25. Possible workaround ⢠RAMbacked media as spark scratch space
- 26. Our Solution Use specifiedvolumes mount as spark local directory. SPARK-27499
- 27. Result CreateTime(ms) TPCDS@1000 lower isbetter . 0 . . 0.0 200000.0 400000.0 600000.0 800000.0 1000000.0 1200000.0 q4 q11 q17 q25 q29 q64 q74 q78 q80 q93 timeinms TPCDS@1000 lower is better spark on k8s orig spark on k8s optimized
- 28. Summary ⢠Spark onKubernetes practices â scenario in serverless and cloud native ⢠But we hit problems in production - local storage issues, etc. ⢠Work with community for better solution and best practices
- 29. Future Work ⢠Closeperformance gap with yarn mode ⢠Data locality awareness for Kubernetes scheduler ⢠Prometheus/Grafana integration
- 30.