Download as PDF, PPTX
Uploaded bySpark Summit
Spark Summit EU talk by William Benton
The document discusses containerizing Spark clusters on Kubernetes. It describes how the author's Spark cluster looked in 2014 running on Mesos with networked storage. It then covers motivations for microservices architectures and how Spark fits into this. The document outlines architectures for analytics and applications, including responsibilities like transformation, aggregation, training models, and more. It also discusses legacy architectures like data warehouses and Hadoop-style data lakes. Finally, it covers practical considerations and potential pitfalls of containerized Spark clusters like scheduling, security, and storage options.
More Related Content
Similar to Spark Summit EU talk by William Benton
![[DSC Europe 25] Josip Saban - Career building for data professionals.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/zroflcttkm1vmli0txea-josip-saban-career-building-for-data-professionals-260123083019-587cdb8c-thumbnail.jpg?width=640&height=640&fit=bounds)
Spark Summit EU talk by William Benton
- 1. William Benton Red Hat,Inc. @willb • willb@redhat.com CONTAINERIZED SPARK ON KUBERNETES
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10. WHAT OUR SPARKCLUSTER LOOKED LIKE IN 2014
- 11. WHAT OUR SPARKCLUSTER LOOKED LIKE IN 2014 Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor
- 12. WHAT OUR SPARKCLUSTER LOOKED LIKE IN 2014 Networked POSIX FS Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor
- 13. Mesos WHAT OUR SPARKCLUSTER LOOKED LIKE IN 2014 Networked POSIX FS Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor
- 14. Mesos WHAT OUR SPARKCLUSTER LOOKED LIKE IN 2014 Networked POSIX FS Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor 1 2 3 4
- 15. Mesos WHAT OUR SPARKCLUSTER LOOKED LIKE IN 2014 Networked POSIX FS Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor 1 2 3 4 1 1 2 3 3 4
- 16. Mesos WHAT OUR SPARKCLUSTER LOOKED LIKE IN 2014 Networked POSIX FS Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor 1 2 3 4 1 1 2 3 3 4
- 17. Analytics is nolonger a separate workload.
- 18. Analytics is anessential component of modern data- driven applications.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. FORECAST Motivating containerized microservices Architecturesfor analytics and applications Spark clusters in containers: practicalities and pitfalls Play along at home Future work
- 26.
- 27. A microservice architecture employslightweight, modular, and typically stateless components with well-defined interfaces and contracts.
- 28. BENEFITS OF MICROSERVICEARCHITECTURES
- 29. BENEFITS OF MICROSERVICEARCHITECTURES
- 30. BENEFITS OF MICROSERVICEARCHITECTURES
- 31. BENEFITS OF MICROSERVICEARCHITECTURES
- 32. BENEFITS OF MICROSERVICEARCHITECTURES
- 33. BENEFITS OF MICROSERVICEARCHITECTURES
- 34. BENEFITS OF MICROSERVICEARCHITECTURES
- 35. BENEFITS OF MICROSERVICEARCHITECTURES 2 + 2
- 36. BENEFITS OF MICROSERVICEARCHITECTURES 2 + 2 5
- 37. BENEFITS OF MICROSERVICEARCHITECTURES 2 + 2 5
- 38. BENEFITS OF MICROSERVICEARCHITECTURES
- 39. BENEFITS OF MICROSERVICEARCHITECTURES
- 40. BENEFITS OF MICROSERVICEARCHITECTURES ?
- 41. BENEFITS OF MICROSERVICEARCHITECTURES ?
- 42. BENEFITS OF MICROSERVICEARCHITECTURES ?
- 43. MICROSERVICES AND SPARK executor 12 3 executor 4 5 6 executor 7 8 9 executor 10 11 12 master
- 44. MICROSERVICES AND SPARK executor 12 3 executor 4 5 6 executor 7 8 9 executor 10 11 12 master λ x: x * 2
- 45. MICROSERVICES AND SPARK executor 12 3 executor 4 5 6 executor 7 8 9 executor 10 11 12 master λ x: x * 22 4 6 8 10 12 14 16 18 20 22 24 λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2
- 46. MICROSERVICES AND SPARK executor 12 3 executor 4 5 6 executor 7 8 9 executor 10 11 12 master λ x: x * 22 4 6 8 10 12 14 16 18 20 22 24 λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2
- 47. MICROSERVICES AND SPARK executor 12 3 executor 4 5 6 executor 7 8 9 executor 10 11 12 master λ x: x * 22 4 6 8 10 12 14 16 18 20 22 24 λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2
- 48. ARCHITECTURES FOR ANALYTICSAND APPLICATIONS
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55. APPLICATION RESPONSIBILITIES archive trainmodels transform transform transform aggregate events databases file, object storage weband mobile reporting developer UI
- 56.
- 57.
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67. transaction processing CONVENTIONAL DATA WAREHOUSE transformevents UI business logic RDBMSRDBMS analysis reporting
- 68. transaction processing CONVENTIONAL DATA WAREHOUSE transformevents UI business logic RDBMSRDBMS analysis reporting
- 69. transaction processing CONVENTIONAL DATA WAREHOUSE transformevents UI business logic RDBMSRDBMS analysis interactive query reporting
- 70. transaction processing CONVENTIONAL DATA WAREHOUSE transformevents UI business logic RDBMSanalytic processing RDBMS analysis interactive query reporting
- 71. HADOOP-STYLE “DATA LAKE” HDFSHDFS HDFS
- 72. HADOOP-STYLE “DATA LAKE” HDFSHDFS HDFS HDFS HDFS
- 73. HADOOP-STYLE “DATA LAKE” HDFS events HDFSHDFS HDFS HDFS
- 74. HADOOP-STYLE “DATA LAKE” HDFS events HDFSHDFS HDFS HDFS
- 75. HADOOP-STYLE “DATA LAKE” HDFS compute events HDFS compute HDFS computecompute compute HDFS HDFS
- 76.
- 77.
- 78.
- 79. speed layer THE LAMBDAARCHITECTURE events (imprecise) analysistransform DFS
- 80. speed layer THE LAMBDAARCHITECTURE events (precise) analysistransform (imprecise) analysistransform DFS
- 81. speed layer THE LAMBDAARCHITECTURE events batch layer (precise) analysistransform (imprecise) analysistransform DFS
- 82. speed layer THE LAMBDAARCHITECTURE events batch layer federate (precise) analysistransform (imprecise) analysistransform DFS
- 83. speed layer THE LAMBDAARCHITECTURE events batch layer UIfederate (precise) analysistransform (imprecise) analysistransform DFS
- 84. serving layerspeed layer THELAMBDA ARCHITECTURE events batch layer UIfederate (precise) analysistransform (imprecise) analysistransform DFS
- 85.
- 86. queue for “rawdata” topic THE KAPPA ARCHITECTURE events
- 87. queue for “rawdata” topic THE KAPPA ARCHITECTURE events transform queue for “preprocessed data” topic
- 88. queue for “rawdata” topic THE KAPPA ARCHITECTURE events transform analysis queue for “preprocessed data” topic queue for “analysis results” topic
- 89. queue for “rawdata” topic THE KAPPA ARCHITECTURE events transform analysis queue for “preprocessed data” topic queue for “analysis results” topic reporting end-user UI
- 90. DATA FEDERATION INTHE COMPUTE LAYER aggregate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
- 91. DATA FEDERATION INTHE COMPUTE LAYER aggregate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
- 92. DATA FEDERATION INTHE COMPUTE LAYER aggregate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
- 93. Cluster scheduler SIDEBAR: THEMONOLITHIC SPARK ANTIPATTERN Shared FS Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3
- 94. Cluster scheduler SIDEBAR: THEMONOLITHIC SPARK ANTIPATTERN Shared FS Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3
- 95. Resource manager ONE CLUSTERPER APPLICATION Object stores app 1 app 2 app 5app 4 app 3 app 6 Databases
- 96. Resource manager ONE CLUSTERPER APPLICATION Object stores app 1 app 2 app 5app 4 app 3 app 6 app 2 app 4 Databases
- 97.
- 98. SCHEDULING Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 Object stores Databases
- 99. SCHEDULING Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1
- 100. SCHEDULING Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1
- 101. SCHEDULING Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1
- 102. SCHEDULING Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1
- 103. SCHEDULING Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1
- 104.
- 105.
- 106.
- 107.
- 108.
- 109.
- 110.
- 111. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6
- 112. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 POSIX filesystem
- 113. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 POSIX filesystem ✓ familiar interface ✓ interoperability with other programs
- 114. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 POSIX filesystem ✓ familiar interface ✓ interoperability with other programs ✗ unnecessary semantic guarantees ✗ difficult to manage
- 115. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 HDFS HDFS HDFS HDFS HDFS HDFS
- 116. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 HDFS HDFS HDFS HDFS HDFS HDFS ✓ support for legacy Hadoop installations
- 117. ✗ inelastic ✗ stateful ✗can’t collocate compute and data STORAGE Kubernetes app 1 app 2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 HDFS HDFS HDFS HDFS HDFS HDFS ✓ support for legacy Hadoop installations
- 118. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 object store
- 119. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 object store ✓ interoperability ✓ fine-grained AC ✓ many implementations
- 120. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 object store ✓ interoperability ✓ fine-grained AC ✓ many implementations ✗ consistency model ✗ performance (?)
- 121. STORAGE Kubernetes app 1 app2 app 5app 4 app 3 app 6 app 1 app 2 app 5app 4 app 3 app 6 object store ✓ interoperability ✓ fine-grained AC ✓ many implementations ✗ consistency model ✗ performance
- 122. “…in a cloudnative architecture, the benefit of HDFS is actually very small and that is why many cloud-first organizations no longer run HDFS, or only run it as a caching layer for S3.” —Reynold Xin on Quora (http://qr.ae/TAF4cN)
- 123.
- 124.
- 125.
- 126.
- 127. NETWORKING http://app1:8080 ✗ can’t accessworker web UI (but wait for Spark 2.1!)
- 128. NETWORKING http://app1:8080 ✗ can’t accessworker web UI (but wait for Spark 2.1!)
- 129. NETWORKING http://app1:8080 ✗ can’t accessworker web UI (but wait for Spark 2.1!) http://app1:80
- 130. NEXT STEPS: FUTUREWORK & PLAYING ALONG AT HOME
- 131. NEXT STEPS Further performanceevaluation Better developer experience Improved scheduling of Spark tasks on Kubernetes
- 132. TRY IT OUTYOURSELF Kubernetes standalone Spark example: https://github.com/kubernetes/kubernetes/tree/master/examples/spark Enabling Spark on OpenShift: https://github.com/radanalyticsio Native Spark on Kubernetes proposal: https://github.com/kubernetes/kubernetes/issues/34377
- 133.