Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deploying Apache Spark Jobs on Kubernetes with Helm and Spark Operator

90 views

Published on

Using a live coding demonstration attendee’s will learn how to deploy scala spark jobs onto any kubernetes environment using helm and learn how to make their deployments more scalable and less need for custom configurations, resulting into a boilerplate free, highly flexible and stress free deployments.

Published in: Data & Analytics
  • Be the first to comment

Deploying Apache Spark Jobs on Kubernetes with Helm and Spark Operator

  1. 1. Deploying Apache Spark Jobs on Kubernetes with Helm and Spark Operator Tom Lous Freelance Data Engineer @ Shell @tomlous
  2. 2. Why?
  3. 3. Bad Idea?
  4. 4. 1. Build it! 2. Run it?
  5. 5. Challenge!
  6. 6. Solution?
  7. 7. Kubernetes!?
  8. 8. minikube https://carbon.now.sh/5pwVel5DBKj0cO3ZNCRh
  9. 9. Application
  10. 10. Dependencies & App
  11. 11. Base Image
  12. 12. Dockerize
  13. 13. Deploy?
  14. 14. Spark Operator!
  15. 15. Helm Template
  16. 16. Helm Values
  17. 17. Chart Museum
  18. 18. Deploy!
  19. 19. Success!
  20. 20. Next Steps
  21. 21. Links ▪ HowTo: https://medium.com/@tomlous/deploying-apache-spark- jobs-on-kubernetes-with-helm-and-spark-operator-eb1455930435 ▪ SparkOperator: https://github.com/GoogleCloudPlatform/spark-on- k8s-operator ▪ SparkOperator Helm: https://github.com/helm/charts/tree/master/incubator/sparkoperat or ▪ Code: https://github.com/TomLous/medium-spark-k8s ▪ Chart Museum: https://github.com/helm/chartmuseum
  22. 22. Questions?
  23. 23. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.

×