Successfully reported this slideshow.

Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton

1

Share

1 of 74
1 of 74

Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton

1

Share

Download to read offline

Developers love Linux containers, which neatly package up an application and its dependencies and are easy to create and share. However, this unbeatable developer experience hides some deployment challenges for real applications: how do you wire together pieces of a multi-container application? Where do you store your persistent data if your containers are ephemeral? Do containers really contain and isolate your application, or are they merely hiding potential security vulnerabilities? Are your containers scheduled across your compute resources efficiently, or are they trampling on one another?

Container application platforms like Kubernetes provide the answers to some of these questions. We’ll draw on expertise in Linux security, distributed scheduling, and the Java Virtual Machine to dig deep on the performance and security implications of running in containers. This talk will provide a deep dive into tuning and orchestrating containerized Spark applications. You’ll leave this talk with an understanding of the relevant issues, best practices for containerizing data-processing workloads, and tips for taking advantage of the latest features and fixes in Linux Containers, the JDK, and Kubernetes. You’ll leave inspired and enabled to deploy high-performance Spark applications without giving up the security you need or the developer-friendly workflow you want.

Developers love Linux containers, which neatly package up an application and its dependencies and are easy to create and share. However, this unbeatable developer experience hides some deployment challenges for real applications: how do you wire together pieces of a multi-container application? Where do you store your persistent data if your containers are ephemeral? Do containers really contain and isolate your application, or are they merely hiding potential security vulnerabilities? Are your containers scheduled across your compute resources efficiently, or are they trampling on one another?

Container application platforms like Kubernetes provide the answers to some of these questions. We’ll draw on expertise in Linux security, distributed scheduling, and the Java Virtual Machine to dig deep on the performance and security implications of running in containers. This talk will provide a deep dive into tuning and orchestrating containerized Spark applications. You’ll leave this talk with an understanding of the relevant issues, best practices for containerizing data-processing workloads, and tips for taking advantage of the latest features and fixes in Linux Containers, the JDK, and Kubernetes. You’ll leave inspired and enabled to deploy high-performance Spark applications without giving up the security you need or the developer-friendly workflow you want.

More Related Content

More from Spark Summit

Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton

  1. 1. OPTIMIZING SPARK DEPLOYMENTS FOR CONTAINERS: ISOLATION, SAFETY, AND PERFORMANCE William Benton • @willb Red Hat, Inc.
  2. 2. OPTIMIZING SPARK DEPLOYMENTS FOR CONTAINERS: ISOLATION, SAFETY, AND PERFORMANCE William Benton • @willb Red Hat, Inc.
  3. 3. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions
  4. 4. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions Architectural concerns
  5. 5. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions Architectural concerns Security concerns
  6. 6. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways
  7. 7. Preliminaries
  8. 8. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform?
  9. 9. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  10. 10. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  11. 11. pid root / net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  12. 12. pid root / net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  13. 13. pid root /tmp/foo net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 container runtime
  14. 14. pid root /tmp/foo net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 container runtime SPEED LIMIT 55
  15. 15. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 /
  16. 16. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 /
  17. 17. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform?
  18. 18. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs.
  19. 19. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs. …a way to provide reasonable, not exhaustive application isolation.
  20. 20. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs. …a way to provide reasonable, not exhaustive application isolation. …yes, but really just any Linux process with some special settings!
  21. 21. Architectural considerations
  22. 22. Microservice architectures
  23. 23. Microservice architectures
  24. 24. Microservice architectures
  25. 25. Microservice architectures
  26. 26. High-level app architecture federate events databases file, object storage transform transform transform archive
  27. 27. High-level app architecture federate trainmodels events databases file, object storage transform transform transform archive
  28. 28. High-level app architecture federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
  29. 29. High-level app architecture federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
  30. 30. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
  31. 31. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
  32. 32. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform Spark is a natural fit for microservice architectures, since executors are microservices!
  33. 33. Monolithic Spark clusters Cluster scheduler Shared FS / object store Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3 Databases
  34. 34. Monolithic Spark clusters Cluster scheduler Shared FS / object store Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3 Databases
  35. 35. One cluster per application Resource manager Shared FS / object store app 1 app 2 app 5app 4 app 3 app 6 Databases
  36. 36. One cluster per application Resource manager Shared FS / object store app 1 app 2 app 5app 4 app 3 app 6 app 2 app 4 Databases
  37. 37. Security
  38. 38. systemd qemu qemu qemu
  39. 39. systemd nginx mongodb spark-class /tmp/foo /tmp/bar /tmp/blah
  40. 40. systemd nginx mongodb spark-class
  41. 41. spark-class /tmp/foo systemd nginx Use SELinux
  42. 42. spark-class /tmp/foo systemd nginx Use SELinux
  43. 43. spark-class /tmp/foo systemd nginx SELinux limits your exposure to an exploit in a container or a bug in a container runtime. Use SELinux
  44. 44. Root is root … /tmp/foo
  45. 45. Root is root … /
  46. 46. Denials of service … /tmp/foo
  47. 47. Denials of service … /tmp/foo
  48. 48. Kernel panics … /tmp/foo
  49. 49. Kernel panics … /tmp/foo
  50. 50. Keeping secrets … /tmp/foo
  51. 51. Keeping secrets … /tmp/foo Shared FS / object store ACCESS_KEY=… SECRET_KEY=…
  52. 52. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt
  53. 53. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=…
  54. 54. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=… kubectl create secret generic mysecrets --from-file=… --from-file=…
  55. 55. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=… kubectl create secret generic mysecrets --from-file=… --from-file=…
  56. 56. Performance
  57. 57. Potential performance pitfalls
  58. 58. Potential performance pitfalls Hypervisors introduce overhead. Use more lightweight isolation mechanisms to preserve performance.
  59. 59. Potential performance pitfalls
  60. 60. Potential performance pitfalls
  61. 61. Potential performance pitfalls Virtualized networking likely has minimal impact on overall application performance!
  62. 62. Potential performance pitfalls Virtualized networking likely has minimal impact on overall application performance! …but measure the performance of your I/O configuration!
  63. 63. Potential performance pitfalls
  64. 64. Potential performance pitfalls SPEED LIMIT 55
  65. 65. Potential performance pitfalls SPEED LIMIT 55 Quotas mean some ubiquitous techniques can have surprising performance impact. Consider in particular parallel GC and disk buffer cache use.
  66. 66. Potential performance pitfalls SPEED LIMIT 55 Be sure you set your heap sizes based on your resource limits…or wait for OpenJDK 9!
  67. 67. Conclusions and takeaways
  68. 68. Architectural takeaways Spark executors are already microservices. Consider using a single Spark cluster per application for flexible scheduling and easy deployments. Persistent storage lives outside of containers and is probably best accessed via service interfaces rather than through filesystem interfaces.
  69. 69. Security takeaways It isn’t safe to run arbitrary code just because you put it in a container. Use SELinux to minimize your exposure to error and malice. Don’t run as root unless you absolutely have to (and you probably don’t). Ad hoc mechanisms for configuring secrets are likely to leak information and are almost always a bad idea.
  70. 70. Performance takeaways Avoid hypervisor overhead by using different approaches to isolation. Measure everything, but virtualized networking likely has a minimal performance impact on real applications. Artificially throttled performance can be a real problem. Experiment with JVM settings, including serial GC, to reduce your chance of getting limited.
  71. 71. Configuration takeaways If you consume logs from standard output and error, consider using an alternate stack trace formatter to get exceptions in a single log record. If you use ephemeral user IDs, set SPARK_USER or use nss_wrapper so Hadoop file libraries won’t get confused.
  72. 72. Thanks! willb@redhat.com • @willb http://radanalytics.io https://chapeau.freevariable.com

×