Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
OPTIMIZING SPARK DEPLOYMENTS
FOR CONTAINERS: ISOLATION,
SAFETY, AND PERFORMANCE
William Benton • @willb
Red Hat, Inc.
OPTIMIZING SPARK DEPLOYMENTS
FOR CONTAINERS: ISOLATION,
SAFETY, AND PERFORMANCE
William Benton • @willb
Red Hat, Inc.
Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways...
Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways...
Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways...
Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways
Preliminaries
What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtim...
pid
root
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
pid
root
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
pid
root /
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
pid
root /
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
pid
root /tmp/foo
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
container runtime
pid
root /tmp/foo
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
container runtime
SP...
pid
root
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
/
pid
root
net
$SPARK_HOME/bin/spark-class 
org.apache.spark.deploy.worker.Worker 
master:7077
/
What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtim...
What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtim...
What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtim...
What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtim...
Architectural considerations
Microservice architectures
Microservice architectures
Microservice architectures
Microservice architectures
High-level app architecture
federate
events
databases
file, object
storage
transform
transform
transform
archive
High-level app architecture
federate
trainmodels
events
databases
file, object
storage
transform
transform
transform
archive
High-level app architecture
federate
trainmodels
events
databases
file, object
storage
management
web and mobile
reporting
...
High-level app architecture
federate
trainmodels
events
databases
file, object
storage
management
web and mobile
reporting
...
High-level app architecture
federate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
re...
High-level app architecture
federate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
re...
High-level app architecture
federate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
re...
Monolithic Spark clusters
Cluster scheduler
Shared FS /
object store
Spark executor
Spark executor
Spark executor
Spark ex...
Monolithic Spark clusters
Cluster scheduler
Shared FS /
object store
Spark executor
Spark executor
Spark executor
Spark ex...
One cluster per application
Resource manager
Shared FS /
object store
app 1 app 2
app 5app 4
app 3
app 6
Databases
One cluster per application
Resource manager
Shared FS /
object store
app 1 app 2
app 5app 4
app 3
app 6
app 2
app 4
Datab...
Security
systemd
qemu
qemu
qemu
systemd
nginx
mongodb
spark-class
/tmp/foo
/tmp/bar
/tmp/blah
systemd
nginx
mongodb
spark-class
spark-class
/tmp/foo
systemd
nginx
Use SELinux
spark-class
/tmp/foo
systemd
nginx
Use SELinux
spark-class
/tmp/foo
systemd
nginx
SELinux limits your exposure to an exploit in a
container or a bug in a container runti...
Root is root
…
/tmp/foo
Root is root
…
/
Denials of service
…
/tmp/foo
Denials of service
…
/tmp/foo
Kernel panics
…
/tmp/foo
Kernel panics
…
/tmp/foo
Keeping secrets
…
/tmp/foo
Keeping secrets
…
/tmp/foo
Shared FS /
object store
ACCESS_KEY=…
SECRET_KEY=…
Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt
Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt
export ACCESS_KEY=…
export SECRET_...
Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt
export ACCESS_KEY=…
export SECRET_...
Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt
export ACCESS_KEY=…
export SECRET_...
Performance
Potential performance pitfalls
Potential performance pitfalls
Hypervisors introduce overhead.
Use more lightweight isolation
mechanisms to preserve perfo...
Potential performance pitfalls
Potential performance pitfalls
Potential performance pitfalls
Virtualized networking likely
has minimal impact on overall
application performance!
Potential performance pitfalls
Virtualized networking likely
has minimal impact on overall
application performance!
…but m...
Potential performance pitfalls
Potential performance pitfalls
SPEED
LIMIT
55
Potential performance pitfalls
SPEED
LIMIT
55
Quotas mean some ubiquitous techniques can have
surprising performance impac...
Potential performance pitfalls
SPEED
LIMIT
55
Be sure you set your heap sizes based on your
resource limits…or wait for Op...
Conclusions and takeaways
Architectural takeaways
Spark executors are already microservices.
Consider using a single Spark cluster per application f...
Security takeaways
It isn’t safe to run arbitrary code just because you put it in a container.
Use SELinux to minimize you...
Performance takeaways
Avoid hypervisor overhead by using different approaches to isolation.
Measure everything, but virtua...
Configuration takeaways
If you consume logs from standard output and error, consider using an
alternate stack trace formatt...
Thanks!
willb@redhat.com • @willb
http://radanalytics.io
https://chapeau.freevariable.com
Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton
Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton
Upcoming SlideShare
Loading in …5
×

Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton

919 views

Published on

Developers love Linux containers, which neatly package up an application and its dependencies and are easy to create and share. However, this unbeatable developer experience hides some deployment challenges for real applications: how do you wire together pieces of a multi-container application? Where do you store your persistent data if your containers are ephemeral? Do containers really contain and isolate your application, or are they merely hiding potential security vulnerabilities? Are your containers scheduled across your compute resources efficiently, or are they trampling on one another?

Container application platforms like Kubernetes provide the answers to some of these questions. We’ll draw on expertise in Linux security, distributed scheduling, and the Java Virtual Machine to dig deep on the performance and security implications of running in containers. This talk will provide a deep dive into tuning and orchestrating containerized Spark applications. You’ll leave this talk with an understanding of the relevant issues, best practices for containerizing data-processing workloads, and tips for taking advantage of the latest features and fixes in Linux Containers, the JDK, and Kubernetes. You’ll leave inspired and enabled to deploy high-performance Spark applications without giving up the security you need or the developer-friendly workflow you want.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton

  1. 1. OPTIMIZING SPARK DEPLOYMENTS FOR CONTAINERS: ISOLATION, SAFETY, AND PERFORMANCE William Benton • @willb Red Hat, Inc.
  2. 2. OPTIMIZING SPARK DEPLOYMENTS FOR CONTAINERS: ISOLATION, SAFETY, AND PERFORMANCE William Benton • @willb Red Hat, Inc.
  3. 3. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions
  4. 4. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions Architectural concerns
  5. 5. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions Architectural concerns Security concerns
  6. 6. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways
  7. 7. Preliminaries
  8. 8. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform?
  9. 9. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  10. 10. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  11. 11. pid root / net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  12. 12. pid root / net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077
  13. 13. pid root /tmp/foo net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 container runtime
  14. 14. pid root /tmp/foo net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 container runtime SPEED LIMIT 55
  15. 15. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 /
  16. 16. pid root net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 /
  17. 17. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform?
  18. 18. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs.
  19. 19. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs. …a way to provide reasonable, not exhaustive application isolation.
  20. 20. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs. …a way to provide reasonable, not exhaustive application isolation. …yes, but really just any Linux process with some special settings!
  21. 21. Architectural considerations
  22. 22. Microservice architectures
  23. 23. Microservice architectures
  24. 24. Microservice architectures
  25. 25. Microservice architectures
  26. 26. High-level app architecture federate events databases file, object storage transform transform transform archive
  27. 27. High-level app architecture federate trainmodels events databases file, object storage transform transform transform archive
  28. 28. High-level app architecture federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
  29. 29. High-level app architecture federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
  30. 30. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
  31. 31. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
  32. 32. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform Spark is a natural fit for microservice architectures, since executors are microservices!
  33. 33. Monolithic Spark clusters Cluster scheduler Shared FS / object store Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3 Databases
  34. 34. Monolithic Spark clusters Cluster scheduler Shared FS / object store Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3 Databases
  35. 35. One cluster per application Resource manager Shared FS / object store app 1 app 2 app 5app 4 app 3 app 6 Databases
  36. 36. One cluster per application Resource manager Shared FS / object store app 1 app 2 app 5app 4 app 3 app 6 app 2 app 4 Databases
  37. 37. Security
  38. 38. systemd qemu qemu qemu
  39. 39. systemd nginx mongodb spark-class /tmp/foo /tmp/bar /tmp/blah
  40. 40. systemd nginx mongodb spark-class
  41. 41. spark-class /tmp/foo systemd nginx Use SELinux
  42. 42. spark-class /tmp/foo systemd nginx Use SELinux
  43. 43. spark-class /tmp/foo systemd nginx SELinux limits your exposure to an exploit in a container or a bug in a container runtime. Use SELinux
  44. 44. Root is root … /tmp/foo
  45. 45. Root is root … /
  46. 46. Denials of service … /tmp/foo
  47. 47. Denials of service … /tmp/foo
  48. 48. Kernel panics … /tmp/foo
  49. 49. Kernel panics … /tmp/foo
  50. 50. Keeping secrets … /tmp/foo
  51. 51. Keeping secrets … /tmp/foo Shared FS / object store ACCESS_KEY=… SECRET_KEY=…
  52. 52. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt
  53. 53. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=…
  54. 54. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=… kubectl create secret generic mysecrets --from-file=… --from-file=…
  55. 55. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=… kubectl create secret generic mysecrets --from-file=… --from-file=…
  56. 56. Performance
  57. 57. Potential performance pitfalls
  58. 58. Potential performance pitfalls Hypervisors introduce overhead. Use more lightweight isolation mechanisms to preserve performance.
  59. 59. Potential performance pitfalls
  60. 60. Potential performance pitfalls
  61. 61. Potential performance pitfalls Virtualized networking likely has minimal impact on overall application performance!
  62. 62. Potential performance pitfalls Virtualized networking likely has minimal impact on overall application performance! …but measure the performance of your I/O configuration!
  63. 63. Potential performance pitfalls
  64. 64. Potential performance pitfalls SPEED LIMIT 55
  65. 65. Potential performance pitfalls SPEED LIMIT 55 Quotas mean some ubiquitous techniques can have surprising performance impact. Consider in particular parallel GC and disk buffer cache use.
  66. 66. Potential performance pitfalls SPEED LIMIT 55 Be sure you set your heap sizes based on your resource limits…or wait for OpenJDK 9!
  67. 67. Conclusions and takeaways
  68. 68. Architectural takeaways Spark executors are already microservices. Consider using a single Spark cluster per application for flexible scheduling and easy deployments. Persistent storage lives outside of containers and is probably best accessed via service interfaces rather than through filesystem interfaces.
  69. 69. Security takeaways It isn’t safe to run arbitrary code just because you put it in a container. Use SELinux to minimize your exposure to error and malice. Don’t run as root unless you absolutely have to (and you probably don’t). Ad hoc mechanisms for configuring secrets are likely to leak information and are almost always a bad idea.
  70. 70. Performance takeaways Avoid hypervisor overhead by using different approaches to isolation. Measure everything, but virtualized networking likely has a minimal performance impact on real applications. Artificially throttled performance can be a real problem. Experiment with JVM settings, including serial GC, to reduce your chance of getting limited.
  71. 71. Configuration takeaways If you consume logs from standard output and error, consider using an alternate stack trace formatter to get exceptions in a single log record. If you use ephemeral user IDs, set SPARK_USER or use nss_wrapper so Hadoop file libraries won’t get confused.
  72. 72. Thanks! willb@redhat.com • @willb http://radanalytics.io https://chapeau.freevariable.com

×