Developers love Linux containers, which neatly package up an application and its dependencies and are easy to create and share. However, this unbeatable developer experience hides some deployment challenges for real applications: how do you wire together pieces of a multi-container application? Where do you store your persistent data if your containers are ephemeral? Do containers really contain and isolate your application, or are they merely hiding potential security vulnerabilities? Are your containers scheduled across your compute resources efficiently, or are they trampling on one another? Container application platforms like Kubernetes provide the answers to some of these questions. We’ll draw on expertise in Linux security, distributed scheduling, and the Java Virtual Machine to dig deep on the performance and security implications of running in containers. This talk will provide a deep dive into tuning and orchestrating containerized Spark applications. You’ll leave this talk with an understanding of the relevant issues, best practices for containerizing data-processing workloads, and tips for taking advantage of the latest features and fixes in Linux Containers, the JDK, and Kubernetes. You’ll leave inspired and enabled to deploy high-performance Spark applications without giving up the security you need or the developer-friendly workflow you want.

1. OPTIMIZING SPARK DEPLOYMENTS
FOR CONTAINERS: ISOLATION,
SAFETY, AND PERFORMANCE
William Benton • @willb
Red Hat, Inc.

2. OPTIMIZING SPARK DEPLOYMENTS
FOR CONTAINERS: ISOLATION,
SAFETY, AND PERFORMANCE
William Benton • @willb
Red Hat, Inc.

3. Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways
Background and definitions

4. Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways
Background and definitions
Architectural concerns

5. Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways
Background and definitions
Architectural concerns
Security concerns

6. Forecast
Background and definitions
Architectural concerns
Security concerns
Performance concerns
Conclusions and takeaways

7. Preliminaries

8. What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtime or orchestration platform?

9. pid
root
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077

10. pid
root
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077

11. pid
root /
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077

12. pid
root /
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077

13. pid
root /tmp/foo
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077
container runtime

14. pid
root /tmp/foo
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077
container runtime
SPEED
LIMIT
55

15. pid
root
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077
/

16. pid
root
net
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077
/

17. What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtime or orchestration platform?

18. What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtime or orchestration platform?
…a lightweight means to address some of the same use cases as VMs.

19. What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtime or orchestration platform?
…a lightweight means to address some of the same use cases as VMs.
…a way to provide reasonable, not exhaustive application isolation.

20. What is a container?
…a lightweight VM?
…a way to totally isolate applications?
…a packaging format for a container runtime or orchestration platform?
…a lightweight means to address some of the same use cases as VMs.
…a way to provide reasonable, not exhaustive application isolation.
…yes, but really just any Linux process with some special settings!

21. Architectural considerations

22. Microservice architectures

23. Microservice architectures

24. Microservice architectures

25. Microservice architectures

26. High-level app architecture
federate
events
databases
file, object
storage
transform
transform
transform
archive

27. High-level app architecture
federate
trainmodels
events
databases
file, object
storage
transform
transform
transform
archive

28. High-level app architecture
federate
trainmodels
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform
archive

29. High-level app architecture
federate
trainmodels
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform
archive

30. High-level app architecture
federate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform

31. High-level app architecture
federate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform

32. High-level app architecture
federate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform
Spark is a natural fit for
microservice architectures, since
executors are microservices!

33. Monolithic Spark clusters
Cluster scheduler
Shared FS /
object store
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Resource manager
app 1 app 2
app 4app 3
Databases

34. Monolithic Spark clusters
Cluster scheduler
Shared FS /
object store
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Resource manager
app 1 app 2
app 4app 3
Databases

35. One cluster per application
Resource manager
Shared FS /
object store
app 1 app 2
app 5app 4
app 3
app 6
Databases

36. One cluster per application
Resource manager
Shared FS /
object store
app 1 app 2
app 5app 4
app 3
app 6
app 2
app 4
Databases

37. Security

40. systemd
qemu
qemu
qemu

41. systemd
nginx
mongodb
spark-class
/tmp/foo
/tmp/bar
/tmp/blah

42. systemd
nginx
mongodb
spark-class

43. spark-class
/tmp/foo
systemd
nginx
Use SELinux

44. spark-class
/tmp/foo
systemd
nginx
Use SELinux

45. spark-class
/tmp/foo
systemd
nginx
SELinux limits your exposure to an exploit in a
container or a bug in a container runtime.
Use SELinux

46. Root is root
…
/tmp/foo

47. Root is root
…
/

48. Denials of service
…
/tmp/foo

49. Denials of service
…
/tmp/foo

50. Kernel panics
…
/tmp/foo

51. Kernel panics
…
/tmp/foo

52. Keeping secrets
…
/tmp/foo

53. Keeping secrets
…
/tmp/foo
Shared FS /
object store
ACCESS_KEY=…
SECRET_KEY=…

54. Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt

55. Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt
export ACCESS_KEY=…
export SECRET_KEY=…

56. Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt
export ACCESS_KEY=…
export SECRET_KEY=…
kubectl create secret
generic mysecrets
--from-file=…
--from-file=…

57. Keeping secrets
cat <<EOF > secret.txt
ACCESS_KEY=…
SECRET_KEY=…
EOF
git add secret.txt
export ACCESS_KEY=…
export SECRET_KEY=…
kubectl create secret
generic mysecrets
--from-file=…
--from-file=…

58. Performance

59. Potential performance pitfalls

60. Potential performance pitfalls
Hypervisors introduce overhead.
Use more lightweight isolation
mechanisms to preserve performance.

61. Potential performance pitfalls

62. Potential performance pitfalls

63. Potential performance pitfalls
Virtualized networking likely
has minimal impact on overall
application performance!

64. Potential performance pitfalls
Virtualized networking likely
has minimal impact on overall
application performance!
…but measure the
performance of your
I/O configuration!

65. Potential performance pitfalls

66. Potential performance pitfalls
SPEED
LIMIT
55

67. Potential performance pitfalls
SPEED
LIMIT
55
Quotas mean some ubiquitous techniques can have
surprising performance impact. Consider in particular
parallel GC and disk buffer cache use.

68. Potential performance pitfalls
SPEED
LIMIT
55
Be sure you set your heap sizes based on your
resource limits…or wait for OpenJDK 9!

69. Conclusions and takeaways

70. Architectural takeaways
Spark executors are already microservices.
Consider using a single Spark cluster per application for flexible
scheduling and easy deployments.
Persistent storage lives outside of containers and is probably best
accessed via service interfaces rather than through filesystem interfaces.

71. Security takeaways
It isn’t safe to run arbitrary code just because you put it in a container.
Use SELinux to minimize your exposure to error and malice.
Don’t run as root unless you absolutely have to (and you probably don’t).
Ad hoc mechanisms for configuring secrets are likely to leak information
and are almost always a bad idea.

72. Performance takeaways
Avoid hypervisor overhead by using different approaches to isolation.
Measure everything, but virtualized networking likely has a minimal
performance impact on real applications.
Artificially throttled performance can be a real problem. Experiment with
JVM settings, including serial GC, to reduce your chance of getting limited.

73. Configuration takeaways
If you consume logs from standard output and error, consider using an
alternate stack trace formatter to get exceptions in a single log record.
If you use ephemeral user IDs, set SPARK_USER or use nss_wrapper so
Hadoop file libraries won’t get confused.

74. Thanks!
willb@redhat.com • @willb
http://radanalytics.io
https://chapeau.freevariable.com