The document discusses containerizing Spark clusters on Kubernetes. It describes how the author's Spark cluster looked in 2014 running on Mesos with networked storage. It then covers motivations for microservices architectures and how Spark fits into this. The document outlines architectures for analytics and applications, including responsibilities like transformation, aggregation, training models, and more. It also discusses legacy architectures like data warehouses and Hadoop-style data lakes. Finally, it covers practical considerations and potential pitfalls of containerized Spark clusters like scheduling, security, and storage options.

1. William Benton
Red Hat, Inc.
@willb • willb@redhat.com
CONTAINERIZED SPARK
ON KUBERNETES

2. BACKGROUND

3. BACKGROUND

4. BACKGROUND

5. BACKGROUND

6. BACKGROUND

7. BACKGROUND

8. BACKGROUND

9. BACKGROUND

10. WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014

11. WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor

12. WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014
Networked
POSIX FS
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor

13. Mesos
WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014
Networked
POSIX FS
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor

14. Mesos
WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014
Networked
POSIX FS
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
1
2
3
4

15. Mesos
WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014
Networked
POSIX FS
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
1
2
3
4
1
1
2
3
3
4

16. Mesos
WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014
Networked
POSIX FS
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
1
2
3
4
1
1
2
3
3
4

17. Analytics is no longer a
separate workload.

18. Analytics is an essential
component of modern data-
driven applications.

19. OUR GOALS

20. OUR GOALS

21. OUR GOALS

22. OUR GOALS

23. OUR GOALS
git

24. OUR GOALS
git

25. FORECAST
Motivating containerized microservices
Architectures for analytics and applications
Spark clusters in containers: practicalities and pitfalls
Play along at home
Future work

26. MOTIVATING MICROSERVICES

27. A microservice architecture
employs lightweight, modular,
and typically stateless
components with well-defined
interfaces and contracts.

28. BENEFITS OF MICROSERVICE ARCHITECTURES

29. BENEFITS OF MICROSERVICE ARCHITECTURES

30. BENEFITS OF MICROSERVICE ARCHITECTURES

31. BENEFITS OF MICROSERVICE ARCHITECTURES

32. BENEFITS OF MICROSERVICE ARCHITECTURES

33. BENEFITS OF MICROSERVICE ARCHITECTURES

34. BENEFITS OF MICROSERVICE ARCHITECTURES

35. BENEFITS OF MICROSERVICE ARCHITECTURES
2 + 2

36. BENEFITS OF MICROSERVICE ARCHITECTURES
2 + 2 5

37. BENEFITS OF MICROSERVICE ARCHITECTURES
2 + 2 5

38. BENEFITS OF MICROSERVICE ARCHITECTURES

39. BENEFITS OF MICROSERVICE ARCHITECTURES

40. BENEFITS OF MICROSERVICE ARCHITECTURES
?

41. BENEFITS OF MICROSERVICE ARCHITECTURES
?

42. BENEFITS OF MICROSERVICE ARCHITECTURES
?

43. MICROSERVICES AND SPARK
executor
1 2 3
executor
4 5 6
executor
7 8 9
executor
10 11 12
master

44. MICROSERVICES AND SPARK
executor
1 2 3
executor
4 5 6
executor
7 8 9
executor
10 11 12
master
λ x: x * 2

45. MICROSERVICES AND SPARK
executor
1 2 3
executor
4 5 6
executor
7 8 9
executor
10 11 12
master
λ x: x * 22 4 6 8 10 12 14 16 18 20 22 24
λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2

46. MICROSERVICES AND SPARK
executor
1 2 3
executor
4 5 6
executor
7 8 9
executor
10 11 12
master
λ x: x * 22 4 6 8 10 12 14 16 18 20 22 24
λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2

47. MICROSERVICES AND SPARK
executor
1 2 3
executor
4 5 6
executor
7 8 9
executor
10 11 12
master
λ x: x * 22 4 6 8 10 12 14 16 18 20 22 24
λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2

48. ARCHITECTURES FOR
ANALYTICS AND APPLICATIONS

49. APPLICATION RESPONSIBILITIES
transform
transform
transform
events
databases
file, object
storage

50. APPLICATION RESPONSIBILITIES
transform
transform
transform
aggregate
events
databases
file, object
storage

51. APPLICATION RESPONSIBILITIES
trainmodels
transform
transform
transform
aggregate
events
databases
file, object
storage

52. APPLICATION RESPONSIBILITIES
trainmodels
transform
transform
transform
aggregate
events
databases
file, object
storage

53. APPLICATION RESPONSIBILITIES
archive
trainmodels
transform
transform
transform
aggregate
events
databases
file, object
storage

54. APPLICATION RESPONSIBILITIES
archive
trainmodels
transform
transform
transform
aggregate
events
databases
file, object
storage

55. APPLICATION RESPONSIBILITIES
archive
trainmodels
transform
transform
transform
aggregate
events
databases
file, object
storage
web and mobile
reporting
developer UI

56. APPLICATION RESPONSIBILITIES
archive
trainmodels
transform
transform
transform
aggregate
events
databases
file, object
storage
management
web and mobile
reporting
developer UI

57. LEGACY ARCHITECTURES

58. CONVENTIONAL DATA WAREHOUSE
events

59. CONVENTIONAL DATA WAREHOUSE
transformevents

60. CONVENTIONAL DATA WAREHOUSE
transformevents
UI

61. CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic

62. CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS

63. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS

64. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS

65. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS RDBMS

66. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS RDBMS
analysis

67. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS RDBMS
analysis
reporting

68. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS RDBMS
analysis
reporting

69. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS RDBMS
analysis
interactive
query
reporting

70. transaction
processing
CONVENTIONAL DATA WAREHOUSE
transformevents
UI
business
logic
RDBMS analytic
processing
RDBMS
analysis
interactive
query
reporting

71. HADOOP-STYLE “DATA LAKE”
HDFS HDFS HDFS

72. HADOOP-STYLE “DATA LAKE”
HDFS HDFS HDFS HDFS HDFS

73. HADOOP-STYLE “DATA LAKE”
HDFS
events
HDFS HDFS HDFS HDFS

74. HADOOP-STYLE “DATA LAKE”
HDFS
events
HDFS HDFS HDFS HDFS

75. HADOOP-STYLE “DATA LAKE”
HDFS
compute
events
HDFS
compute
HDFS
compute compute compute
HDFS HDFS

76. MODERN ARCHITECTURES

77. THE LAMBDA ARCHITECTURE
events
(imprecise)
analysistransform

78. THE LAMBDA ARCHITECTURE
events
(imprecise)
analysistransform
DFS

79. speed layer
THE LAMBDA ARCHITECTURE
events
(imprecise)
analysistransform
DFS

80. speed layer
THE LAMBDA ARCHITECTURE
events
(precise)
analysistransform
(imprecise)
analysistransform
DFS

81. speed layer
THE LAMBDA ARCHITECTURE
events
batch layer
(precise)
analysistransform
(imprecise)
analysistransform
DFS

82. speed layer
THE LAMBDA ARCHITECTURE
events
batch layer
federate
(precise)
analysistransform
(imprecise)
analysistransform
DFS

83. speed layer
THE LAMBDA ARCHITECTURE
events
batch layer
UIfederate
(precise)
analysistransform
(imprecise)
analysistransform
DFS

84. serving layerspeed layer
THE LAMBDA ARCHITECTURE
events
batch layer
UIfederate
(precise)
analysistransform
(imprecise)
analysistransform
DFS

85. THE KAPPA ARCHITECTURE
events

86. queue for “raw data” topic
THE KAPPA ARCHITECTURE
events

87. queue for “raw data” topic
THE KAPPA ARCHITECTURE
events
transform
queue for “preprocessed data” topic

88. queue for “raw data” topic
THE KAPPA ARCHITECTURE
events
transform analysis
queue for “preprocessed data” topic
queue for “analysis results” topic

89. queue for “raw data” topic
THE KAPPA ARCHITECTURE
events
transform analysis
queue for “preprocessed data” topic
queue for “analysis results” topic
reporting end-user UI

90. DATA FEDERATION IN THE COMPUTE LAYER
aggregate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform

91. DATA FEDERATION IN THE COMPUTE LAYER
aggregate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform

92. DATA FEDERATION IN THE COMPUTE LAYER
aggregate
trainmodels
archive
events
databases
file, object
storage
management
web and mobile
reporting
developer UItransform
transform
transform

93. Cluster scheduler
SIDEBAR: THE MONOLITHIC SPARK ANTIPATTERN
Shared FS
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Resource manager
app 1 app 2
app 4app 3

94. Cluster scheduler
SIDEBAR: THE MONOLITHIC SPARK ANTIPATTERN
Shared FS
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Spark executor
Resource manager
app 1 app 2
app 4app 3

95. Resource manager
ONE CLUSTER PER APPLICATION
Object stores
app 1 app 2
app 5app 4
app 3
app 6
Databases

96. Resource manager
ONE CLUSTER PER APPLICATION
Object stores
app 1 app 2
app 5app 4
app 3
app 6
app 2
app 4
Databases

97. PRACTICALITIES AND
POTENTIAL PITFALLS

98. SCHEDULING
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
Object stores
Databases

99. SCHEDULING
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1

100. SCHEDULING
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1

101. SCHEDULING
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1

102. SCHEDULING
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1

103. SCHEDULING
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1

104. SECURITY

105. SECURITY

106. SECURITY
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077
pid
root
net

107. SECURITY
$SPARK_HOME/bin/spark-class
org.apache.spark.deploy.worker.Worker
master:7077
pid
root
net
/tmp/foo

108. SECURITY

109. SECURITY
k8s namespace

110. SECURITY
k8s namespace*

111. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6

112. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
POSIX
filesystem

113. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
POSIX
filesystem
✓ familiar interface
✓ interoperability with
other programs

114. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
POSIX
filesystem
✓ familiar interface
✓ interoperability with
other programs
✗ unnecessary
semantic guarantees
✗ difficult to manage

115. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
HDFS HDFS
HDFS HDFS
HDFS
HDFS

116. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
HDFS HDFS
HDFS HDFS
HDFS
HDFS
✓ support for legacy
Hadoop installations

117. ✗ inelastic
✗ stateful
✗ can’t collocate
compute and data
STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
HDFS HDFS
HDFS HDFS
HDFS
HDFS
✓ support for legacy
Hadoop installations

118. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
object store

119. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
object store
✓ interoperability
✓ fine-grained AC
✓ many implementations

120. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
object store
✓ interoperability
✓ fine-grained AC
✓ many implementations
✗ consistency model
✗ performance (?)

121. STORAGE
Kubernetes
app 1 app 2
app 5app 4
app 3
app 6
app 1 app 2
app 5app 4
app 3
app 6
object store
✓ interoperability
✓ fine-grained AC
✓ many implementations
✗ consistency model
✗ performance

122. “…in a cloud native architecture, the benefit of
HDFS is actually very small and that is why
many cloud-first organizations no longer run
HDFS, or only run it as a caching layer for S3.”
—Reynold Xin on Quora (http://qr.ae/TAF4cN)

123. NETWORKING

124. NETWORKING

125. NETWORKING

126. NETWORKING
http://app1:8080

127. NETWORKING
http://app1:8080
✗ can’t access worker web UI
(but wait for Spark 2.1!)

128. NETWORKING
http://app1:8080
✗ can’t access worker web UI
(but wait for Spark 2.1!)

129. NETWORKING
http://app1:8080
✗ can’t access worker web UI
(but wait for Spark 2.1!)
http://app1:80

130. NEXT STEPS: FUTURE WORK &
PLAYING ALONG AT HOME

131. NEXT STEPS
Further performance evaluation
Better developer experience
Improved scheduling of Spark tasks on Kubernetes

132. TRY IT OUT YOURSELF
Kubernetes standalone Spark example:
https://github.com/kubernetes/kubernetes/tree/master/examples/spark
Enabling Spark on OpenShift: https://github.com/radanalyticsio
Native Spark on Kubernetes proposal:
https://github.com/kubernetes/kubernetes/issues/34377

133. @willb • willb@redhat.com
https://chapeau.freevariable.com
THANKS!