Notebooks: they enable our users, but they can cripple our clusters. Let’s fix that. Notebooks have soared in popularity at companies world-wide because they provide an easy, user-friendly way of accessing the cluster-computing power of Spark. But the more users you have hitting a cluster, the harder it is to manage the cluster resources as big, long-running jobs start to starve out small, short-running jobs. While you could have users spin up EMR-style clusters, this reduces the ability to take advantage of the collaborative nature of notebooks. It also quickly becomes expensive as clusters sit idle for long periods of time waiting on single users. What we want is fair, efficient resource utilization on a large single cluster for a large number of users. In this talk we’ll discuss dynamic allocation and the best practices for configuring the current version of Spark as-is to help solve this problem. We’ll also present new improvements we’ve made to address this use case. These include: decommissioning executors without losing cached data, proactively shutting down executors to prevent starvation, and improving the start times of new executors.

1. Brad Kaiser, IBM/TWC
Craig Ingram, IBM/TWC
Supporting Highly
Multitenant Spark
Notebook Workloads
#EUdev8

2. Hosting Multitenant Spark Notebooks Is
Hard But It Doesn't Have To Be
• Our Journey
• Best Practices
• New Work
2#EUdev8

3. Our Journey
3#EUdev8

4. Who we are
4#EUdev8

5. IBM's Commitment to Open Source
5#EUdev8
• Contribute intellectual and technical
capital to the Apache Spark
community.
• Make the core technology enterprise-
and cloud-ready.
• Build data science skills to drive
intelligence into business applications
— https://cognitiveclass.ai/
• http://spark.tc

6. Mission
• Provide:
– a secure, performant, stable cluster.
– interactive analytics, visualizations, and reports.
– collaboration and sharing with other data scientists,
engineers, and consumers.
– job scheduling capabilities.
– a quick and easy way to get started with Spark.
6#EUdev8

7. Goals
• Support hundreds of analysts/data scientists
using Spark
– Quick kernel creation (<10s from notebook creation to
available sc).
– Utilize cluster resources efficiently.
– Elastically scale based on load.
7#EUdev8

8. Lessons Learned at TWC
8#EUdev8

9. One Big Cluster
9#EUdev8
• Spark collocated with
Cassandra
• Fast
• Stable

10. One Big Cluster - Cons
10#EUdev8
• Outside analysts start using
our cluster
• Provided notebook services
• Interrupted our perfectly
scheduled jobs
• Used a lot of resources
causing Cassandra to crash

11. Add Smaller Clusters
11#EUdev8
• We built some smaller
clusters
• Still platform agnostic
• Other teams couldn't
affect our production
cluster

12. Add Smaller Clusters - Cons
12#EUdev8
• Hassle to set up
• Required a lot of
maintenance
• Sat idle

13. EMR
• Analysts make ad hoc clusters for their needs
• No maintenance from us
• Learning curve for analysts
• They tend to leave them running
13#EUdev8

14. Lessons Learned - IBM
14#EUdev8

15. Data Science Experience
• Collaborative environment on the front end
– Collaboration Tools
– Shared Data Sets
– Flows
– GitHub Integration
• Multiple compute environments on the back end
– DSX on the cloud: compute runs on IBM cloud
– DSX Local: compute runs on private cloud or Z
15#EUdev8

16. Lessons Learned - IBM
• You need kernel remoting
– Allows advanced collaborative tools in the application tier
– Allows resource consolidation in the analytics tier
• Resource consolidation puts stress on the analytics
tier
– Starvation
– Management of cached data
– Performance bottlenecks (example: Spark web UI)
16#EUdev8

17. Best Practices
17#EUdev8

18. Best Practices
• Use kernel remoting
• Use fewer, bigger clusters
• Know your workloads
• Isolate users
• Schedule resources efficiently
18#EUdev8

19. Use Kernel Remoting
• Running all of your notebook kernels on the
same server is a bottle neck
• Run your kernels distributed on the cluster
• You can run a lot more notebooks
19#EUdev8

20. Jupyter Enterprise Gateway
• New Open Source project from IBM
• Goals:
– Allow hundreds of notebook users to share a single Spark
cluster…
– …with enterprise-level security and performance.
• Used in IBM Analytics Engine (GA)
• developer.ibm.com/code/openprojects/jupyter-
enterprise-gateway-2
20#EUdev8

21. Jupyter Enterprise Gateway:
How it Works
21#EUdev8
Spark Cluster
Security
Layer
Jupyter Enterprise Gateway
• Multitenant
• Remote Kernel Lifecycle Management
YARN
Workers
Impersonation:
Alice’s kernel
runs under
Alice’s user ID.
Spark
ExecutorsSpark Executors
Spark Executors
Yarn Container
Jupyter Kernel
Spark Driver
Spark
ExecutorsSpark Executors
Spark Executors
Yarn Container
Jupyter Kernel
Spark Driver
Secure
Secure
Secure Secure

22. Use Fewer, Bigger Clusters
• Better resource utilization through statistical multiplexing
• Improved security and auditing due to centralization
– Hive, Ranger, and Atlas are common in the ecosystem.
– Many new, platform specific solutions to address this problem.
• Easier collaboration between users
– Shared notebooks with interactive visualizations and markdown
support.
– GitHub integration for versioning and external sharing.
– Catalog based data discovery and sharing.
– Governance and auditing support.
22#EUdev8

23. Know your workloads
• What is the main resource they use?
• Overprovision the hardware
• When to scale up and down?
– CPU load
– YARN/RM Queue Stats (depth, waiting jobs, available
CPU/mem, preemptions)
• If containers are getting preempted, it’s due to queues filling
up.
23#EUdev8

24. Isolate Users
• Don't have a generic user account
• Catalog and Governance
– Hive
– Atlas
– Ranger
• You don't want users embedding keys and
passwords in their notebooks.
24#EUdev8

25. Schedule Resources Efficiently
25#EUdev8

26. YARN Queues
• Take advantage of YARN’s hierarchical queue system to manage
and organize resources.
– Over-allocate queues for better resource utilization and sharing.
– Take advantage of node labels for users that have priority jobs
that require an SLA.
– Intra-queue preemption and asynchronous container allocation
should be available in YARN 3.0.
26#EUdev8

27. YARN Queues
27#EUdev8

28. Dynamic Allocation
• Dynamic allocation lets you take advantage of varying activity
– Proactively scales the number of executors based on the
scheduler's backlog.
– Removes idle executors after a timeout.
– Be sure to set a sensible number of initial executors
and minimum executor floor and let them ramp up on
demand.
• Static allocation best for known workloads
28#EUdev8

29. New Work
29#EUdev8

30. Improvements to Spark
• Alleviate the tradeoffs inherent in current best
practices.
– Recover cached data when shutting down idle
executors
– Proactively shut down executors to prevent
starvation
30#EUdev8

31. Recovering Cached Data
• Replicates cached data to remaining executors
before shutting them down.
• Ameliorates cache issues with dynamic
allocation
• Useful in shared spark notebook environments
31#EUdev8

32. Benchmarks
32#EUdev8

33. Benchmarks
33#EUdev8

34. Check out my PR
– SPARK-21097
– github.com/apache/spark/pull/19041
34#EUdev8

35. Preventing Starvation
• Eliminate issues where users are unable to run
anything due to other users taking up all of the
cluster’s resources.
• Especially useful in shared spark notebook
environments where idle resources can be
reclaimed easily.
• Preemption can solve this.
35#EUdev8

36. Enter Preemption
• Requests containers associated with over-
allocated queues to shut down.
• Handle YARN's PreemptionMessage in a way
that best suits the workload.
• Pick the right executors to terminate.
36#EUdev8

37. Keep up with the JIRA
• SPARK-21122
• PR coming soon
37#EUdev8

38. Call to action
• Look at our JIRAs
• Try out our PRs
38#EUdev8

39. Shout-outs!!!
• Our notebook workload simulator, benchmark,
and tracing tools.
– spark-bench - github.com/SparkTC/spark-bench
• Check out Emily Curtin’s talk tomorrow about spark-bench.
– spark-tracing - github.com/SparkTC/spark-tracing
• Matthew Schauer's baby awaiting open-source approval.
• Special thanks to Vijay Bommireddipalli and
Fred Reiss for their guidance and support!
39#EUdev8

40. Contact Info
Brad Kaiser
kaiserb@us.ibm.com
Craig Ingram
cingram@us.ibm.com
40#EUdev8

41. Extra Material
41#EUdev8

42. YARN Asynchronous Scheduling
• Enable asynchronous scheduling of containers in YARN.
– yarn.scheduler.capacity.schedule-asynchronously.enable
– YARN-7327 and YARN-5139
42#EUdev8

43. References
• Some Icons provided by Icons8
43#EUdev8

44. Quick Settings
• Use spark.yarn.jars or
spark.yarn.archive.
• Running Spark from tmpfs did not improve
performance.
• Support for multiple/new versions of Spark.
44#EUdev8
Move to backup

45. Disable unused credential providers
• spark.yarn.security.credentials.hive.enabled
• spark.yarn.security.credentials.hbase.enabled
45#EUdev8
conf avg stdev
default 8.745 0.051
nohive 8.7 0.05
nohbase 7.87 0.05
Move to backup

46. Problem Domain
• Security
– UI and service protection
– Data governance and auditing
• Stability
• Performance
46#EUdev8

47. What’s next…
• spark-on-k8s
• Scheduler improvements
• Executor startup time reduction
47#EUdev8

48. • Hundreds of notebook users leads to a
highly multitenant and interactive workload.
• Challenge: Give each user the illusion of having
a large cluster all to herself
48#EUdev8
Many users
at once
Bursty
offered load
Latency is
important

49. What platforms are out there…
Hosted Solutions
• Data Science Experience
(DSX)/Watson Data Platform (WDP)
• databricks
• Google Cloud Platform
• Microsoft Azure HDInsight
• and others!!!
49#EUdev8

50. What platforms are out there…
Self-Hosted Solutions
• HDP – Hadoop Data Platform
• CDH – Cloudera Distribution
Including Hadoop
• MapR
• Mesosphere
50#EUdev8

51. Reasons to Self Host at TWC
• Cloud Agnostic
• Flexibility
• Sensitive Data
• Fewer Options in 2014
• Cassandra Colocation
• Cost…maybe?
51#EUdev8

52. Potential Downfalls
• In a self-hosted environment, everything is up to
you.
– Security
– Stability and Performance
– Scalability
• Compute
• Storage
– Monitoring
– Alerting
– Logging
52#EUdev8