The document discusses integrating Docker, Mesos, Spark, Marathon, and Chronos into a unified big data platform. Docker provides containerization capabilities, while Mesos is a distributed resource manager that supports running Docker containers. Spark can run natively on Mesos by running Spark as a Docker container within Mesos. Marathon and Chronos help manage long-running services and cron jobs on Mesos. The author will demonstrate how to put these technologies together into an integrated system and address running Spark on Mesos in Docker containers.

1. DOCKER, MESOS, SPARK
LET’S PUT THEM TOGETHER
FIRST SPARK MEETUP SINGAPORE 2016

2. WHO AM I
• I am spark user,
• I am big data platform builder,
• I am a Docker fan.

3. WHY, I AM HERE
• I am here not because of I am super good at
Spark,
• I am here just in case, someone don’t know that,
Spark works with Mesos and Docker quite well,
• I am here just in case, you have the similar issues
we have previously.

4. WHAT AM GOING TO TALK
• DOCKER
• MESOS
• SPARK
• MARATHON
• CHRONOS
• DATA PLATFORM AS A SYSTEM

5. DOCKER
Docker is containerizer Technology built
on top of three important techniques:
- Namespace
- Cgroup
- Union Filesystem
Multi-host docker container manager
- Kubernetes
- Mesos
- Swarm
- Many other projects

6. MESOS

7. SPARK
• Natively, SPARK can run on top of Mesos.
• Since, mesos support docker container as task executor.
Spark can run as docker container in mesos.

8. MARATHON
• I ran MARATHON twice. It’s not easy.
• Marathon help you run long-term running
services on mesos.
• Your RESTful API services, your long-term
running workers can be driven by Marathon.

9. CHRONOS
• Chronos is run as a distributed cron-job
manager.
• If your cron-jobs are not strict time-sensitive,
chronos can help you manage all these jobs.

10. INTEGRATED SYSTEM

11. INTEGRATED SYSTEM

12. INTEGRATED SYSTEM

13. SOME POINTS I MISSED
• Benefits for running spark in docker with mesos.
• Scenario 1: You want to use different version of spark running in the same infra
without teardown instances or multiple version of spark running at the same time in
the infra for different team.
• Scenario 2: Want to use different dependencies (jar, python packages) in different
spark context/projects in the same cluster.
• How to check the spark ui, when spark run on mesos.
• Each mesos framework has a link in mesos dashboard frameworks tab. That link for
spark is spark ui link.

14. DEMO
• SPARK DEMO REPO
• CUSTOMIZED MANTL