Developers are moving away from their host-based patterns and adopting a new mindset around the idea that the datacenter is the computer. It?s quickly becoming a mainstream model that you can view a warehouse full of servers as a single computer (with terabytes of memory and tens of thousands of cores). There is a key missing piece, which is an operating system for the datacenter (DCOS), which would provide the same OS functionality and core OS abstractions across thousands of machines that an OS provides on a single machine today. In this session, we will discuss: How the abstraction of an OS has evolved over time and can cleanly scale to spand thousands of machines in a datacenter. How key open source technologies like the Apache Mesos distributed systems kernel provide the key underpinnings for a DCOS. How developers can layer core system services on top of a distributed systems kernel, including an init system (Marathon), cron (Chronos), service discovery (DNS), and storage (HDFS) What would the interface to the DCOS look like? How would you use it? How you would install and operate datacenter services, including Apache Spark, Apache Cassandra, Apache Kafka, Apache Hadoop, Apache YARN, Apache HDFS, and Google's Kubernetes. How will developers build datacenter-scale apps, programmed against the datacenter OS like it?s a single machine?

1. Dr. Bernd Mathiske
Senior Software Architect
Mesosphere
Why the Datacenter needs an
Operating System
1

2. Bringing
Google-Scale
Computing
to Everybody

3. A Slice of Google Tech Transfer History
2005: MapReduce -> Hadoop (Yahoo)
2007: Linux cgroups for lightweight isolation (Google)
2009: BigTable -> MongoDB
2009: “The Datacenter as a Computer” - Barroso, Hölzle (Google)
2009: Mesos - a distributed operating system kernel (UC Berkeley)
2010: Large scale production Mesos deployment (Twitter)
since 2010: Many more frameworks and quite a few meta-frameworks

4. Notable Operating System Developments
Single-something => multi-something: user, tasking, threading, core, …
More: bits, memory, storage, bandwidth…
OS virtualization => lightweight virtualization (cgroups, LXCs, jails, …)
Packaging => containers (docker, rkt, lmctfy, …)
Static libraries => dynamic libraries => static libraries
4

5. Cluster Operating Systems (Hardware Clustering)
Researched since the 1980s
Trying to provide (the illusion of) a single system image
Aiming at HA, load balancing, location transparency (e.g. for storage)
Many systems: Amoeba, ChorusOS, GLUnix, Hurricane, MOSIX, Plan9, RHCS,
Spring, Sprite, Sumo, QNX, Solaris MC, UnixWare, VAXclusters, …
Relatively low scale (up to 100s of nodes)
Complicated to manage, less dynamic than software clustering
5

6. From HPC Grid to Enterprise Cloud
Condor, LSF, Maui, Moab, Quartz, SLURM, …
Typically for batch jobs
Also cover services => SOA => more job schedulers
=> grid computing => grid middleware … => cloud stacks
6

7. From Server Virtualization to App Aggregation
Cloud Era:
Big apps, small servers
Client-Server Era:
Small apps, big servers
Server
Virtualization
App App App App
App
Aggregation
Serv Serv Serv Serv

8. Cloud Computing
SaaS: Salesforce demonstrated success, then many followed
PaaS: Deis, Dotcloud, OpenShift, Heroku, Pivotal, Stackato, …
IaaS: AWS, Azure, DigitalOcean, GCE…
Private cloud stacks including IaaS: Eucalyptus, CloudStack,
Joyent, OpenStack, SmartCloud, vSphere, …
8

9. Datacenter
✴ A facility used to house computer systems and associated
components (e.g. networking, storage, cooling, sensors)
✴ In this talk we focus on how to manage and use a single
production cluster of networked computers in a datacenter
✴ Such clusters range in size from 10s to 10000s of nodes
✴ Why should we and how can we end up with
just one production cluster?
9

10. Datacenter Services
✴ LAMP (Linux, Apache, MSQL, PHP) or similar
✴ MEAN (MongoDB, Express.js, Angular.js, Node.js) or similar
✴ Cassandra, ElasticSearch, Exelixi, Hadoop, Hypertable, Jenkins,
Kafka, MPI, Spark, Storm, SSSP, Torque, …
✴ Private PaaS: Deis, …
✴ …
10

11. Operate your Laptop like your Datacenter?

12. From Static Partitioning to Elastic Sharing
Static Partitioning
Elastic Sharing
WEB HADOOPCACHE
WASTED
FREEFREE
HADOOP
WEB
CACHE
WASTED WASTED
100% —
100% —

13. Software Clustering
Layer between node OS and
application frameworks
Scale
Multi-tenancy
High availability

14. Available Open Source Components
✴ 2-level scheduler: Apache Mesos
✴ Meta-frameworks / schedulers: Aurora, Chronos, Marathon,
Kubernetes, Swarm, …
✴ Service discovery: Consul, HAProxy, Mesos DNS, …
✴ Highly available configuration: zk, etcd, …
✴ Storage: HDFS, Ceph, …
✴ Node OSs: lots of Linux variants
✴ Lots of app frameworks: Sparc, Storm, Cassandra, Kafka, …14

15. 2-Level Scheduling
Scale: from 1 node to at least 10000s of nodes
Optimizing resource management
End-to-end principle: “application-specific functions ought to reside
in the end nodes of a network rather than intermediary nodes”
-> Requirement for general multi-tenancy
-> Requirement for having only one production cluster
15

16. App
How Mesos Works
16
Framework
Scheduler Master Slave
Master
Master
Master
Executor
Executor
Task
Task
Task
Task
zk/etcd

17. Ways to Run an Application
1. Vanilla job
• Employ meta-framework for invocation: Chronos, Aurora, Kubernetes, …
2. Application of an adapted framework
• Hadoop, Sparc, Storm, ElasticSearch, Cassandra, Kafka, many more…
3. Non-adapted services
• Employ meta-framework for invocation: Marathon, Aurora, Kubernetes, …
• Provide (select) a service discovery solution
4. Program your own scheduler (and executor)
17

18. The Mesos Framework API
✴ Currently like internal Mesos communication:
• protobuf messages over HTTP
✴ Soon:
• JSON messages over HTTP (stream)
=> no need to link with binary Mesos library and/or less to reimplement
ca. a dozen programming languages => any language
18

19. How to implement a framework
✴ Scheduler interface: 1 half of 2-level scheduling
• The framework knows best when to do what with what kind of resources
• About a dozen callbacks, main functionality in 2 of them:
- receive resource offers
- receive task status updates
✴ Executor interface: task life-cycle management and monitoring
• Command line executor included in Mesos
• Docker executor included in Mesos
• Custom executors often not needed
19

20. Scheduler SPI (implemented by Framework)
20
public interface Scheduler {
void registered(SchedulerDriver driver, FrameworkID frameworkId,
MasterInfo masterInfo);
void reregistered(SchedulerDriver driver, MasterInfo masterInfo);
void resourceOffers(SchedulerDriver driver, List<Offer> offers);
void offerRescinded(SchedulerDriver driver, OfferID offerId);
void statusUpdate(SchedulerDriver driver, TaskStatus status);
void frameworkMessage(SchedulerDriver driver, ExecutorID executorId,
SlaveID slaveId, byte[] data);
void disconnected(SchedulerDriver driver);
void slaveLost(SchedulerDriver driver, SlaveID slaveId);
void executorLost(SchedulerDriver driver, ExecutorID executorId,
SlaveID slaveId, int status);
void error(SchedulerDriver driver, String message);
}

21. Minimal Scheduler Implementation
class MyFrameworkScheduler implements Scheduler {
…
private TaskGenerator _taskGen;
public void resourceOffers(SchedulerDriver driver, List<Offer> offers) {
if (_taskGen.doneCreatingTasks()) {
for (offer : offers) {
driver.declineOffer(offer.getId());
}
} else {
for (offer : offers) {
List<TaskInfo> taskInfos = _taskGen.generateTaskInfos(offer);
driver.launchTasks(offer.getId(), taskInfos, _filters);
}
}
}
public void statusUpdate(SchedulerDriver driver, TaskStatus status) {
_taskGen.observeTaskStatusUpdate(taskStatus);
if (_taskGen.done()) {
driver.stop();
}
}
…
}
21

22. The Developer’s Perspective
✴ Focus on application logic, not datacenter structure
✴ Avoid networking-related code
✴ Reuse of built-in fault-tolerance and high availability
✴ Reuse distributed (infrastructure) frameworks (e.g., storage)
=> API, SDK for datacenter services
22

23. The Operations Engineer’s Perspective
✴ Ease of deployment/management
✴ Uniformity of deployment/management
✴ Hardware utilization rate
✴ Scaling up as business grows
✴ Scaling out sporadically
✴ Cost and time for moving to a different datacenter
✴ High availability and fault-tolerance of system services
✴ Monitoring
✴ Trouble shooting
23

24. Necessary Multi-Tenancy Features
Task containerization
Resource isolation
Resource and task attributes
Static and dynamic resource reservations
Reservation levels
Meta-frameworks
Dynamic scheduler update and reconfiguration
Security
24

25. Desirable Multi-Tenancy Features
Optimistic offers
Oversubscription
Task preemption, migration, resizing, reconfiguration
Rate limiting
Auto-scaling => hybrid cloud
Infrastructure frameworks
25

26. Using Docker Containers in Mesos
26
Mesos Master Server
init
|
+ mesos-master
|
+ marathon
|
Mesos Slave Server
init
|
+ docker
| |
| + lxc
| |
| + (user task, under container init system)
| |
|
+ mesos-slave
| |
| + /var/lib/mesos/executors/docker
| | |
| | + docker run …
| | |
Docker
Registry
When a user requests
a container…
Mesos, LXC, and
Docker are tied
together for launch
2
1
3
4
5
6
7
8

27. Other Schedulers as Meta-Frameworks in a 2-level Scheduler
YARN => https://github.com/mesos/myriad
Kubernetes => https://github.com/mesosphere/kubernetes-mesos
Swarm => Swarm on Mesos (new project)
=> run everything in one cluster
27

28. Myriad : Virtual YARN Clusters on Mesos
28
◦ POST /api/clusters: Registers a new YARN
◦ GET /api/clusters: Lists all registered clusters
◦ GET /api/clusters/{clusterId}: Lists the cluster with {clusterId}
◦ PUT /api/clusters/{clusterId}/flexup: Expands the size of cluster with {clusterId}
◦ PUT /api/clusters/{clusterId}/flexdown: Shrinks the size of cluster with {clusterId}
◦ DELETE /api/clusters/{clusterId}: Unregisters YARN cluster with {clusterId}. Also, kills all the nodes.
Node
Master
Mesos
Slave
Mesos
YARN
Myriad
Scheduler RM
Myriad
Executor
1. Launch NodeManager
1
1
1
2.5 CPU
2.5 GB
1
NM
YARN
flexUp
2.0 CPU
2.0 GB
C1
C2

29. 29
Kubernetes in Mesos

30. Portability
30
Mesos
Public Cloud Managed Cloud Your Own DC
Framework Apps
Meta-Frameworks
Vanilla Apps
Infrastructure Frameworks

31. The Application User’s Perspective
✴ Focus on apps, services, parameters, results
✴ Avoid dealing with datacenter operations/management
✴ Avoid adjusting system settings
✴ High availability
✴ Throughput
✴ Responsiveness
✴ Predictiveness
✴ Run everything I need
✴ Return on and safety of investment
31

32. The Datacenter is the new form factor
✴ 2-level scheduler => single production cluster
✴ scalability and portability => avoiding hardware/cloud lock-in
✴ built-in container support => running containers at scale
✴ automation => operator efficiency
✴ repositories => apps/services readily available
✴ API and SDK => productive/quick app/service development
32

33. 33
Above the Clouds
with Open Source!