Mesos provides a distributed systems kernel that allows organizations to dynamically share resources between distributed applications like Hadoop, Spark, and Storm. It addresses issues with static resource partitioning, like increased complexity and poor resource utilization. Mesos introduces an abstraction layer that bundles all machines in a cluster into a single shared pool. It provides APIs for building frameworks to run applications that leverage the shared resources.

1. Kernel for Distributed
Applications
Sam BESSALAH
@samklr
ROOM 9 #devoxx #DV14 #mesos #docker @samklr

2. #Devoxx #DV14 #mesos #docker @samklr

3. Example of a statically partionned cluster
#Devoxx #DV14 #mesos #docker
@samklr

4. Akka/Play Jenkins Spark/Hadoop Storm Redis
#Devoxx #DV14 #mesos #docker @samklr

5. Issues with static partitionning
- Increased Complexity :
Machines/vm sprawl
Software failures gets harder to handle as
machines number grow
- Huge overheard, on development,
- Planning for failure and maintenance gets harder
- Not really developer friendly. Long cycles to
deployments.
#Devoxx #DV14 #mesos #docker
@samklr

6. Resources utilisation under static partitionning
Play/Akka Redis Hadoop
#Devoxx #DV14 #mesos #docker
@samklr

7. Resources utilisation under static partitionning
Play/Akka Redis
#Devoxx #DV14 #mesos #docker
@samklr
Hadoop
Put together, we would get

8. Enters Google stage left ...
Google “Borg” .Unofficial name.
Moved to a new system called Omega
Bundles all the cluster’s machines into a single pooled
view of resources.
Evolved to Omega.
#Devoxx #DV14 #mesos #docker @samklr

9. #Devoxx #DV14 #mesos #docker @samklr

10. Mesos : Level of indirection
1. Run multiple types of distributed systems on the same cluster of
machines, and dynamically share resources more eficiently.
2. Provide failure detection, task distribution, task tracking, task
monitoring, low level resources management, and fine grained
resources sharing. And scale to the thousands of nodes.
#Devoxx #DV14 #mesos #docker @samklr

11. #Devoxx #DV14 #mesos #docker
@samklr

12. #Devoxx #DV14 #mesos #docker
@samklr
Quick View.

13. Mesos Architecture
#Devoxx #DV14 #mesos #docker @samklr

14. http://mesos.apache.org/documentation/latest/mesos-architecture/
#Devoxx #DV14 #mesos #docker @samklr

15. #Devoxx #DV14 #mesos #docker
@samklr
Mesos Framework
A framework is an application that runs distributed applications on
Mesos.
It has two components :
The Scheduler : Interacts with Mesos, to book resources and
launch tasks within the mesos slaves.
Executors : gets information from the framework like
environmental variables,configuration information
and run tasks inside the mesos slaves.

16. Mesos Framework
A framework is an application that runs distributed applications on
Mesos.
It has two components :
The Scheduler : Interacts with Mesos, to book resources and
launch tasks within the mesos slaves.
Executors : run tasks in mesos slaves.
Mesos provides an API to build its own frameworks. And there are SDKs
in mulitple languages.
#Devoxx #DV14 #mesos #docker
@samklr

17. #Devoxx #DV14 #mesos #docker @samklr

18. Resources allocation process
- Mesos allocate resources via
“resources offers”
- Resources are a snapshot of
available resources that a
scheduler will use to run
tasks on mesos slaves.
#Devoxx #DV14 #mesos #docker @samklr

19. #Devoxx #DV14 #mesos #docker @samklr

20. #Devoxx #DV14 #mesos #docker @samklr

21. #Devoxx #DV14 #mesos #docker @samklr

22. 1- resourceOffers()
#Devoxx #DV14 #mesos #docker @samklr

23. 1- resourceOffers() 2. launchTasks()
#Devoxx #DV14 #mesos #docker @samklr

24. 1- resourceOffers() 2. launchTasks()
3. lauchTask()
#Devoxx #DV14 #mesos #docker @samklr

25. 1- resourceOffers() 2. launchTasks()
3. lauchTask() 4. statusUpdate()
#Devoxx #DV14 #mesos #docker @samklr

26. 1- resourceOffers()
2. launchTasks()
5. statusUpdate()
3. lauchTask() 4. statusUpdate()
#Devoxx #DV14 #mesos #docker @samklr

27. #Devoxx #DV14 #mesos #docker
@samklr
Mesos slaves

28. #Devoxx #DV14 #mesos #docker
@samklr
Mesos slaves
Linux Containers

29. Mesos uses Linux Containers for resource isolation.
Mesos containers are built around cgroups , to provide isolation of
CPU, memory, disk … and namespaces for network isolation.
Started leveraging containerirsation since 2010
#Devoxx #DV14 #mesos #docker
@samklr

30. #Devoxx #DV14 #mesos #docker
@samklr

31. #Devoxx #DV14 #mesos #docker
@samklr
Linux Containers

32. Linux Containers
#Devoxx #DV14 #mesos #docker
@samklr
LXC

33. Linux Containers
#Devoxx #DV14 #mesos #docker
@samklr
LXC
Cgroups

34. #Devoxx #DV14 #mesos #docker
@samklr

35. Mesos provides a native support for Docker
#Devoxx #DV14 #mesos #docker
@samklr
Containers since 0.20

36. #Devoxx #DV14 #mesos #docker
@samklr

37. Fault recovery & HA
- Mesos Master uses Zookeeper for leader election and leader
discovery.
- Mesos Master have a Registrar to log all the tasks and
slaves info, implemented as a replicated log using
MultiPaxos for consensus
#Devoxx #DV14 #mesos #docker
@samklr

38. Fault recovery & HA
- Mesos Master uses Zookeeper for leader election and leader
discovery.
- Mesos Master have a Registrar to log all the tasks and
slaves info, implemented as a replicated log using
MultiPaxos for consensus
- Mesos has a slave recovery mechanism. So user’s tasks keep
running whenever a slave proces goes under.
#Devoxx #DV14 #mesos #docker
@samklr

39. Fault recovery & HA
- Mesos Master uses Zookeeper for leader election and leader
discovery.
- Mesos Master have a Registrar to log all the tasks and
slaves info, implemented as a replicated log using
MultiPaxos for consensus
- Mesos has a slave recovery mechanism. So user’s tasks keep
running whenever a slave proces goes under.
Slaves checkpoint information to their local disks (task
info, status updates, etc ), they recover checkpointed data
and reconnect with old tasks when they restart.
#Devoxx #DV14 #mesos #docker
@samklr

40. #Devoxx #DV14 #mesos #docker @samklr

41. Mesos Framework
A framework is an application that runs distributed applications on
Mesos.
It has two components :
The Scheduler : Interacts with Mesos, to book resources and
launch tasks within the mesos slaves.
Executors : run tasks in mesos slaves.
Mesos provides an API to build its own frameworks. And there are SDKs
in mulitple languages.
#Devoxx #DV14 #mesos #docker
@samklr

42. #Devoxx #DV14 #mesos #docker
@samklr
Marathon

43. What is Marathon ?
- A generic mesos framework to run long running services (web apps,
etc …)
- A distributed Init.d for the cluster.
- Runs any linux binary without modification (Tomcat, Play, …)
- Cluster wide process supervisor
-Private PaaS
- Service discovery
- Provide a self service Rest API for deployment
- Authentification & SSL
- Placement constrainst (nodes, racks, etc. (Like with Spark) )
- Service discovery and load balancing via HAProxy
#Devoxx #DV14 #mesos #docker
@samklr

44. Marathon workflow
#Devoxx #DV14 #mesos #docker
@samklr

45. Marathon workflow
#Devoxx #DV14 #mesos #docker
@samklr

46. POST /v2/apps {
Marathon Rest APi
“id” : “IPython Notebook”
“container “ : {
“image “ :
“docker://ipython/notebook”
“options” : [ . . . ]
}
}
#Devoxx #DV14 #mesos #docker
@samklr

47. Marathon Rest APi
Wants to scale by addind more instances ..
PATCH /v2/apps {
“instances” : 4
}
#Devoxx #DV14 #mesos #docker
@samklr

48. #Devoxx #DV14 #mesos #docker
@samklr
Conclusion

49. #Devoxx #DV14 #mesos #docker
@samklr
Conclusion

50. #Devoxx #DV14 #mesos #docker
@samklr
Conclusion
Aurora

51. #Devoxx #DV14 #mesos #docker
@samklr
Conclusion
Kubernetes

52. #Devoxx #DV14 #mesos #docker
@samklr

53. #Devoxx #DV14 #mesos #docker
@samklr

54. #Devoxx #DV14 #mesos #docker
@samklr

55. speakerdeck.com/samklr
#Devoxx #DV14 #mesos #docker
@samklr