SlideShare a Scribd company logo
Kernel for Distributed 
Applications 
Sam BESSALAH 
@samklr 
ROOM 9 #devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker @samklr
Example of a statically partionned cluster 
#Devoxx #DV14 #mesos #docker 
@samklr
Akka/Play Jenkins Spark/Hadoop Storm Redis 
#Devoxx #DV14 #mesos #docker @samklr
Issues with static partitionning 
- Increased Complexity : 
Machines/vm sprawl 
Software failures gets harder to handle as 
machines number grow 
- Huge overheard, on development, 
- Planning for failure and maintenance gets harder 
- Not really developer friendly. Long cycles to 
deployments. 
#Devoxx #DV14 #mesos #docker 
@samklr
Resources utilisation under static partitionning 
Play/Akka Redis Hadoop 
#Devoxx #DV14 #mesos #docker 
@samklr
Resources utilisation under static partitionning 
Play/Akka Redis 
#Devoxx #DV14 #mesos #docker 
@samklr 
Hadoop 
Put together, we would get
Enters Google stage left ... 
Google “Borg” .Unofficial name. 
Moved to a new system called Omega 
Bundles all the cluster’s machines into a single pooled 
view of resources. 
Evolved to Omega. 
#Devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker @samklr
Mesos : Level of indirection 
1. Run multiple types of distributed systems on the same cluster of 
machines, and dynamically share resources more eficiently. 
2. Provide failure detection, task distribution, task tracking, task 
monitoring, low level resources management, and fine grained 
resources sharing. And scale to the thousands of nodes. 
#Devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker 
@samklr 
Quick View.
Mesos Architecture 
#Devoxx #DV14 #mesos #docker @samklr
http://mesos.apache.org/documentation/latest/mesos-architecture/ 
#Devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker 
@samklr 
Mesos Framework 
A framework is an application that runs distributed applications on 
Mesos. 
It has two components : 
The Scheduler : Interacts with Mesos, to book resources and 
launch tasks within the mesos slaves. 
Executors : gets information from the framework like 
environmental variables,configuration information 
and run tasks inside the mesos slaves.
Mesos Framework 
A framework is an application that runs distributed applications on 
Mesos. 
It has two components : 
The Scheduler : Interacts with Mesos, to book resources and 
launch tasks within the mesos slaves. 
Executors : run tasks in mesos slaves. 
Mesos provides an API to build its own frameworks. And there are SDKs 
in mulitple languages. 
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker @samklr
Resources allocation process 
- Mesos allocate resources via 
“resources offers” 
- Resources are a snapshot of 
available resources that a 
scheduler will use to run 
tasks on mesos slaves. 
#Devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 2. launchTasks() 
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 2. launchTasks() 
3. lauchTask() 
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 2. launchTasks() 
3. lauchTask() 4. statusUpdate() 
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 
2. launchTasks() 
5. statusUpdate() 
3. lauchTask() 4. statusUpdate() 
#Devoxx #DV14 #mesos #docker @samklr
#Devoxx #DV14 #mesos #docker 
@samklr 
Mesos slaves
#Devoxx #DV14 #mesos #docker 
@samklr 
Mesos slaves 
Linux Containers
Mesos uses Linux Containers for resource isolation. 
Mesos containers are built around cgroups , to provide isolation of 
CPU, memory, disk … and namespaces for network isolation. 
Started leveraging containerirsation since 2010 
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker 
@samklr 
Linux Containers
Linux Containers 
#Devoxx #DV14 #mesos #docker 
@samklr 
LXC
Linux Containers 
#Devoxx #DV14 #mesos #docker 
@samklr 
LXC 
Cgroups
#Devoxx #DV14 #mesos #docker 
@samklr
Mesos provides a native support for Docker 
#Devoxx #DV14 #mesos #docker 
@samklr 
Containers since 0.20
#Devoxx #DV14 #mesos #docker 
@samklr
Fault recovery & HA 
- Mesos Master uses Zookeeper for leader election and leader 
discovery. 
- Mesos Master have a Registrar to log all the tasks and 
slaves info, implemented as a replicated log using 
MultiPaxos for consensus 
#Devoxx #DV14 #mesos #docker 
@samklr
Fault recovery & HA 
- Mesos Master uses Zookeeper for leader election and leader 
discovery. 
- Mesos Master have a Registrar to log all the tasks and 
slaves info, implemented as a replicated log using 
MultiPaxos for consensus 
- Mesos has a slave recovery mechanism. So user’s tasks keep 
running whenever a slave proces goes under. 
#Devoxx #DV14 #mesos #docker 
@samklr
Fault recovery & HA 
- Mesos Master uses Zookeeper for leader election and leader 
discovery. 
- Mesos Master have a Registrar to log all the tasks and 
slaves info, implemented as a replicated log using 
MultiPaxos for consensus 
- Mesos has a slave recovery mechanism. So user’s tasks keep 
running whenever a slave proces goes under. 
Slaves checkpoint information to their local disks (task 
info, status updates, etc ), they recover checkpointed data 
and reconnect with old tasks when they restart. 
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker @samklr
Mesos Framework 
A framework is an application that runs distributed applications on 
Mesos. 
It has two components : 
The Scheduler : Interacts with Mesos, to book resources and 
launch tasks within the mesos slaves. 
Executors : run tasks in mesos slaves. 
Mesos provides an API to build its own frameworks. And there are SDKs 
in mulitple languages. 
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker 
@samklr 
Marathon
What is Marathon ? 
- A generic mesos framework to run long running services (web apps, 
etc …) 
- A distributed Init.d for the cluster. 
- Runs any linux binary without modification (Tomcat, Play, …) 
- Cluster wide process supervisor 
-Private PaaS 
- Service discovery 
- Provide a self service Rest API for deployment 
- Authentification & SSL 
- Placement constrainst (nodes, racks, etc. (Like with Spark) ) 
- Service discovery and load balancing via HAProxy 
#Devoxx #DV14 #mesos #docker 
@samklr
Marathon workflow 
#Devoxx #DV14 #mesos #docker 
@samklr
Marathon workflow 
#Devoxx #DV14 #mesos #docker 
@samklr
POST /v2/apps { 
Marathon Rest APi 
“id” : “IPython Notebook” 
“container “ : { 
“image “ : 
“docker://ipython/notebook” 
“options” : [ . . . ] 
} 
} 
#Devoxx #DV14 #mesos #docker 
@samklr
Marathon Rest APi 
Wants to scale by addind more instances .. 
PATCH /v2/apps { 
“instances” : 4 
} 
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker 
@samklr 
Conclusion
#Devoxx #DV14 #mesos #docker 
@samklr 
Conclusion
#Devoxx #DV14 #mesos #docker 
@samklr 
Conclusion 
Aurora
#Devoxx #DV14 #mesos #docker 
@samklr 
Conclusion 
Kubernetes
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker 
@samklr
#Devoxx #DV14 #mesos #docker 
@samklr
speakerdeck.com/samklr 
#Devoxx #DV14 #mesos #docker 
@samklr

More Related Content

What's hot

Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...
CloudxLab
 
A deeper-understanding-of-spark-internals
A deeper-understanding-of-spark-internalsA deeper-understanding-of-spark-internals
A deeper-understanding-of-spark-internals
Cheng Min Chi
 
MongoDB & Hadoop: Flexible Hourly Batch Processing Model
MongoDB & Hadoop: Flexible Hourly Batch Processing ModelMongoDB & Hadoop: Flexible Hourly Batch Processing Model
MongoDB & Hadoop: Flexible Hourly Batch Processing Model
Takahiro Inoue
 
Hadoop
HadoopHadoop
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLabApache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
Apache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLabApache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...
Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...
Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...
CloudxLab
 
Upgrading To The New Map Reduce API
Upgrading To The New Map Reduce APIUpgrading To The New Map Reduce API
Upgrading To The New Map Reduce API
Tom Croucher
 
Sparkstreaming
SparkstreamingSparkstreaming
Sparkstreaming
Marilyn Waldman
 
Data analysis scala_spark
Data analysis scala_sparkData analysis scala_spark
Data analysis scala_spark
Yiguang Hu
 
Cascalog internal dsl_preso
Cascalog internal dsl_presoCascalog internal dsl_preso
Cascalog internal dsl_preso
Hadoop User Group
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
Mohamed hedi Abidi
 
Apache Spark with Scala
Apache Spark with ScalaApache Spark with Scala
Apache Spark with Scala
Fernando Rodriguez
 
Cassandra Explained
Cassandra ExplainedCassandra Explained
Cassandra Explained
Eric Evans
 
Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...
Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...
Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...
Spark Summit
 
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
Hadoop Integration in Cassandra
Hadoop Integration in CassandraHadoop Integration in Cassandra
Hadoop Integration in Cassandra
Jairam Chandar
 
Introduction to Spark with Scala
Introduction to Spark with ScalaIntroduction to Spark with Scala
Introduction to Spark with Scala
Himanshu Gupta
 
Spark 2.x Troubleshooting Guide
Spark 2.x Troubleshooting GuideSpark 2.x Troubleshooting Guide
Spark 2.x Troubleshooting Guide
IBM
 
Cassandra+Hadoop
Cassandra+HadoopCassandra+Hadoop
Cassandra+Hadoop
Jeremy Hanna
 

What's hot (20)

Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 2 | Big Data Hadoop Spark Tutori...
 
A deeper-understanding-of-spark-internals
A deeper-understanding-of-spark-internalsA deeper-understanding-of-spark-internals
A deeper-understanding-of-spark-internals
 
MongoDB & Hadoop: Flexible Hourly Batch Processing Model
MongoDB & Hadoop: Flexible Hourly Batch Processing ModelMongoDB & Hadoop: Flexible Hourly Batch Processing Model
MongoDB & Hadoop: Flexible Hourly Batch Processing Model
 
Hadoop
HadoopHadoop
Hadoop
 
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLabApache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
 
Apache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLabApache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark - Running on a Cluster | Big Data Hadoop Spark Tutorial | CloudxLab
 
Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...
Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...
Introduction to MapReduce - Hadoop Streaming | Big Data Hadoop Spark Tutorial...
 
Upgrading To The New Map Reduce API
Upgrading To The New Map Reduce APIUpgrading To The New Map Reduce API
Upgrading To The New Map Reduce API
 
Sparkstreaming
SparkstreamingSparkstreaming
Sparkstreaming
 
Data analysis scala_spark
Data analysis scala_sparkData analysis scala_spark
Data analysis scala_spark
 
Cascalog internal dsl_preso
Cascalog internal dsl_presoCascalog internal dsl_preso
Cascalog internal dsl_preso
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
 
Apache Spark with Scala
Apache Spark with ScalaApache Spark with Scala
Apache Spark with Scala
 
Cassandra Explained
Cassandra ExplainedCassandra Explained
Cassandra Explained
 
Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...
Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...
Taming GC Pauses for Humongous Java Heaps in Spark Graph Computing-(Eric Kacz...
 
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLab
 
Hadoop Integration in Cassandra
Hadoop Integration in CassandraHadoop Integration in Cassandra
Hadoop Integration in Cassandra
 
Introduction to Spark with Scala
Introduction to Spark with ScalaIntroduction to Spark with Scala
Introduction to Spark with Scala
 
Spark 2.x Troubleshooting Guide
Spark 2.x Troubleshooting GuideSpark 2.x Troubleshooting Guide
Spark 2.x Troubleshooting Guide
 
Cassandra+Hadoop
Cassandra+HadoopCassandra+Hadoop
Cassandra+Hadoop
 

Similar to mesos-devoxx14

Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos
Rahul Kumar
 
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
Carlos Sanchez
 
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)
Docker, Inc.
 
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Carlos Sanchez
 
Docker
DockerDocker
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed ApplicationsFrom Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
Carlos Sanchez
 
Mesos on coreOS
Mesos on coreOSMesos on coreOS
Mesos on coreOS
충섭 김
 
Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on AzureDevoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Patrick Chanezon
 
Best Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker ContainersBest Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker Containers
BlueData, Inc.
 
Docker 1.9 Workshop
Docker 1.9 WorkshopDocker 1.9 Workshop
Docker 1.9 Workshop
{code}
 
Docker - Frank Maounis
Docker - Frank MaounisDocker - Frank Maounis
Docker - Frank Maounis
Frank Maounis
 
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
Patrick Chanezon
 
Choosing PaaS: Cisco and Open Source Options: an overview
Choosing PaaS:  Cisco and Open Source Options: an overviewChoosing PaaS:  Cisco and Open Source Options: an overview
Choosing PaaS: Cisco and Open Source Options: an overview
Cisco DevNet
 
Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14
Simon Storm
 
Axigen on docker
Axigen on dockerAxigen on docker
Docker DANS workshop
Docker DANS workshopDocker DANS workshop
Docker DANS workshop
vty
 
Docker In Brief
Docker In BriefDocker In Brief
Docker In Brief
Ritu Kamthan
 
Docker
DockerDocker
Docker
Narato
 
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
Codemotion
 
Dockerization of Azure Platform
Dockerization of Azure PlatformDockerization of Azure Platform
Dockerization of Azure Platform
nirajrules
 

Similar to mesos-devoxx14 (20)

Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos
 
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
 
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)
 
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
 
Docker
DockerDocker
Docker
 
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed ApplicationsFrom Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
 
Mesos on coreOS
Mesos on coreOSMesos on coreOS
Mesos on coreOS
 
Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on AzureDevoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure
 
Best Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker ContainersBest Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker Containers
 
Docker 1.9 Workshop
Docker 1.9 WorkshopDocker 1.9 Workshop
Docker 1.9 Workshop
 
Docker - Frank Maounis
Docker - Frank MaounisDocker - Frank Maounis
Docker - Frank Maounis
 
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
 
Choosing PaaS: Cisco and Open Source Options: an overview
Choosing PaaS:  Cisco and Open Source Options: an overviewChoosing PaaS:  Cisco and Open Source Options: an overview
Choosing PaaS: Cisco and Open Source Options: an overview
 
Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14
 
Axigen on docker
Axigen on dockerAxigen on docker
Axigen on docker
 
Docker DANS workshop
Docker DANS workshopDocker DANS workshop
Docker DANS workshop
 
Docker In Brief
Docker In BriefDocker In Brief
Docker In Brief
 
Docker
DockerDocker
Docker
 
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
 
Dockerization of Azure Platform
Dockerization of Azure PlatformDockerization of Azure Platform
Dockerization of Azure Platform
 

More from Samir Bessalah

Machine Learning In Production
Machine Learning In ProductionMachine Learning In Production
Machine Learning In Production
Samir Bessalah
 
Eventual Consitency with CRDTS
Eventual Consitency with CRDTSEventual Consitency with CRDTS
Eventual Consitency with CRDTS
Samir Bessalah
 
Deep learning for mere mortals - Devoxx Belgium 2015
Deep learning for mere mortals - Devoxx Belgium 2015Deep learning for mere mortals - Devoxx Belgium 2015
Deep learning for mere mortals - Devoxx Belgium 2015
Samir Bessalah
 
High Performance RPC with Finagle
High Performance RPC with FinagleHigh Performance RPC with Finagle
High Performance RPC with Finagle
Samir Bessalah
 
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Samir Bessalah
 
Big Data Analytics with Scala at SCALA.IO 2013
Big Data Analytics with Scala at SCALA.IO 2013Big Data Analytics with Scala at SCALA.IO 2013
Big Data Analytics with Scala at SCALA.IO 2013
Samir Bessalah
 
Structures de données exotiques
Structures de données exotiquesStructures de données exotiques
Structures de données exotiques
Samir Bessalah
 

More from Samir Bessalah (7)

Machine Learning In Production
Machine Learning In ProductionMachine Learning In Production
Machine Learning In Production
 
Eventual Consitency with CRDTS
Eventual Consitency with CRDTSEventual Consitency with CRDTS
Eventual Consitency with CRDTS
 
Deep learning for mere mortals - Devoxx Belgium 2015
Deep learning for mere mortals - Devoxx Belgium 2015Deep learning for mere mortals - Devoxx Belgium 2015
Deep learning for mere mortals - Devoxx Belgium 2015
 
High Performance RPC with Finagle
High Performance RPC with FinagleHigh Performance RPC with Finagle
High Performance RPC with Finagle
 
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
 
Big Data Analytics with Scala at SCALA.IO 2013
Big Data Analytics with Scala at SCALA.IO 2013Big Data Analytics with Scala at SCALA.IO 2013
Big Data Analytics with Scala at SCALA.IO 2013
 
Structures de données exotiques
Structures de données exotiquesStructures de données exotiques
Structures de données exotiques
 

Recently uploaded

BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
ZachWylie3
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
DianaGray10
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
Ivanti
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
janagijoythi
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
AmandaCheung15
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Zilliz
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
Enterprise Knowledge
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
SelfMade bd
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 

Recently uploaded (20)

BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 

mesos-devoxx14

  • 1. Kernel for Distributed Applications Sam BESSALAH @samklr ROOM 9 #devoxx #DV14 #mesos #docker @samklr
  • 2. #Devoxx #DV14 #mesos #docker @samklr
  • 3. Example of a statically partionned cluster #Devoxx #DV14 #mesos #docker @samklr
  • 4. Akka/Play Jenkins Spark/Hadoop Storm Redis #Devoxx #DV14 #mesos #docker @samklr
  • 5. Issues with static partitionning - Increased Complexity : Machines/vm sprawl Software failures gets harder to handle as machines number grow - Huge overheard, on development, - Planning for failure and maintenance gets harder - Not really developer friendly. Long cycles to deployments. #Devoxx #DV14 #mesos #docker @samklr
  • 6. Resources utilisation under static partitionning Play/Akka Redis Hadoop #Devoxx #DV14 #mesos #docker @samklr
  • 7. Resources utilisation under static partitionning Play/Akka Redis #Devoxx #DV14 #mesos #docker @samklr Hadoop Put together, we would get
  • 8. Enters Google stage left ... Google “Borg” .Unofficial name. Moved to a new system called Omega Bundles all the cluster’s machines into a single pooled view of resources. Evolved to Omega. #Devoxx #DV14 #mesos #docker @samklr
  • 9. #Devoxx #DV14 #mesos #docker @samklr
  • 10. Mesos : Level of indirection 1. Run multiple types of distributed systems on the same cluster of machines, and dynamically share resources more eficiently. 2. Provide failure detection, task distribution, task tracking, task monitoring, low level resources management, and fine grained resources sharing. And scale to the thousands of nodes. #Devoxx #DV14 #mesos #docker @samklr
  • 11. #Devoxx #DV14 #mesos #docker @samklr
  • 12. #Devoxx #DV14 #mesos #docker @samklr Quick View.
  • 13. Mesos Architecture #Devoxx #DV14 #mesos #docker @samklr
  • 15. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : gets information from the framework like environmental variables,configuration information and run tasks inside the mesos slaves.
  • 16. Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages. #Devoxx #DV14 #mesos #docker @samklr
  • 17. #Devoxx #DV14 #mesos #docker @samklr
  • 18. Resources allocation process - Mesos allocate resources via “resources offers” - Resources are a snapshot of available resources that a scheduler will use to run tasks on mesos slaves. #Devoxx #DV14 #mesos #docker @samklr
  • 19. #Devoxx #DV14 #mesos #docker @samklr
  • 20. #Devoxx #DV14 #mesos #docker @samklr
  • 21. #Devoxx #DV14 #mesos #docker @samklr
  • 22. 1- resourceOffers() #Devoxx #DV14 #mesos #docker @samklr
  • 23. 1- resourceOffers() 2. launchTasks() #Devoxx #DV14 #mesos #docker @samklr
  • 24. 1- resourceOffers() 2. launchTasks() 3. lauchTask() #Devoxx #DV14 #mesos #docker @samklr
  • 25. 1- resourceOffers() 2. launchTasks() 3. lauchTask() 4. statusUpdate() #Devoxx #DV14 #mesos #docker @samklr
  • 26. 1- resourceOffers() 2. launchTasks() 5. statusUpdate() 3. lauchTask() 4. statusUpdate() #Devoxx #DV14 #mesos #docker @samklr
  • 27. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves
  • 28. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves Linux Containers
  • 29. Mesos uses Linux Containers for resource isolation. Mesos containers are built around cgroups , to provide isolation of CPU, memory, disk … and namespaces for network isolation. Started leveraging containerirsation since 2010 #Devoxx #DV14 #mesos #docker @samklr
  • 30. #Devoxx #DV14 #mesos #docker @samklr
  • 31. #Devoxx #DV14 #mesos #docker @samklr Linux Containers
  • 32. Linux Containers #Devoxx #DV14 #mesos #docker @samklr LXC
  • 33. Linux Containers #Devoxx #DV14 #mesos #docker @samklr LXC Cgroups
  • 34. #Devoxx #DV14 #mesos #docker @samklr
  • 35. Mesos provides a native support for Docker #Devoxx #DV14 #mesos #docker @samklr Containers since 0.20
  • 36. #Devoxx #DV14 #mesos #docker @samklr
  • 37. Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus #Devoxx #DV14 #mesos #docker @samklr
  • 38. Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. #Devoxx #DV14 #mesos #docker @samklr
  • 39. Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. Slaves checkpoint information to their local disks (task info, status updates, etc ), they recover checkpointed data and reconnect with old tasks when they restart. #Devoxx #DV14 #mesos #docker @samklr
  • 40. #Devoxx #DV14 #mesos #docker @samklr
  • 41. Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages. #Devoxx #DV14 #mesos #docker @samklr
  • 42. #Devoxx #DV14 #mesos #docker @samklr Marathon
  • 43. What is Marathon ? - A generic mesos framework to run long running services (web apps, etc …) - A distributed Init.d for the cluster. - Runs any linux binary without modification (Tomcat, Play, …) - Cluster wide process supervisor -Private PaaS - Service discovery - Provide a self service Rest API for deployment - Authentification & SSL - Placement constrainst (nodes, racks, etc. (Like with Spark) ) - Service discovery and load balancing via HAProxy #Devoxx #DV14 #mesos #docker @samklr
  • 44. Marathon workflow #Devoxx #DV14 #mesos #docker @samklr
  • 45. Marathon workflow #Devoxx #DV14 #mesos #docker @samklr
  • 46. POST /v2/apps { Marathon Rest APi “id” : “IPython Notebook” “container “ : { “image “ : “docker://ipython/notebook” “options” : [ . . . ] } } #Devoxx #DV14 #mesos #docker @samklr
  • 47. Marathon Rest APi Wants to scale by addind more instances .. PATCH /v2/apps { “instances” : 4 } #Devoxx #DV14 #mesos #docker @samklr
  • 48. #Devoxx #DV14 #mesos #docker @samklr Conclusion
  • 49. #Devoxx #DV14 #mesos #docker @samklr Conclusion
  • 50. #Devoxx #DV14 #mesos #docker @samklr Conclusion Aurora
  • 51. #Devoxx #DV14 #mesos #docker @samklr Conclusion Kubernetes
  • 52. #Devoxx #DV14 #mesos #docker @samklr
  • 53. #Devoxx #DV14 #mesos #docker @samklr
  • 54. #Devoxx #DV14 #mesos #docker @samklr
  • 55. speakerdeck.com/samklr #Devoxx #DV14 #mesos #docker @samklr