MESOS UNIVERSITY
•
1 like•1,052 views
Apache Mesos apporte une abstraction des ressources CPU, mémoire et stockage, d'un ensemble de machines (physiques ou virtuelles). Un cluster Mesos forme un système distribué, tolérant à la panne et permettant de scaler les services déployés. Venez découvrir les fonctionnalités de Mesos, ses stratégies de déploiement, et l'écosystème Mesosphere qui apporte scalabilité et découverte de services. Nous visiterons un cluster hébergeant une application JS et un backend REST conteneurisés. Pour finir, nous jouerons au Chaos Monkey afin de prouver la résilience du système mis en place. Speaker : Pablo Lopez, Aurélien Maury, Jean-Baptiste Claramonte et Jean-Pascal Thiery à Devoxx France 2015
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (6)
Similar to MESOS UNIVERSITY
Similar to MESOS UNIVERSITY (20)
More from Publicis Sapient Engineering
More from Publicis Sapient Engineering (20)
Recently uploaded
Recently uploaded (20)
MESOS UNIVERSITY
- 2. @XebiaFr @YesWeScale#MesosUniversity Agenda • Mesos • Frameworks • Mesosphere • Marathon
- 4. @XebiaFr @YesWeScale#MesosUniversity Ressources partagées Master Slave Slave Slave Slave cpu: 2 mem: 2048 cpu: 4 mem: 16384 cpu: 2 mem: 4096 cpu: 32 mem: 131072
- 5. @XebiaFr @YesWeScale#MesosUniversity Resources • 2 méthodes permettent de décrire les slaves du cluster • Resources • Tags (ou « Attributes »)
- 6. @XebiaFr @YesWeScale#MesosUniversity Resources • Certaines resources types sont pré définies par Mesos • cpu • mem • disk • port
- 7. @XebiaFr @YesWeScale#MesosUniversity Resources • Certaines resources types sont pré définies par Mesos • cpu • mem • disk • port OBLIGATOIRE !
- 8. @XebiaFr @YesWeScale#MesosUniversity Resources • Trois différentes types de resources • scalars • ranges • sets
- 12. @XebiaFr @YesWeScale#MesosUniversity Resources cpu:24;mem:24576;disk:409600;ports:[21000-24000];disks:{1,2,3,4,5,6,7,8,9} Set par exemple /mnt/disk/{diskNum}
- 19. @XebiaFr @YesWeScale#MesosUniversity Framework Framework = Application
- 23. @XebiaFr @YesWeScale#MesosUniversity API Scheduler • Est notifié par le master de la mise à disposition de ressource • Demande l’exécution de tâches au master Executor • Réalise les taches soumises par le master
- 24. @XebiaFr @YesWeScale#MesosUniversity API Scheduler Mesos (master) register time Executor
- 25. @XebiaFr @YesWeScale#MesosUniversity API Scheduler Mesos (master) register registered time Executor
- 26. @XebiaFr @YesWeScale#MesosUniversity API Scheduler Mesos (master) register registered offer time Executor
- 27. @XebiaFr @YesWeScale#MesosUniversity API Scheduler Mesos (master) register registered offer launch time launch Executor
- 28. @XebiaFr @YesWeScale#MesosUniversity API Scheduler Mesos (master) register registered offer launch update time launch Executor update
- 29. @XebiaFr @YesWeScale#MesosUniversity Framework : API API du Scheduler
- 30. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Lifecycle management Executor Master Slave Scheduler registered(…) void registered(SchedulerDriver driver, FrameworkID fwkId, MasterInfo masterInfo); Master Standby
- 31. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Lifecycle management Executor Master Slave Scheduler Master Standby disconnected(…) void disconnected(SchedulerDriver driver);
- 32. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Lifecycle management Executor Slave Scheduler reregistered(…) Master Master void reregistered(SchedulerDriver driver, MasterInfo masterInfo);
- 33. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Resource Allocation Executor Master Slave Hadoop Scheduler resourceOffers(…) Master Standby Cassandra Scheduler void resourceOffers(SchedulerDriver driver, List<Offer> offers); Offre de ressources
- 34. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Resource Allocation Cassandra Executor Master Slave Hadoop Scheduler offerRescinded(…) Master Standby Cassandra Scheduler Task Le scheduler Cassandra exécute le premier la tâche void offerRescinded(SchedulerDriver driver, OfferID offerId);
- 35. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Communication Cassandra Executor Master Slave Hadoop Scheduler statusUpdate(…) Master Standby Cassandra Scheduler Task void statusUpdate(SchedulerDriver driver, TaskStatus taskStatus);
- 36. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Communication Cassandra Executor Master Slave Hadoop Scheduler frameworkMessage(…) Master Standby Cassandra Scheduler Task void frameworkMessage(SchedulerDriver driver, ExecutorID execId, SlaveID slaveId, byte[] message);
- 37. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Error events Cassandra Executor Master Slave Hadoop Scheduler slaveLost(…) Master Standby Cassandra Scheduler Task void slaveLost(SchedulerDriver driver, SlaveID slaveId);
- 38. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Error events Cassandra Executor Master Slave Hadoop Scheduler executorLost(…) Master Standby Cassandra Scheduler Task void executorLost(SchedulerDriver driver,ExecutorID execId, SlaveID slaveId, int status);
- 39. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Error events Executor Master Slave Scheduler error(…) Master Standby void error(SchedulerDriver driver, String message);
- 40. @XebiaFr @YesWeScale#MesosUniversity Framework : API API de l’Executor
- 41. @XebiaFr @YesWeScale#MesosUniversity Executor : Lifecycle management Executor Master Slave Scheduler registered(…) Master Standby void registered (ExecutorDriver driver, ExecutorInfo execInfo, FrameworkInfo fwkInfo, SlaveInfo slaveInfo);
- 42. @XebiaFr @YesWeScale#MesosUniversity Executor : Lifecycle management Executor Master Slave Scheduler Master Standby disconnected(…) void disconnected(ExecutorDriver driver);
- 43. @XebiaFr @YesWeScale#MesosUniversity Executor : Lifecycle management Executor Master Slave Scheduler Master Standby reregistered(…) void reregistered(ExecutorDriver driver, SlaveInfo slaveInfo);
- 44. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Task management Cassandra Executor Master Slave launchTask(…) Master Standby Cassandra Scheduler Task void launchTask(ExecutorDriver driver, TaskInfo taskInfo);
- 45. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Task management Cassandra Executor Master Slave killTask(…) Master Standby Cassandra Scheduler Task void killTask(ExecutorDriver driver, TaskID taskId); schedulerDriver.killTask(…)
- 46. @XebiaFr @YesWeScale#MesosUniversity Scheduler : Communication Cassandra Executor Master Slave frameworkMessage(…) Master Standby Cassandra Scheduler Task void frameworkMessage(ExecutorDriver driver, byte[] msg);
- 47. @YourTwitterHandle@YourTwitterHandle@XebiaFr @YesWeScale#MesosUniversity Mesosphere : Chronos & Marathon
- 48. @XebiaFr @YesWeScale#MesosUniversity Mesosphere Florian Leibert Benjamin Hindman Tobi Knaup
- 49. @XebiaFr @YesWeScale#MesosUniversity Mesosphere Our mission is to make building and running distributed systems as easy as building or running an app on your smartphone.
- 52. @XebiaFr @YesWeScale#MesosUniversity Quoi ? “Init Daemon” pour Data Center Private PaaS
- 53. @XebiaFr @YesWeScale#MesosUniversity Pourquoi ? Simplifier Automatiser API pour les développeurs Améliorer l’efficacité
- 55. @XebiaFr @YesWeScale#MesosUniversity UI : liste des applications
- 56. @XebiaFr @YesWeScale#MesosUniversity UI : créer une application
- 57. @XebiaFr @YesWeScale#MesosUniversity Cycle de vie Master Slave Slave Slave Slave Marathon Tâches Évènements / Offres
- 59. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "myrestapi", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT0" } Démarrer une “Self Contained App”
- 60. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "myrestapi", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT0" } Démarrer une “Self Contained App” Slave myrestapi $PORT0 ports:[0]
- 61. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "ubuntu", "container": { "type": "DOCKER", "docker": { "image": "libmesos/ubuntu" } }, "instances": 1, "cpus": 0.5, "mem": 512, "uris": [], "cmd": "while sleep 10; do date -u +%T; done" } Démarrer une application Dockerisée
- 62. @XebiaFr @YesWeScale#MesosUniversity Health check • Par défaut, information provenant de Mesos TASK_RUNNING => Healthy • Configurable via l’API REST de Marathon
- 63. @XebiaFr @YesWeScale#MesosUniversity { "id": "my-restapi-example-app", ... "healthChecks": [ { "protocol": "COMMAND", "command": { "value": "curl -f -X GET http://$HOST:$PORT0/simpleapi/info" }, "gracePeriodSeconds": 300, "intervalSeconds": 5, "timeoutSeconds": 3, "maxConsecutiveFailures": 3 } ] } Health check
- 64. @XebiaFr @YesWeScale#MesosUniversity { "id": "my-restapi-example-app", ... "healthChecks": [ { "protocol": "HTTP", "path": "/simpleapi/info", "gracePeriodSeconds": 300, "intervalSeconds": 5, "timeoutSeconds": 3, "maxConsecutiveFailures": 3 } ] } Health check
- 65. @XebiaFr @YesWeScale#MesosUniversity Les contraintes Contrôler le “où” lors du déploiement
- 66. @XebiaFr @YesWeScale#MesosUniversity Les contraintes : tags Identifier les ressources par des tags : • type de hardware • emplacement physique (datacenter , rack) • … Positionnées au démarrage du slave $ mesos-slave … --attributes="rack_id:2"
- 67. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT", "constraints": [["hostname","UNIQUE"] } Contrainte : UNIQUE
- 68. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT", "constraints": [["hostname","UNIQUE"] } Contrainte : UNIQUE Slave 1 UserMgt UserMgt Slave 2 UserMgt CreditApp Slave 3 Slave 4 UserMgt CreditApp
- 69. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT", "constraints": [["harddrive","CLUSTER", "ssd"] } Contrainte : CLUSTER Slave 1 UserMgt UserMgt Slave 2 UserMgt CreditApp Slave 3 Slave 4 UserMgt UserMgt ssd hdd
- 70. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT", "constraints":[["hostname","CLUSTER", “slave-3”] } Contrainte : CLUSTER Slave 1 UserMgt Slave 2 UserMgt CreditApp Slave 3 Slave 4 UserMgt UserMgt CreditApp
- 71. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT", "constraints": [["rack_id","GROUP_BY"] } Contrainte : GROUP_BY Slave 1 UserMgt Slave 2 CreditApp Slave 3 Slave 4 Rack 1 Rack 2 UserMgt CreditApp UserMgt UserMgt
- 72. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT", "constraints": [["hostname","UNLIKE", "slave-[2-3]"] } Contrainte : UNLIKE Slave 1 UserMgt UserMgt Slave 2 UserMgt CreditApp Slave 3 Slave 4 UserMgt UserMgt
- 73. @XebiaFr @YesWeScale#MesosUniversity POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": ["http://storage.googleapis.com/restapi-jb/ RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$PORT", "constraints": [["hostname","LIKE", "slave-[1-3]"] } Contrainte : LIKE Slave 1 UserMgt UserMgt Slave 2 UserMgt CreditApp Slave 3 Slave 4 UserMgt UserMgt
- 74. @XebiaFr @YesWeScale#MesosUniversity Application groups • Définir des liens de dépendance entre les groupes • Scaler tout un groupe à la fois
- 75. @XebiaFr @YesWeScale#MesosUniversity Marathon : Application groups /myapp/bdd postgresql instance=3 redis instance=4 /myapp/service apache instance=5 tomcat instance=8 myapp Dépendance
- 76. @XebiaFr @YesWeScale#MesosUniversity { "id": "/myapp", "groups": [ { "id": "/myapp/bdd", "apps": [ { "id": "/myapp/postgresql", ... }, { "id": "/myapp/redis", ... } ] },{ "id": "/myapp/service", "dependencies": ["/myapp/bdd"], "apps": [ { "id": "/myapp/apache", ... }, { "id": "/myapp/tomcat", ... } ] } ] } Application groups
- 77. @XebiaFr @YesWeScale#MesosUniversity Service Discovery Où est mon application ?
- 78. @XebiaFr @YesWeScale#MesosUniversity HaProxy : reload configuration • HaProxy - Marathon bridge (script) • Bamboo
- 79. @XebiaFr @YesWeScale#MesosUniversity Service Discovery Slave 1 Slave 2 Marathon HA Proxy HA Proxy Marathon Bridge HA Proxy HA Proxy Marathon Bridge
- 80. @XebiaFr @YesWeScale#MesosUniversity Service Discovery Slave 1 Slave 2 UserMgt Port 31100 UserMgt Port 31200 UserMgt Port 31100 CreditApp Port 31200 UserMgt Port : 10000 CreditApp Port : 20000Marathon HA Proxy HA Proxy Marathon Bridge HA Proxy HA Proxy Marathon Bridge
- 81. @XebiaFr @YesWeScale#MesosUniversity Service Discovery Slave 1 Slave 2 UserMgt Port 31100 UserMgt Port 31200 UserMgt Port 31100 CreditApp Port 31200 UserMgt Port : 10000 CreditApp Port : 20000Marathon HA Proxy HA Proxy Marathon Bridge HA Proxy HA Proxy Marathon Bridge
- 82. @XebiaFr @YesWeScale#MesosUniversity Service Discovery Slave 1 Marathon HA Proxy HA Proxy Marathon Bridge Slave 2 HA Proxy HA Proxy Marathon Bridge UserMgt Port 31100 UserMgt Port 31200 UserMgt Port 31100 CreditApp Port 31200 localhost:10000 { slave1:31100 slave1:31200 slave2:31100 } localhost:20000 { slave2:31200 } UserMgt Port : 10000 CreditApp Port : 20000
- 83. @XebiaFr @YesWeScale#MesosUniversity Service Discovery Slave 1 Marathon HA Proxy HA Proxy Marathon Bridge Slave 2 HA Proxy HA Proxy Marathon Bridge UserMgt Port 31100 UserMgt Port 31200 UserMgt Port 31100 CreditApp Port 31200 localhost:10000 { slave1:31100 slave1:31200 slave2:31100 } localhost:20000 { slave2:31200 } UserMgt Port : 10000 CreditApp Port : 20000