Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)

22,156 views

Published on

Operating apps at web scale has become the new normal, but has been out of reach for most companies. Join us as we show you how to deploy and manage your Docker containers at scale. See how easy it is to build highly-available, fault-tolerant web scale apps using Docker with the Mesos cluster scheduler. Docker plus Mesos is a new way to scale applications. Together they give you capabilities similar to Google’s Borg, the Googleplex’s secret weapon of scalability and fault tolerance.

Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)

  1. 1. Building Web Scale Apps with Docker and Mesos Alexander Rukletsov Software Engineer @ Mesosphere
  2. 2. Why should we care?
  3. 3. Applications in the Cloud Era App App App App Client-Server Era: Small apps, big servers App Serv Serv Serv Serv Cloud Era: Big apps, small servers Virtualization Aggregation Server
  4. 4. Major Components ● Hardware ● Orchestration, deployment and isolation ● Cluster and resource management ● Scale-aware applications, service discovery, etc.
  5. 5. Major Components ● Hardware ● Orchestration, deployment and isolation ● Cluster and resource management ● Scale-aware applications, service discovery
  6. 6. Applications in the Cloud Era App App App App Client-Server Era: Small apps, big servers App Serv Serv Serv Serv Cloud Era: Big apps, small servers Virtualization Aggregation Server
  7. 7. “Divide et impera [et sarcina]” “Divide and rule [and package]” —Philip II of Macedon, paraphrased
  8. 8. Containers ● Lightweight Linux execution environment ● Static application composition ● Reliable deployment ● Unit of resource isolation ● Execution isolation ● Multi-tenancy without heavyweight VMs
  9. 9. Docker ● Open source ● Configurable layers ● Reproducible ● Version-controlled ● Plenty of other people’s containers
  10. 10. Docker ● Open source ● Configurable layers ● Reproducible ● Version-controlled ● Plenty of other people’s containers ● First-class citizen in Mesos and the Mesosphere stack ● Kubernetes employs and promotes Docker
  11. 11. © Gerard Julien/AFP Run everything in containers!
  12. 12. What about container management?
  13. 13. © Unknown
  14. 14. © Unknown
  15. 15. Apache Mesos facts ● Created in 2009 at UC Berkeley, hardened in Twitter ● Top-level Apache project ● Mesosphere, Twitter, and Airbnb are major users / contributors ● Scales to 10 000s of nodes, production grade ● Packages and support through Mesosphere ● Google officially endorsed Mesos for Kubernetes ● Built-in containerization, including Docker
  16. 16. Mesos as a Distributed OS kernel ● Two level resource scheduling ● Launch tasks across the cluster ● Communication between tasks (like IPC) ● APIs for building “native” applications (aka frameworks): program against the datacenter ● APIs in C++, Python, JVM-languages, Go and counting ● Pluggable CPU, memory, IO isolation ● Multi-tenant workloads ● Failure detection ● Easy failover and HA
  17. 17. How Mesos works (HA mode) Application Scheduler Zookeeper Mesos MasteMr esos Master Mesos Master Mesos Master Mesos Slave Executor Task Task Executor Framework Task Task Task Executor Mesos Slave
  18. 18. Manage containers with Mesos! © ect.nl
  19. 19. Mesos + Docker =
  20. 20. Native Docker support in Mesos message DockerInfo { required string image = 1; // Network options. enum Network { HOST = 1; BRIDGE = 2; NONE = 3; } message PortMapping { required uint32 host_port = 1; required uint32 container_port = 2; optional string protocol = 3; // Protocol to expose (ie: tcp, udp). } optional Network network = 2 [default = HOST]; repeated PortMapping port_mappings = 3; optional bool privileged = 4 [default = false]; // Allowing arbitrary parameters to be passed to docker CLI. repeated Parameter parameters = 5; }
  21. 21. Native Docker support in Mesos CommandInfo command; command.set_value("dd if=/dev/zero of=/dev/null"); ContainerInfo::DockerInfo dockerInfo; dockerInfo.set_image("mesosphere/inky"); ContainerInfo containerInfo; containerInfo.set_type(ContainerInfo::DOCKER); containerInfo.mutable_docker()->CopyFrom(dockerInfo); TaskInfo task; task.set_name(""); task.mutable_task_id()->set_value("1"); task.mutable_slave_id()->CopyFrom(offer.slave_id()); task.mutable_resources()->CopyFrom(offer.resource()); task.mutable_command()->CopyFrom(command); task.mutable_container()->CopyFrom(containerInfo); vector<TaskInfo> tasks; tasks.push_back(task); driver.launchTasks(offer.id(), tasks);
  22. 22. Native Docker support in Marathon // nginx-task.json { "container": { "type": "DOCKER", "docker": { "image": "nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0, "servicePort": 80, "protocol": "tcp" } ] } }, "id": "nginx", "instances": "1", "cpus": "0.25", "mem": "256", "uris": [] } $ cat nginx-task.json | http http://dev1.mesosphere.com:8080/v2/apps
  23. 23. Future Docker Swarm API support App App App Mesos Framework Mesos App Serv Docker Docker Swarm API Docker App Docker Docker Docker App Docker App Docker Serv Serv Serv Serv
  24. 24. What Mesos contributes ● Multi-framework: weighted fair sharing, roles, etc. ● Run Docker containers alongside other popular frameworks (e.g. Spark, Rails, Hadoop, …) ● Run services and batch apps in the same cluster ● Advanced scheduling: resources, constraints, global view ● High resource availability, cluster self-healing ● Proven at scale, battle-tested in production ● GUI / CLI cluster management console
  25. 25. Ways of running Dockers ● Marathon easy to setup, reliable orchestration ● Multiple Marathons benefit from two level scheduling, reservations, framework roles ● Custom framework fine-grained management
  26. 26. “L’homme est libre au moment qu’il veut l’être” “Man is free at the instant he wants to be” —Voltaire
  27. 27. Cluster configuration example #1 Services Mesos App App AWS, DigitalOcean, GCE Apps API Kernel Hardware Docker Docker Services REST API “Marathon” App Docker App App Docker App
  28. 28. Cluster configuration example #2 Mesos App App AWS, DigitalOcean, GCE Apps API Kernel Hardware Services Services REST API “Marathon” App App Kubernetes Docker Docker Docker App Docker App
  29. 29. Cluster configuration example #3 Docker Services REST API “Marathon” (init) Serv Serv Serv Serv Mesos Spark, MPI, Hadoop, Storm Mesos SDK App Java, Python, C++, Go Serv Serv Serv Serv Batch REST API “Chronos” (cron) Serv App Recurring Jobs (ETL, backups) Hardware Native Long running Batch Apps API Kernel App Docker
  30. 30. Cluster configuration example #4 Docker AWS, DO, GCE Mesos Spark, MPI, Hadoop, Storm Mesos SDK App Java, Python, C++, Go Serv Serv Services REST API “Marathon” (init) Batch REST API “Chronos” (cron) Serv App Recurring Jobs (ETL, backups) Hardware Native Long running Batch Apps API Kernel App Docker Serv
  31. 31. Mesos is not only for “big” players! © Reuters
  32. 32. Tiny clusters for everybody ● Mesosphere on GCE ● Mesosphere on DigitalOcean ● Mesosphere on AWS ● Portable Mesosphere on a USB stick
  33. 33. google.mesosphere.com
  34. 34. digitalocean.mesosphere.com
  35. 35. elastic.mesosphere.io
  36. 36. Demo: Mesos-on-Mesosphere
  37. 37. Mesos-on-Mesosphere Task Task Task Zookeeper Mesos Marathon Slave Mesos Master Mesos MasMteersos MasteMresos Master Mesos Slave ... Task
  38. 38. Mesos-on-Mesosphere Mesos Slave Mesos Slave Docker Docker Docker Docker Zookeeper Mesos Marathon Slave Mesos Master Mesos MasMteersos MasteMresos Master Mesos Slave ... Mesos Master Docker Mesos Slave Docker Mesos Slave Mesos Master
  39. 39. Summary ● Complete stack for large (and small) distributed apps ● Multi-tenancy ● Resource optimizations ● Easy to deploy ● No vendor lock-in
  40. 40. Thank You. alex@mesosphere.io #mesos on irc.freenode.net

×