Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert

12,992 views

Published on

In this slide deck you will learn:
- How Twitter used Mesos to eliminate the "fail whale"
- Why Mesos is important to companies of any size
- How Mesosphere Mesos reduces operational complexity
- How to use Mesos to increase developer velocity
- How Mesos helps you reduce total cost of ownership and ultimately increase the competitiveness of your business
- How to use Mesos on your own datacenter, or virtualized in a public cloud like Amazon Web Services (AWS) and Google Compute Engine (GCE)

You will also gain knowledge about how Apache Mesos is being used by well-known brands, including Twitter, Airbnb, and Hubspot to improve their operational effectiveness and save money. Learn how Twitter has scaled its service, running Mesos on tens of thousands of nodes. Learn how Airbnb has reduced its operational complexity, running it's entire data stack on Mesos. And learn how Hubspot has figured out how to use Mesos to reduce it's Amazon Web Services bill by 65%.

Try Mesosphere for Free: https://mesosphere.com/try

Published in: Software

Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert

  1. 1. Software for the hyperscale datacenter Florian Leibert, CEO & Founder flo@mesosphere.io
  2. 2. Imagine if…
  3. 3. All your servers in your datacenter and cloud
  4. 4. were pooled together
  5. 5. So they behave like one big computer
  6. 6. as easy as
  7. 7. …and building new datacenter apps is as easy as building an app for one machine PowerBook G4
  8. 8. Fantasy?
  9. 9. Mesos Users Today Sigmoid Analytics
  10. 10. Today’s Legacy Datacenter
  11. 11. Provision VMs in the cloud or on physical servers Today’s Legacy Datacenter
  12. 12. Install Hadoop on a static set of machines Installing an Application with Static Partitioning
  13. 13. Install Web Server on a static set of machines Installing an Application with Static Partitioning
  14. 14. Scale up Hadoop manually Resizing an Application with Static Partitioning
  15. 15. What if your Laptop was operated like your Data
  16. 16. Issues with Statically Partitioned Data Centers Complex
 Machine sprawl, manual resize/scale Limited
 No software failure handling, “black box” Inefficient
 Static partitioning, overhead Not Developer-Friendly
 Long time to roll out software, development starts at the machine level
  17. 17. Aggregation Mesosphere aggregates resources, makes a data center look like one big computer Mesosphere runs on top of a VM or on bare metal
  18. 18. Applications in the Cloud Era Cloud Era:
 Big apps, small servers Client-Server Era:
 Small apps, big servers Server Virtualization App App App App App Aggregation Serv Serv Serv Serv
  19. 19. From Static Partitioning to Elastic Sharing Static Partitioning Elastic Sharing WEB HADOOPCACHE WASTED FREEFREE HADOOP WEB CACHE WASTED WASTED 100% — 100% —
  20. 20. Applications are Changing
  21. 21. Deployments
  22. 22. Mesos Facts Scales to 10,000s of nodes Top-level Apache project Twitter and Airbnb are major users and contributors APIs for C++, Python, JVM, Go Pluggable CPU, memory, IO isolation Packages and commercial support through Mesosphere Highly available, scalable, elastic
  23. 23. Apache Mesos Features Multi-tenancy Improved resource utilization Resource Isolation Fault-tolerance, HA Scalability + Elasticity 26
  24. 24. Mesos Slave Hadoop task-tracker Mesos Executor Task #1 Task #2 ./ruby XYZ Mesos Slave Docker Executor Docker Executor java -jar XYZ.jar ./xyz Mesos Master Mesos Master Mesos Master Hadoop scheduler Marathon scheduler Zookeeper quorum
  25. 25. Mesos Slave Hadoop task-tracker Mesos Executor Task #1 Task #2 ./ruby XYZ Mesos Slave Docker Executor Docker Executor java -jar XYZ.jar ./xyz Mesos Master Mesos Master Mesos Master Hadoop scheduler Marathon scheduler Zookeeper quorum 28
  26. 26. Mesos Slave Hadoop task-tracker Mesos Executor Task #1 Task #2 ./ruby XYZ Mesos Slave Docker Executor Docker Executor java -jar XYZ.jar ./xyz Mesos Master Mesos Master Mesos Master Hadoop scheduler Marathon scheduler Zookeeper quorum 29
  27. 27. Mesos Slave Hadoop task-tracker Mesos Executor Task #1 Task #2 ./ruby XYZ Mesos Slave Docker Executor Docker Executor java -jar XYZ.jar ./xyz Mesos Master Mesos Master Mesos Master Hadoop scheduler Marathon scheduler Zookeeper quorum 30
  28. 28. Resource Offers and Launching a Task 1. resourceOffers() 2. launchTasks() 3. launchTask() 4. statusUpdate() 5. statusUpdate() Mesos Slave Executor Mesos Master Scheduler Task Task
  29. 29. Response times and overhead are significantly Timetoprovision(seconds) 1 100 10000 Bare metal VM Container Inspired by Tomas Barton’s Mesos talk at InstallFest in Prague
  30. 30. Mesos provides fine-grained resource isolation Compute Node Mesos Slave Process Hadoop task-tracker Mesos Executor Task #1 Task #2 ruby XYZ Container (Cgroups) Executor
  31. 31. Mesos provides fine-grained resource isolation Compute Node Mesos Slave Process Hadoop task-tracker Task #1 Task #2 Container (Cgroups) Task #3
  32. 32. Mesos provides componentized resource Mesos Slave Process Mesos Containerizer CGroups CPU isolator CGroups Memory isolator Launcher Container foo Task baz Containerizer API Executor bar
  33. 33. Mesos provides pluggable resource isolation External Containerizer External Containerizer API Mesos Slave Process External Containerizer Program Container foo MySQL Containerizer API Ubuntu 13.10 Container bar Ruby Centos 6.4 github.com/mesosphere/deimos
  34. 34. From Static Partitioning to Elastic Sharing Static Partitioning Elastic Sharing WEB HADOOPCACHE WASTED FREEFREE HADOOP WEB CACHE WASTED WASTED 100% — 100% —
  35. 35. Mesos has no single point of failure Tasks keep running! Framework Masters
  36. 36. Master node can fail-over Tasks keep running! Framework Masters
  37. 37. Slave processes can fail-over Tasks keep running! Compute Node Mesos Slave Process Mesos Executor Mesos Executor
  38. 38. The UNIX Operating System Stack SSHd Linux, BSD MySQLApache Kernel ApplicationsMemcached Init, Upstart, Systemd Init System
  39. 39. The Mesos Stack Memcached Mesos RedisRails Kernel ApplicationsElasticsearch Marathon Init System
  40. 40. Mesosphere Demo
  41. 41. Marathon
  42. 42. What is Marathon? “Init Daemon” for the data center •Runs any Linux binary without modification (e.g. Rails, Tomcat, …) •Cluster-wide process supervisor ! Private PaaS •Service discovery •Automated software and hardware failure handling •Deployment and scaling
  43. 43. Marathon Design Goals Simplify
 Fewer things to manage = fewer sources of error
 All machines have the same configuration Automate
 Nobody likes to get paged at night
 Automatically respond to hardware & software failures Improve efficiency
 Humans are bad at estimating resource requirements
 Use software-controlled elastic resource sharing Self-serve API
 Developers want to have control over their apps
 Give them direct access to cluster resources
  44. 44. Marathon Key Features REST/JSON API Easy to use web interface Authentication & SSL Highly available - no single point of failure Placement constraints (nodes, racks, etc.) Service discovery & load balancing via HAProxy Event system for integration with 3rd party components, like load balancers !
  45. 45. Marathon Workflow 3.foo.com1.foo.com 2.foo.com Rails Rails Mesos Marathon POST /v2/apps
  46. 46. Marathon Workflow 1.foo.com 3.foo.com2.foo.com Rails Rails Mesos Marathon POST /v2/apps Play Play
  47. 47. Marathon API - Launching Self-Contained Apps •Command to start the app •URL(s) to the app archive/configuration •Environment variables
 ! POST /v2/apps { “id”: “Play”, “uris”: [“http://downloads.mesosphere.io/tutorials/ PlayHello.zip”] “cmd”: “./Hello-*/bin/hello -Dhttp.port=$PORT”, “env”: {“SECRET”: “password123”} }
  48. 48. Marathon API - Launching Dockers •Starting with Mesos 0.19 containers are 1st class citizens •Deimos is the Docker containerizer ! POST /v2/apps { “id”: “Cassandra”, “container”: { “image”: “docker:///mesosphere/cassandra:2.0.6”, “options”: [“-v”, “/mnt:/mnt:rw”, “-e”, “CLUSTER_NAME=prod”] } }
  49. 49. Marathon API - Integration with Deimos Mesos Master Marathon Launch 3x Redis Mesos Slave Deimos Task 1 Task 2 docker run ... Deimos docker run ... Mesos Slave Deimos docker run ... Task 3
  50. 50. Marathon API - Scaling Apps •Just tell Marathon how many you want! ! ! PATCH /v2/apps/Play { “instances”: 4 }
  51. 51. Marathon Service Discovery with HAProxy Apps available on localhost & known port HAProxy updates via Marathon REST API HAProxy runs on every cluster node Configurable policies Rails HAProxy Rails HAProxy SQL HAProxy Search Search HAProxy Cache Marathon
  52. 52. Mesosphere Products & Services Infrastructure • Try out Mesos on Amazon Web Services
 https://elastic.mesosphere.io • On premise and cloud provisioning tools • Linux Packages • Mesosphere Plugins and Tools • Professional Services • Training, Installation, Support Contracts
  53. 53. Thank you.

×