Mesos, DC/OS and the Architecture of the New Datacenter

5,167 views

Published on

Talk on Mesos and DC/OS by Benjamin Bannier (Mesosphere). Given at the QAware Meetup Cloud Native Night Munich #1, June 2016.

Published in: Data & Analytics
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,167
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
24
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Mesos, DC/OS and the Architecture of the New Datacenter

  1. 1. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS, DC/ OS AND THE ARCHITECTUR E OF THE NEW DATACENTER 1
  2. 2. © 2016 Mesosphere, Inc. All Rights Reserved. WHOAMI 2 benjamin.bannier@mesosphere.io Mesosphere Distributed databases PhD, High energy nuclear physics2007 - 2014 2014 - 2015 2015 -
  3. 3. 3 PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME SERVER VIRTUAL MACHINE PARTITION (LPAR) UNIT OF INTERACTION ! ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER ! LINUX, WINDOWS ! DATA / TRANSACTION PROCESSING ! UNIX, IBM OS/360 DEFINITIVE APPS AND OS ! ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER ! HYPERVISOR + GUEST OS ! BIG DATA, INTERNET OF THINGS, MOBILE APPS ! ??? ???DATACENTER NEW FORM FACTOR FOR 
 DEVELOPING AND RUNNING APPS ! BIG DATA, INTERNET OF THINGS, MOBILE APPS ! DATACENTER OPERATING SYSTEM (DC/OS) EVOLUTION OF APPLICATIONS
  4. 4. © 2016 Mesosphere, Inc. All Rights Reserved. STATIC PARTITIONING 4 Deals well with heterogeneous workloads. Headroom left for peak workloads will lead to poor overall resource utilization. When resource requirements change potentially costly reconfigurations.
  5. 5. © 2016 Mesosphere, Inc. All Rights Reserved. SHARED RESOURCES 5 Multiple frameworks can use the same cluster resources, with their share adjusting dynamically.
  6. 6. © Gerard Julien/ Run everything in containers!
  7. 7. © 2016 Mesosphere, Inc. All Rights Reserved. What about container management? 7
  8. 8. © 2016 Mesosphere, Inc. All Rights Reserved. What about container management? 8
  9. 9. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS 9
  10. 10. © 2016 Mesosphere, Inc. All Rights Reserved. THE BIRTH OF MESOS 10 TWITTER TECH TALK The grad students working on Mesos give a tech talk at Twitter. March 2010 APACHE INCUBATION Mesos enters the Apache Incubator. Spring 2009 CS262B Ben Hindman, Andy Konwinski and Matei Zaharia create “Nexus” as their CS262B class project. MESOS PUBLISHED Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center is published as a technical report. September 2010 December 2010 DC/OS April 2016
  11. 11. © 2016 Mesosphere, Inc. All Rights Reserved. VISION 11 Sharing resources between batch processing frameworks • Hadoop • MPI • Spark What does an operating system provide? • Resource management • Programming abstractions • Security • Monitoring, debugging, logging
  12. 12. © 2015 Mesosphere, Inc. All Rights Reserved. • A top-level Apache project • A cluster resource negotiator • Scalable to 10,000s of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support 12 Apache Mesos
  13. 13. © 2016 Mesosphere, Inc. All Rights Reserved. ARCHITECTURE 13 Mesos Fundamentals Now called Agent
  14. 14. © 2016 Mesosphere, Inc. All Rights Reserved. TWO-LEVEL SCHEDULING 14 Scheduler(s) AllocationUser Wor Resource Launch
  15. 15. © 2016 Mesosphere, Inc. All Rights Reserved. TWO-LEVEL SCHEDULING 15
  16. 16. © 2016 Mesosphere, Inc. All Rights Reserved. 16 YARN
  17. 17. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAIN ER NETWO RKING 17
  18. 18. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER NETWORKING 18 Containers isolate tasks on the agent, but what about their communication? The status quo in a Mesos cluster: one IP per agent. Many containers per agent: they must share a single IP. Agent ContainerContainer ContainerContainer ContainerContainer
  19. 19. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER NETWORKING 19 This causes headaches: Port conflicts Security compromises Performance Service discovery Agent ContainerContainer ContainerContainer Web serviceWeb service
  20. 20. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER NETWORKING 20 This causes headaches: Port conflicts Security compromises Performance Service discovery Agent ContainerContainer ContainerContainer Test serviceProd. service
  21. 21. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER NETWORKING 21 This causes headaches: Port conflicts Security compromises Performance Service discovery Agent ContainerContainer ContainerContainer ContainerContainer
  22. 22. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER NETWORKING 22 This causes headaches: Port conflicts Security compromises Performance Service discovery Agent Container Container Agent Container Container
  23. 23. © 2016 Mesosphere, Inc. All Rights Reserved. IP PER CONTAIN ER 23 Segregating containers’ network traffic can solve these problems in an elegant, maintainable way.
  24. 24. © 2016 Mesosphere, Inc. All Rights Reserved. CALICO NETWORK ISOLATION 24 Agent iptables Kernel routing Container Calico modul
  25. 25. © 2016 Mesosphere, Inc. All Rights Reserved. CALICO NETWORK ISOLATION 25
  26. 26. © 2016 Mesosphere, Inc. All Rights Reserved. DISTRIBU TED SYSTEMS SDK 26 “Developers will ask for 
 an API not for a VM to
 run their software”
  27. 27. © 2016 Mesosphere, Inc. All Rights Reserved. Distributed System SDK 27 Focus on application logic, not on data-center structure • Avoid networking-related code • Reuse of built-in fault-tolerance and high availability • Reuse distributed (infrastructure) frameworks (e.g., storage) Mesos
  28. 28. © 2016 Mesosphere, Inc. All Rights Reserved. FIRST FRAMEWORK/SERVICE 28 Mesos Utilize resources ResourceOffer React on Tasks Events TaskStatus Update Scheduler Interface
  29. 29. © 2016 Mesosphere, Inc. All Rights Reserved. RENDLER 29 Mesos Scheduler Interface Examples implementations for different languages: C++, Go, Haskell, Java, Python, Scala https://github.com/mesosphere/RENDLER
  30. 30. © 2016 Mesosphere, Inc. All Rights Reserved. 30
  31. 31. Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) Big Data + Analytics EnginesMicroservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Distributed systems kernel to abstract resources Ecosystem of frameworks & apps Consistent architecture to run on top of kernel User Interface (GUI & CLI) Core system services 
 (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Any Infrastructure (Physical, Virtual, Cloud) 31 DC/OS ENABLES MODERN DISTRIBUTED APPS
  32. 32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 DC/OS is … ‣ 100% open source (ASL2.0) • A big, diverse community ‣ An umbrella for ~30 OSS projects • Roadmap and designs • The build tool chain • Docs and tutorials ‣ Not limited in any way ‣ Familiar, with a few new features
  33. 33. © 2016 Mesosphere, Inc. All Rights Reserved. http://dcos.io 33
  34. 34. © 2016 Mesosphere, Inc. All Rights Reserved. QUESTIONS? 34 http://apache.mesos.org http://dcos.io http://mesosphere.com

×