1
Apache Mesos and
the New Open Source
Architecture of the
Modern Datacenter
April 20, 2016 - LA Big Data User Group
2
AN INTRO TO
OR ...
3
HELLO
Aaron Williams
Head of Advocacy
aaron.williams@mesosphere.io
@_arw_
4
“Every business will become a
software business, build applications,
use advanced analytics and provide
SaaS services.”
- Satya Nadella
Smart CEO Guy
5
DC/OS ORIGIN STORY
PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME
SERVER VIRTUAL MACHINEPARTITION (LPAR)
UNIT OF
INTERACTION
● ERP, CRM, PRODUCTIVITY,
MAIL & WEB SERVER
● LINUX, WINDOWS
● DATA / TRANSACTION
PROCESSING
● UNIX, IBM OS/360
DEFINITIVE
APPS AND OS
● ERP, CRM, PRODUCTIVITY,
MAIL & WEB SERVER
● HYPERVISOR + GUEST OS
● BIG DATA, INTERNET OF
THINGS, MOBILE APPS
● ???
???DATACENTER
NEW FORM FACTOR FOR
DEVELOPING AND RUNNING APPS
● BIG DATA, INTERNET OF
THINGS, MOBILE APPS
● DATACENTER OPERATING
SYSTEM (DC/OS)
6
Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
MODERN DISTRIBUTED APPS
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions &
Logic
Search
Time Series
SQL / NoSQL
Databases
Modern App Components
Distributed systems kernel to
abstract resources
Ecosystem of frameworks & apps
Consistent architecture to run on
top of kernel
User Interface (GUI & CLI)
Core system services
(e.g., distributed init, cron, service
discovery, package mgt & installer,
storage)
Any Infrastructure (Physical, Virtual, Cloud)
7
8
LAYERS **
Application Orchestration
Container Orchestration
Job Scheduling
Containerization
Resource Management
Provisioning
CloudFoundry, Heroku,
OpenShift
Kubernetes, Marathon,
Swarm, Fleet, Lattice, ECS
Chronos, Kubernetes
Docker, Rkt, Garden, Mesos
Mesos
Puppet, Chef, Ansible, Salt,
Vagrant, Otto
** obviously not exhaustive
Machine Management
AWS, Azure, GCE, OpenStack,
vSphere, VirtualBox, Fusion
9
THE
BASICS
So … DC/OS is:
● 100% open source (ASL2.0)
+ A big, diverse community
● An umbrella for ~30 OSS projects
+ Roadmap and designs
+ The build tool chain
+ Docs and tutorials
● Not limited in any way
● Familiar, with a few new features
10
THE COMMUNITY
11
EARLY
RESULTS
Forging Ahead with Mesos, Containers and
DC/OS
Having now run our event streaming and big data
ingestion pipeline services in production on DC/OS,
across 3 regions, over the last year, we've achieved the
following results:
● A 66% reduction in AWS Instances
● Cost Improvements up to 57%
● An impressive 40 sec time to deploy a new build
with zero downtime
● A 3 min time to stand up a new region
● 100% Uptime
● Total Resources needed: 1 DevOps Engineer
http://cloudengineering.autodesk.
com/blog/2016/04/autodesk-is-forging-ahead-with-
dcos.html
12
Give it a try:
https://dcos.io/install
https://github.com/mesosphere/time-series-demo
13
dcos.io
14
15
THE
UNIVERSE
16
17
THANK
YOU
18
MORE
NEWS
Introducing open source DC/OS: The best way
to run containers (and everything else)
http://mesosphere.com/blog/2016/04/19/open-source-dcos/
Microsoft joins the new DC/OS open source
project
https://azure.microsoft.com/en-us/blog/microsoft-joins-the-new-dc-os-open-
source-project/
Mesosphere Open Sources the Mesos-based
Data Center Operating System
http://thenewstack.io/inevitable-mesosphere-open-sources-mesos-based-data-
center-operating-system/
19
DEMO
VIDEO
20
PARTNERS
VIDEO

Apache Mesos and the new Open Source Architecture of the Modern Datacenter

  • 1.
    1 Apache Mesos and theNew Open Source Architecture of the Modern Datacenter April 20, 2016 - LA Big Data User Group
  • 2.
  • 3.
    3 HELLO Aaron Williams Head ofAdvocacy aaron.williams@mesosphere.io @_arw_
  • 4.
    4 “Every business willbecome a software business, build applications, use advanced analytics and provide SaaS services.” - Satya Nadella Smart CEO Guy
  • 5.
    5 DC/OS ORIGIN STORY PHYSICAL(x86) VIRTUAL HYPERSCALEMAINFRAME SERVER VIRTUAL MACHINEPARTITION (LPAR) UNIT OF INTERACTION ● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER ● LINUX, WINDOWS ● DATA / TRANSACTION PROCESSING ● UNIX, IBM OS/360 DEFINITIVE APPS AND OS ● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER ● HYPERVISOR + GUEST OS ● BIG DATA, INTERNET OF THINGS, MOBILE APPS ● ??? ???DATACENTER NEW FORM FACTOR FOR DEVELOPING AND RUNNING APPS ● BIG DATA, INTERNET OF THINGS, MOBILE APPS ● DATACENTER OPERATING SYSTEM (DC/OS)
  • 6.
  • 7.
    Datacenter Operating System(DC/OS) Distributed Systems Kernel (Mesos) MODERN DISTRIBUTED APPS Big Data + Analytics EnginesMicroservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Distributed systems kernel to abstract resources Ecosystem of frameworks & apps Consistent architecture to run on top of kernel User Interface (GUI & CLI) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Any Infrastructure (Physical, Virtual, Cloud) 7
  • 8.
    8 LAYERS ** Application Orchestration ContainerOrchestration Job Scheduling Containerization Resource Management Provisioning CloudFoundry, Heroku, OpenShift Kubernetes, Marathon, Swarm, Fleet, Lattice, ECS Chronos, Kubernetes Docker, Rkt, Garden, Mesos Mesos Puppet, Chef, Ansible, Salt, Vagrant, Otto ** obviously not exhaustive Machine Management AWS, Azure, GCE, OpenStack, vSphere, VirtualBox, Fusion
  • 9.
    9 THE BASICS So … DC/OSis: ● 100% open source (ASL2.0) + A big, diverse community ● An umbrella for ~30 OSS projects + Roadmap and designs + The build tool chain + Docs and tutorials ● Not limited in any way ● Familiar, with a few new features
  • 10.
  • 11.
    11 EARLY RESULTS Forging Ahead withMesos, Containers and DC/OS Having now run our event streaming and big data ingestion pipeline services in production on DC/OS, across 3 regions, over the last year, we've achieved the following results: ● A 66% reduction in AWS Instances ● Cost Improvements up to 57% ● An impressive 40 sec time to deploy a new build with zero downtime ● A 3 min time to stand up a new region ● 100% Uptime ● Total Resources needed: 1 DevOps Engineer http://cloudengineering.autodesk. com/blog/2016/04/autodesk-is-forging-ahead-with- dcos.html
  • 12.
    12 Give it atry: https://dcos.io/install https://github.com/mesosphere/time-series-demo
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
    18 MORE NEWS Introducing open sourceDC/OS: The best way to run containers (and everything else) http://mesosphere.com/blog/2016/04/19/open-source-dcos/ Microsoft joins the new DC/OS open source project https://azure.microsoft.com/en-us/blog/microsoft-joins-the-new-dc-os-open- source-project/ Mesosphere Open Sources the Mesos-based Data Center Operating System http://thenewstack.io/inevitable-mesosphere-open-sources-mesos-based-data- center-operating-system/
  • 19.
  • 20.