© 2017 Mesosphere, Inc. All Rights Reserved.
Eryn Muetzel, Product Marketing for Big Data
@errbear3
JOURNEY TO THE MODERN APP WITH
CONTAINERS, MICROSERVICES AND BIG DATA
Datacenter Operating System (DC/OS)
The simplest way to build, run, scale and manage modern enterprise apps
© 2017 Mesosphere, Inc. All Rights Reserved.
© 2017 Mesosphere, Inc. All Rights Reserved.
INDUSTRY TRANSITIONS
Customer analytics
2000s
Electronic customer
records
1980s
Online customer
engagement
1990s
Real-time & predictive
customer engagement
2013+
© 2017 Mesosphere, Inc. All Rights Reserved.
© 2017 Mesosphere, Inc. All Rights Reserved.
WE LIVE IN THE
ALWAYS
CONNECTED
ECONOMY.
3.7BMobile Broadband Users
80%New Data is Unstructured
13Connected Devices per Household
(N. Amer)
40%Annual Enterprise Data Growth
IN THE ALWAYS CONNECTED ECONOMY
SOFTWARE IS YOUR BUSINESS.
Build more apps.
Personal and meaningful
customer experiences
Innovate faster.
Capture value from new
value streams and
actionable data insights
Modernize IT
Always-available services
that are secure and cost
effective
1M trips daily across 311 cities,
disrupting multiple industries.
Personal behavior-based
insurance using IoT.
Real-time, personalized clinical
care.
MODERN ENTERPRISE APPS IN ACTION.
© 2017 Mesosphere, Inc. All Rights Reserved.
ARCHITECTURAL SHIFT
TRADITIONAL
ENTERPRISE
APPLICATIONS
MODERN
ENTERPRISE
APPLICATIONS
Data
Code
Latency
Users
2+ Billion smartphone users
3+ Billion internet users
Enterprise data growth (CAGR): 40%+
Source: KPCB Internet Trends 2017, EMC Digital Universe 2014
© 2017 Mesosphere, Inc. All Rights Reserved.
ARCHITECTING FOR SCALE AT TWITTER
BEFORE AFTER
© 2017 Mesosphere, Inc. All Rights Reserved.
Streaming
11
THE MODERN ENTERPRISE APP
PLATFORM
SERVICES
MICROSERVICES
STATELESS CONTAINERS ANALYTICS
Batch
Machine
Learning
Search
DATABASES
Time
Series
SQL /
NoSQL
© 2017 Mesosphere, Inc. All Rights Reserved.
Web Apps and
Microservices
Connected
Devices
Machine Learning
and Simulation
Industrial IoT Connected CarFraud DetectionSocial Media
DIGITAL TRANSFORMATION REQUIRES MODERN APPS
Modern Apps
Operational
expertise
Continuous
integration
and delivery
Big Data
Analytics
Message
Queue
Distributed
Database
Search
Future
services...
Platform
Services
Container
Orchestration
How do you develop these skills in your organization?
MODERN
ENTERPRISE APPS
REQUIRE NEW
CAPABILITIES. Distributed computing
expertise.
Engineering and operations of secure
& highly reliable services at scale
DevOps process & culture.
Frequent & reliable releases supported
by an automated CI/CD toolchain
Cloud-native technologies.
Containers, microservices, data servies
(e.g., Spark, Kafka, Cassandra)
© 2017 Mesosphere, Inc. All Rights Reserved.
MODERN ENTERPRISE APPS
Containers Data Services
MicroservicesOpen source & Social coding1
2
3
4
© 2017 Mesosphere, Inc. All Rights Reserved.
OPEN SOURCE SOFTWARE PROVIDES LEVERAGE1
© 2017 Mesosphere, Inc. All Rights Reserved.
CONTAINERS SIMPLIFY PACKAGING AND DEPLOYMENT
Private Copy
Shared
App 3
Libraries
Guest
Operating
System
Infrastructure
App 1
Libs
Host Operating System
Start / stop time One minute to few seconds Milliseconds
Workload density 10 - 100x1x
App 2
Libraries
Guest
Operating
System
App 1
Libraries
Guest
Operating
System
App 2
Libs
App 3
Libs
Virtual Machines Containers
Host Operating System
Hypervisor
Infrastructure
App 4
Libs
App 5
Libs
2
© 2017 Mesosphere, Inc. All Rights Reserved.
JUST HOW FAST ARE CONTAINERS?
Larry Rau from @Verizon
with @flo Launching 50,000
containers in seconds with
@mesosphere #DC/OS
© 2017 Mesosphere, Inc. All Rights Reserved.
MICROSERVICES ENABLE FAST AND FREQUENT RELEASES
Traditional Architecture
Many functions
in a single process
Siloed
teams
RESTAPIs
Microservices Architecture
Cross-functional
teams organized
around capabilities
Scales
individually
Each element of
functionality defined as
“microservices”
Scales
monolithically
3
© 2017 Mesosphere, Inc. All Rights Reserved.
DATA SERVICES PROVIDE CONNECTION AND PERSISTENCE
Data Ingestion
Response
Devices
ClientSensors
Event Bus
Reactive
App
Database
Analytics
Use Case Examples
● Anomaly detection
● Personalization
● IoT Applications
● Predictive Analytics
● Machine Learning
4
© 2017 Mesosphere, Inc. All Rights Reserved. 20
BUT WHAT ABOUT THE INFRASTRUCTURE?
CaaSPaaS
Container
AppContainer App
Stateful
Service #1
V1
Big Data
Analytics #1
V1
Stateful
Service #2
V1
Stateful
Service #2
V2
PaaS
Container
App
Team A
Big Data
Analytics #1
V2
Big Data
Analytics
#2
V1
Team B
© 2017 Mesosphere, Inc. All Rights Reserved. 21
BUT WHAT ABOUT THE INFRASTRUCTURE?
CaaSPaaS
Container
AppContainer App
Stateful
Service #1
V1
Big Data
Analytics #1
V1
Stateful
Service #2
V1
Stateful
Service #2
V2
PaaS
Container
App
Team A
Big Data
Analytics #1
V2
Big Data
Analytics
#2
V1
Team B
● Extremely low utilization - less than 10%, due to static
partitioning
● Manual operations - Weeks to provision, wasted time
managing individual machines
● Difficult to experiment with new tech - months to
provision environments to evaluate new technologies
● High risk - Failure from manual operations and
difficulties maintaining high availability
© 2015 Mesosphere, Inc. All Rights Reserved.
WHERE HAVE WE SEEN THIS PROBLEM BEFORE?
You are launching Google Chrome.
Which core would you like to use?
Core 1 Core 2
Core 3 Core 4
You are launching a distributed app requiring 500 cores.
Which of your 50,000 cores would you like to use?
Core 1 Core 2 Core 3 Core 4 Core 5
Core 6 Core 8 Core 9 Core 10Core 7
Core 11 Core 13 Core 14 Core 15Core 12
Core 16 Core 18 Core 19 Core 20Core 17
Core 21 Core 23 Core 24 Core 25Core 22
Core 26 Core 28 Core 29 Core 30Core 27
Core 31 Core 33 Core 34 Core 35Core 32
Core 36 Core 38 Core 39 Core 40Core 37
Core 41 Core 43 Core 44 Core 45Core 42
© 2015 Mesosphere, Inc. All Rights Reserved.
MODERN APPS REQUIRE OPERATIONS AT THE DC LEVEL
Traditional App Modern Enterprise App
© 2015 Mesosphere, Inc. All Rights Reserved.
YES, THIS TOO CAN BE SOLVED WITH ABSTRACTION
Scheduler
Executor
Task
Launch
Task
Launch
Task Status
Task Status
Resource
Offer
Executor
Executor
Executor
Scheduler
Executor
Task
Launch
Executor
Executor
Task Status
Scheduler
Executor
Task
Launch
Executor
Executor
Task Status
Distributed System A Distributed System B Datacenter Operating System
Approach
Distributed
Systems
A+B+C+...
Apache Mesos Two-level Scheduling
© 2017 Mesosphere, Inc. All Rights Reserved. 25
MODERN APPS WITH THE DC/OS MODEL
Traditional Approach
Big Data
Analytics
Stateful
Service
DC/OS Model
Container
App
Container
App
CaaS PaaS
● Turnkey install of datacenter-wide services
● Simplified operations & efficiency
● Lowers barriers to experiment with new tech (e.g., Spark)
Container
App
Datacenter-scale operating system
© 2017 Mesosphere, Inc. All Rights Reserved.
BRINGING
MODERN
APPS TO
EVERYONE
● 100% Open source
● No limits
27
THE DC/OS COMMUNITY
YOUR DATACENTER OR CLOUD IS NOW
ONE GIANT COMPUTER
© 2017 Mesosphere, Inc. All Rights Reserved.
DC/OS
SERVICE
INSTALL
© 2017 Mesosphere, Inc. All Rights Reserved.
DC/OS
SERVICE
INSTALL
© 2017 Mesosphere, Inc. All Rights Reserved. 31
DC/OS
SERVICE
OPS
PLATFORM SERVICES
INSTALLED ACROSS
DATACENTER OR CLOUD IN
A SINGLE COMMAND
© 2017 Mesosphere, Inc. All Rights Reserved.
TRY DC/OS
IN UNDER 20
MINUTES
dcos.io
BUSINESS APPS
MESOSPHERE ENTERPRISE DC/OS
CONTAINER
ORCHESTRATION
CI/CD
BIG DATA
ANALYTICS
MESSAGE
QUEUE
DISTRIBUTED
DATABASE
SEARCH
PLATFORM SERVICES
RESOURCE
MANAGEMENT
OPERATIONS SECURITY MULTI-TENANCY
DATACENTER-CLOUD OPERATING SYSTEM
APACHE MESOS
PRIVATE CLOUD AWS INFRASTRUCTURE (EC2) GOOGLE CLOUD MICROSOFT AZURE
MESOSPHERE
ENTERPRISE
DC/OS
Container Orchestration
PaaS
Stateful Data Services
Analytics
Hybrid Cloud
Platform Services Ecosystem
Production Proven at Scale
DATACENTER
MESOSPHERE DC/OS
CUSTOMER IMPACT
Accelerate
Time-to-Market.
Capture value from data insights and
quickly roll out new services to stay
competitive.
Ensure Resilience
and Automation.
Provide always available and secure
business services on hybrid clouds.
Save Money.
Get the most out of infrastructure with
high utilization and cost predictability.
© 2017 Mesosphere, Inc. All Rights Reserved.
“We’re running real time analytics and
visualization of data from connected sensors
Before DC/OS we had no where near the scale
we had today, from thousands of to millions of
events per second.
Next, we’re doing predictive GIS”
Adam Mollenkopf, Real Time Geospatial
Information Systems Capability Lead
© 2017 Mesosphere, Inc. All Rights Reserved.
“We are building a PaaS and services on an
existing OpenStack cloud to enable developers to
build microservices on Docker containers and use
big data technologies...
DC/OS was a force multiplier for us to get these
new services out the door fast...
We transitioned the full enterprise infrastructure
at 2X the speed of previous efforts.”
Tim Pletcher
Engineering Director, Application Services
© 2017 Mesosphere, Inc. All Rights Reserved.
“We came across Mesosphere while planning for the
coming IoT market..
If I build everything in a silo, I have no chance...
With DC/OS, I have one single contiguous cluster
and I can bring my data and store it and I can run all
my applications as well..
and I think that's a huge advantage”
Larry Rau, Director Architecture & Infrastructure,
Verizon Labs
© 2017 Mesosphere, Inc. All Rights Reserved.
McKinsey Quarterly 2015: “The perils of ignoring software development”
Based on assessment of software development projects at 1,300 companies
Top companies bring new
software to market
3x faster
© 2017 Mesosphere, Inc. All Rights Reserved.
Thank You
Questions?
@errbear3
Learn more at mesosphere.com, including our whitepaper:
Modern Enterprise App Operations with DC/OS
Lessons from Running Containers, Microservices, and
Stateful Big Data Services in Production

Journey to the Modern App with Containers, Microservices and Big Data

  • 1.
    © 2017 Mesosphere,Inc. All Rights Reserved. Eryn Muetzel, Product Marketing for Big Data @errbear3 JOURNEY TO THE MODERN APP WITH CONTAINERS, MICROSERVICES AND BIG DATA Datacenter Operating System (DC/OS) The simplest way to build, run, scale and manage modern enterprise apps
  • 2.
    © 2017 Mesosphere,Inc. All Rights Reserved.
  • 3.
    © 2017 Mesosphere,Inc. All Rights Reserved. INDUSTRY TRANSITIONS Customer analytics 2000s Electronic customer records 1980s Online customer engagement 1990s Real-time & predictive customer engagement 2013+
  • 4.
    © 2017 Mesosphere,Inc. All Rights Reserved.
  • 5.
    © 2017 Mesosphere,Inc. All Rights Reserved.
  • 6.
    WE LIVE INTHE ALWAYS CONNECTED ECONOMY. 3.7BMobile Broadband Users 80%New Data is Unstructured 13Connected Devices per Household (N. Amer) 40%Annual Enterprise Data Growth
  • 7.
    IN THE ALWAYSCONNECTED ECONOMY SOFTWARE IS YOUR BUSINESS. Build more apps. Personal and meaningful customer experiences Innovate faster. Capture value from new value streams and actionable data insights Modernize IT Always-available services that are secure and cost effective
  • 8.
    1M trips dailyacross 311 cities, disrupting multiple industries. Personal behavior-based insurance using IoT. Real-time, personalized clinical care. MODERN ENTERPRISE APPS IN ACTION.
  • 9.
    © 2017 Mesosphere,Inc. All Rights Reserved. ARCHITECTURAL SHIFT TRADITIONAL ENTERPRISE APPLICATIONS MODERN ENTERPRISE APPLICATIONS Data Code Latency Users 2+ Billion smartphone users 3+ Billion internet users Enterprise data growth (CAGR): 40%+ Source: KPCB Internet Trends 2017, EMC Digital Universe 2014
  • 10.
    © 2017 Mesosphere,Inc. All Rights Reserved. ARCHITECTING FOR SCALE AT TWITTER BEFORE AFTER
  • 11.
    © 2017 Mesosphere,Inc. All Rights Reserved. Streaming 11 THE MODERN ENTERPRISE APP PLATFORM SERVICES MICROSERVICES STATELESS CONTAINERS ANALYTICS Batch Machine Learning Search DATABASES Time Series SQL / NoSQL
  • 12.
    © 2017 Mesosphere,Inc. All Rights Reserved. Web Apps and Microservices Connected Devices Machine Learning and Simulation Industrial IoT Connected CarFraud DetectionSocial Media DIGITAL TRANSFORMATION REQUIRES MODERN APPS Modern Apps Operational expertise Continuous integration and delivery Big Data Analytics Message Queue Distributed Database Search Future services... Platform Services Container Orchestration How do you develop these skills in your organization?
  • 13.
    MODERN ENTERPRISE APPS REQUIRE NEW CAPABILITIES.Distributed computing expertise. Engineering and operations of secure & highly reliable services at scale DevOps process & culture. Frequent & reliable releases supported by an automated CI/CD toolchain Cloud-native technologies. Containers, microservices, data servies (e.g., Spark, Kafka, Cassandra)
  • 14.
    © 2017 Mesosphere,Inc. All Rights Reserved. MODERN ENTERPRISE APPS Containers Data Services MicroservicesOpen source & Social coding1 2 3 4
  • 15.
    © 2017 Mesosphere,Inc. All Rights Reserved. OPEN SOURCE SOFTWARE PROVIDES LEVERAGE1
  • 16.
    © 2017 Mesosphere,Inc. All Rights Reserved. CONTAINERS SIMPLIFY PACKAGING AND DEPLOYMENT Private Copy Shared App 3 Libraries Guest Operating System Infrastructure App 1 Libs Host Operating System Start / stop time One minute to few seconds Milliseconds Workload density 10 - 100x1x App 2 Libraries Guest Operating System App 1 Libraries Guest Operating System App 2 Libs App 3 Libs Virtual Machines Containers Host Operating System Hypervisor Infrastructure App 4 Libs App 5 Libs 2
  • 17.
    © 2017 Mesosphere,Inc. All Rights Reserved. JUST HOW FAST ARE CONTAINERS? Larry Rau from @Verizon with @flo Launching 50,000 containers in seconds with @mesosphere #DC/OS
  • 18.
    © 2017 Mesosphere,Inc. All Rights Reserved. MICROSERVICES ENABLE FAST AND FREQUENT RELEASES Traditional Architecture Many functions in a single process Siloed teams RESTAPIs Microservices Architecture Cross-functional teams organized around capabilities Scales individually Each element of functionality defined as “microservices” Scales monolithically 3
  • 19.
    © 2017 Mesosphere,Inc. All Rights Reserved. DATA SERVICES PROVIDE CONNECTION AND PERSISTENCE Data Ingestion Response Devices ClientSensors Event Bus Reactive App Database Analytics Use Case Examples ● Anomaly detection ● Personalization ● IoT Applications ● Predictive Analytics ● Machine Learning 4
  • 20.
    © 2017 Mesosphere,Inc. All Rights Reserved. 20 BUT WHAT ABOUT THE INFRASTRUCTURE? CaaSPaaS Container AppContainer App Stateful Service #1 V1 Big Data Analytics #1 V1 Stateful Service #2 V1 Stateful Service #2 V2 PaaS Container App Team A Big Data Analytics #1 V2 Big Data Analytics #2 V1 Team B
  • 21.
    © 2017 Mesosphere,Inc. All Rights Reserved. 21 BUT WHAT ABOUT THE INFRASTRUCTURE? CaaSPaaS Container AppContainer App Stateful Service #1 V1 Big Data Analytics #1 V1 Stateful Service #2 V1 Stateful Service #2 V2 PaaS Container App Team A Big Data Analytics #1 V2 Big Data Analytics #2 V1 Team B ● Extremely low utilization - less than 10%, due to static partitioning ● Manual operations - Weeks to provision, wasted time managing individual machines ● Difficult to experiment with new tech - months to provision environments to evaluate new technologies ● High risk - Failure from manual operations and difficulties maintaining high availability
  • 22.
    © 2015 Mesosphere,Inc. All Rights Reserved. WHERE HAVE WE SEEN THIS PROBLEM BEFORE? You are launching Google Chrome. Which core would you like to use? Core 1 Core 2 Core 3 Core 4 You are launching a distributed app requiring 500 cores. Which of your 50,000 cores would you like to use? Core 1 Core 2 Core 3 Core 4 Core 5 Core 6 Core 8 Core 9 Core 10Core 7 Core 11 Core 13 Core 14 Core 15Core 12 Core 16 Core 18 Core 19 Core 20Core 17 Core 21 Core 23 Core 24 Core 25Core 22 Core 26 Core 28 Core 29 Core 30Core 27 Core 31 Core 33 Core 34 Core 35Core 32 Core 36 Core 38 Core 39 Core 40Core 37 Core 41 Core 43 Core 44 Core 45Core 42
  • 23.
    © 2015 Mesosphere,Inc. All Rights Reserved. MODERN APPS REQUIRE OPERATIONS AT THE DC LEVEL Traditional App Modern Enterprise App
  • 24.
    © 2015 Mesosphere,Inc. All Rights Reserved. YES, THIS TOO CAN BE SOLVED WITH ABSTRACTION Scheduler Executor Task Launch Task Launch Task Status Task Status Resource Offer Executor Executor Executor Scheduler Executor Task Launch Executor Executor Task Status Scheduler Executor Task Launch Executor Executor Task Status Distributed System A Distributed System B Datacenter Operating System Approach Distributed Systems A+B+C+... Apache Mesos Two-level Scheduling
  • 25.
    © 2017 Mesosphere,Inc. All Rights Reserved. 25 MODERN APPS WITH THE DC/OS MODEL Traditional Approach Big Data Analytics Stateful Service DC/OS Model Container App Container App CaaS PaaS ● Turnkey install of datacenter-wide services ● Simplified operations & efficiency ● Lowers barriers to experiment with new tech (e.g., Spark) Container App Datacenter-scale operating system
  • 26.
    © 2017 Mesosphere,Inc. All Rights Reserved. BRINGING MODERN APPS TO EVERYONE ● 100% Open source ● No limits
  • 27.
  • 28.
    YOUR DATACENTER ORCLOUD IS NOW ONE GIANT COMPUTER
  • 29.
    © 2017 Mesosphere,Inc. All Rights Reserved. DC/OS SERVICE INSTALL
  • 30.
    © 2017 Mesosphere,Inc. All Rights Reserved. DC/OS SERVICE INSTALL
  • 31.
    © 2017 Mesosphere,Inc. All Rights Reserved. 31 DC/OS SERVICE OPS
  • 32.
  • 33.
    © 2017 Mesosphere,Inc. All Rights Reserved. TRY DC/OS IN UNDER 20 MINUTES dcos.io
  • 34.
    BUSINESS APPS MESOSPHERE ENTERPRISEDC/OS CONTAINER ORCHESTRATION CI/CD BIG DATA ANALYTICS MESSAGE QUEUE DISTRIBUTED DATABASE SEARCH PLATFORM SERVICES RESOURCE MANAGEMENT OPERATIONS SECURITY MULTI-TENANCY DATACENTER-CLOUD OPERATING SYSTEM APACHE MESOS PRIVATE CLOUD AWS INFRASTRUCTURE (EC2) GOOGLE CLOUD MICROSOFT AZURE MESOSPHERE ENTERPRISE DC/OS
  • 35.
    Container Orchestration PaaS Stateful DataServices Analytics Hybrid Cloud Platform Services Ecosystem Production Proven at Scale DATACENTER
  • 36.
    MESOSPHERE DC/OS CUSTOMER IMPACT Accelerate Time-to-Market. Capturevalue from data insights and quickly roll out new services to stay competitive. Ensure Resilience and Automation. Provide always available and secure business services on hybrid clouds. Save Money. Get the most out of infrastructure with high utilization and cost predictability.
  • 37.
    © 2017 Mesosphere,Inc. All Rights Reserved. “We’re running real time analytics and visualization of data from connected sensors Before DC/OS we had no where near the scale we had today, from thousands of to millions of events per second. Next, we’re doing predictive GIS” Adam Mollenkopf, Real Time Geospatial Information Systems Capability Lead
  • 38.
    © 2017 Mesosphere,Inc. All Rights Reserved. “We are building a PaaS and services on an existing OpenStack cloud to enable developers to build microservices on Docker containers and use big data technologies... DC/OS was a force multiplier for us to get these new services out the door fast... We transitioned the full enterprise infrastructure at 2X the speed of previous efforts.” Tim Pletcher Engineering Director, Application Services
  • 39.
    © 2017 Mesosphere,Inc. All Rights Reserved. “We came across Mesosphere while planning for the coming IoT market.. If I build everything in a silo, I have no chance... With DC/OS, I have one single contiguous cluster and I can bring my data and store it and I can run all my applications as well.. and I think that's a huge advantage” Larry Rau, Director Architecture & Infrastructure, Verizon Labs
  • 40.
    © 2017 Mesosphere,Inc. All Rights Reserved. McKinsey Quarterly 2015: “The perils of ignoring software development” Based on assessment of software development projects at 1,300 companies Top companies bring new software to market 3x faster
  • 41.
    © 2017 Mesosphere,Inc. All Rights Reserved. Thank You Questions? @errbear3 Learn more at mesosphere.com, including our whitepaper: Modern Enterprise App Operations with DC/OS Lessons from Running Containers, Microservices, and Stateful Big Data Services in Production