Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Journey to Modern Apps with Containers, Microservices and Big Data

326 views

Published on

2016-10-04 Reactive Summit - Mesosphere Keynote

Enterprises hear about the promise of application containers, but realizing meaningful business results from containers requires more than abandoning virtual machines. In order to implement containers correctly, businesses must consider the operational implications, as well as the new types of applications they want to build using microservices. In this session, Ed Hsu, Vice President of Enterprise DC/OS at Mesosphere, discusses how to capitalize on new opportunities that can accelerate your IT modernization initiatives.

Published in: Technology
  • Be the first to comment

A Journey to Modern Apps with Containers, Microservices and Big Data

  1. 1. © 2016 Mesosphere, Inc. All Rights Reserved. Edward Hsu, VP of Enterprise DC/OS @edwardihsu JOURNEY TO THE MODERN APP WITH CONTAINERS, MICROSERVICES AND BIG DATA Datacenter Operating System (DC/OS) The simplest way to build, run, scale and manage modern enterprise apps
  2. 2. © 2016 Mesosphere, Inc. All Rights Reserved.
  3. 3. © 2016 Mesosphere, Inc. All Rights Reserved. INDUSTRY TRANSITIONS Customer analytics 2000s Electronic customer records 1980s Online customer engagement 1990s Real-time & predictive customer engagement 2013+
  4. 4. © 2016 Mesosphere, Inc. All Rights Reserved.
  5. 5. © 2016 Mesosphere, Inc. All Rights Reserved.
  6. 6. © 2016 Mesosphere, Inc. All Rights Reserved. MAYO CLINIC REVOLUTIONIZING HEALTHCARE Individualized care recommendations based on 5 million patient records Key technologies: ● Real-time data ingestion ● Distributed search engine ● Large-scale processing ● Distributed database
  7. 7. © 2016 Mesosphere, Inc. All Rights Reserved. UBER DISRUPTING MULTIPLE INDUSTRIES Operations in over 300 cities around the world Key technologies: ● Distributed logging ● Large-scale data processing ● Distributed datastore
  8. 8. © 2016 Mesosphere, Inc. All Rights Reserved. ARCHITECTURAL SHIFT TRADITIONAL ENTERPRISE APPLICATIONS MODERN ENTERPRISE APPLICATIONS Data Code Latency Users 2+ Billion smartphone users 3+ Billion internet users Enterprise data growth (CAGR): 40%+ Source: KPCB Internet Trends 2016, EMC Digital Universe 2014
  9. 9. © 2016 Mesosphere, Inc. All Rights Reserved. ARCHITECTING FOR SCALE AT TWITTER BEFORE AFTER
  10. 10. © 2016 Mesosphere, Inc. All Rights Reserved. Streaming 10 THE MODERN ENTERPRISE APP BIG DATA SERVICESMICROSERVICES STATELESS CONTAINERS ANALYTICS Batch Machine Learning Search DATABASES Time Series SQL / NoSQL
  11. 11. © 2016 Mesosphere, Inc. All Rights Reserved. MODERN ENTERPRISE APPS Containers Data Services MicroservicesOpen source & Social coding1 2 3 4
  12. 12. © 2016 Mesosphere, Inc. All Rights Reserved. OPEN SOURCE SOFTWARE PROVIDES LEVERAGE1
  13. 13. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINERS SIMPLIFY PACKAGING AND DEPLOYMENT Private Copy Shared App 3 Libraries Guest Operating System Infrastructure App 1 Libs Host Operating System Start / stop time One minute to few seconds Milliseconds Workload density 10 - 100x1x App 2 Libraries Guest Operating System App 1 Libraries Guest Operating System App 2 Libs App 3 Libs Virtual Machines Containers Host Operating System Hypervisor Infrastructure App 4 Libs App 5 Libs 2
  14. 14. © 2016 Mesosphere, Inc. All Rights Reserved. JUST HOW FAST ARE CONTAINERS? Larry Rau from @Verizon with @flo Launching 50,000 containers in seconds with @mesosphere #DC/OS
  15. 15. © 2016 Mesosphere, Inc. All Rights Reserved. MICROSERVICES ENABLE FAST AND FREQUENT RELEASES Traditional Architecture Many functions in a single process Siloed teams RESTAPIs Microservices Architecture Cross-functional teams organized around capabilities Scales individually Each element of functionality defined as “microservices” Scales monolithically 3
  16. 16. © 2016 Mesosphere, Inc. All Rights Reserved. DATA SERVICES PROVIDE CONNECTION AND PERSISTENCE Data Ingestion Response Devices ClientSensors Event Bus Reactive App Database Analytics Use Case Examples ● Anomaly detection ● Personalization ● IoT Applications ● Predictive Analytics ● Machine Learning 4
  17. 17. © 2016 Mesosphere, Inc. All Rights Reserved. 17 BUT WHAT ABOUT THE INFRASTRUCTURE? CaaSPaaS Container AppContainer App Stateful Service #1 V1 Big Data Analytics #1 V1 Stateful Service #2 V1 Stateful Service #2 V2 PaaS Container App Team A Big Data Analytics #1 V2 Big Data Analytics #2 V1 Team B
  18. 18. © 2016 Mesosphere, Inc. All Rights Reserved. 18 BUT WHAT ABOUT THE INFRASTRUCTURE? CaaSPaaS Container AppContainer App Stateful Service #1 V1 Big Data Analytics #1 V1 Stateful Service #2 V1 Stateful Service #2 V2 PaaS Container App Team A Big Data Analytics #1 V2 Big Data Analytics #2 V1 Team B ● Extremely low utilization - less than 10%, due to static partitioning ● Manual operations - Weeks to provision, wasted time managing individual machines ● Difficult to experiment with new tech - months to provision environments to evaluate new technologies ● High risk - Failure from manual operations and difficulties maintaining high availability
  19. 19. © 2015 Mesosphere, Inc. All Rights Reserved. WHERE HAVE WE SEEN THIS PROBLEM BEFORE? You are launching Google Chrome. Which core would you like to use? Core 1 Core 2 Core 3 Core 4 You are launching a distributed app requiring 500 cores. Which of your 50,000 cores would you like to use? Core 1 Core 2 Core 3 Core 4 Core 5 Core 6 Core 8 Core 9 Core 10Core 7 Core 11 Core 13 Core 14 Core 15Core 12 Core 16 Core 18 Core 19 Core 20Core 17 Core 21 Core 23 Core 24 Core 25Core 22 Core 26 Core 28 Core 29 Core 30Core 27 Core 31 Core 33 Core 34 Core 35Core 32 Core 36 Core 38 Core 39 Core 40Core 37 Core 41 Core 43 Core 44 Core 45Core 42
  20. 20. © 2015 Mesosphere, Inc. All Rights Reserved. MODERN ENTERPRISE APPS REQUIRE WE OPERATE AT THE LOGICAL DATACENTER LEVEL Traditional App Modern Enterprise App
  21. 21. © 2015 Mesosphere, Inc. All Rights Reserved. YES, THIS TOO CAN BE SOLVED WITH ABSTRACTION Scheduler Executor Task Launch Task Launch Task Status Task Status Resource Offer Executor Executor Executor Scheduler Executor Task Launch Executor Executor Task Status Scheduler Executor Task Launch Executor Executor Task Status Distributed System A Distributed System B Datacenter Operating System Approach Distributed Systems A+B+C+... Apache Mesos Two-level Scheduling
  22. 22. © 2015 Mesosphere, Inc. All Rights Reserved. Datacenter Operating System (DC/OS) DC/OS ... Agent 1 Agent 2 Agent 3 Agent n Server VM Cloud VM ... Modern App Components FROM SERVER TO DATACENTER-SCALE Server Operating System OS (e.g., Windows, Linux) ... Core 0 Core 1 Core 2 Core n Server Process
  23. 23. © 2016 Mesosphere, Inc. All Rights Reserved. 23 MODERN APPS WITH THE DC/OS MODEL Traditional Approach Big Data Analytics Stateful Service DC/OS Model Container App Container App CaaS PaaS ● Turnkey install of datacenter-wide services ● Simplified operations & efficiency ● Lowers barriers to experiment with new tech (e.g., Spark) Container App Datacenter-scale operating system
  24. 24. © 2016 Mesosphere, Inc. All Rights Reserved. BRINGING MODERN APPS TO EVERYONE ● 100% Open source ● No limits
  25. 25. 25 THE DC/OS COMMUNITY
  26. 26. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS Operations
  27. 27. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS SERVICE INSTALL: SPARK
  28. 28. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS SERVICE INSTALL: KAFKA
  29. 29. © 2016 Mesosphere, Inc. All Rights Reserved. 29 DC/OS SERVICE UPDATE & CONFIG: KAFKA
  30. 30. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS UNIVERSE: STORE FOR APP SERVICES DC/OS Anyone can publish on DC/OS Universe app store - partners & OSS contributors Popular services include Spark, Cassandra, Jenkins, and Kafka Services include distributed systems that run elastically across the datacenter Install these services with a single command Spark Jenkins Riak DataStax Enterprise Confluent Kafka ArangoDB GitLab Cassandra JFrog Artifactory Elasticsearch MariaDB Storm HDFS Zeppelin MemSQL Over 40 Services Made For DC/OS
  31. 31. © 2016 Mesosphere, Inc. All Rights Reserved. TRY DC/OS IN UNDER 20 MINUTES dcos.io
  32. 32. © 2016 Mesosphere, Inc. All Rights Reserved. MESOSPHERE ENTERPRISE DC/OS
  33. 33. © 2016 Mesosphere, Inc. All Rights Reserved. “We’re running real time analytics and visualization of data from connected sensors Before DC/OS we had no where near the scale we had today, from thousands of to millions of events per second. Next, we’re doing predictive GIS” Adam Mollenkopf, Real Time Geospatial Information Systems Capability Lead
  34. 34. © 2016 Mesosphere, Inc. All Rights Reserved. “We are building a PaaS and services on an existing OpenStack cloud to enable developers to build microservices on Docker containers and use big data technologies... DC/OS was a force multiplier for us to get these new services out the door fast... We transitioned the full enterprise infrastructure at 2X the speed of previous efforts.” Tim Pletcher Engineering Director, Application Services
  35. 35. © 2016 Mesosphere, Inc. All Rights Reserved. “We came across Mesosphere while planning for the coming IoT market.. If I build everything in a silo, I have no chance... With DC/OS, I have one single contiguous cluster and I can bring my data and store it and I can run all my applications as well.. and I think that's a huge advantage” Larry Rau, Director Architecture & Infrastructure, Verizon Labs
  36. 36. © 2016 Mesosphere, Inc. All Rights Reserved. SPEED MATTERS McKinsey Quarterly 2015: “The perils of ignoring software development” Based on assessment of software development projects at 1,300 companies Top quartile companies bring new software features to market 3x faster than bottom quartile
  37. 37. © 2016 Mesosphere, Inc. All Rights Reserved. Thank You Questions? @edwardihsu Learn more at mesosphere.com, including our whitepaper: Modern Enterprise App Operations with DC/OS Lessons from Running Containers, Microservices, and Stateful Big Data Services in Production

×