Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Episode 3: Kubernetes and Big Data Services

421 views

Published on

Kubernetes is great for deploying stateless containers, but what about the big data ecosystem? Episode 3 of our Kubernetes series covers how DC/OS enables you to connect your Kubernetes-based applications to co-located big data services.

Slides cover:

1. Why persistence is challenging in distributed architectures
How DC/OS helps you take advantage of the services available in the big data ecosystem
2. How to connect Kubernetes to your data services through networking
3. How Apache Flink and Apache Spark work with Kubernetes to enable real-time data processing on DC/OS

Published in: Technology
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Episode 3: Kubernetes and Big Data Services

  1. 1. Kubernetes and Big Data Services @joerg_schad @gaunetes @dcos
  2. 2. Chris Gaun PMM at Mesosphere / Kubernetes Expert / CNCF Ambasador ● Previous to that Gartner analyst covering public IaaS ● Kubernetes community for 3 years
  3. 3. © 2018 Mesosphere, Inc. All Rights Reserved. Mesosphere DC/OS at KubeCon EU ● Mesosphere - Platinum Sponsor ● Many presentation: container storage, ML, HDFS ● Demoing smart city application 3
  4. 4. Jörg Schad Technical Community Lead / Developer ● Core Mesos developer at Mesosphere ● Passions are deep learning, distributed data systems, and data analytics
  5. 5. Bootcamp: Building Kubernetes-as-a-Service at Scale, Anywhere ● Episode 1: Building Kubernetes-as-a-Service at Scale ● Episode 2: Deploying Kubernetes at Scale with DC/OS ● Episode 3: Kubernetes and Big Data Services ● Episode 4: Operating Kubernetes at Scale with DC/OS ● End-to-end components and best practices ● Automated management of Kubernetes ● Connecting Kubernetes to Big Data services ● Delivering an entire Kubernetes solution
  6. 6. 6 Star / Clone Github 1. Go to Kubernetes DC/OS quickstart 2. Search “DC/OS Kubernetes Quickstart Github” or https://github.com/mesospher e/dcos-kubernetes-quickstart 3. Live demo https://github.com/dcos/demos /tree/master/flink-k8s/1.11
  7. 7. 7 Sign Up For Slack 1. Slack URL: https://chat.dcos.io/ 2. Join #kubernetes channel 3. OSS support / feedback
  8. 8. © 2017 Mesosphere, Inc. All Rights Reserved. 8 MapReduce is crunching Data Ancient Times...
  9. 9. © 2016 Mesosphere, Inc. All Rights Reserved. 9 But then business demanded FAST DATA We need to turn faster! Today...
  10. 10. © 2016 Mesosphere, Inc. All Rights Reserved. 10 Fast Data Batch Event ProcessingMicro-Batch Days Hours Minutes Seconds Microseconds Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations
  11. 11. © 2016 Mesosphere, Inc. All Rights Reserved. 11 The SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Apache Mesos/ DC/OS Sensors Devices Clients
  12. 12. © 2016 Mesosphere, Inc. All Rights Reserved. 13 The SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Flink ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Apache Mesos/ DC/OS Sensors Devices Clients
  13. 13. © 2017 Mesosphere, Inc. All Rights Reserved. 14 Challenges
  14. 14. © 2018 Mesosphere, Inc. All Rights Reserved. 15 Datacenter Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Spark Cassandra
  15. 15. 3 AM Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Spark Cassandra
  16. 16. DC1 DC2 DC3 3 AM
  17. 17. © 2017 Mesosphere, Inc. All Rights Reserved. 18
  18. 18. © 2018 Mesosphere, Inc. All Rights Reserved. 19 Datacenter Typical Datacenter siloed, over-provisioned servers, low utilization Mesos/ DC/OS automated schedulers, workload multiplexing onto the same machines Kubernetes Jenkins Kafka Spark Cassandra
  19. 19. • Brings “as-a-Service” automation to any application technology on any infrastructure • Organizations Run All Types of Container Management as-a- Service Using Mesos: "(Netflix) launches up to 500,000 containers and 200,000 clusters/day" -Netflix OSS, on using Titus container management ontop Mesos
  20. 20. © 2018 Mesosphere, Inc. All Rights Reserved. 21 DC/OS PHYSICAL INFRASTRUCTURE MICROSERVICES, CONTAINERS, & DEV TOOLS VIRTUAL MACHINES PUBLIC CLOUDS DATA SERVICES, MACHINE LEARNING, & AI Security & Compliance Application-Aware Automation Multitenancy Hybrid Cloud Management 100+ MORE DatacenterEdge Datacenter and Cloud as a Single Computing Resource Powered by Apache Mesos 20+ MORE
  21. 21. © 2018 Mesosphere, Inc. All Rights Reserved. 22
  22. 22. © 2018 Mesosphere, Inc. All Rights Reserved. Two-level Scheduling 1. Agents advertise resources to Master 2. Master offers resources to Framework 3. Framework rejects / uses resources 4. Agent reports task status to Master 23 MESOS ARCHITECTURE Mesos Master Mesos Master Mesos Master Mesos AgentMesos Agent Service Cassandra Executor Cassandra Task Kubernetes Scheduler Spark Executor Spark Task Mesos AgentMesos Agent Service Docker Executor Docker Task K8s Executor Kubelet Task Marathon Scheduler Kafka Scheduler
  23. 23. © 2018 Mesosphere, Inc. All Rights Reserved. Distributed Systems are ... HDFS Scheduler
  24. 24. © 2018 Mesosphere, Inc. All Rights Reserved. Plans dcos hdfs --name=hdfs plan status deploy deploy (serial strategy) (COMPLETE) ├─ journal (serial strategy) (COMPLETE) │ ├─ journal-0:[node] (COMPLETE) │ ├─ journal-1:[node] (COMPLETE) │ └─ journal-2:[node] (COMPLETE) ├─ name (serial strategy) (COMPLETE) │ ├─ name-0:[node, zkfc] (COMPLETE) │ └─ name-1:[node, zkfc] (COMPLETE) └─ data (serial strategy) (COMPLETE) ├─ data-0:[node] (COMPLETE) ├─ data-1:[node] (COMPLETE) └─ data-2:[node] (COMPLETE)
  25. 25. Server Server Server Server Server Server Server Server Server Server as-a-Service Installation KubernetesOne-Click 20+ MORE Cloud Native Services
  26. 26. ServerServer Server as-a-Service Automated Self Healing Server Server Server Server Server Server Server Server KubernetesZero Touch 20+ MORE Cloud Native Services
  27. 27. © 2018 Mesosphere, Inc. All Rights Reserved. 28 Why {Spark, HDFS, ..} on K8s today? Kelsey Hightower Kubernetes Thought Leader Ranked #1 K8s Influencer Staff Developer Advocate PM & Chief Advocate Today Big Data on K8s is more DIY Top Kubernetes Advocate
  28. 28. © 2018 Mesosphere, Inc. All Rights Reserved. 29 SMACK Stack Generator Display 1. Financial data created by generator 2. Written to Kafka topics 3. Kafka Topics consumed by Spark or Flink 4. Results written back into Kafka stream (another topic) 7. Results displayed
  29. 29. 30 Star / Clone Github 1. Go to Kubernetes DC/OS quickstart 2. Search “DC/OS Kubernetes Quickstart Github” or https://github.com/mesospher e/dcos-kubernetes-quickstart 3. Live demo https://github.com/dcos/demos /tree/master/flink-k8s/1.11
  30. 30. © 2018 Mesosphere, Inc. All Rights Reserved. 31 SMACK Stack Generator Display 1. Financial data created by generator 2. Written to Kafka topics 4. Results written back into Kafka stream (another topic) 5. Results displayed 3. Kafka Topics consumed and analyzed by Flink Kubernetes Cluster (running on top of DC/OS)
  31. 31. Download Now https://mesosphere.com/resource/category/ebook/
  32. 32. © 2018 Mesosphere, Inc. All Rights Reserved. 33 THANK YOU! ANY QUESTION S? @dcos users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io https://github.com/mesosphere/dcos-kubernetes-quickstart https://mesosphere.com/blog/another-kubernetes-service/

×