You’ve installed your Kubernetes cluster on DC/OS — now what? Operating Kubernetes efficiently can be challenging. In the final episode of our Kubernetes series, we will share best practices for operating your DC/OS Kubernetes cluster and maintaining performance. During this presentation, Joerg Schad and Chris Gaun show you how to successfully operate Kubernetes at scale in your environment. During this session, we discuss: 1. How to upgrade DC/OS and Kubernetes with no downtime 2. How DC/OS guards against failure and enables fault domains that are resistant to outages within racks, availability zones, or cloud environments 3. How the monitoring and metrics capabilities on DC/OS improve operational analytics and help you get the most from your cluster 4. How cloud bursting extends your on-prem environment with resources from the cloud to handle spikes in your workload

1. Operating Kubernetes at
Scale
@joerg_schad @gaunetes @dcos

2. Chris Gaun
PMM at Mesosphere /
Kubernetes Expert /
CNCF Ambasador
● Previous to that
Gartner analyst
covering public IaaS
● Kubernetes
community for 3 years

3. Jörg Schad
Technical Community
Lead / Developer
● Core Mesos
developer at
Mesosphere
● Passions are deep
learning, distributed
data systems, and
data analytics

4. Bootcamp: Building Kubernetes-as-a-Service at
Scale, Anywhere
● Episode 1: Building Kubernetes-as-a-Service
at Scale
● Episode 2: Deploying Kubernetes at Scale
with DC/OS
● Episode 3: Kubernetes and Big Data
Services
● Episode 4: Operating Kubernetes at Scale
with DC/OS
● End-to-end components
and best practices
● Automated management
of Kubernetes
● Connecting Kubernetes
to Big Data services
● Delivering an entire
Kubernetes solution

5. 5
Star / Clone Github
1. Go to Kubernetes DC/OS
quickstart
2. Search “DC/OS Kubernetes
Quickstart Github” or
https://github.com/mesospher
e/dcos-kubernetes-quickstart
3. Live demo
https://github.com/dcos/demos
/tree/master/flink-k8s/1.11

6. 6
Sign Up For Slack
1. Slack URL: https://chat.dcos.io/
2. Join #kubernetes channel
3. OSS support / feedback

7. Server Server Server Server Server
Server Server Server Server Server
as-a-Service
Installation
KubernetesOne-Click
20+
MORE
Cloud Native
Services

8. How to operate?

9. © 2018 Mesosphere, Inc. All Rights Reserved. 10
DC/OS
PHYSICAL INFRASTRUCTURE
MICROSERVICES, CONTAINERS, & DEV TOOLS
VIRTUAL MACHINES PUBLIC CLOUDS
DATA SERVICES, MACHINE LEARNING, & AI
Security &
Compliance
Application-Aware
Automation
Multitenancy
Hybrid Cloud
Management
100+
MORE
DatacenterEdge
Datacenter and Cloud as a Single Computing Resource
Powered by Apache Mesos
20+
MORE

10. • Brings “as-a-Service”
automation to any application
technology on any
infrastructure
• Organizations Run All Types of
Container Management as-a-
Service Using Mesos:
"(Netflix) launches up to 500,000
containers and 200,000
clusters/day"
-Netflix OSS, on using Titus container
management ontop Mesos

11. © 2017 Mesosphere, Inc. All Rights Reserved. 12
Day 2
Challenges

12. © 2018 Mesosphere, Inc. All Rights Reserved. 13
Day 2 Operations
Maintenance
- Cluster Upgrades
- Cluster Resizing
- Capacity Planning
- User & Package Management
- Networking Policies
- Auditing
- Backups & Disaster Recovery
Troubleshooting
- Debugging
- Services
- System
- Tracing
- Chaos engineering

13. © 2018 Mesosphere, Inc. All Rights Reserved. 14
DAY 2 OPERATIONS
Metrics and Monitoring
- Collecting metrics
- Downstream processing
○ Alerting
○ Dashboards
○ Storage (long-term retention)
Logging
- Scopes
- Local vs. centralized
- Security considerations

14. ServerServer Server
as-a-Service
Automated Self Healing
Server Server Server Server Server
Server Server Server
KubernetesZero
Touch
20+
MORE
Cloud Native
Services

15. © 2018 Mesosphere, Inc. All Rights Reserved.
Two-level Scheduling
1. Agents advertise resources to Master
2. Master offers resources to Framework
3. Framework rejects / uses resources
4. Agent reports task status to Master
16
MESOS ARCHITECTURE
Mesos
Master
Mesos
Master
Mesos
Master
Mesos AgentMesos Agent Service
Cassandra
Executor
Cassandra
Task
Kubernetes
Scheduler
Spark
Executor
Spark
Task
Mesos AgentMesos Agent Service
Docker
Executor
Docker
Task
K8s Executor
Kubelet
Task
Marathon
Scheduler
Kafka
Scheduler

16. © 2018 Mesosphere, Inc. All Rights Reserved.
Plans
dcos hdfs --name=hdfs plan status deploy
deploy (serial strategy) (COMPLETE)
├─ journal (serial strategy) (COMPLETE)
│ ├─ journal-0:[node] (COMPLETE)
│ ├─ journal-1:[node] (COMPLETE)
│ └─ journal-2:[node] (COMPLETE)
├─ name (serial strategy) (COMPLETE)
│ ├─ name-0:[node, zkfc] (COMPLETE)
│ └─ name-1:[node, zkfc] (COMPLETE)
└─ data (serial strategy) (COMPLETE)
├─ data-0:[node] (COMPLETE)
├─ data-1:[node] (COMPLETE)
└─ data-2:[node] (COMPLETE)

17. © 2018 Mesosphere, Inc. All Rights Reserved. 18
TROUBLES
HOOTING

18. © 2017 Mesosphere, Inc. All Rights Reserved. 19
Debuggin
g
Dashboar
d

19. © 2018 Mesosphere, Inc. All Rights Reserved. 20
Demo
Failure Handling Configuration Updates

20. Download Now
https://mesosphere.com/resource/category/ebook/

21. © 2018 Mesosphere, Inc. All Rights Reserved. 22
THANK YOU!
ANY
QUESTION
S?
@dcos
users@dcos.io
/groups/8295652
/dcos
/dcos/examples
/dcos/demos
chat.dcos.io
https://github.com/mesosphere/dcos-kubernetes-quickstart
https://mesosphere.com/blog/another-kubernetes-service/

22. © 2017 Mesosphere, Inc. All Rights Reserved. 23
MapReduce is
crunching Data
Ancient
Times...

23. © 2016 Mesosphere, Inc. All Rights Reserved. 24
But then business
demanded
FAST DATA
We need to turn faster!
Today...

24. © 2016 Mesosphere, Inc. All Rights Reserved. 25
Fast Data
Batch Event ProcessingMicro-Batch
Days Hours Minutes Seconds Microseconds
Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics
Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations

25. © 2016 Mesosphere, Inc. All Rights Reserved. 26
The SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
Apache Kafka
STORE
Apache Spark
ANALYZE
Apache Cassandra
ACT
Akka
Ingest millions of events
per second
Distributed & highly
scalable database
Real-time and batch
process data
Visualize data and build
data driven applications
Apache Mesos/ DC/OS
Sensors
Devices
Clients

27. © 2016 Mesosphere, Inc. All Rights Reserved. 28
The SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
Apache Kafka
STORE
Apache Flink
ANALYZE
Apache Cassandra
ACT
Akka
Ingest millions of events
per second
Distributed & highly
scalable database
Real-time and batch
process data
Visualize data and build
data driven applications
Apache Mesos/ DC/OS
Sensors
Devices
Clients

28. © 2018 Mesosphere, Inc. All Rights Reserved. 29
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Kubernetes
Jenkins
Kafka
Spark
Cassandra

29. 3 AM
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Kubernetes
Jenkins
Kafka
Spark
Cassandra

30. DC1
DC2
DC3
3 AM

31. © 2017 Mesosphere, Inc. All Rights Reserved. 32

32. © 2018 Mesosphere, Inc. All Rights Reserved. 33
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
Kubernetes
Jenkins
Kafka
Spark
Cassandra

33. © 2018 Mesosphere, Inc. All Rights Reserved. 34

34. © 2018 Mesosphere, Inc. All Rights Reserved.
Distributed Systems are ...
HDFS Scheduler

35. Server Server Server Server Server
Server Server Server Server Server
as-a-Service
Installation
KubernetesOne-Click
20+
MORE
Cloud Native
Services

36. © 2018 Mesosphere, Inc. All Rights Reserved. 37
Why {Spark, HDFS, ..} on K8s today?
Kelsey Hightower
Kubernetes Thought
Leader
Ranked #1 K8s Influencer
Staff Developer
Advocate
PM & Chief Advocate
Today Big Data on K8s is more DIY
Top
Kubernetes
Advocate

37. © 2018 Mesosphere, Inc. All Rights Reserved. 38
SMACK Stack
Generator Display
1. Financial data created
by generator
2. Written to
Kafka topics
3. Kafka Topics
consumed by Spark or
Flink
4. Results written back into Kafka
stream (another topic)
7. Results displayed

38. 39
Star / Clone Github
1. Go to Kubernetes DC/OS
quickstart
2. Search “DC/OS Kubernetes
Quickstart Github” or
https://github.com/mesospher
e/dcos-kubernetes-quickstart
3. Live demo
https://github.com/dcos/demos
/tree/master/flink-k8s/1.11