SlideShare a Scribd company logo
1 of 13
Download to read offline
Robust Containers
Eric Brewer
VP, Infrastructure
DockerCon Keynote, June 10, 2014
@eric_brewer #dockercon
1) Application-centric, not machine-centric view
It is easier, more natural, and more productive
Why we Love Containers
Over 2B containers launched per week
(even our VMs run inside containers)
We evolved here over the last decade…
but Docker made it exciting and much easier to use (thanks!)
2) Essentially the way Google works internally:
Signed static bundles + Linux containers (resolve dependencies up front)
Containers interfere with each other
• Unimportant things break important things
• We want fair use among equally important things
Solution: resource & performance isolation
Series of open-source solutions:
2005: cpusets + “fake” NUMA to partition cores, memory
2006: cgroups for general task hierarchies
2009: bandwidth fair use, QoS levels
2010: memcg for better memory accounting, enforcement
Status: isolation works well in practice (if you use these tools)
First Problem: Unpredictable Interference
Second Problem: Low Utilization
Tier 1: Live services (e.g. search engine)
• Provision for peak load (2-10x higher than average)
• High priority, always get resources when needed
Tier 2: Batch jobs (e.g. MapReduce)
• Run in the leftovers, never displace Tier 1
• Lots of capacity — rarely at peak load
If you partition resources, utilization goes down…
Solution: controlled use of slack resources (free $$)
Status: Our OSS container solutions support this well
Note: Google does not overcommit customer VMs — you get the whole VM all the time
Third Problem: Hard to Enforce Isolation
Bad way: control loop (see LPC 2011)
• Read stats, verify allocation, tune knobs, repeat
• Slow response time, fragile
Right way:
• Direct enforcement in the kernel
• Many patches to make this happen… (e.g. memcg)
Status: enforcement now mostly in the kernel
• Caches, memory bandwidth can still cause interference
• Challenges getting these changes accepted upstream
• Meta control loop: detect interference and migrate tasks (see CPI2)
“Let Me Contain That For You” (LMCTFY = “L-M-C-T-
fee”)
You want this, but didn’t know it
• Declarative allocation, prioritization of resources
• Enforces resource isolation, with multiple hierarchies
• Many resources: CPU, memory, bandwidth, latency, disk I/O, …
• Enables better utilization
• Stable API, as kernel mechanisms continue to evolve
• Released as OSS in 2013 (see LPC 2013)
OSS containers based on Docker are a core foundation for the future
• Many contributors over the decade: SGI, LXC, RedHat, Parallels, Docker, …
• We want to move LMCTFY functionality into Docker’s libcontainer
• Released for Docker Hackathon: cAdvisor for container stats & alerts (written in Go)
Pods (or how we really use containers)
We actually use groups of nested containers = pods
• Use LMCTFY for nesting, isolation & utilization
• Many things implemented as helpers:
• Logging and log rotation
• Content management system + webserver
Pod attributes:
• Deployed together (in a parent container!)
• Shared local volumes
• Individual IP address (even if multiple pods per VM)
• Ensures clean port allocation
OK, we don’t use a single IP per pod, but we should have…
Without this, need to track/distribute port allocations, since they must be late bound...
Kubernetes “koo ber NAY tace” — Greek for “helmsman”
New OSS release: orchestrating replicated pods across multiple nodes
Craig McLuckie, Brendan Burns to cover at 2pm today
Master:
• Manages worker pods dynamically
• Uses etcd to track desired configuration API Server
Replica
Controller
etcd
k Workers:
• Replicated Docker image
• Parameterized: arguments passed in via
environment variables
• Shared view of load-balanced services
Kubelet
Service
Proxy
Docker
Concept 1: Labels and Services
Service = load-balanced replica set
• Pod labels ⇒ the services they implement
• Pods access services via localhost:<port>
• (Local) proxy sends traffic to member of set
• Ports are the service “names”
{
"id": "redisslave",
"port": 10001,
"labels": {
"name": "redisslave"
}
}
Service Definition (JSON)
"labels": {
"name": "redisslave"
}
Partial pod definition (JSON)
Pods have labels
Many overlapping sets of labels:
stage: production name: redis
zone: west version: 2.6
Replica set = a group of pods with the same labels
The set is defined by a query (not a static list)
(because entropy happens)
Having an explicit desired state is a good idea!
Otherwise can’t tell if the desire changed, or the actual state changes
Concept 2: The Reconciler Model
Key idea: Declare the desired state
Loop { // the reconciler loop, run by master
• Query the actual state of the system
• Compare with desired state
• Implement corrections (if any) // reconcile reality with desired state
}
In Kubernetes
desiredState: if we lose a replica for some reason, add one
replicas: 2
Robust Containers
Docker (used well) ⇒ clean, repeatable images
Single Node (pods):
• Allocate ports per pod (conflict free!)
• Attach data-only containers to the pod (as volumes)
(clean sharing of data)
• “Parameterized containers” using environment variables
Multi-Node:
• Labels for time-varying overlapping sets
• Services are load-balanced groups of replicated pods
• The Reconciler Model recovers from changes (expected or not)
(actually used at worker level and master level)
Containers are the Path to “Cloud Native”
Pods as a building block
• Clean port namespace
• Shared volumes
• Isolation, prioritization, tools for utilization
• Auto restart (don’t run supervisord k times)
• Liveness probes, stats for load balancing
• sshd in environment (not in your container)
Application-level cloud events per container or pod
• Start, stop, restart
• Notification of migration, resizing, new shards, ...
• Resource alerts, OOM management
Services and labels
Reconciliation
Summary
We are standardizing around the Docker container image
• Pushing for usable, scalable, open containers
• Isolation, nesting, utilization, enforcement
• Moving to Go to simplify integration (and because we like it)
Thanks to Docker…
for making containers lightweight, easy to use, and exciting!
We look forward to creating a great robust space together
News today:
• Kubernetes: see Craig & Brendan at 2pm today
• Docker on GAE: see Ekaterina Volkova at 2:50pm today
• cAdvisor: stats & alerts for containers

More Related Content

What's hot

Supporting bioinformatics applications with hybrid multi-cloud services
Supporting bioinformatics applications with hybrid multi-cloud servicesSupporting bioinformatics applications with hybrid multi-cloud services
Supporting bioinformatics applications with hybrid multi-cloud servicesAhmed Abdullah
 
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.Opcito Technologies
 
The (mutable) config management showdown
The (mutable) config management showdownThe (mutable) config management showdown
The (mutable) config management showdownBob Killen
 
Open stack and k8s(v4)
Open stack and k8s(v4)Open stack and k8s(v4)
Open stack and k8s(v4)H K Yoon
 
Ansible, integration testing, and you.
Ansible, integration testing, and you.Ansible, integration testing, and you.
Ansible, integration testing, and you.Bob Killen
 
Optimising nfv service chains on open stack using docker
Optimising nfv service chains on open stack using dockerOptimising nfv service chains on open stack using docker
Optimising nfv service chains on open stack using dockerAnanth Padmanabhan
 
The Case For Docker In Multi-Cloud Enabled Bioinformatics Applications
The Case For Docker In Multi-Cloud Enabled Bioinformatics ApplicationsThe Case For Docker In Multi-Cloud Enabled Bioinformatics Applications
The Case For Docker In Multi-Cloud Enabled Bioinformatics ApplicationsAhmed Abdullah
 
Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...
Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...
Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...Quinton Hoole
 
Kubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKevin Lynch
 
Kubernetes - A Comprehensive Overview
Kubernetes - A Comprehensive OverviewKubernetes - A Comprehensive Overview
Kubernetes - A Comprehensive OverviewBob Killen
 
Deploying Microservice on Docker
Deploying Microservice on DockerDeploying Microservice on Docker
Deploying Microservice on DockerKnoldus Inc.
 
Dockerizing OpenStack for High Availability
Dockerizing OpenStack for High AvailabilityDockerizing OpenStack for High Availability
Dockerizing OpenStack for High AvailabilityDaniel Krook
 
Deep Dive into Kubernetes - Part 2
Deep Dive into Kubernetes - Part 2Deep Dive into Kubernetes - Part 2
Deep Dive into Kubernetes - Part 2Imesh Gunaratne
 
Deploying WSO2 Middleware on Kubernetes
Deploying WSO2 Middleware on KubernetesDeploying WSO2 Middleware on Kubernetes
Deploying WSO2 Middleware on KubernetesWSO2
 
Cloud nativeday duane (final)
Cloud nativeday duane (final)Cloud nativeday duane (final)
Cloud nativeday duane (final)DuaneDeC7
 
Container Orchestration Integration: OpenStack Kuryr
Container Orchestration Integration: OpenStack KuryrContainer Orchestration Integration: OpenStack Kuryr
Container Orchestration Integration: OpenStack KuryrTaku Fukushima
 
Containers kuberenetes
Containers kuberenetesContainers kuberenetes
Containers kuberenetescsegayan
 
Live migrating a container: pros, cons and gotchas
Live migrating a container: pros, cons and gotchasLive migrating a container: pros, cons and gotchas
Live migrating a container: pros, cons and gotchasDocker, Inc.
 

What's hot (20)

Supporting bioinformatics applications with hybrid multi-cloud services
Supporting bioinformatics applications with hybrid multi-cloud servicesSupporting bioinformatics applications with hybrid multi-cloud services
Supporting bioinformatics applications with hybrid multi-cloud services
 
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.
 
The (mutable) config management showdown
The (mutable) config management showdownThe (mutable) config management showdown
The (mutable) config management showdown
 
rtnetlink
rtnetlinkrtnetlink
rtnetlink
 
Open stack and k8s(v4)
Open stack and k8s(v4)Open stack and k8s(v4)
Open stack and k8s(v4)
 
Ansible, integration testing, and you.
Ansible, integration testing, and you.Ansible, integration testing, and you.
Ansible, integration testing, and you.
 
Optimising nfv service chains on open stack using docker
Optimising nfv service chains on open stack using dockerOptimising nfv service chains on open stack using docker
Optimising nfv service chains on open stack using docker
 
The Case For Docker In Multi-Cloud Enabled Bioinformatics Applications
The Case For Docker In Multi-Cloud Enabled Bioinformatics ApplicationsThe Case For Docker In Multi-Cloud Enabled Bioinformatics Applications
The Case For Docker In Multi-Cloud Enabled Bioinformatics Applications
 
Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...
Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...
Federation of Kubernetes Clusters (Ubernetes) KubeCon 2015 slides - Quinton H...
 
Kubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the Datacenter
 
Kubernetes - A Comprehensive Overview
Kubernetes - A Comprehensive OverviewKubernetes - A Comprehensive Overview
Kubernetes - A Comprehensive Overview
 
Multi tenancy for docker
Multi tenancy for dockerMulti tenancy for docker
Multi tenancy for docker
 
Deploying Microservice on Docker
Deploying Microservice on DockerDeploying Microservice on Docker
Deploying Microservice on Docker
 
Dockerizing OpenStack for High Availability
Dockerizing OpenStack for High AvailabilityDockerizing OpenStack for High Availability
Dockerizing OpenStack for High Availability
 
Deep Dive into Kubernetes - Part 2
Deep Dive into Kubernetes - Part 2Deep Dive into Kubernetes - Part 2
Deep Dive into Kubernetes - Part 2
 
Deploying WSO2 Middleware on Kubernetes
Deploying WSO2 Middleware on KubernetesDeploying WSO2 Middleware on Kubernetes
Deploying WSO2 Middleware on Kubernetes
 
Cloud nativeday duane (final)
Cloud nativeday duane (final)Cloud nativeday duane (final)
Cloud nativeday duane (final)
 
Container Orchestration Integration: OpenStack Kuryr
Container Orchestration Integration: OpenStack KuryrContainer Orchestration Integration: OpenStack Kuryr
Container Orchestration Integration: OpenStack Kuryr
 
Containers kuberenetes
Containers kuberenetesContainers kuberenetes
Containers kuberenetes
 
Live migrating a container: pros, cons and gotchas
Live migrating a container: pros, cons and gotchasLive migrating a container: pros, cons and gotchas
Live migrating a container: pros, cons and gotchas
 

Viewers also liked

DockerCon SF 2015: Docker Community in China
DockerCon SF 2015: Docker Community in ChinaDockerCon SF 2015: Docker Community in China
DockerCon SF 2015: Docker Community in ChinaDocker, Inc.
 
Infinit: Modern Storage Platform for Container Environments
Infinit: Modern Storage Platform for Container EnvironmentsInfinit: Modern Storage Platform for Container Environments
Infinit: Modern Storage Platform for Container EnvironmentsDocker, Inc.
 
Learnersourcing: Improving Learning with Collective Learner Activity
Learnersourcing: Improving Learning with Collective Learner ActivityLearnersourcing: Improving Learning with Collective Learner Activity
Learnersourcing: Improving Learning with Collective Learner ActivityJuho Kim
 
Ph.D. Research Update: Year#4 Annual Progress and Planned Activities
Ph.D. Research Update: Year#4 Annual Progress and Planned ActivitiesPh.D. Research Update: Year#4 Annual Progress and Planned Activities
Ph.D. Research Update: Year#4 Annual Progress and Planned ActivitiesLighton Phiri
 
Efficient Online Evaluation of Big Data Stream Classifiers
Efficient Online Evaluation of Big Data Stream ClassifiersEfficient Online Evaluation of Big Data Stream Classifiers
Efficient Online Evaluation of Big Data Stream ClassifiersAlbert Bifet
 
Vom Utility Computing zum Cloud Computing
Vom Utility Computing zum Cloud ComputingVom Utility Computing zum Cloud Computing
Vom Utility Computing zum Cloud ComputingLothar Wieske
 
Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015
Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015
Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015Chris Jang
 
Moa: Real Time Analytics for Data Streams
Moa: Real Time Analytics for Data StreamsMoa: Real Time Analytics for Data Streams
Moa: Real Time Analytics for Data StreamsAlbert Bifet
 
DockerCon SF 2015: MomOps in DevOps w/ Mukta Aphale
DockerCon SF 2015: MomOps in DevOps w/ Mukta AphaleDockerCon SF 2015: MomOps in DevOps w/ Mukta Aphale
DockerCon SF 2015: MomOps in DevOps w/ Mukta AphaleDocker, Inc.
 
Test What You Write, Ship What You Test
Test What You Write, Ship What You TestTest What You Write, Ship What You Test
Test What You Write, Ship What You TestDocker, Inc.
 
Tyrion Cannister Neural Styles by Dora Korpar and Siphan Bou
Tyrion Cannister Neural Styles by Dora Korpar and Siphan BouTyrion Cannister Neural Styles by Dora Korpar and Siphan Bou
Tyrion Cannister Neural Styles by Dora Korpar and Siphan BouDocker, Inc.
 
DockerCon SF 2015: Cultural Change using Docker
DockerCon SF 2015: Cultural Change using Docker DockerCon SF 2015: Cultural Change using Docker
DockerCon SF 2015: Cultural Change using Docker Docker, Inc.
 
Dockerizing Stashboard
Dockerizing StashboardDockerizing Stashboard
Dockerizing StashboardDocker, Inc.
 
Building a Smarter Application Stack
Building a Smarter Application StackBuilding a Smarter Application Stack
Building a Smarter Application StackDocker, Inc.
 
DockerCon EU 2015: From Local Development to Production Deployments using Ama...
DockerCon EU 2015: From Local Development to Production Deployments using Ama...DockerCon EU 2015: From Local Development to Production Deployments using Ama...
DockerCon EU 2015: From Local Development to Production Deployments using Ama...Docker, Inc.
 
Contribute and Collaborate 101
Contribute and Collaborate 101Contribute and Collaborate 101
Contribute and Collaborate 101Docker, Inc.
 
WOT Cloud Computing Architect Summit
WOT Cloud Computing Architect SummitWOT Cloud Computing Architect Summit
WOT Cloud Computing Architect SummitDocker, Inc.
 
Building a Platform with Django, Docker and Salt
Building a Platform with Django, Docker and SaltBuilding a Platform with Django, Docker and Salt
Building a Platform with Django, Docker and SaltDocker, Inc.
 
Securing your Containers (Meetup at Docker HQ 4/7)
Securing your Containers (Meetup at Docker HQ 4/7)Securing your Containers (Meetup at Docker HQ 4/7)
Securing your Containers (Meetup at Docker HQ 4/7)Docker, Inc.
 
Dockerizing WordPress
Dockerizing WordPressDockerizing WordPress
Dockerizing WordPressDocker, Inc.
 

Viewers also liked (20)

DockerCon SF 2015: Docker Community in China
DockerCon SF 2015: Docker Community in ChinaDockerCon SF 2015: Docker Community in China
DockerCon SF 2015: Docker Community in China
 
Infinit: Modern Storage Platform for Container Environments
Infinit: Modern Storage Platform for Container EnvironmentsInfinit: Modern Storage Platform for Container Environments
Infinit: Modern Storage Platform for Container Environments
 
Learnersourcing: Improving Learning with Collective Learner Activity
Learnersourcing: Improving Learning with Collective Learner ActivityLearnersourcing: Improving Learning with Collective Learner Activity
Learnersourcing: Improving Learning with Collective Learner Activity
 
Ph.D. Research Update: Year#4 Annual Progress and Planned Activities
Ph.D. Research Update: Year#4 Annual Progress and Planned ActivitiesPh.D. Research Update: Year#4 Annual Progress and Planned Activities
Ph.D. Research Update: Year#4 Annual Progress and Planned Activities
 
Efficient Online Evaluation of Big Data Stream Classifiers
Efficient Online Evaluation of Big Data Stream ClassifiersEfficient Online Evaluation of Big Data Stream Classifiers
Efficient Online Evaluation of Big Data Stream Classifiers
 
Vom Utility Computing zum Cloud Computing
Vom Utility Computing zum Cloud ComputingVom Utility Computing zum Cloud Computing
Vom Utility Computing zum Cloud Computing
 
Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015
Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015
Google Tech Talk with Dr. Eric Brewer in Korea Apr.27.2015
 
Moa: Real Time Analytics for Data Streams
Moa: Real Time Analytics for Data StreamsMoa: Real Time Analytics for Data Streams
Moa: Real Time Analytics for Data Streams
 
DockerCon SF 2015: MomOps in DevOps w/ Mukta Aphale
DockerCon SF 2015: MomOps in DevOps w/ Mukta AphaleDockerCon SF 2015: MomOps in DevOps w/ Mukta Aphale
DockerCon SF 2015: MomOps in DevOps w/ Mukta Aphale
 
Test What You Write, Ship What You Test
Test What You Write, Ship What You TestTest What You Write, Ship What You Test
Test What You Write, Ship What You Test
 
Tyrion Cannister Neural Styles by Dora Korpar and Siphan Bou
Tyrion Cannister Neural Styles by Dora Korpar and Siphan BouTyrion Cannister Neural Styles by Dora Korpar and Siphan Bou
Tyrion Cannister Neural Styles by Dora Korpar and Siphan Bou
 
DockerCon SF 2015: Cultural Change using Docker
DockerCon SF 2015: Cultural Change using Docker DockerCon SF 2015: Cultural Change using Docker
DockerCon SF 2015: Cultural Change using Docker
 
Dockerizing Stashboard
Dockerizing StashboardDockerizing Stashboard
Dockerizing Stashboard
 
Building a Smarter Application Stack
Building a Smarter Application StackBuilding a Smarter Application Stack
Building a Smarter Application Stack
 
DockerCon EU 2015: From Local Development to Production Deployments using Ama...
DockerCon EU 2015: From Local Development to Production Deployments using Ama...DockerCon EU 2015: From Local Development to Production Deployments using Ama...
DockerCon EU 2015: From Local Development to Production Deployments using Ama...
 
Contribute and Collaborate 101
Contribute and Collaborate 101Contribute and Collaborate 101
Contribute and Collaborate 101
 
WOT Cloud Computing Architect Summit
WOT Cloud Computing Architect SummitWOT Cloud Computing Architect Summit
WOT Cloud Computing Architect Summit
 
Building a Platform with Django, Docker and Salt
Building a Platform with Django, Docker and SaltBuilding a Platform with Django, Docker and Salt
Building a Platform with Django, Docker and Salt
 
Securing your Containers (Meetup at Docker HQ 4/7)
Securing your Containers (Meetup at Docker HQ 4/7)Securing your Containers (Meetup at Docker HQ 4/7)
Securing your Containers (Meetup at Docker HQ 4/7)
 
Dockerizing WordPress
Dockerizing WordPressDockerizing WordPress
Dockerizing WordPress
 

Similar to Robust Containers by Eric Brewer

Kubernetes Basics - ICP Workshop Batch II
Kubernetes Basics - ICP Workshop Batch IIKubernetes Basics - ICP Workshop Batch II
Kubernetes Basics - ICP Workshop Batch IIPT Datacomm Diangraha
 
Introducing Kubernetes
Introducing Kubernetes Introducing Kubernetes
Introducing Kubernetes VikRam S
 
Kubernetes workshop
Kubernetes workshopKubernetes workshop
Kubernetes workshopKumar Gaurav
 
Webinar: Enabling Microservices with Containers, Orchestration, and MongoDB
Webinar: Enabling Microservices with Containers, Orchestration, and MongoDBWebinar: Enabling Microservices with Containers, Orchestration, and MongoDB
Webinar: Enabling Microservices with Containers, Orchestration, and MongoDBMongoDB
 
Brief Introduction To Kubernetes
Brief Introduction To KubernetesBrief Introduction To Kubernetes
Brief Introduction To KubernetesAvinash Ketkar
 
Kubernetes Immersion
Kubernetes ImmersionKubernetes Immersion
Kubernetes ImmersionJuan Larriba
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101Huy Vo
 
Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications OpenEBS
 
Kubernetes presentation
Kubernetes presentationKubernetes presentation
Kubernetes presentationGauranG Bajpai
 
Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...All Things Open
 
Intro to kubernetes
Intro to kubernetesIntro to kubernetes
Intro to kubernetesElad Hirsch
 
DevJam 2019 - Introduction to Kubernetes
DevJam 2019 - Introduction to KubernetesDevJam 2019 - Introduction to Kubernetes
DevJam 2019 - Introduction to KubernetesRonny Trommer
 
To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…Sergey Dzyuban
 
PlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин Владев
PlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин ВладевPlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин Владев
PlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин ВладевPlovDev Conference
 
(Draft) Kubernetes - A Comprehensive Overview
(Draft) Kubernetes - A Comprehensive Overview(Draft) Kubernetes - A Comprehensive Overview
(Draft) Kubernetes - A Comprehensive OverviewBob Killen
 
Containerization - The DevOps Revolution
Containerization - The DevOps RevolutionContainerization - The DevOps Revolution
Containerization - The DevOps RevolutionYulian Slobodyan
 

Similar to Robust Containers by Eric Brewer (20)

Kubernetes2
Kubernetes2Kubernetes2
Kubernetes2
 
Introduction to Kubernetes
Introduction to KubernetesIntroduction to Kubernetes
Introduction to Kubernetes
 
Kubernetes Basics - ICP Workshop Batch II
Kubernetes Basics - ICP Workshop Batch IIKubernetes Basics - ICP Workshop Batch II
Kubernetes Basics - ICP Workshop Batch II
 
Introducing Kubernetes
Introducing Kubernetes Introducing Kubernetes
Introducing Kubernetes
 
Kubernetes workshop
Kubernetes workshopKubernetes workshop
Kubernetes workshop
 
Webinar: Enabling Microservices with Containers, Orchestration, and MongoDB
Webinar: Enabling Microservices with Containers, Orchestration, and MongoDBWebinar: Enabling Microservices with Containers, Orchestration, and MongoDB
Webinar: Enabling Microservices with Containers, Orchestration, and MongoDB
 
Brief Introduction To Kubernetes
Brief Introduction To KubernetesBrief Introduction To Kubernetes
Brief Introduction To Kubernetes
 
Kubernetes Immersion
Kubernetes ImmersionKubernetes Immersion
Kubernetes Immersion
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
 
Openshift Workshop
Openshift Workshop Openshift Workshop
Openshift Workshop
 
Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications
 
Kubernetes intro
Kubernetes introKubernetes intro
Kubernetes intro
 
Kubernetes presentation
Kubernetes presentationKubernetes presentation
Kubernetes presentation
 
Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...
 
Intro to kubernetes
Intro to kubernetesIntro to kubernetes
Intro to kubernetes
 
DevJam 2019 - Introduction to Kubernetes
DevJam 2019 - Introduction to KubernetesDevJam 2019 - Introduction to Kubernetes
DevJam 2019 - Introduction to Kubernetes
 
To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…
 
PlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин Владев
PlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин ВладевPlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин Владев
PlovDev 2016: Оркестрация на контейнери с Kubernetes - Мартин Владев
 
(Draft) Kubernetes - A Comprehensive Overview
(Draft) Kubernetes - A Comprehensive Overview(Draft) Kubernetes - A Comprehensive Overview
(Draft) Kubernetes - A Comprehensive Overview
 
Containerization - The DevOps Revolution
Containerization - The DevOps RevolutionContainerization - The DevOps Revolution
Containerization - The DevOps Revolution
 

More from Docker, Inc.

Containerize Your Game Server for the Best Multiplayer Experience
Containerize Your Game Server for the Best Multiplayer Experience Containerize Your Game Server for the Best Multiplayer Experience
Containerize Your Game Server for the Best Multiplayer Experience Docker, Inc.
 
How to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker BuildHow to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker BuildDocker, Inc.
 
Build & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWSBuild & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWSDocker, Inc.
 
Securing Your Containerized Applications with NGINX
Securing Your Containerized Applications with NGINXSecuring Your Containerized Applications with NGINX
Securing Your Containerized Applications with NGINXDocker, Inc.
 
How To Build and Run Node Apps with Docker and Compose
How To Build and Run Node Apps with Docker and ComposeHow To Build and Run Node Apps with Docker and Compose
How To Build and Run Node Apps with Docker and ComposeDocker, Inc.
 
Distributed Deep Learning with Docker at Salesforce
Distributed Deep Learning with Docker at SalesforceDistributed Deep Learning with Docker at Salesforce
Distributed Deep Learning with Docker at SalesforceDocker, Inc.
 
The First 10M Pulls: Building The Official Curl Image for Docker Hub
The First 10M Pulls: Building The Official Curl Image for Docker HubThe First 10M Pulls: Building The Official Curl Image for Docker Hub
The First 10M Pulls: Building The Official Curl Image for Docker HubDocker, Inc.
 
Monitoring in a Microservices World
Monitoring in a Microservices WorldMonitoring in a Microservices World
Monitoring in a Microservices WorldDocker, Inc.
 
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...Docker, Inc.
 
Predicting Space Weather with Docker
Predicting Space Weather with DockerPredicting Space Weather with Docker
Predicting Space Weather with DockerDocker, Inc.
 
Become a Docker Power User With Microsoft Visual Studio Code
Become a Docker Power User With Microsoft Visual Studio CodeBecome a Docker Power User With Microsoft Visual Studio Code
Become a Docker Power User With Microsoft Visual Studio CodeDocker, Inc.
 
How to Use Mirroring and Caching to Optimize your Container Registry
How to Use Mirroring and Caching to Optimize your Container RegistryHow to Use Mirroring and Caching to Optimize your Container Registry
How to Use Mirroring and Caching to Optimize your Container RegistryDocker, Inc.
 
Monolithic to Microservices + Docker = SDLC on Steroids!
Monolithic to Microservices + Docker = SDLC on Steroids!Monolithic to Microservices + Docker = SDLC on Steroids!
Monolithic to Microservices + Docker = SDLC on Steroids!Docker, Inc.
 
Kubernetes at Datadog Scale
Kubernetes at Datadog ScaleKubernetes at Datadog Scale
Kubernetes at Datadog ScaleDocker, Inc.
 
Labels, Labels, Labels
Labels, Labels, Labels Labels, Labels, Labels
Labels, Labels, Labels Docker, Inc.
 
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment ModelUsing Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment ModelDocker, Inc.
 
Build & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWSBuild & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWSDocker, Inc.
 
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...Docker, Inc.
 
Developing with Docker for the Arm Architecture
Developing with Docker for the Arm ArchitectureDeveloping with Docker for the Arm Architecture
Developing with Docker for the Arm ArchitectureDocker, Inc.
 

More from Docker, Inc. (20)

Containerize Your Game Server for the Best Multiplayer Experience
Containerize Your Game Server for the Best Multiplayer Experience Containerize Your Game Server for the Best Multiplayer Experience
Containerize Your Game Server for the Best Multiplayer Experience
 
How to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker BuildHow to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker Build
 
Build & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWSBuild & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWS
 
Securing Your Containerized Applications with NGINX
Securing Your Containerized Applications with NGINXSecuring Your Containerized Applications with NGINX
Securing Your Containerized Applications with NGINX
 
How To Build and Run Node Apps with Docker and Compose
How To Build and Run Node Apps with Docker and ComposeHow To Build and Run Node Apps with Docker and Compose
How To Build and Run Node Apps with Docker and Compose
 
Hands-on Helm
Hands-on Helm Hands-on Helm
Hands-on Helm
 
Distributed Deep Learning with Docker at Salesforce
Distributed Deep Learning with Docker at SalesforceDistributed Deep Learning with Docker at Salesforce
Distributed Deep Learning with Docker at Salesforce
 
The First 10M Pulls: Building The Official Curl Image for Docker Hub
The First 10M Pulls: Building The Official Curl Image for Docker HubThe First 10M Pulls: Building The Official Curl Image for Docker Hub
The First 10M Pulls: Building The Official Curl Image for Docker Hub
 
Monitoring in a Microservices World
Monitoring in a Microservices WorldMonitoring in a Microservices World
Monitoring in a Microservices World
 
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...
 
Predicting Space Weather with Docker
Predicting Space Weather with DockerPredicting Space Weather with Docker
Predicting Space Weather with Docker
 
Become a Docker Power User With Microsoft Visual Studio Code
Become a Docker Power User With Microsoft Visual Studio CodeBecome a Docker Power User With Microsoft Visual Studio Code
Become a Docker Power User With Microsoft Visual Studio Code
 
How to Use Mirroring and Caching to Optimize your Container Registry
How to Use Mirroring and Caching to Optimize your Container RegistryHow to Use Mirroring and Caching to Optimize your Container Registry
How to Use Mirroring and Caching to Optimize your Container Registry
 
Monolithic to Microservices + Docker = SDLC on Steroids!
Monolithic to Microservices + Docker = SDLC on Steroids!Monolithic to Microservices + Docker = SDLC on Steroids!
Monolithic to Microservices + Docker = SDLC on Steroids!
 
Kubernetes at Datadog Scale
Kubernetes at Datadog ScaleKubernetes at Datadog Scale
Kubernetes at Datadog Scale
 
Labels, Labels, Labels
Labels, Labels, Labels Labels, Labels, Labels
Labels, Labels, Labels
 
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment ModelUsing Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
 
Build & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWSBuild & Deploy Multi-Container Applications to AWS
Build & Deploy Multi-Container Applications to AWS
 
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...
 
Developing with Docker for the Arm Architecture
Developing with Docker for the Arm ArchitectureDeveloping with Docker for the Arm Architecture
Developing with Docker for the Arm Architecture
 

Robust Containers by Eric Brewer

  • 1. Robust Containers Eric Brewer VP, Infrastructure DockerCon Keynote, June 10, 2014 @eric_brewer #dockercon
  • 2. 1) Application-centric, not machine-centric view It is easier, more natural, and more productive Why we Love Containers Over 2B containers launched per week (even our VMs run inside containers) We evolved here over the last decade… but Docker made it exciting and much easier to use (thanks!) 2) Essentially the way Google works internally: Signed static bundles + Linux containers (resolve dependencies up front)
  • 3. Containers interfere with each other • Unimportant things break important things • We want fair use among equally important things Solution: resource & performance isolation Series of open-source solutions: 2005: cpusets + “fake” NUMA to partition cores, memory 2006: cgroups for general task hierarchies 2009: bandwidth fair use, QoS levels 2010: memcg for better memory accounting, enforcement Status: isolation works well in practice (if you use these tools) First Problem: Unpredictable Interference
  • 4. Second Problem: Low Utilization Tier 1: Live services (e.g. search engine) • Provision for peak load (2-10x higher than average) • High priority, always get resources when needed Tier 2: Batch jobs (e.g. MapReduce) • Run in the leftovers, never displace Tier 1 • Lots of capacity — rarely at peak load If you partition resources, utilization goes down… Solution: controlled use of slack resources (free $$) Status: Our OSS container solutions support this well Note: Google does not overcommit customer VMs — you get the whole VM all the time
  • 5. Third Problem: Hard to Enforce Isolation Bad way: control loop (see LPC 2011) • Read stats, verify allocation, tune knobs, repeat • Slow response time, fragile Right way: • Direct enforcement in the kernel • Many patches to make this happen… (e.g. memcg) Status: enforcement now mostly in the kernel • Caches, memory bandwidth can still cause interference • Challenges getting these changes accepted upstream • Meta control loop: detect interference and migrate tasks (see CPI2)
  • 6. “Let Me Contain That For You” (LMCTFY = “L-M-C-T- fee”) You want this, but didn’t know it • Declarative allocation, prioritization of resources • Enforces resource isolation, with multiple hierarchies • Many resources: CPU, memory, bandwidth, latency, disk I/O, … • Enables better utilization • Stable API, as kernel mechanisms continue to evolve • Released as OSS in 2013 (see LPC 2013) OSS containers based on Docker are a core foundation for the future • Many contributors over the decade: SGI, LXC, RedHat, Parallels, Docker, … • We want to move LMCTFY functionality into Docker’s libcontainer • Released for Docker Hackathon: cAdvisor for container stats & alerts (written in Go)
  • 7. Pods (or how we really use containers) We actually use groups of nested containers = pods • Use LMCTFY for nesting, isolation & utilization • Many things implemented as helpers: • Logging and log rotation • Content management system + webserver Pod attributes: • Deployed together (in a parent container!) • Shared local volumes • Individual IP address (even if multiple pods per VM) • Ensures clean port allocation OK, we don’t use a single IP per pod, but we should have… Without this, need to track/distribute port allocations, since they must be late bound...
  • 8. Kubernetes “koo ber NAY tace” — Greek for “helmsman” New OSS release: orchestrating replicated pods across multiple nodes Craig McLuckie, Brendan Burns to cover at 2pm today Master: • Manages worker pods dynamically • Uses etcd to track desired configuration API Server Replica Controller etcd k Workers: • Replicated Docker image • Parameterized: arguments passed in via environment variables • Shared view of load-balanced services Kubelet Service Proxy Docker
  • 9. Concept 1: Labels and Services Service = load-balanced replica set • Pod labels ⇒ the services they implement • Pods access services via localhost:<port> • (Local) proxy sends traffic to member of set • Ports are the service “names” { "id": "redisslave", "port": 10001, "labels": { "name": "redisslave" } } Service Definition (JSON) "labels": { "name": "redisslave" } Partial pod definition (JSON) Pods have labels Many overlapping sets of labels: stage: production name: redis zone: west version: 2.6 Replica set = a group of pods with the same labels The set is defined by a query (not a static list) (because entropy happens)
  • 10. Having an explicit desired state is a good idea! Otherwise can’t tell if the desire changed, or the actual state changes Concept 2: The Reconciler Model Key idea: Declare the desired state Loop { // the reconciler loop, run by master • Query the actual state of the system • Compare with desired state • Implement corrections (if any) // reconcile reality with desired state } In Kubernetes desiredState: if we lose a replica for some reason, add one replicas: 2
  • 11. Robust Containers Docker (used well) ⇒ clean, repeatable images Single Node (pods): • Allocate ports per pod (conflict free!) • Attach data-only containers to the pod (as volumes) (clean sharing of data) • “Parameterized containers” using environment variables Multi-Node: • Labels for time-varying overlapping sets • Services are load-balanced groups of replicated pods • The Reconciler Model recovers from changes (expected or not) (actually used at worker level and master level)
  • 12. Containers are the Path to “Cloud Native” Pods as a building block • Clean port namespace • Shared volumes • Isolation, prioritization, tools for utilization • Auto restart (don’t run supervisord k times) • Liveness probes, stats for load balancing • sshd in environment (not in your container) Application-level cloud events per container or pod • Start, stop, restart • Notification of migration, resizing, new shards, ... • Resource alerts, OOM management Services and labels Reconciliation
  • 13. Summary We are standardizing around the Docker container image • Pushing for usable, scalable, open containers • Isolation, nesting, utilization, enforcement • Moving to Go to simplify integration (and because we like it) Thanks to Docker… for making containers lightweight, easy to use, and exciting! We look forward to creating a great robust space together News today: • Kubernetes: see Craig & Brendan at 2pm today • Docker on GAE: see Ekaterina Volkova at 2:50pm today • cAdvisor: stats & alerts for containers

Editor's Notes

  1. Static binaries prevent changes in behavior due to changes in libraries, much like Docker containers pre-resolve the file system and packages. Signing the binaries adds security by preventing tampering with the binaries once produced.
  2. 2004: SGI started cpusets, influenced by Solaris and others. They were the first to need to deal with more cores. http://man7.org/linux/man-pages/man7/cpuset.7.html 2005: NUMA aware memory usage essentially put physical memory into different groups so that you could allocate memory from the right place (nearby). Fake NUMA makes up artificial groups as a way to limit memory usage to a group. 2006: cpusets had some support for hierarchies for CPUs; cgroups generalized it. Then each subsystem was modified to make use of cgroups for allocation and/or accounting. 2010: memcg (=memory cgroup) adds better control over memory allocation and accounting and put enforcement in the kernel.
  3. We actually use more than two levels; LMCTFY supports four. Two gets across the core idea, but you get higher utilization as you get more sophisticated with policies.
  4. By the way, we have to fight in general to get these container changes accepted upstream. For example, we would like OOM handling to be more flexible. See http://lwn.net/Articles/591990/ for a discussion about how to handle OOM management in the kernel (or not). See http://lwn.net/Articles/589404/ for a representative patch set user user-level OOM handling and its motivation.
  5. Really important to have a declarative, stable API about how to allocate and prioritize resources. The direct use of kernel knobs, although more powerful, leads to a wide, fragile API, extra complexity, and poor evolution. It hinders both the app AND the kernel, since the kernel must support the wide complex API for a very long time. cAdvisor released on June 6th, 2014 for Docker Hackathon as part of LMCTFY: https://github.com/google/lmctfy/tree/master/cadvisor It may move to its own project to facilitate integration.
  6. Pods that offer the service “myService” need to have the labels “stage: production” AND “zone: west” (simple conjunction for now). A more realistic use would have labels for a service name and version as well. The service “myService” can be found on localhost:8000. That request goes to the local proxy (nginx), which forwards it to the best pod.