Cloud-native Deployment
Kubernetes Colorado June 2016
What is “Cloud Native”
How do you deploy
applications built for the Cloud?
Don’t break production
Complex upgrades
Secrets and reuse
Kit Merker
Product Manager
Google Cloud Platform
@kitmerker | kitm@google.com
What is Kubernetes?
Old way: install applications on host
kernel
libs
app
app app
Application and OS share filesystem
Use OS distribution package manager
Entangled with each other and with host
• Executables
• Configuration
• Shared libraries
• Process and lifecycle management
Immutable VM images provide predictable
rollouts and rollbacks
• but are heavyweight and not portable
app
New way: deploy containers
libs
app
kernel
libs
app
libs
app
libs
app
OS-level virtualization
Isolated, from each other and from the host
• filesystems
• processes
• resources
Small and fast ⇒ enables 1:1 app to image
• Unlocks benefits of microservices
• Decouple build (Dev) from deployment (Ops)
• Consistency from development to production
• Portable across OS distros and clouds
• Application-centric management
Everything at Google
runs in containers
Everything at Google
runs in containers
Launch over 2 billion
containers per week.
job hello_world = {
runtime = { cell = 'ic' } // Cell (cluster) to run in
binary = '.../hello_world_webserver' // Program to run
args = { port = '%port%' } // Command line parameters
requirements = { // Resource requirements
ram = 100M
disk = 100M
cpu = 0.1
}
replicas = 5 // Number of tasks
}
10000
Developer View
web browsers
BorgMaster
link shard
UI shardBorgMaster
link shard
UI shardBorgMaster
link shard
UI shardBorgMaster
link shard
UI shard
Scheduler
borgcfg web browsers
scheduler
Borglet Borglet Borglet Borglet
Config
file
BorgMaster
link shard
UI shard
persistent store
(Paxos)
Binary
Developer View
What just
happened?
Hello
world!
Hello
world!
Hello
world!
Hello
world!Hello
world! Hello
world! Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world! Hello
world!
Hello
world!
Hello
world!
Hello
world!
Image by Connie
Zhou
Hello
world!
Hello
world!
Hello
world! Hello
world!
Hello
world! Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world! Hello
world!
Hello
world! Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world!
Hello
world! Hello
world!
Hello
world! Hello
world!
Hello
world!
Hello
world!
Kubernetes
Greek for “Helmsman”; also the root of the
words “governor” and “cybernetic”
● Infrastructure for containers
● Schedules, runs, and manages
containers on virtual and physical
machines
● Platform for automating deployment,
scaling, and operations
● Inspired and informed by Google’s
experiences and internal systems
● 100% Open source, written in Go
gcr.io
web browsers
Scheduler
kubectl web browsers
scheduler
Kubelet Kubelet Kubelet Kubelet
Config
file
Kubernetes Master
Container
Image
Developer View
What just
happened?
Cloud Native Deployment
Google Cloud Platform 15
Building block: Sets of Containers
Pod
/data
Containers:
● Handle package dependencies
● Different versions, same machine
● No “DLL hell”
python 3.4.2
glibc 2.21
MySite
python 2.7.9
glibc 2.19
MyCachePods:
● Co-locate containers
● Shared volumes
● IP address, independent port space
● Unit of deployment, migration
Google Cloud Platform 16
Unit of deployment: Services Service IP
Service:
● Replicated pods
○ Source pod is a template
● Auto-restart member pods
● Abstract name (DNS)
● IP address for the service
○ in addition to the members
● Load balancing among replicas
Load
Balancer
Google Cloud Platform 17
The Flow of Deployment
Deploy
flags, scripts,
keys, …
Package
lock binary dependencies
(containers)
Build
lock binary version
source
control
binary
libraries
resources
static config
image
?
Google Cloud Platform 18
Example: Rolling Upgrade with Labels
Pods:
Labels:
frontend
v1.2
frontend
v1.2
frontend
v1.2
frontend
v1.2
frontend
v1.3
frontend
v1.3
frontend
v1.3
frontend
v1.3
frontend
Replication
Controller
replicas: 4
v1.2
Replication
Controller
replicas: 1
v1.3
replicas: 3 replicas: 2replicas: 3replicas: 2replicas: 1 replicas: 4replicas: 0
Google Cloud Platform 19
The Flow of Configuration: Immutable + Cloud
libraries
resources
static config
Package
lock binary
dependencies
source
control
binary
image
Build
lock binary version
load balancer
Deploy
lock runtime state
(flags, keys, …)
load balancer
auto-scaler
each step removes degrees of freedom
Google Cloud Platform 20
Need Secrets & Deploy-time Configuration
Some things cannot be baked into a container
1. Credentials/key material
a. Kubernetes has solved this with the secret model
b. A volume, containing secrets, that is mapped into a pod
2. Environment-specific metadata
a. Kubernetes has solved this with the configMap model
b. A volume mapped into a pod with key/value pairs
c. Can also be used to via environment variables
Google Cloud Platform 21
The Flow of Configuration - with Secrets/Config
libraries
resources
static config
Package
lock binary
dependencies
source
control
binary
image
Build
lock binary version
load balancer
Deploy
lock runtime state
(flags, keys, …)
load balancer
auto-scaler
each step removes degrees of freedom
Mounted
Volumes
Google Cloud Platform 22
Holy War:
Scripting vs. DSL
Holy War:
Scripting vs. DSL
Image credit - larping.com
Google Cloud Platform 23
Holy War: Scripting vs. DSL
Scripting
● Pros:
○ Simple
○ Turing Complete
● Cons:
○ No model to support
introspection
○ No declaration of intent
○ Fragile - needs to be
coordinated with app changes
Configuration language (DSL)
● Pros:
○ Less verbose
○ Establishes a model
● Cons:
○ DSL lags resource types
○ Becomes a language without full
tooling
○ Interpreted in production
environment: many moving parts
Image credit - larping.com
Google Cloud Platform 24
Solution: Construction as deployment step
1. Want a simple declarative framework to construct deployments
a. If you need deep logic, it should be written in a first-class language
b. Code, if needed, generates the (immutable) configuration
2. We need to support encapsulation and composition
a. Much like abstract data types or objects
The entire deployment graph becomes immutable.
Google Cloud Platform 25
The Flow of Configuration
libraries
resources
static config
Package
lock binary
dependencies
source
control
binary
image
Build
lock binary version
load balancer
Deploy
lock runtime state
(flags, keys, …)
load balancer
auto-scaler
load balancer
Construct
lock topology
compose,
physical resources
each step removes degrees of freedom
load balancer
auto-scaler
Google Cloud Platform 26
Immutable VM spec:
VM(cores, RAM, image, zone)
REST “create” call to instantiate
Start with Cloud Primitive Types
load balancer
auto-scaler
network routes
kubernetes cluster
vm
disk
managed instance group
...
Google Cloud Platform 27
“Frontend” builds on Cloud Primitive Types
FE
Template
Frontend
FE.yaml
...
resources:
- name: FE_App
type: FE
properties:
zone:us-central1-a
FEimage:https://www.googleapis…
publish: true
load balancer
auto-scaler
network routes
managed instance group
config type
Google Cloud Platform 28
Nested Deployment model
imports:
path: myapp.jinja
resources:
- name: MyApp_1
type: MyApp.jinja
properties:
zone:us-central1-a
FEimage:https://www.googleapis…
BEImage:https://www.googleapis…
deployment:production
...
MyApp
Template
MyApp.yaml
Frontend
Backend
Encapsulated
Nested Types
Google Cloud Platform 29
Nested resource model - fully expanded
imports:
path: myapp.jinja
resources:
- name: MyApp_1
type: MyApp.jinja
properties:
zone:us-central1-a
FEimage:https://www.googleapis…
BEImage:https://www.googleapis…
deployment:production
...
MyApp
Template
FE
TemplateMyApp.yaml type
Frontend
FE.yaml
...
resources:
- name: FE_App
type: FE
properties:
zone:us-central1-a
FEimage:https://www.googleapis…
publish: true
BE
Template
FE.yaml
...
resources:
- name: FE_App
type: FE
properties:
zone:us-central1-a
BEimage:https://www.googleapis…
numberinstances: 3
instancetype: n1-standard-8
load balancer
load balancer
auto-scaler
load balancer
auto-scaler
network routes
managed instance group
load balancer
network routes
vm 1
vm 2
vm 3
disk 1
disk 2
disk 3
config type
Backend
construction deployment
Google Cloud Platform 30
Kubernetes Velocity Top 0.01% of all
GitHub projects
1200+ external
projects based
on Kubernetes
720+
unique
contributors
1.0
1.1
1.2
45+ commits / day over the last year! 100+ Meetup
groups around
the world
Google Cloud Platform 31
Chat: slack.k8s.io
Visit: kubernetes.io
Share: @kubernetesio
Code: github.com/kubernetes/kubernetes
open community
open design
open source
open to ideas
Invitation: Kubernetes is Open
Try out Google Container Engine
https://cloud.google.com/container-engine/
What is “Cloud Native”
How do you deploy
applications built for the Cloud?
Don’t break production
Complex upgrades
Secrets and reuse
Construct deploy graph offline, roll it out immutably
Use a real language to generate declarative instructions
Mount volumes for secrets & runtime config
(Hint: use Helm & Kubernetes!)
THANK YOU!

Kubernetes Boulder - Kit Merker - Cloud Native Deployment

  • 1.
  • 2.
    What is “CloudNative” How do you deploy applications built for the Cloud? Don’t break production Complex upgrades Secrets and reuse
  • 3.
    Kit Merker Product Manager GoogleCloud Platform @kitmerker | kitm@google.com
  • 4.
  • 5.
    Old way: installapplications on host kernel libs app app app Application and OS share filesystem Use OS distribution package manager Entangled with each other and with host • Executables • Configuration • Shared libraries • Process and lifecycle management Immutable VM images provide predictable rollouts and rollbacks • but are heavyweight and not portable app
  • 6.
    New way: deploycontainers libs app kernel libs app libs app libs app OS-level virtualization Isolated, from each other and from the host • filesystems • processes • resources Small and fast ⇒ enables 1:1 app to image • Unlocks benefits of microservices • Decouple build (Dev) from deployment (Ops) • Consistency from development to production • Portable across OS distros and clouds • Application-centric management
  • 7.
  • 8.
    Everything at Google runsin containers Launch over 2 billion containers per week.
  • 9.
    job hello_world ={ runtime = { cell = 'ic' } // Cell (cluster) to run in binary = '.../hello_world_webserver' // Program to run args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements ram = 100M disk = 100M cpu = 0.1 } replicas = 5 // Number of tasks } 10000 Developer View
  • 10.
    web browsers BorgMaster link shard UIshardBorgMaster link shard UI shardBorgMaster link shard UI shardBorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Binary Developer View What just happened?
  • 11.
    Hello world! Hello world! Hello world! Hello world!Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!Hello world! Hello world! Hello world! Hello world! Image by Connie Zhou Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!
  • 12.
    Kubernetes Greek for “Helmsman”;also the root of the words “governor” and “cybernetic” ● Infrastructure for containers ● Schedules, runs, and manages containers on virtual and physical machines ● Platform for automating deployment, scaling, and operations ● Inspired and informed by Google’s experiences and internal systems ● 100% Open source, written in Go
  • 13.
    gcr.io web browsers Scheduler kubectl webbrowsers scheduler Kubelet Kubelet Kubelet Kubelet Config file Kubernetes Master Container Image Developer View What just happened?
  • 14.
  • 15.
    Google Cloud Platform15 Building block: Sets of Containers Pod /data Containers: ● Handle package dependencies ● Different versions, same machine ● No “DLL hell” python 3.4.2 glibc 2.21 MySite python 2.7.9 glibc 2.19 MyCachePods: ● Co-locate containers ● Shared volumes ● IP address, independent port space ● Unit of deployment, migration
  • 16.
    Google Cloud Platform16 Unit of deployment: Services Service IP Service: ● Replicated pods ○ Source pod is a template ● Auto-restart member pods ● Abstract name (DNS) ● IP address for the service ○ in addition to the members ● Load balancing among replicas Load Balancer
  • 17.
    Google Cloud Platform17 The Flow of Deployment Deploy flags, scripts, keys, … Package lock binary dependencies (containers) Build lock binary version source control binary libraries resources static config image ?
  • 18.
    Google Cloud Platform18 Example: Rolling Upgrade with Labels Pods: Labels: frontend v1.2 frontend v1.2 frontend v1.2 frontend v1.2 frontend v1.3 frontend v1.3 frontend v1.3 frontend v1.3 frontend Replication Controller replicas: 4 v1.2 Replication Controller replicas: 1 v1.3 replicas: 3 replicas: 2replicas: 3replicas: 2replicas: 1 replicas: 4replicas: 0
  • 19.
    Google Cloud Platform19 The Flow of Configuration: Immutable + Cloud libraries resources static config Package lock binary dependencies source control binary image Build lock binary version load balancer Deploy lock runtime state (flags, keys, …) load balancer auto-scaler each step removes degrees of freedom
  • 20.
    Google Cloud Platform20 Need Secrets & Deploy-time Configuration Some things cannot be baked into a container 1. Credentials/key material a. Kubernetes has solved this with the secret model b. A volume, containing secrets, that is mapped into a pod 2. Environment-specific metadata a. Kubernetes has solved this with the configMap model b. A volume mapped into a pod with key/value pairs c. Can also be used to via environment variables
  • 21.
    Google Cloud Platform21 The Flow of Configuration - with Secrets/Config libraries resources static config Package lock binary dependencies source control binary image Build lock binary version load balancer Deploy lock runtime state (flags, keys, …) load balancer auto-scaler each step removes degrees of freedom Mounted Volumes
  • 22.
    Google Cloud Platform22 Holy War: Scripting vs. DSL Holy War: Scripting vs. DSL Image credit - larping.com
  • 23.
    Google Cloud Platform23 Holy War: Scripting vs. DSL Scripting ● Pros: ○ Simple ○ Turing Complete ● Cons: ○ No model to support introspection ○ No declaration of intent ○ Fragile - needs to be coordinated with app changes Configuration language (DSL) ● Pros: ○ Less verbose ○ Establishes a model ● Cons: ○ DSL lags resource types ○ Becomes a language without full tooling ○ Interpreted in production environment: many moving parts Image credit - larping.com
  • 24.
    Google Cloud Platform24 Solution: Construction as deployment step 1. Want a simple declarative framework to construct deployments a. If you need deep logic, it should be written in a first-class language b. Code, if needed, generates the (immutable) configuration 2. We need to support encapsulation and composition a. Much like abstract data types or objects The entire deployment graph becomes immutable.
  • 25.
    Google Cloud Platform25 The Flow of Configuration libraries resources static config Package lock binary dependencies source control binary image Build lock binary version load balancer Deploy lock runtime state (flags, keys, …) load balancer auto-scaler load balancer Construct lock topology compose, physical resources each step removes degrees of freedom load balancer auto-scaler
  • 26.
    Google Cloud Platform26 Immutable VM spec: VM(cores, RAM, image, zone) REST “create” call to instantiate Start with Cloud Primitive Types load balancer auto-scaler network routes kubernetes cluster vm disk managed instance group ...
  • 27.
    Google Cloud Platform27 “Frontend” builds on Cloud Primitive Types FE Template Frontend FE.yaml ... resources: - name: FE_App type: FE properties: zone:us-central1-a FEimage:https://www.googleapis… publish: true load balancer auto-scaler network routes managed instance group config type
  • 28.
    Google Cloud Platform28 Nested Deployment model imports: path: myapp.jinja resources: - name: MyApp_1 type: MyApp.jinja properties: zone:us-central1-a FEimage:https://www.googleapis… BEImage:https://www.googleapis… deployment:production ... MyApp Template MyApp.yaml Frontend Backend Encapsulated Nested Types
  • 29.
    Google Cloud Platform29 Nested resource model - fully expanded imports: path: myapp.jinja resources: - name: MyApp_1 type: MyApp.jinja properties: zone:us-central1-a FEimage:https://www.googleapis… BEImage:https://www.googleapis… deployment:production ... MyApp Template FE TemplateMyApp.yaml type Frontend FE.yaml ... resources: - name: FE_App type: FE properties: zone:us-central1-a FEimage:https://www.googleapis… publish: true BE Template FE.yaml ... resources: - name: FE_App type: FE properties: zone:us-central1-a BEimage:https://www.googleapis… numberinstances: 3 instancetype: n1-standard-8 load balancer load balancer auto-scaler load balancer auto-scaler network routes managed instance group load balancer network routes vm 1 vm 2 vm 3 disk 1 disk 2 disk 3 config type Backend construction deployment
  • 30.
    Google Cloud Platform30 Kubernetes Velocity Top 0.01% of all GitHub projects 1200+ external projects based on Kubernetes 720+ unique contributors 1.0 1.1 1.2 45+ commits / day over the last year! 100+ Meetup groups around the world
  • 31.
    Google Cloud Platform31 Chat: slack.k8s.io Visit: kubernetes.io Share: @kubernetesio Code: github.com/kubernetes/kubernetes open community open design open source open to ideas Invitation: Kubernetes is Open
  • 32.
    Try out GoogleContainer Engine https://cloud.google.com/container-engine/
  • 33.
    What is “CloudNative” How do you deploy applications built for the Cloud? Don’t break production Complex upgrades Secrets and reuse Construct deploy graph offline, roll it out immutably Use a real language to generate declarative instructions Mount volumes for secrets & runtime config (Hint: use Helm & Kubernetes!)
  • 34.