Bringing Docker to the Cloud

Bringing Docker to the Cloud
Andrew Kennedy, 30 June 2015

@grkvlt
Agenda
• Clocker Introduction
• What is Clocker?
• Application Management
• Container Management
• Clocker Networking
• Demonstration
• Clocker 1.0.0
• Summary
Copyright 2015 by CloudsoftCorporationLimited

Clocker
IntroductionClocker
Introduction

@grkvlt
Clocker Statistics
• 1 Year Old
• 636 Commits
• 131 Pull Requests
• 10 Contributors
• 2 External
• 14 KLOC
• 20 Releases
• 17 Conferences
http://www.redotheweb.com/CodeFlower/http://www.redotheweb.com/CodeFlower/

@grkvlt
What does it do?
1. Spins up and Manages Docker
Clouds
2. Serves up Containers on Demand
3. Manages Composite Application
Deployments on Docker

@grkvlt
What does it provide?
• Infrastructure Management
• Docker Hosts
• Swarm Controller
• Multi Host and Multi Container Applications
• Seamless Networking
• Communication Between Services
• Orchestration and Clustering
• Control of Containers

@grkvlt
Who is using it?
• Proof of Concept Stage
• Financial Services
• Insurance
• Production
• Multi-tenant Application Trial
• Container per service
• Testing
• Clocker

@grkvlt
Where can I find it?
• Open Source
• Apache 2.0 Licensed
• http://clocker.io/
• Status
• Working towards 1.0.0
• 1.0.0-PRE.20150630
• 1.1.0-SNAPSHOT
• Testers welcome!

What
is
Clocker?What
is
Clocker?

@grkvlt
Clocker and Brooklyn
• What is it?
• Brooklyn Application
• Brooklyn Location
• What does it provide?
• First Class Docker Support in Brooklyn
• Optimized Brooklyn Blueprints for Docker
• Container Orchestration

@grkvlt
Apache Brooklyn
• Apache Brooklyn
• Open Source
• Java
• Donated to the Apache Software Foundation
• ASF Incubator Status
• 0.7.0-incubating

@grkvlt
Apache Brooklyn
• Application Management Platform
• Deploy, Manage and Monitor Blueprints
• Provisioning, Installation and Customization
• Management
• AutoScaling
• Resilience
• Security

@grkvlt
Apache jclouds
• Java Cloud Library
• API Agnostic
• CloudStack, OpenStack, AWS EC2, GCE…
• Create Virtual Machines
• Return SSH Endpoint
• Manage Properties

@grkvlt
Apache jclouds
• Drivers for REST APIs
• Docker Driver
• Written by @turlinux
• Virtual Container
• Using SSH Daemon or native Docker
• Same ComputeNode abstraction as VM
• Can use any Image or Dockerfile

@grkvlt
Docker
• Popular
• Large Ecosystem
• Growing
• Complex
• Containers
• Isolation
• Performance
• Composable
• Ubiquitous

@grkvlt
Docker

Application
ManagementApplication
Management

@grkvlt
Brooklyn Blueprints
• Describe Applications
• OASIS CAMP Standard
• List of Services
• NoSQL Database Clusters
• Web Servers and Load Balancers
• Shell or Python Scripts
• Tree Structure
• Sensors, Effectors and Policies

@grkvlt
Blueprint Example
name: appserver-‐w-‐policy
services:
-‐ type: brooklyn.entity.webapp.ControlledDynamicWebAppCluster
initialSize: 1
memberSpec:
$brooklyn:entitySpec:
type: brooklyn.entity.webapp.jboss.JBoss7Server
brooklyn.config:
wars.root:
http://search.maven.org/remotecontent?filepath=io/brooklyn/example/brooklyn-‐example-‐hello-‐world-‐sql-‐webapp/0.6.0/brooklyn-‐example-‐hello-‐
world-‐sql-‐webapp-‐0.6.0.war
http.port: 8080+
java.sysprops:
brooklyn.example.db.url: $brooklyn:formatString("jdbc:%s%s?user=%s&password=%s",
component("db").attributeWhenReady("datastore.url"), "visitors", "brooklyn", "br00k11n")
brooklyn.policies:
-‐ policyType: brooklyn.policy.autoscaling.AutoScalerPolicy
brooklyn.config:
metric: $brooklyn:sensor("brooklyn.entity.webapp.DynamicWebAppCluster", "webapp.reqs.perSec.windowed.perNode")
metricLowerBound: 10
metricUpperBound: 100
minPoolSize: 1
maxPoolSize: 5
-‐ type: brooklyn.entity.database.mysql.MySqlNode
id: db
name: DB HelloWorld Visitors
brooklyn.config:
datastore.creation.script.url:
https://github.com/apache/incubator-‐brooklyn/raw/master/usage/launcher/src/test/resources/visitors-‐creation-‐script.sql

@grkvlt
Application Management
• Policies
• Sensor Data Driven
• Auto Scaling
• Cluster Management
• Resilience and Failover
• Follow the Sun

@grkvlt
Docker Cloud
• Brooklyn Blueprint for Docker Cluster
• Docker Engine on Cloud VM or Bare Metal
• Configuration for Host
• TLS Certificates
• Setup Volumes
• Logging
• Install SDN Agents
• Manage Capacity or Headroom

@grkvlt
Clocker Blueprints

@grkvlt
Docker Cloud
1. On-demand
2. Multi-Tenant
3. Hardware Independent
4. Application Level

@grkvlt
Simple Architecture

@grkvlt
Clocker 1.x Architecture

@grkvlt
Clocker Features
• Docker Extensions to Brooklyn
–Docker Image as First-Class Service Type
–Placement Strategies for Containers
–Create Docker Images and Networks
• Manages Docker Engine and Swarm
–Deployment and Management
–Installation and Configuration
–Software-Defined Networking

Container
ManagementContainer
Management

@grkvlt
Clocker Features
• Application Deployment
–Oasis CAMP YAML Blueprint
–TOSCA and Compose in Development
–Same as Core Brooklyn
• Mixed Destinations
–Some Virtual Machines
–Some Bare Metal
–Some Containers

@grkvlt
Clocker Orchestration
Docker
Engine
Virtual
Machine
Container
Clocker
Network
Segment
SDN
Provider
Cloud
Provider
Brooklyn

@grkvlt
Docker Cloud Internals
• Placement and Provisioning
• Constraints and Labels
• Autonomics
• Policies and Scaling
• Headroom
• Networks

@grkvlt
Container Management
• Sources
• Docker Image Definition
• Docker Hub
• Dockerfile
• Brooklyn Entity Definition
• Create Image Automatically
• Commit or Push for Reuse

@grkvlt
Container Management
• Installation of Service
• Defined by Brooklyn or Dockerfile
• Common to all Entity Instances
• Commit Image
• Available for next Entity
• Push Image
• Available for all Hosts

@grkvlt
Application Blueprints
id: docker-‐mysql
name: "Docker Hub MySQL Server"
origin: "https://registry.hub.docker.com/_/mysql/"
locations:
-‐ my-‐docker-‐cloud
services:
-‐ type: docker:mysql:5.7
openPorts:
-‐ 3306
directPorts:
-‐ 3306
env:
MYSQL_ROOT_PASSWORD: "s3kr3t"
id: docker-‐mysql
name: "Docker Hub MySQL Server"
origin: "https://registry.hub.docker.com/_/mysql/"
locations:
services:
-‐ type: docker:mysql:5.7
openPorts:
-‐ 3306
directPorts:
-‐ 3306
env:
MYSQL_ROOT_PASSWORD: "s3kr3t"
id: docker-‐jboss
name: "Docker Hub JBoss Server"
origin: "https://registry.hub.docker.com/_/jboss/"
locations:
services:
-‐ type: docker:jboss/wildfly:8.2.0.Final
openPorts:
-‐ 8080
-‐ 9990
directPorts:
-‐ 8080
id: docker-‐jboss
name: "Docker Hub JBoss Server"
origin: "https://registry.hub.docker.com/_/jboss/"
locations:
services:
-‐ type: docker:jboss/wildfly:8.2.0.Final
openPorts:
-‐ 8080
-‐ 9990
directPorts:
-‐ 8080
id: docker-‐redis
name: "Docker Hub Redis Service"
origin: "https://registry.hub.docker.com/_/redis/"
locations:
services:
-‐ type: docker:redis
openPorts:
-‐ 6379
directPorts:
-‐ 6379
id: docker-‐redis
name: "Docker Hub Redis Service"
origin: "https://registry.hub.docker.com/_/redis/"
locations:
services:
-‐ type: docker:redis
openPorts:
-‐ 6379
directPorts:
-‐ 6379

@grkvlt
Application Blueprints
id: dockerfile-‐mysql
name: "Docker Hub LAMP Stack"
locations:
-‐my-‐docker-‐cloud
services:
-‐ type: docker:mysql:5.7.5
id: mysql
env:
MYSQL_ROOT_PASSWORD: "s3cr3t"
-‐ type: docker:grkvlt/myapp:latest
id: application
env:
MYSQL_HOST:
component(”mysql").attributeWhenReady("host.hostname")
id: dockerfile-‐mysql
name: "Docker Hub LAMP Stack"
locations:
-‐my-‐docker-‐cloud
services:
-‐ type: docker:mysql:5.7.5
id: mysql
env:
MYSQL_ROOT_PASSWORD: "s3cr3t"
-‐ type: docker:grkvlt/myapp:latest
id: application
env:
MYSQL_HOST:
component(”mysql").attributeWhenReady("host.hostname")

@grkvlt
Container Placement
• Demand
• Adding an Application
• Scaling existing Application
• Requirements
• Host Location
• Service Resources
• CPU, Memory, Network
• Labels

@grkvlt
Container Placement
• Supply
• Choose a Host from available
• Create new Host
• Start Container
• Set CPU and Memory
• Attach to Network

@grkvlt
Container Placement
• Placement Strategies
• Filter Hosts
• Sort Hosts
• Information from
• Docker
• Underlying Machine
• Cloud Environment
• Entity or Service

@grkvlt
Container Placement
• Placement Strategies
• Random, Depth or Breadth First
• CPU or Memory Usage
• Memory, CPU or Container Limits
• Geographic Constraints
• User Defined
• Java Predicate

@grkvlt
Autonomics
• Brooklyn Policies
• Attached to Entities in Application
• Nothing Docker Specific
• Elastic Scaling
• Cluster Resizing
• Sensor Driven
• Service Resilience and Replacement

@grkvlt
Headroom
• Ensure resources available
• Based on MaxContainers strategy limit
• Or Percentage Utilisation
• Or CPU and RAM allocation
• Scale Docker Host Cluster Automatically
• Add new Docker hosts
• Remove empty Docker hosts

Clocker
NetworkingClocker
Networking

@grkvlt
Software-Defined Networking
• Needed for Seamless Provisioning
• Host to Host Communication
• Same LAN Segment
• No Port Forwarding
• Natural Application Configuration
• Initial Driver was EPMD Applications

@grkvlt
Networking Providers
• Standardized Interfaces
• Pluggable Providers
• Weave
• Metaswitch Calico
• IBM OpenDOVE
• Same Basic Features
• Extensions provided by configuration

@grkvlt
Networking Providers
• Implementation Agnostic
• L2 over L3 etc.
• Similar to Hypervisor in Clouds
• Generic Interfaces
• Host Component
• Service Component (or Endpoint)

@grkvlt
Networking Capabilities
• Provide Multiple Networks
• Single Application or Shared
• Private Addresses
• Segmented by CIDR
• Docker Port Forwarding Access
• Debug Mechanism

@grkvlt
Metaswitch Project Calico
• SDN for Bare Metal, VMs and Containers
• Layer 3 (with adorable kittens)
• Uses OS IP routing and forwarding
• Configuration in an etcd Cluster
• Version 0.4 now available
• Uses profiles for container ACLs
• Spans VMs and Containers
• OpenStack Neutron network driver

@grkvlt
Clocker Networking
ContainerHost
SDN
Bridge
Container
Internet
SDN
Gateway

@grkvlt
• IP Pool Controlled by Clocker
• Clocker Controls Subnet Allocation
• Applications Segmented by CIDR
• Delegate to SDN or Cloud
• Bring your own IP
• Both Weave and Calico offer this now

@grkvlt
• Wide Area and Multi Region SDN
• Cross Platform SDN
• Both VMs and Containers on one VLAN
• Name Resolution
• Contributing to Weave DNS for orchestration
• Use traditional external BIND service entity

@grkvltCopyright 2015 by CloudsoftCorporationLimited
Cross-Target Deployment

@grkvlt
Virtual Network Entity
• Clocker Specific Entity
• Provisions a Virtual Network or Subnet
• Looks for NetworkProvisioningExtension
• Registered in DockerLocation by SDN Provider
• Also defined with JcloudsLocation for OpenStack
• Part of Application Blueprint

@grkvlt
Network Provisioning
• Create Named Virtual Networks
• Uses Available Provider
• Currently Weave, Calico and OpenStack Neutron Supported
-‐ type: brooklyn.networking.VirtualNetwork
networkId: my-‐application
cidr: 192.168.12.0/24
gateway: 192.168.12.1
dnsServers:
-‐ 8.8.8.8
brooklyn.config:
sdn.example.securityGroup: "my-‐security-‐group"

@grkvlt
• Allow Minimal (Zero!) Configuration
• Use Sensible Defaults
• Also SDN or Cloud Specific Configuration, e.g. Calico or
OpenStack
• Allocate Address Space on Demand
-‐ type: brooklyn.networking.OpenStackVirtualNetwork
networkId: database-‐net
cidr: 192.168.34.0/24
-‐ type: brooklyn.networking.VirtualNetwork
networkId: couchbase-‐net

@grkvlt
• Attach Containers to Networks
brooklyn.config:
network.list:
-‐ couchbase-‐net
-‐ management-‐net
• Create Networks as Required
• Also Attach to VMs and Metal
• Only supported with Calico at present

Clocker Console 1

Clocker Console 2

@grkvlt
• Orchestrated Docker 1.7.0 deployment with
Calico SDN integration
• Automated attachment of containers to
multiple dynamic networks
• Brooklyn application blueprints with network
topology
Features

Clocker
1.0.0Clocker
1.0.0

@grkvlt
• Latest Docker 1.7.x Feature Support
• Stats command
• Read-only containers
• Named Dockerfiles
• Swarm
• Native API for access to managed Docker Cloud
• Clocker as an extension to Swarm using Powerstrip
• Docker Registry
New Features

@grkvlt
• Lightweight Images
• No SSH server
• Minimal Alpine base distribution
• Getting Started
• Dockerfile and images on Docker Hub
• Vagrantfile for local or remote VMs
New Features

@grkvlt
New Features
• Multi Region
• Use Swarm and labels to decide where to run
• More than one Cloud provider (modulo SDN)
• Better Native Docker Support
• No more SSH per container
• Minimize images with Alpine
• Cross-target Applications
• Virtual Machine and Container services

@grkvlt
Roadmap
• Mesos Integration
• Deploy the Mesos Infrastructure
• Provide Mesos as another Brooklyn endpoint
• Container Mobility
• Stateless Services
• Defined in Blueprint
• Brooklyn Core Integration

@grkvlt
Roadmap

@grkvlt
Roadmap
• Docker Experimental Features
• Separate release for now
• Not yet stable
• More networking providers
• Integrated Flocker plugin
• Clocker Experimental Release
• Coming soon...

@grkvlt
Roadmap
• Docker Enterprise Hub
• Logging with ELK
• Application Definition
• Docker Compose via Application Definition WG
• Networking
• Weave DNS Integration
• Kubernetes Support

Solves:
– Composite Application Management
– Docker Cloud Networking
– Container Placement and
Provisioning
Solves:
– Composite Application Management
– Docker Cloud Networking
– Container Placement and
Provisioning
SummarySummary

@grkvlt
Resources
http://clocker.io/
http://brooklyn.io/
https://github.com/brooklyncentral/clocker/
https://github.com/apache/incubator-‐brooklyn/
https://github.com/weaveworks/weave/
https://github.com/Metaswitch/calico-‐docker/
http://blog.abstractvisitorpattern.co.uk/

Bringing Docker to the Cloud

More Related Content

What's hot

Viewers also liked

Similar to Bringing Docker to the Cloud

More from Andrew Kennedy

Recently uploaded

Bringing Docker to the Cloud