Container
Microservices:
A Holistic Look at
Managing Containers for
Microservices
© Copyright 2015 Pivotal. All rights reserved.
Overview
• Why Containers?
• Exploring Docker and Cloud Foundry
• Docker and Cloud Foundry Workflow
• Diego Cell and Brain
Why Containers?
© Copyright 2015 Pivotal. All rights reserved.
Why Containers?
© Copyright 2015 Pivotal. All rights reserved.
• a psychological condition of an operator who has suffered (usually
persistent) emotional, or psychological abuse from a problematic CI
system.
Why Containers?
Battered Continuous Delivery
Ops Syndrome
© Copyright 2015 Pivotal. All rights reserved.
Environment Parity
- run tasks locally before
committing
Configurable
- declarative CI (no more
snowflakes)
- store in version control
Unpolluted builds
- previous builds do not
affect subsequent builds
Usability
- visualize pipeline
- simple ui (click less!)
Scalability
- scale up to increase
performance
- scale down to decrease
cost
Example: What if we could...
Exploring
Container
Platforms
© Copyright 2015 Pivotal. All rights reserved.
Swarm
Router
Cloud Controller
BOSH Director BOSH Agent
UAA/Login Servers
Messaging (NATS)
IaaS
Cloud Foundry Elastic Runtime Stack
Router
Cloud Controller
BOSH Director BOSH Agent
UAA/Login Servers
Messaging (NATS)
IaaS
Cloud Foundry Elastic Runtime Stack
Container Cells
Doppler (Metrics & Logs)
Router
Cloud Controller
BOSH Director BOSH Agent
UAA/Login Servers
Messaging (NATS)
IaaS
Cloud Foundry Elastic Runtime Stack
Container Cells
Doppler (Metrics & Logs)
Router
Cloud Controller
BOSH Director BOSH Agent
UAA/Login Servers Brain
Messaging (NATS)
IaaS
Cloud Foundry Elastic Runtime Stack
Container Cells
© Copyright 2015 Pivotal. All rights reserved.
The Container: Runtime vs No-Runtime
Platform buildpacks
provide standard runtime
Platform provides fixed
OS container image
Developer brings
customized app
Developer brings runtime
container image
Developer brings
container OS image
Developer brings
customized app
Platform provides fixed
host OS Kernel
Platform provides fixed
host OS Kernel
App container
© Copyright 2015 Pivotal. All rights reserved.
Back-end Container Stuff
Container Orchestrator Container Engine
docker/docker-engine runC*
Kubernetes/kublet docker-engine -> runC*
Cloudfoundry/garden-linux custom code
Cloudfoundry/garden-runc runC*
*runC – Open Container Initiative reference implementation
© Copyright 2015 Pivotal. All rights reserved.
CF Root Filesystem
cflinuxfs2
Droplet
Docker
Write layer
• Layered file system in the
container namespace
• RootFS changed by pivot root
• RootFS can be either
cflinuxfs2 or from a Docker
image
• Buildpacks are the Droplet
added to the write layer
• Write layer is ephemeral
Write layer
BuildPack Docker
© Copyright 2015 Pivotal. All rights reserved.
CF Buildpacks Provide Runtime
© Copyright 2015 Pivotal. All rights reserved.
Concourse.ci Pipelines: Docker-Native CI/CD
© Copyright 2015 Pivotal. All rights reserved.
Finding The Runtime
© Copyright 2015 Pivotal. All rights reserved.
Figuring Out How to Add My Application
© Copyright 2015 Pivotal. All rights reserved.
Creating The Dockerfile
© Copyright 2015 Pivotal. All rights reserved.
Running the Image
• Cloud Foundry
– cf push
• Docker:
– docker build
– docker push
– docker run
© Copyright 2015 Pivotal. All rights reserved.
Orchestrating PCF Demo Application
© Copyright 2015 Pivotal. All rights reserved.
Cloud Foundry Manifest
© Copyright 2015 Pivotal. All rights reserved.
Docker Compose (On Docker-Machine)
© Copyright 2015 Pivotal. All rights reserved.
Kubernetes
© Copyright 2015 Pivotal. All rights reserved.
Deploying the Application
• Cloud Foundry
– cf create-service
– cf push
• Docker
– docker-compose up
• Kubernetes
– kubectl create -f ./pcfdemo.yml
– kubectl create –f ./pcfdemo-service.yml
– kubectl create –f ./rabbitmq.yml
– kubectl create –f ./rabbitmq-service.yml
© Copyright 2015 Pivotal. All rights reserved.
Ease of Platform Management
• Installing/Deploying Platform
• Monitoring
• Configuration
• Upgrades
© Copyright 2015 Pivotal. All rights reserved.
BOSH
• Cloud Foundry is a distributed
system that runs on many
infrastructures
• BOSH was mainly designed as
a holistic approach to
deploying and managing
Cloud Foundry-managed VMs
– BOSH keeps the Cloud
Foundry runtime running
...
BOSH
Cloud Foundry VMs
Infrastructure
© Copyright 2015 Pivotal. All rights reserved.
BOSH Architecture
© Copyright 2015 Pivotal. All rights reserved.
Deploying New Services
• Managing Credentials
• Connection Configuration and Metadata
• Ease of Adding New Capabilities
• Command Line Access
• Accessible API
© Copyright 2015 Pivotal. All rights reserved.
Service Broker API
© Copyright 2015 Pivotal. All rights reserved.
Architectural Diagram of Services
© Copyright 2015 Pivotal. All rights reserved.
Examples of Services
• MySQL, RabbitMQ, Redis, and Kafka
• Spring Cloud Services: (i.e. Netflix OSS)
– Eureka – Service Registry
– Hysterix – Circuit Breaker
– ConfigServer – Configuration Service
• Route Services (contributed by GE Predix)
– Rate Limiting
– Authentication
– Caching Services
© Copyright 2015 Pivotal. All rights reserved.
Logging and Metrics: Two Perspectives
• Application Developers
–Access to logs with logging a ticket
–Self-service access for diagnostics
• Operations
–Understanding the health of the Platform
–Insights for Analysis and Capacity Planning
© Copyright 2015 Pivotal. All rights reserved.
Application Dashboard
• Display current
app status,
scale, and route
• Streaming Data
– last five
minutes of
container and
network
metrics
– most recent
app events
© Copyright 2015 Pivotal. All rights reserved.
Container Metrics Detail
• View app events
in context with
streaming:
– CPU
– Disk
– Memory
• Viewable by day,
hour and minute
© Copyright 2015 Pivotal. All rights reserved.
Network Metrics Detail
• View app events
in context with
streaming:
– HTTP requests
– HTTP errors
– Latency
• Viewable by day,
hour and minute
© Copyright 2015 Pivotal. All rights reserved.
Loggregator
• Applications write logs
to STDOUT and
STDERR
• Relevant Cloud
Foundry component
events added to
application stream
• Log events collected,
collated, and expose
via API and the
Firehose
Developer
Workflow
© Copyright 2015 Pivotal. All rights reserved.
Focus of Cloud Foundry
• Increase Developer Productivity
• Speed Application Development
• Shorten Feedback Loop
• Product Development Focused on Large
Enterprises
© Copyright 2015 Pivotal. All rights reserved.
PCF Workflow
• Docker and PCF Workflow
• PCF (Pivotal Cloud Foundry) Workflow
© Copyright 2015 Pivotal. All rights reserved.
Initial Docker and PCF Workflow
Select base find/
images
Build Artifact Create Dockerfile
Create Manifest
cf create-service
Build Docker
Image
cf push Check App/Service
Push Docker
Image to Registry
© Copyright 2015 Pivotal. All rights reserved.
Subsequent Docker/PCF Workflow
Build Artifact
Rebuild Docker
Image
cf restage
Push Docker
Image to Registry
Check App/Service
© Copyright 2015 Pivotal. All rights reserved.
Initial Cloud Foundry Workflow
cf create-service Build Artifact
Create Manifestcf push
Check Service/App
© Copyright 2015 Pivotal. All rights reserved.
Subsequent Cloud Foundry Workflow
Build Artifact cf push
Check App/Service
© Copyright 2015 Pivotal. All rights reserved.
Services of PCF
• Container and Runtime Management
– Native: Complete Container and Runtime
– Docker: User brings own Runtime
• Orchestration
• Logging and Metrics
• Services
– Databases
– Message Queues
• Routing
– Routing Services
© Copyright 2015 Pivotal. All rights reserved.
Diego Brain And Cell
• Rep
– Auction
– Commands to Executor
• Executor
– Manages the Container
– Decoupled from Container Format
• Garden
– Interface that defines Container
– Garden-Linux
– Garden-runC*
– Can be Garden-Windows
• Metron
– Manages Logging and Metrics
© Copyright 2015 Pivotal. All rights reserved.
Garden
allows Diego to programmatically say
“make me a container” “put this in it” “then run this”
via a platform-agnostic API
© Copyright 2015 Pivotal. All rights reserved.
Garden
allows Diego’s abstractions to be flexible
© Copyright 2015 Pivotal. All rights reserved.
The Open Container Initiative
© Copyright 2015 Pivotal. All rights reserved.
Values Guiding the OCI Specification
• Composable - Container format not bound to runtime or
clients
• Portable – Usable across different HW, OS, Clouds etc
• Decentralized – Standard format for referencing images
• Minimalist – Spec should focus on doing a few things well
• Backward Compatible – Strive to be as backward
compatible as possible with Docker container format
Thank You:
Stuart Charlton
@svrc
With thanks to
@jyidiego for much
of this content

Cloud Foundry Vancouver Meetup July 2016

  • 1.
    Container Microservices: A Holistic Lookat Managing Containers for Microservices
  • 2.
    © Copyright 2015Pivotal. All rights reserved. Overview • Why Containers? • Exploring Docker and Cloud Foundry • Docker and Cloud Foundry Workflow • Diego Cell and Brain
  • 3.
  • 4.
    © Copyright 2015Pivotal. All rights reserved. Why Containers?
  • 5.
    © Copyright 2015Pivotal. All rights reserved. • a psychological condition of an operator who has suffered (usually persistent) emotional, or psychological abuse from a problematic CI system. Why Containers? Battered Continuous Delivery Ops Syndrome
  • 6.
    © Copyright 2015Pivotal. All rights reserved. Environment Parity - run tasks locally before committing Configurable - declarative CI (no more snowflakes) - store in version control Unpolluted builds - previous builds do not affect subsequent builds Usability - visualize pipeline - simple ui (click less!) Scalability - scale up to increase performance - scale down to decrease cost Example: What if we could...
  • 7.
  • 8.
    © Copyright 2015Pivotal. All rights reserved. Swarm
  • 9.
    Router Cloud Controller BOSH DirectorBOSH Agent UAA/Login Servers Messaging (NATS) IaaS Cloud Foundry Elastic Runtime Stack
  • 10.
    Router Cloud Controller BOSH DirectorBOSH Agent UAA/Login Servers Messaging (NATS) IaaS Cloud Foundry Elastic Runtime Stack Container Cells
  • 11.
    Doppler (Metrics &Logs) Router Cloud Controller BOSH Director BOSH Agent UAA/Login Servers Messaging (NATS) IaaS Cloud Foundry Elastic Runtime Stack Container Cells
  • 12.
    Doppler (Metrics &Logs) Router Cloud Controller BOSH Director BOSH Agent UAA/Login Servers Brain Messaging (NATS) IaaS Cloud Foundry Elastic Runtime Stack Container Cells
  • 13.
    © Copyright 2015Pivotal. All rights reserved. The Container: Runtime vs No-Runtime Platform buildpacks provide standard runtime Platform provides fixed OS container image Developer brings customized app Developer brings runtime container image Developer brings container OS image Developer brings customized app Platform provides fixed host OS Kernel Platform provides fixed host OS Kernel App container
  • 14.
    © Copyright 2015Pivotal. All rights reserved. Back-end Container Stuff Container Orchestrator Container Engine docker/docker-engine runC* Kubernetes/kublet docker-engine -> runC* Cloudfoundry/garden-linux custom code Cloudfoundry/garden-runc runC* *runC – Open Container Initiative reference implementation
  • 15.
    © Copyright 2015Pivotal. All rights reserved. CF Root Filesystem cflinuxfs2 Droplet Docker Write layer • Layered file system in the container namespace • RootFS changed by pivot root • RootFS can be either cflinuxfs2 or from a Docker image • Buildpacks are the Droplet added to the write layer • Write layer is ephemeral Write layer BuildPack Docker
  • 16.
    © Copyright 2015Pivotal. All rights reserved. CF Buildpacks Provide Runtime
  • 17.
    © Copyright 2015Pivotal. All rights reserved. Concourse.ci Pipelines: Docker-Native CI/CD
  • 18.
    © Copyright 2015Pivotal. All rights reserved. Finding The Runtime
  • 19.
    © Copyright 2015Pivotal. All rights reserved. Figuring Out How to Add My Application
  • 20.
    © Copyright 2015Pivotal. All rights reserved. Creating The Dockerfile
  • 21.
    © Copyright 2015Pivotal. All rights reserved. Running the Image • Cloud Foundry – cf push • Docker: – docker build – docker push – docker run
  • 22.
    © Copyright 2015Pivotal. All rights reserved. Orchestrating PCF Demo Application
  • 23.
    © Copyright 2015Pivotal. All rights reserved. Cloud Foundry Manifest
  • 24.
    © Copyright 2015Pivotal. All rights reserved. Docker Compose (On Docker-Machine)
  • 25.
    © Copyright 2015Pivotal. All rights reserved. Kubernetes
  • 26.
    © Copyright 2015Pivotal. All rights reserved. Deploying the Application • Cloud Foundry – cf create-service – cf push • Docker – docker-compose up • Kubernetes – kubectl create -f ./pcfdemo.yml – kubectl create –f ./pcfdemo-service.yml – kubectl create –f ./rabbitmq.yml – kubectl create –f ./rabbitmq-service.yml
  • 27.
    © Copyright 2015Pivotal. All rights reserved. Ease of Platform Management • Installing/Deploying Platform • Monitoring • Configuration • Upgrades
  • 28.
    © Copyright 2015Pivotal. All rights reserved. BOSH • Cloud Foundry is a distributed system that runs on many infrastructures • BOSH was mainly designed as a holistic approach to deploying and managing Cloud Foundry-managed VMs – BOSH keeps the Cloud Foundry runtime running ... BOSH Cloud Foundry VMs Infrastructure
  • 29.
    © Copyright 2015Pivotal. All rights reserved. BOSH Architecture
  • 30.
    © Copyright 2015Pivotal. All rights reserved. Deploying New Services • Managing Credentials • Connection Configuration and Metadata • Ease of Adding New Capabilities • Command Line Access • Accessible API
  • 31.
    © Copyright 2015Pivotal. All rights reserved. Service Broker API
  • 32.
    © Copyright 2015Pivotal. All rights reserved. Architectural Diagram of Services
  • 33.
    © Copyright 2015Pivotal. All rights reserved. Examples of Services • MySQL, RabbitMQ, Redis, and Kafka • Spring Cloud Services: (i.e. Netflix OSS) – Eureka – Service Registry – Hysterix – Circuit Breaker – ConfigServer – Configuration Service • Route Services (contributed by GE Predix) – Rate Limiting – Authentication – Caching Services
  • 34.
    © Copyright 2015Pivotal. All rights reserved. Logging and Metrics: Two Perspectives • Application Developers –Access to logs with logging a ticket –Self-service access for diagnostics • Operations –Understanding the health of the Platform –Insights for Analysis and Capacity Planning
  • 35.
    © Copyright 2015Pivotal. All rights reserved. Application Dashboard • Display current app status, scale, and route • Streaming Data – last five minutes of container and network metrics – most recent app events
  • 36.
    © Copyright 2015Pivotal. All rights reserved. Container Metrics Detail • View app events in context with streaming: – CPU – Disk – Memory • Viewable by day, hour and minute
  • 37.
    © Copyright 2015Pivotal. All rights reserved. Network Metrics Detail • View app events in context with streaming: – HTTP requests – HTTP errors – Latency • Viewable by day, hour and minute
  • 38.
    © Copyright 2015Pivotal. All rights reserved. Loggregator • Applications write logs to STDOUT and STDERR • Relevant Cloud Foundry component events added to application stream • Log events collected, collated, and expose via API and the Firehose
  • 39.
  • 40.
    © Copyright 2015Pivotal. All rights reserved. Focus of Cloud Foundry • Increase Developer Productivity • Speed Application Development • Shorten Feedback Loop • Product Development Focused on Large Enterprises
  • 41.
    © Copyright 2015Pivotal. All rights reserved. PCF Workflow • Docker and PCF Workflow • PCF (Pivotal Cloud Foundry) Workflow
  • 42.
    © Copyright 2015Pivotal. All rights reserved. Initial Docker and PCF Workflow Select base find/ images Build Artifact Create Dockerfile Create Manifest cf create-service Build Docker Image cf push Check App/Service Push Docker Image to Registry
  • 43.
    © Copyright 2015Pivotal. All rights reserved. Subsequent Docker/PCF Workflow Build Artifact Rebuild Docker Image cf restage Push Docker Image to Registry Check App/Service
  • 44.
    © Copyright 2015Pivotal. All rights reserved. Initial Cloud Foundry Workflow cf create-service Build Artifact Create Manifestcf push Check Service/App
  • 45.
    © Copyright 2015Pivotal. All rights reserved. Subsequent Cloud Foundry Workflow Build Artifact cf push Check App/Service
  • 46.
    © Copyright 2015Pivotal. All rights reserved. Services of PCF • Container and Runtime Management – Native: Complete Container and Runtime – Docker: User brings own Runtime • Orchestration • Logging and Metrics • Services – Databases – Message Queues • Routing – Routing Services
  • 47.
    © Copyright 2015Pivotal. All rights reserved. Diego Brain And Cell • Rep – Auction – Commands to Executor • Executor – Manages the Container – Decoupled from Container Format • Garden – Interface that defines Container – Garden-Linux – Garden-runC* – Can be Garden-Windows • Metron – Manages Logging and Metrics
  • 48.
    © Copyright 2015Pivotal. All rights reserved. Garden allows Diego to programmatically say “make me a container” “put this in it” “then run this” via a platform-agnostic API
  • 49.
    © Copyright 2015Pivotal. All rights reserved. Garden allows Diego’s abstractions to be flexible
  • 50.
    © Copyright 2015Pivotal. All rights reserved. The Open Container Initiative
  • 51.
    © Copyright 2015Pivotal. All rights reserved. Values Guiding the OCI Specification • Composable - Container format not bound to runtime or clients • Portable – Usable across different HW, OS, Clouds etc • Decentralized – Standard format for referencing images • Minimalist – Spec should focus on doing a few things well • Backward Compatible – Strive to be as backward compatible as possible with Docker container format
  • 52.
    Thank You: Stuart Charlton @svrc Withthanks to @jyidiego for much of this content