Effective DevOps by using Docker and Chef together !

WhiteHedge
@thewhitehedgeinfo@WhiteHedge.comWhiteHedge.com/docker-microservices/
Baking Docker using
Chef
WhiteHedge Technologies

WhiteHedge
WhiteHedge
2
TABLE OF CONTENTS - AGENDA
Welcome
About WhiteHedge
and Me
1
Docker
A quick introduction
2
Chef + Docker
Getting best of both worlds
3
Push Jobs
5
Chef Cookbook
6
Chef Containers
7
Our Story
8
CD Pipeline
Use of knife + ssh
4

WhiteHedge
WhiteHedge
3
- HELLO
Shorten the path between Innovation and Value
WhiteHedge is an agile Product Development
company with deep experience in the Cloud
Automation, DevOps and Big Data Analytics.
Experience of building 50+ successful products across
the globe in various industries and the knowledge of
latest trends and technologies, make us stand out as
the best and the coolest software product
development company.
INTRODUCTION

WhiteHedge
WhiteHedge
4
WHITEHEDGE - AN OVERVIEW
Global Presence
Agile + Flexible
Thorough + Quick Learner
Competitive + Comprehensive
Honest + Transparent
Young + Mature
Innovative + Creative
More about us …
What defines us ?
California
New Jersey Rotterdam
Pune
EnvisionProducts|
Convertintobusinesses
100+employees|50+live
productsworld-wide
ThebestoftheTalentand
Infrastructure
Started2003|FocusedAgile
ProductDevelopment
Selffunded|Wellfunded|
Profitable

WhiteHedge
Docker
A Quick Introduction

WhiteHedge
What is Docker?
Linux Container
3 Components:
Docker Engine
Docker Hub
Docker Images
Benefits:
Speed
Portability
Density
Open Source
sufficient containers from any

WhiteHedge
Docker is not a VM
Virtual Machine Docker

WhiteHedge
FROM ubuntu:14.04
RUN apt-get update
RUN apt-get install
libfuse-dev
ADD dev.conf/etc/myapp-
config/
Dockerfiles
oCodify your configuration
oSet of bash commands
oExample:
• HelloScala
 Dockerfile
 dev.conf
• Docker build HelloScala

WhiteHedge
Use Cases of Docker
o Shared Hosting PaaS
o Microservices
o Lightweight Testing

WhiteHedge
Chef and Docker
Getting the best of both worlds!

WhiteHedge
THE CHALLENGE
Automate
Make Whole
Enchilada Deliver!

WhiteHedge
Config Management Vs Golden Images
o Control the environment Vs System Image / Runtime image
o Tradeoff between flexibility and manageability
o CM is the vein of DevOps
• Shell scripts -> Chef
o Immutable Infrastructure

WhiteHedge
Docker
Chef
Awesomeness

WhiteHedge
Chef and Docker
Replaces Human Tasks,
Idempotence, Thick
client - thin servers,
Order Matters, Huge
Community Support
An improved Robot,
Fast, Easy, Fresh fish
in the market, ready
to be baked!

WhiteHedge
Simple CD Pipeline
Because simple things can bring the
most happiness!

WhiteHedge
Simple CI/CD Pipeline
Deploy using
knife-ssh or
Push Jobs
docker pull
docker stop
docker run
Docker
Registry
Unique tag
Docker Image
Save image
Build Process
Build tools have
docker support
Build tools generate
a docker image
Code
git push
Triggers Build
CI Server

WhiteHedge
The Simple Steps
o git push to https://github.com/WHDevOpsDev/HelloScala
o Triggers a build on your CI server
• sbt docker
• docker push WHDevOpsDev/hello-scala
• knife ssh 'role:test' 'deploy.sh' -x ssh-user -i ssh-key -c knife.rb
o Build tools offer docker integration
o Eg: Maven has docker-maven-plugin
• https://github.com/spotify/docker-maven-plugin
• mvn clean package docker:build -DpushImage

WhiteHedge
~/github/HelloScala > sbt docker
[info] Loading project definition from
/Users/WHDevOpsDevphale/github/HelloScala/project
[info] Set current project to hello-scala (in build
file:/Users/WHDevOpsDevphale/github/HelloScala/)
[info] Creating docker image with name: 'WHDevOpsDev/hello-
scala'
:
[info] Sending build context to Docker daemon
[info] Step 0 : FROM dockerfile/java
[info] ---> 1126c85d8a06
[info] Step 1 : ADD /app/hello-scala_2.11-1.4-one-jar.jar /app/hello-
scala_2.11-1.4-one-jar.jar
[info] ---> Using cache
[info] ---> 61871958f108
[info] Step 2 : ENTRYPOINT java -jar /app/hello-scala_2.11-1.4-
one-jar.jar
[info] ---> Using cache
[info] ---> a8005b32ddc4
[info] Successfully built a8005b32ddc4
[info] Successfully built Docker image: WHDevOpsDev/hello-scala
[success] Total time: 1 s, completed Mar 3, 2015 2:10:04 PM
~/github/HelloScala > docker images | grep hello-scala
WHDevOpsDev/hello-scala latest a8005b32ddc4 12 hours
ago 715 MB
~/github/HelloScala > docker run WHDevOpsDev/hello-scala
Hello, world! #1
Hello, world! #2
Hello, world! #3

WhiteHedge
Docker Registry
Docker Hub
Link:
https://registry.hub.docker.com
/u/WHDevOpsDev/hello-scala
Automated Build in Docker:
https://registry.hub.docker.com
/u/WHDevOpsDev/helloscala-
automated-build/

WhiteHedge
Push Jobs
Do you need to push harder?

WhiteHedge
Push Jobs
o Knife-ssh
o Journey from pull to push
o
run against nodes independently of a chef-
o Job: set of commands to be run on node
• Docker pull
• Docker stop
• Docker run

WhiteHedge
Push Jobs
oUse message bus (zeromq)
oClaims to attack the
scalability issue
oDeployment status is relayed
back
oNew born baby
oComplex at the moment,
ready with just the basic
foundation
Knife SSH
oParallel ssh
oSSH Protocol is slow and
CPU hungry at scale
oFeedback on deployment
status is not as easy
oBeen in the market for long
oEasy to use
How are Push Jobs different from knife-ssh?

WhiteHedge
Chef Push Jobs Server
oEnterprise Chef 11 or Chef server 12
oStandalone or HA
oRun the commands on Chef Server:
• chef-server-ctl install opscode-push-jobs-server
• opscode-push-jobs-server-ctl reconfigure
• chef-server-ctl reconfigure

WhiteHedge
Setup Workstation
o Install knife push plugin
• Gem install knife-jobs
o Knife cookbook site download push-jobs
o Extract and save to your cookbook path
o Edit the attributes file (push-jobs/attributes/default.rb)
• default['push_jobs']['package_url'] = 'https://opscode-private-
chef.s3.amazonaws.com/ubuntu/12.04/x86_64/opscode-push-jobs-client_1.1.5-
1_amd64.deb'
• default['push_jobs']['package_checksum'] =
o Upload the push-jobs cookbook to your ChefServer

WhiteHedge
Create Groups & Setup Node
o Create 2 groups
• Pushy_job_writers
• Pushy_job_readers
o Add user to the groups
o Sudo chef-client -
o From Workstation:
• Knife node status
• Knife node status <node-name>

WhiteHedge
Run
o -client r recipe[run-docker -name>
o my_node
o Where docker.sh:
• Docker pull WHDevOpsDev/hello-scala
• docker ps | grep WHDevOpsDev/hello-scala| awk -
• Docker run WHDevOpsDev/hello-scala

WhiteHedge
When Reality Strikes…
If only applications were Hello World programs!

WhiteHedge
Docker Image
Application Configuration Docker Image

WhiteHedge
What is Configuration?
Packages Custom SetupsCredentials
Softwares Database
Files
Environment
Specific
Configuration
Ports

WhiteHedge
ENVIRONMENTS
DEV
Docker
Container
Docker
Container
Docker
Container
PRE
PROD
Docker
Container
Docker
Container
Docker
Container
PROD
Docker
Container
Docker
Container
Docker
Container

WhiteHedge
Secure Credential Management
oUnsolved problem with Docker today
oCredentials inside docker containers
•Hard codes
•Set environment variables

WhiteHedge
Workaround?
Create Base Image Manually, with
configuration embedded
Build Tool uses the custom Base Image
Deploy using knife-ssh

WhiteHedge
Docker Chef Cookbook
To manage docker images and deployment

WhiteHedge
Docker Cookbook
o Available in Supermarket: https://supermarket.chef.io/cookbooks/docker
o Install docker
o Build docker image
o Pull image and run container
o Push docker image to registry
o LWRPs
• Docker_container
• Docker_image
• Docker_registry
o https://github.com/bflad/chef-docker/blob/master/README.md

WhiteHedge
Credential Management
secret = Chef::EncryptedDataBagItem.load_secret
@docker_cred = Chef::EncryptedDataBagItem.load(
node['docker']['creds']['databag'],
node['docker']['user'],
secret
)
docker_registry WHDevOpsDev/hello-scala
email docker_cred['email']
username docker_cred['username']
password docker_cred['password']
end

WhiteHedge
Docker_image
# Build a docker image using docker_image resource
docker_image node['docker']['image'] do
tag node['docker']['image']['tag']
source '/var/docker'
action :build
end
# Push the image to docker registery
action :push
end
# Delete the image from the machine
action :remove
end

WhiteHedge
Docker_container
# Run Container
docker_container WHDevOpsDev/hello-scala
detach true
env -
mnt/docker/docker-
action :run
end

WhiteHedge
GENERATE DOCKERFILE
# Generate a docker file using template.
template "#{node['docker']['directory']}/Dockerfile" do
source 'dockerfile.erb'
variables image: node['docker']['base']['image']['name'],
maintainer: @docker_cred['maintainer'],
email: docker_cred['email'],
build_cmd: node['docker']['build']['commands'],
entry_point: node['docker']['build']['entry_point']
action :create
end

WhiteHedge
WORKFLOW
Build
Application
• Save the Artifact to a Repository Manager
Build Docker
Image
• Docker cookbook would build and save the docker image
Deploy
• Docker cookbook runs the container
on the nodes

WhiteHedge
Chef Containers
Contains Awesome.

WhiteHedge
What is a Chef Container?
oPackage
oProvides Configuration Management for
containers

WhiteHedge
Chef Container Components
chef-client
runit
chef-init

WhiteHedge
Why Chef Containers?
oBootstrap chef-client without SSH
connection
oManage multiple services inside your
container
oManage running state of your container
oConsistency across Architectures
oMixed Architecture Applications

WhiteHedge
Best Suited For
oTransitioning traditional architecture to containers
oHandling last mile configuration when container boots
oGetting the best of two worlds without complexity

WhiteHedge
Knife container docker init
oGem install knife-container
oknife container docker init NAMESPACE/IMAGE_NAME
[options]
• -f base docker image (default is ubuntu 12.04) - chef container should
be already installed on it
• -r runlist
• -z chef client local mode
• -b use berkshelf

WhiteHedge
EXample
$ sudo knife container docker init WHDevOpsDev/hello-scala-cc
Compiling Cookbooks...
Recipe: knife_container::docker_init
* directory[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc] action create
* template[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/Dockerfile] action
create
- update content in file /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-
cc/Dockerfile from none to 943017
- * template[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/.dockerignore]
action create
- create new file /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/.dockerignore
cc/.dockerignore from none to e3b0c4
* directory[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef] action create
- create new directory /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef
* template[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef/client.rb] action
create
- create new file /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef/client.rb
cc/chef/client.rb from none to 7de61f
* file[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef/first-boot.json] action
create
- create new file /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef/first-
boot.json
cc/chef/first-boot.json from none to 5269ef
* template[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef/.node_name]
action create
- create new file /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-
cc/chef/.node_name
cc/chef/.node_name from none to 4764d2
* template[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/Berksfile] action
create (skipped due to only_if)
* directory[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc/chef/secure] action
create
- create new directory /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-
cc/chef/secure
* file[/home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-
cc/chef/secure/validation.pem] action create
- create new file /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-
cc/chef/secure/validation.pem
cc/chef/secure/validation.pem from none to ec1f3e
- change mode from '' to '0600'
Downloading base image: chef/ubuntu-12.04:latest. This process may take awhile...
Tagging base image chef/ubuntu-12.04 as WHDevOpsDev/hello-scala-cc
Context Created: /home/ubuntu/chef-repo/dockerfiles/WHDevOpsDev/hello-scala-cc

WhiteHedge
Knife container docker build
orun command docker images
oknife container docker build
• resolve docker dependencies
• build docker image
• cleanup chef artifacts

WhiteHedge
EXAMPLE
$ sudo knife container docker build WHDevOpsDev/hello-scala-cc
Sending build context to Docker daemon 9.728 kB
Sending build context to Docker daemon
Step 0 : FROM WHDevOpsDev/hello-scala-cc
---> 50d3c5c9e133
Step 1 : ADD chef/ /etc/chef/
---> 4933cc9e13e0
Removing intermediate container da0a08413a91
Step 2 : RUN chef-init --bootstrap
---> Running in add27db609cc
[2015-03-31T21:44:44+00:00] INFO: Starting Supervisor...
[2015-03-31T21:44:44+00:00] INFO: Supervisor pid: 9
[2015-03-31T21:44:49+00:00] INFO: Starting chef-client run...
[2015-03-31T21:44:50+00:00] INFO: Forking chef instance to converge...
[2015-03-31T21:44:50+00:00] INFO: *** Chef 11.16.2 ***
[2015-03-31T21:44:50+00:00] INFO: Chef-client pid: 16
[2015-03-31T21:44:53+00:00] INFO: Client key /etc/chef/secure/client.pem is not present - registering
[2015-03-31T21:44:53+00:00] INFO: HTTP Request Returned 404 Object Not Found: error
[2015-03-31T21:44:54+00:00] INFO: Setting the run_list to [] from CLI options
[2015-03-31T21:44:54+00:00] INFO: Run List is []
[2015-03-31T21:44:54+00:00] INFO: Run List expands to []
[2015-03-31T21:44:54+00:00] INFO: Starting Chef Run for WHDevOpsDev-hello-scala-cc-build
[2015-03-31T21:44:54+00:00] INFO: Running start handlers
[2015-03-31T21:44:54+00:00] INFO: Start handlers complete.
[2015-03-31T21:44:55+00:00] INFO: Loading cookbooks []
[2015-03-31T21:44:55+00:00] WARN: Node WHDevOpsDev-hello-scala-cc-build has an empty run list.
[2015-03-31T21:44:55+00:00] INFO: Chef Run complete in 1.121705004 seconds
[2015-03-31T21:44:55+00:00] INFO: Running report handlers
[2015-03-31T21:44:55+00:00] INFO: Report handlers complete
[2015-03-31T21:44:55+00:00] INFO: Sending resource update report (run-id: 6f637baf-18cc-4620-b3e2-
9afc90e8cd6b)
---> 2c2ec6fab1ef
Removing intermediate container add27db609cc
Step 3 : RUN rm -rf /etc/chef/secure/*
---> Running in 30a3611b083f
---> cab28d6eed90
Removing intermediate container 30a3611b083f
Step 4 : ENTRYPOINT ["chef-init"]
---> Running in 0a9f4e96bbf7
---> a8577b66b103
Removing intermediate container 0a9f4e96bbf7
Step 5 : CMD ["--onboot"]
---> Running in f9a444817229
---> 21b3800bc9b3
Removing intermediate container f9a444817229
Successfully built 21b3800bc9b3

WhiteHedge
Docker images
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
WHDevOpsDev/hello-scala-cc latest 21b3800bc9b3 2 hours ago 311.9 MB
<none> <none> b343c8301cc8 2 hours ago 311.9 MB
chef/ubuntu-12.04 latest 50d3c5c9e133 6 months ago 311.9 MB
$ sudo docker push WHDevOpsDev/hello-scala-cc
$ sudo docker d run WHDevOpsDev/hello-scala-cc

WhiteHedge
Our Story
Product under Development. Super Cool DevOps Culture.

WhiteHedge
Lessons Learnt
oRunning apps in containers is easy
oDebugging apps in containers is difficult
oYou can very well run multiple services inside a docker
container
oAh the woes of Docker networking!
oSequential Progression
o

WhiteHedge
WhiteHedgeFOLLOW US
52
Questions?
http://www.whitehedge.com/docker-microservices/
info@whitehedge.com

WhiteHedge
THANK YOU!
Have a Nice Day!

Effective DevOps by using Docker and Chef together !

More Related Content

What's hot

Viewers also liked

Similar to Effective DevOps by using Docker and Chef together !

More from WhiteHedge Technologies Inc.

Recently uploaded

Effective DevOps by using Docker and Chef together !