ACHIEVING CONTINUOUS
DELIVERY:
AN AUTOMATION STORY
ABOUT ME
JAMES CAMMARATA, DIRECTOR - CORE ENGINEERING
Python developer since 2003
Started contributingto Cobbler in September of 2008, and took
over the projectleadership in 2010
Joined Ansible in July, 2013
WHAT IS CONTINUOUS
DELIVERY?
Continuous deliveryaims to reduce the impactof releasing
software bydoingitmore frequently.
Popularized in the book 'Continuous Delivery: Reliable Software
Releases through Build, Test, and DeploymentAutomation',
written byJez Humble and David Farleyin 2010.
(VIDEO)
WHY USE ANSIBLE FOR
CONTINUOUS DELIVERY?
SIMPLICITY
No agents, and aclean playbook syntax are undoubtably
Ansible's two greateststrengths.
AUTOMATION OF COMPLEX DEPLOYMENT
ORCHESTRATION
Rollingreboots of web servers
Migratingprimary/secondary/tertiarydatabases like MySQL
Buildingand deployingnew AMIs/virtualinstances or
containers for immutable infrastructure
Removing/re-enablingmonitoringon servers or services being
updated
REUSABILITY AND CONSISTENCY
Developers usingVagrantor other setups locallycan easily
check outthe exactsame playbooks run in stagingand/or
production to ensure consistency.
Your CI system can use the exactsame playbooks too, rather
than shellscriptingeverything.
EXTENSIBILITY
Ansible has over 250 modules included, butif you need to
interface with an API we don'tsupport, writingmodules to
extend the functionalityis veryeasyand straightforward.
OTHERS HAVE ALREADY DONE THIS
Tons of blogs outthere alreadyoutline how to setallof this up
usingAnsible, so mostof the hard work has been done for you!
Ansible's user communityhas exploded, and gettinghelp is easy.
COMMON PATTERNS
CONTINUOUS INTEGRATION SYSTEM
Jenkins/Hudson, TravisCI, etc.
(source,DevOpsReactions–http://bit.ly/1NkG1ax)
SOURCE CONTROL
Stores both the application code and your Ansible playbooks
(infrastructure as data, notcode).
Allcommits trigger an automatic deploymentto the staging/QA
environment(s), followed byautomated testing.
If the automated testingpasses, the deploymentto production
can triggered byaperson or automatically.
AUTOMATED TESTING
Absolutelycriticalto continuous integration and delivery,
automated testingis whatverifies your code is ok to deploy.
Unittesting
Integration testing
Selenium, Ghost/CasperJS, etc.
MONITORING (ALL THE THINGS!)
Logand graph everything(Graphite and other popular projects
make this nearlytrivial). This is how you diagnose problems with
anygiven deploymentlater.
(source,https://codeascraft.com/2010/12/08/track-every-release/)
An excellentDigitalOcean blogon Graphite+statsd+collectd:
http://bit.ly/1Gx4UM2
CHATOPS
BENEFITS OF CHATOPS:
Everyone knows whateveryone else is doing, leadingto an
increased levelof transparencyacross your teams.
Because of this transparency, these actions also become self-
documentingwhich helps new hires learn the ropes more
quickly, especiallyif they're remote.
Communication is instantaneous and more fluid, as Ops don't
have to flip between emailor head off to a'war room'for
issues.
Time to execute actions is reduced, as there is now asingle
interface to trigger actions on disparate pieces of
infrastructure.
CASE STUDIES
ATLASSIAN
Atlassian is averywellknown software company, with products
such as JIRA, HipChatand Confluence to name afew.
Uses Ansible to address whattheycallthe 'lastmile problem', or
the finalstep in continuous deployment.
From their blog:
Developers perform continous testing/integration against
feature branches
After these feature branches are merged into the release
branch, Bamboo (Atlassian's CI product) is used to create a
build artifact(binary)
Ansible is used to deploythis same artifactto each of the
staging, QAand production environments in turn
ATLASSIAN (CONT.)
Advantages:
1. Promotingthe same build artifactthrough each environment
ensures consistency.
2. Despite usingin-house products, their approach is very
technologyagnostic and could easilybe adapted to other tools.
Disadvantages: Internalteams are verysegregated, with strict
separation of duties (silos) for deployments.
RISINGSTACK
RisingStack is aJavaScriptconsultancygroup, focusingon
NodeJS delivery.
UsingCodeShip (a3rd partyservice) and acustom web hook
service to trigger Docker builds and Ansible deployments to
automate their container-based approach to immutable
infrastructure.
(source:http://blog.risingstack.com/shipping-node-js-applications-with-docker-and-codeship/)
RISINGSTACK (CONT.)
From their blog:
Commits to the master branch trigger trigger builds in the
CodeShip service
If alltests pass, anew docker image is created and uploaded to
Docker Hub
After the upload is complete, another web hook is triggered on
their (custom, in-house) service which executes Ansible
Ansible downloads and deploys the new docker image on each
application server
RISINGSTACK (CONT.)
Advantages: Completelyautomated with no intervention
required to deploy.
Disadvantages: Custom service, when an off-the-shelf CI system
could be used to simplifythings.
BIGPANDA
BigPandais acompanycentered around automatingincident
management, and use ChatOps and Ansible heavilyin their day
to dayoperations
5 Reasons We Love UsingAnsible for Continuous Delivery-
http://bit.ly/1EkbqEi
Their motto is: 'Make the scariestpartof the build the easiest'.
BIGPANDA (CONT.)
Use acombination of HipChatand Hubot(called BeanBot
internally) to automate ops tasks (aone-person ops team).
GRASSWIRE
GrassWire is arecentstartup builtaround crowd-sourcingfact
verification in news reports.
Another ChatOps follower, GrassWire deploys their software via
Slack, which nativelysupports executingwebhooks remotelyand
does notrequire achatbot.
Use Packer to build new AMIs viaJenkins, and use Ansible to
deploythose new images to production alongwith their
application.
UsingTerraform to build initialAWS environments, rather than
CloudFormation or other tools.
GRASSWIRE (CONT.)
Their Packer integration also uses Tower, to take advantage of
the provisioningcallback feature.
"provisioners":[{
"type":"file",
"source":"./{{user`gw_env`}}/setup_callback.sh",
"destination":"/home/ec2-user/setup_callback.sh"
},
{"type":"shell","inline":[
"sudochmod+x/home/ec2-user/setup_callback.sh",
"echosh/home/ec2-user/setup_callback.sh|sudotee--append/etc/rc.local
]}
}]
Since theyalso use Tower for other tasks, theyare able to use the
tower-clicommand directlyin Jenkins to simplifytheir Jenkins
tasks.
TIPS AND TRICKS
USE ANSIBLE TO BUILD YOUR CONTINUOUS DELIVERY
ENVIRONMENT
Roles exist(on Galaxyand generallyon Github) to build allof
these components for you now. Use them to getup and running
quickly.
USE ANSIBLE FROM DAY 1
If you do anything, do itfrom Ansible. Don'tSSHto aserver, just
putwhatever itis you need to do in aplaybook and run that
instead.
QUESTIONS?
ATTRIBUTIONS:
LEGO GreatBallContraption (GBC) Layout2012.9
https://www.youtube.com/watch?v=sUtS52lqL5w
DevOps Reactions: Continuous Delivery
http://bit.ly/1NkG1ax
Code as CraftBlog: TrackingEveryRelease
https://codeascraft.com/2010/12/08/track-every-release/
DigitalOcean Blog: An Introduction to TrackingStatistics with
Graphite, StatsD, and CollectD
http://bit.ly/1Gx4UM2
Atlassian Blog: PracticalContinuous Deployment:
http://www.slideshare.net/tarkasteve/london-atlassian-user-
group-february-2014
ATTRIBUTIONS (CONT.):
RisingStack: ShippingNode.js Applications with Docker and
Codeship
http://blog.risingstack.com/shipping-node-js-applications-with-
docker-and-codeship/
BigPanda:
#ChatOpsFTW: http://www.slideshare.net/chuparkoff/chatops-
ftw
5 Reasons We Love UsingAnsible for Continuous Delivery:
http://bit.ly/1EkbqEi
THANKS!
twitter: jimi1283 github: jimi-c

Achieving Continuous Delivery: An Automation Story

  • 1.
  • 2.
    ABOUT ME JAMES CAMMARATA,DIRECTOR - CORE ENGINEERING Python developer since 2003 Started contributingto Cobbler in September of 2008, and took over the projectleadership in 2010 Joined Ansible in July, 2013
  • 3.
  • 4.
    Continuous deliveryaims toreduce the impactof releasing software bydoingitmore frequently. Popularized in the book 'Continuous Delivery: Reliable Software Releases through Build, Test, and DeploymentAutomation', written byJez Humble and David Farleyin 2010.
  • 5.
  • 6.
    WHY USE ANSIBLEFOR CONTINUOUS DELIVERY?
  • 7.
    SIMPLICITY No agents, andaclean playbook syntax are undoubtably Ansible's two greateststrengths.
  • 8.
    AUTOMATION OF COMPLEXDEPLOYMENT ORCHESTRATION Rollingreboots of web servers Migratingprimary/secondary/tertiarydatabases like MySQL Buildingand deployingnew AMIs/virtualinstances or containers for immutable infrastructure Removing/re-enablingmonitoringon servers or services being updated
  • 9.
    REUSABILITY AND CONSISTENCY DevelopersusingVagrantor other setups locallycan easily check outthe exactsame playbooks run in stagingand/or production to ensure consistency. Your CI system can use the exactsame playbooks too, rather than shellscriptingeverything.
  • 10.
    EXTENSIBILITY Ansible has over250 modules included, butif you need to interface with an API we don'tsupport, writingmodules to extend the functionalityis veryeasyand straightforward.
  • 11.
    OTHERS HAVE ALREADYDONE THIS Tons of blogs outthere alreadyoutline how to setallof this up usingAnsible, so mostof the hard work has been done for you! Ansible's user communityhas exploded, and gettinghelp is easy.
  • 12.
  • 13.
    CONTINUOUS INTEGRATION SYSTEM Jenkins/Hudson,TravisCI, etc. (source,DevOpsReactions–http://bit.ly/1NkG1ax)
  • 14.
    SOURCE CONTROL Stores boththe application code and your Ansible playbooks (infrastructure as data, notcode). Allcommits trigger an automatic deploymentto the staging/QA environment(s), followed byautomated testing. If the automated testingpasses, the deploymentto production can triggered byaperson or automatically.
  • 15.
    AUTOMATED TESTING Absolutelycriticalto continuousintegration and delivery, automated testingis whatverifies your code is ok to deploy. Unittesting Integration testing Selenium, Ghost/CasperJS, etc.
  • 16.
    MONITORING (ALL THETHINGS!) Logand graph everything(Graphite and other popular projects make this nearlytrivial). This is how you diagnose problems with anygiven deploymentlater. (source,https://codeascraft.com/2010/12/08/track-every-release/) An excellentDigitalOcean blogon Graphite+statsd+collectd: http://bit.ly/1Gx4UM2
  • 17.
  • 18.
    BENEFITS OF CHATOPS: Everyoneknows whateveryone else is doing, leadingto an increased levelof transparencyacross your teams. Because of this transparency, these actions also become self- documentingwhich helps new hires learn the ropes more quickly, especiallyif they're remote. Communication is instantaneous and more fluid, as Ops don't have to flip between emailor head off to a'war room'for issues. Time to execute actions is reduced, as there is now asingle interface to trigger actions on disparate pieces of infrastructure.
  • 19.
  • 20.
  • 21.
    Atlassian is averywellknownsoftware company, with products such as JIRA, HipChatand Confluence to name afew. Uses Ansible to address whattheycallthe 'lastmile problem', or the finalstep in continuous deployment. From their blog: Developers perform continous testing/integration against feature branches After these feature branches are merged into the release branch, Bamboo (Atlassian's CI product) is used to create a build artifact(binary) Ansible is used to deploythis same artifactto each of the staging, QAand production environments in turn
  • 22.
    ATLASSIAN (CONT.) Advantages: 1. Promotingthesame build artifactthrough each environment ensures consistency. 2. Despite usingin-house products, their approach is very technologyagnostic and could easilybe adapted to other tools. Disadvantages: Internalteams are verysegregated, with strict separation of duties (silos) for deployments.
  • 23.
  • 24.
    RisingStack is aJavaScriptconsultancygroup,focusingon NodeJS delivery. UsingCodeShip (a3rd partyservice) and acustom web hook service to trigger Docker builds and Ansible deployments to automate their container-based approach to immutable infrastructure. (source:http://blog.risingstack.com/shipping-node-js-applications-with-docker-and-codeship/)
  • 25.
    RISINGSTACK (CONT.) From theirblog: Commits to the master branch trigger trigger builds in the CodeShip service If alltests pass, anew docker image is created and uploaded to Docker Hub After the upload is complete, another web hook is triggered on their (custom, in-house) service which executes Ansible Ansible downloads and deploys the new docker image on each application server
  • 26.
    RISINGSTACK (CONT.) Advantages: Completelyautomatedwith no intervention required to deploy. Disadvantages: Custom service, when an off-the-shelf CI system could be used to simplifythings.
  • 27.
  • 28.
    BigPandais acompanycentered aroundautomatingincident management, and use ChatOps and Ansible heavilyin their day to dayoperations 5 Reasons We Love UsingAnsible for Continuous Delivery- http://bit.ly/1EkbqEi Their motto is: 'Make the scariestpartof the build the easiest'.
  • 29.
    BIGPANDA (CONT.) Use acombinationof HipChatand Hubot(called BeanBot internally) to automate ops tasks (aone-person ops team).
  • 30.
  • 31.
    GrassWire is arecentstartupbuiltaround crowd-sourcingfact verification in news reports. Another ChatOps follower, GrassWire deploys their software via Slack, which nativelysupports executingwebhooks remotelyand does notrequire achatbot. Use Packer to build new AMIs viaJenkins, and use Ansible to deploythose new images to production alongwith their application. UsingTerraform to build initialAWS environments, rather than CloudFormation or other tools.
  • 32.
    GRASSWIRE (CONT.) Their Packerintegration also uses Tower, to take advantage of the provisioningcallback feature. "provisioners":[{ "type":"file", "source":"./{{user`gw_env`}}/setup_callback.sh", "destination":"/home/ec2-user/setup_callback.sh" }, {"type":"shell","inline":[ "sudochmod+x/home/ec2-user/setup_callback.sh", "echosh/home/ec2-user/setup_callback.sh|sudotee--append/etc/rc.local ]} }] Since theyalso use Tower for other tasks, theyare able to use the tower-clicommand directlyin Jenkins to simplifytheir Jenkins tasks.
  • 33.
  • 34.
    USE ANSIBLE TOBUILD YOUR CONTINUOUS DELIVERY ENVIRONMENT Roles exist(on Galaxyand generallyon Github) to build allof these components for you now. Use them to getup and running quickly.
  • 35.
    USE ANSIBLE FROMDAY 1 If you do anything, do itfrom Ansible. Don'tSSHto aserver, just putwhatever itis you need to do in aplaybook and run that instead.
  • 36.
  • 37.
    ATTRIBUTIONS: LEGO GreatBallContraption (GBC)Layout2012.9 https://www.youtube.com/watch?v=sUtS52lqL5w DevOps Reactions: Continuous Delivery http://bit.ly/1NkG1ax Code as CraftBlog: TrackingEveryRelease https://codeascraft.com/2010/12/08/track-every-release/ DigitalOcean Blog: An Introduction to TrackingStatistics with Graphite, StatsD, and CollectD http://bit.ly/1Gx4UM2 Atlassian Blog: PracticalContinuous Deployment: http://www.slideshare.net/tarkasteve/london-atlassian-user- group-february-2014
  • 38.
    ATTRIBUTIONS (CONT.): RisingStack: ShippingNode.jsApplications with Docker and Codeship http://blog.risingstack.com/shipping-node-js-applications-with- docker-and-codeship/ BigPanda: #ChatOpsFTW: http://www.slideshare.net/chuparkoff/chatops- ftw 5 Reasons We Love UsingAnsible for Continuous Delivery: http://bit.ly/1EkbqEi
  • 39.