DTO #ChefConf2012

Using Rundeck and Chef to build DevOps Toolchains Anthony Shortland - anthony@dtosolutions.com Moses Lei - mlei@dtosolutions.com

DTO Solutions

Process Improvement Value Stream Service Delivery Process Program Current state value stream map Committee Eric Switzer Full Full L/T Lead time Projects / Projects / Expedites Expedites P/T Process time H/C Head count EP Business Technical S/R Scrap rate Analysis Analysis Luke Hanson Maya Highsmith Estimates Estimates D Defects and and EP Extra processes requirements requirements M Motion PD Partially done Project Management TS Task switching W Waiting Eric Switzer Business Change Design request Daily Scrum Change Program Management meeting Request QA Control Forms Mx3 build Development Dx3 Develop Technical Specification Daily Scrum Deployment web assets meeting QA Meeting Business Systems Jira issues Bugzilla issues Support Release Engineering L/T 8 weeks Web Assets, Deployment P/T 6 weeks Operations flows and Jira issues Document Stan Walters H/C 15 mockups S/R 8% Bob McNulty Wx2 PDx3 Dx2 RT tickets Change TSx5 Dx2 Development Development W Release Control Development Features and fixes Build and deploy Integration testing Build and deploy Set of EP service L/T 10 weeks L/T 1 days L/T 2 day L/T 11 hours Cut project branch branches P/T 6 weeks P/T 2 hours P/T 2 day P/T 8 hours H/C 4 H/C 2 H/C 11 H/C 2 W W D PD Mx7 Mx3 S/R 38% S/R 20% S/R 96% S/R 5% D PDx4 W D L/T 20 minutes D Joe Harbaugh Joe Harbaugh Joe Harbaugh Fernando Gomez W Deploy to EPx2 P/T 8 minutes Acceptance testing Feature testing Tested Merge project Deploy to Staging Staging testing H/C 2 packages Production branch S/R 0% L/T 5 days L/T 5 weeks L/T 7 days L/T 4 days L/T 10 hours L/T 1 hour Chris Holmes Development P/T 8 hours P/T 3.5 weeks P/T 110 minutes P/T 3.5 days P/T 7 hours P/T 30 minutes Environments H/C 2 H/C 5 H/C 4 H/C 5 H/C 5 H/C 1 Documented S/R 10% S/R 20% S/R 70% S/R 66% S/R 50% S/R 50% test plans Raj Lee Raj Lee Frances Middleton Frances Middleton Allen Cannata Fernando Gomez and cases EP PD Mx3 QA Staging Production Test Requirements Build test plan Build test cases Environments Environment Environment L/T 21 days L/T 8 days L/T 2 weeks P/T 3 days P/T 4 days P/T 8 days H/C 2 H/C 2 H/C 2 S/R 50% S/R 30% S/R 20% Deploy to Performance Luke Hanson Raj Lee Raj Lee Performance testing L/T 6 days L/T 1 week P/T 90 minutes P/T 3 days H/C 3 H/C 1 S/R 66% S/R 66% Frances Middleton Allen Cannata Performance Environment Process Time 5 weeks 5 weeks 5 hours 3 day 8 hours 6 hours 2 weeks 110 minutes = 59 days 4.5 days 9 hours 3 weeks 5 weeks 2.5 days 2 hours 5 days 2.5 weeks 6.75 days 2.5 days 2 hour Lead Time = 136 days Product Milestone RetailTrader - v133 "Add logic updates " 4 Days Remain 10d total Metrics: 1 week, 1 week historical Value-Creating Time (VCT) 65% +5% Environment: Production Stability Delivery Cycle Time Delivery Support Team Utilization 7 Days 3h, -5h 5 10% Support 85%, -5% -5% Scott, Erica, Reigner,… Provision Lead Time Deployments Work Tasks 2 Hours 15m 8 2 versions 4 environments 4/16 Open -30m +2 environments +1 version 15 Closed Features Provision Scrap Rate Delivery Scrap Rate Build Success 4/5 Lead Time 5% 1/20 +1 20% 2/10 -3 80% -5% 34m avg OK retailtr-v133.war

Process Automation • End-to-end automation • Supports business goals • Optimize the system, not the parts

IntroductionTake a toolchain approach E-commerce and software-as-a-service business models have matured quickly, but the quality of the web operations that support these businesses has lagged behind. Outages are all too common. High variability and defect rates are bemoaned but have become an accepted reality. Key engineers spend all day (and sometimes all night) mired in deployment issues and bottlenecks. And topping it all off, what tooling that does exist are usually a custom one-offs that are brittle and expensive to maintain. Today’s business of operating software over the Web as a revenue producing service is a ” Today’s business of dramatic departure from the days when software was primarily produced for delivery on operating software over the physical mediums and IT Operations was considered a back-of-the-house support Web as a revenue producing function. Shouldnt we be completely rethinking our tooling and operational capabilities service is a dramatic to match these new innovations? departure from the days when software was primarily In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated produced for delivery on approaches, and low expectations -- and into Web Operations 2.0 where tools and physical mediums...” procedures are built from the ground up for highly efﬁcient, reliable, and agile Industrial Control Automation operations. Runbook Automation Control Eventing, Alarm Mgmt Charting, History, SPC Measurement Instrumentation System + There are multiple factors that go into achieving excellence in Web Operations, but the linchpin that holds it all together is a fully automated provisioning system. In this paper we will be: 1. Deﬁning what we mean by "fully automated provisioning" 2. Explaining why virtualization and cloud computing efforts fail without fully automated provisioning capabilities 3. Proposing a reference open source tool chain for fully automated provisioning 4. Describing a live implementation where a leading online retailer is actively rolling out a fully automated provisioning system using all open source tools Unix Tool Chain 2 Web Ops 2.0: Achieving Fully Automated Provisioning

Service Delivery Platform Elements

Design points1. Everything is code2. Everything is packaged3. Separate code and conﬁguration4. Separate environment-dependent attributes5. Balance distributed vs local orchestration6. Resolve directed vs convergent orchestration7. Integrate application and infrastructure provisioning8. Design for ﬂow not the organization

Example Service Delivery Platform

Integration Plugins• Jenkins/GitHub - https://wiki.jenkins-ci.org/display/ JENKINS/Github+Plugin• Jenkins/Rundeck - https://wiki.jenkins-ci.org/ display/JENKINS/RunDeck+Plugin - Vincent Behar• Nexus/Rundeck - https://github.com/vbehar/ nexus-rundeck-plugin - Vincent Behar• Nexus/Yum - http://code.google.com/p/nexus- yum-plugin/• Chef/Rundeck - https://github.com/opscode/chef- rundeck - Adam Jacob

Example Service Delivery PlatformLayout Deployment Application Build Console Console Server Database Build ArtifactSource Code Repository Server Client Build Server Deploy Server Development Server

Develop

Build - Projects by artifact

Build - Trace by build number

Build - Template RPM SPEC[ec2-user@centos-ami bank]$ cat SPECS/dukesbank-ear.spec.templateSummary: Dukes Bank ear fileName: dukesbank-earVersion: ${version}Release: ${release}License: LGPLGroup: Applications/SystemPrefix: /opt/jboss-4.0.3SP1BuildArch: noarchRequires: jboss = 4.0.3SP1...

Build - Register packages... <target name="rpmdeploy" depends="resolve,rpmbuild"> <ivy:makepom ivyfile="${basedir}/ivy.xml" pomfile="RPMS/noarch/dukesbank-ear-${version}-${release}.noarch.pom"/> <ivy:publish resolver="nexus-deploy" pubrevision="${version}-${release}"overwrite="true" publishivy="true"> <artifacts pattern="RPMS/noarch/[artifact]-[revision].noarch(-[classifier]).[ext]"/>  </ivy:publish> </target>...</project>

Repository - Build view

Repository - Deploy view

Deploy - Package version list[root@centos-ami yum.repos.d]# cat /etc/yum.repos.d/dukesbank.repo[dukesbank]name=DukesBank demo repobaseurl=http://ec2-75-101-135-205.compute-1.amazonaws.com:8081/nexus/content/repositories/releasesenabled=1gpgcheck=0metadata_expire=1s[root@centos-ami yum.repos.d]# repoquery --repoid=dukesbank --search dukesbank-eardukesbank-ear-0:1.0-14.noarchdukesbank-ear-0:1.0-15.noarchdukesbank-ear-0:1.0-16.noarchdukesbank-ear-0:1.0-18.noarchdukesbank-ear-0:1.0-19.noarchdukesbank-ear-0:1.0-20.noarchdukesbank-ear-0:1.0-21.noarchdukesbank-ear-0:1.0-22.noarchdukesbank-ear-0:1.0-23.noarchdukesbank-ear-0:1.0-24.noarchdukesbank-ear-0:1.0-25.noarchdukesbank-ear-0:1.0-26.noarch[root@centos-ami yum.repos.d]# cat ~jenkins/jobs/dukesbank-ear.rpm/workspace/versions.json[ "1.0-14", "1.0-15", "1.0-16", "1.0-18", "1.0-19", "1.0-20", "1.0-21", "1.0-22","1.0-23", "1.0-24", "1.0-25", "1.0-26", "1.0-27", "1.0-28", "1.0-29", "1.0-30","1.0-31", "1.0-32", "1.0-33", "1.0-34" ][root@centos-ami yum.repos.d]#

Deploy - Nodes & roles

Deploy - Node provider[ec2-user@ip-10-36-119-128 ~]$ curl http://ec2-75-101-135-190.compute-1.amazonaws.com:9980/<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN""project.dtd"><project><node name="ip-10-195-11-159.ec2.internal" type="Node" description="ip-10-195-11-159.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="_default,base,chef_server" username="ec2-user" hostname="ip-10-195-11-159.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-195-11-159.ec2.internal/edit"/><node name="ip-10-36-119-128.ec2.internal" type="Node" description="ip-10-36-119-128.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="dev,base,dukesbank_database_master,dukesbank" username="ec2-user" hostname="ip-10-36-119-128.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-36-119-128.ec2.internal/edit"/></project>

Deploy - Application attributes[ec2-user@centos-ami chef-repo]$ cat data_bags/apps/dukesbank.json{ "id": "dukesbank", "server_roles": [ "dukesbank" ], "type": { "dukesbank": [ "jboss", "jboss_webapp" ] }, "database_master_role": [ "dukesbank_database_master" ], "packages": { "dev": { "dukesbank-ear": "1.0-23" } }, "jboss_state": { "dev": "start" },...

Links• DTO Solutions - http://dtosolutions.com• DTO Labs - http://dtolabs.com• DTO Labs Github - https://github.com/dtolabs• Rundeck - http://rundeck.org• Devops-Toolchain - http://groups.google.com/group/devops- toolchain

http://dev2ops.org	8093
http://www.planetdevops.net	3667
http://www-ig-opensocial.googleusercontent.com	2040
http://lanyrd.com	669
http://feeds.feedburner.com	335
http://dev2ops.local	68
http://dev2ops.dtosolutions.com	40
http://dev2ops.squarespace.com	23
http://www.newsblur.com	19
http://webcache.googleusercontent.com	12
http://newsblur.com	8
http://127.0.0.1	6
http://1kpl.us	4
http://dev.newsblur.com	3
http://translate.googleusercontent.com	3
http://us-w1.rockmelt.com	2
http://dev2ops.org&_=1347001951117 HTTP	1
http://apps.synaptive.net	1
http://www.vrbo.com&_=1352136358122 HTTP	1
http://www.vrbo.com&_=1352136358138 HTTP	1
http://planetdevops.net	1

DTO #ChefConf2012

by AnthonyShortland on May 22, 2012

Accessibility

Categories

Upload Details

Usage Rights

21 Embeds 14,997

Statistics

DTO #ChefConf2012 Presentation Transcript