Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DTO #ChefConf2012


Published on

Published in: Business, Technology
  • Be the first to comment

DTO #ChefConf2012

  1. Using Rundeck and Chef to build DevOps Toolchains Anthony Shortland - Moses Lei -
  2. DTO Solutions
  3. Process Improvement Value Stream Service Delivery Process Program Current state value stream map Committee Eric Switzer Full Full L/T Lead time Projects / Projects / Expedites Expedites P/T Process time H/C Head count EP Business Technical S/R Scrap rate Analysis Analysis Luke Hanson Maya Highsmith Estimates Estimates D Defects and and EP Extra processes requirements requirements M Motion PD Partially done Project Management TS Task switching W Waiting Eric Switzer Business Change Design request Daily Scrum Change Program Management meeting Request QA Control Forms Mx3 build Development Dx3 Develop Technical Specification Daily Scrum Deployment web assets meeting QA Meeting Business Systems Jira issues Bugzilla issues Support Release Engineering L/T 8 weeks Web Assets, Deployment P/T 6 weeks Operations flows and Jira issues Document Stan Walters H/C 15 mockups S/R 8% Bob McNulty Wx2 PDx3 Dx2 RT tickets Change TSx5 Dx2 Development Development W Release Control Development Features and fixes Build and deploy Integration testing Build and deploy Set of EP service L/T 10 weeks L/T 1 days L/T 2 day L/T 11 hours Cut project branch branches P/T 6 weeks P/T 2 hours P/T 2 day P/T 8 hours H/C 4 H/C 2 H/C 11 H/C 2 W W D PD Mx7 Mx3 S/R 38% S/R 20% S/R 96% S/R 5% D PDx4 W D L/T 20 minutes D Joe Harbaugh Joe Harbaugh Joe Harbaugh Fernando Gomez W Deploy to EPx2 P/T 8 minutes Acceptance testing Feature testing Tested Merge project Deploy to Staging Staging testing H/C 2 packages Production branch S/R 0% L/T 5 days L/T 5 weeks L/T 7 days L/T 4 days L/T 10 hours L/T 1 hour Chris Holmes Development P/T 8 hours P/T 3.5 weeks P/T 110 minutes P/T 3.5 days P/T 7 hours P/T 30 minutes Environments H/C 2 H/C 5 H/C 4 H/C 5 H/C 5 H/C 1 Documented S/R 10% S/R 20% S/R 70% S/R 66% S/R 50% S/R 50% test plans Raj Lee Raj Lee Frances Middleton Frances Middleton Allen Cannata Fernando Gomez and cases EP PD Mx3 QA Staging Production Test Requirements Build test plan Build test cases Environments Environment Environment L/T 21 days L/T 8 days L/T 2 weeks P/T 3 days P/T 4 days P/T 8 days H/C 2 H/C 2 H/C 2 S/R 50% S/R 30% S/R 20% Deploy to Performance Luke Hanson Raj Lee Raj Lee Performance testing L/T 6 days L/T 1 week P/T 90 minutes P/T 3 days H/C 3 H/C 1 S/R 66% S/R 66% Frances Middleton Allen Cannata Performance Environment Process Time 5 weeks 5 weeks 5 hours 3 day 8 hours 6 hours 2 weeks 110 minutes = 59 days 4.5 days 9 hours 3 weeks 5 weeks 2.5 days 2 hours 5 days 2.5 weeks 6.75 days 2.5 days 2 hour Lead Time = 136 days Product Milestone RetailTrader - v133 "Add logic updates " 4 Days Remain 10d total Metrics: 1 week, 1 week historical Value-Creating Time (VCT) 65% +5% Environment: Production Stability Delivery Cycle Time Delivery Support Team Utilization 7 Days 3h, -5h 5 10% Support 85%, -5% -5% Scott, Erica, Reigner,… Provision Lead Time Deployments Work Tasks 2 Hours 15m 8 2 versions 4 environments 4/16 Open -30m +2 environments +1 version 15 Closed Features Provision Scrap Rate Delivery Scrap Rate Build Success 4/5 Lead Time 5% 1/20 +1 20% 2/10 -3 80% -5% 34m avg OK retailtr-v133.war
  4. Process Automation • End-to-end automation • Supports business goals • Optimize the system, not the parts
  5. IntroductionTake a toolchain approach E-commerce and software-as-a-service business models have matured quickly, but the quality of the web operations that support these businesses has lagged behind. Outages are all too common. High variability and defect rates are bemoaned but have become an accepted reality. Key engineers spend all day (and sometimes all night) mired in deployment issues and bottlenecks. And topping it all off, what tooling that does exist are usually a custom one-offs that are brittle and expensive to maintain. Today’s business of operating software over the Web as a revenue producing service is a ” Today’s business of dramatic departure from the days when software was primarily produced for delivery on operating software over the physical mediums and IT Operations was considered a back-of-the-house support Web as a revenue producing function. Shouldnt we be completely rethinking our tooling and operational capabilities service is a dramatic to match these new innovations? departure from the days when software was primarily In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated produced for delivery on approaches, and low expectations -- and into Web Operations 2.0 where tools and physical mediums...” procedures are built from the ground up for highly efficient, reliable, and agile Industrial Control Automation operations. Runbook Automation Control Eventing, Alarm Mgmt Charting, History, SPC Measurement Instrumentation System + There are multiple factors that go into achieving excellence in Web Operations, but the linchpin that holds it all together is a fully automated provisioning system. In this paper we will be: 1. Defining what we mean by "fully automated provisioning" 2. Explaining why virtualization and cloud computing efforts fail without fully automated provisioning capabilities 3. Proposing a reference open source tool chain for fully automated provisioning 4. Describing a live implementation where a leading online retailer is actively rolling out a fully automated provisioning system using all open source tools Unix Tool Chain 2 Web Ops 2.0: Achieving Fully Automated Provisioning
  6. Service Delivery Platform Elements
  7. Design points1. Everything is code2. Everything is packaged3. Separate code and configuration4. Separate environment-dependent attributes5. Balance distributed vs local orchestration6. Resolve directed vs convergent orchestration7. Integrate application and infrastructure provisioning8. Design for flow not the organization
  8. Example Service Delivery Platform
  9. Integration Plugins• Jenkins/GitHub - JENKINS/Github+Plugin• Jenkins/Rundeck - display/JENKINS/RunDeck+Plugin - Vincent Behar• Nexus/Rundeck - nexus-rundeck-plugin - Vincent Behar• Nexus/Yum - yum-plugin/• Chef/Rundeck - rundeck - Adam Jacob
  10. Example Service Delivery PlatformLayout Deployment Application Build Console Console Server Database Build ArtifactSource Code Repository Server Client Build Server Deploy Server Development Server
  11. Develop
  12. Build - Projects by artifact
  13. Build - Trace by build number
  14. Build - Template RPM SPEC[ec2-user@centos-ami bank]$ cat SPECS/dukesbank-ear.spec.templateSummary: Dukes Bank ear fileName: dukesbank-earVersion: ${version}Release: ${release}License: LGPLGroup: Applications/SystemPrefix: /opt/jboss-4.0.3SP1BuildArch: noarchRequires: jboss = 4.0.3SP1...
  15. Build - Register packages... <target name="rpmdeploy" depends="resolve,rpmbuild"> <ivy:makepom ivyfile="${basedir}/ivy.xml" pomfile="RPMS/noarch/dukesbank-ear-${version}-${release}.noarch.pom"/> <ivy:publish resolver="nexus-deploy" pubrevision="${version}-${release}"overwrite="true" publishivy="true"> <artifacts pattern="RPMS/noarch/[artifact]-[revision].noarch(-[classifier]).[ext]"/> <!--artifact name="dukesbank-ear" type="rpm" url="${build.dir}/rpm/RPMS/noarch/dukesbank-1.0-2.noarch.rpm" /--> </ivy:publish> </target>...</project>
  16. Repository - Build view
  17. Repository - Deploy view
  18. Deploy - Package version list[root@centos-ami yum.repos.d]# cat /etc/yum.repos.d/dukesbank.repo[dukesbank]name=DukesBank demo repobaseurl=[root@centos-ami yum.repos.d]# repoquery --repoid=dukesbank --search dukesbank-eardukesbank-ear-0:1.0-14.noarchdukesbank-ear-0:1.0-15.noarchdukesbank-ear-0:1.0-16.noarchdukesbank-ear-0:1.0-18.noarchdukesbank-ear-0:1.0-19.noarchdukesbank-ear-0:1.0-20.noarchdukesbank-ear-0:1.0-21.noarchdukesbank-ear-0:1.0-22.noarchdukesbank-ear-0:1.0-23.noarchdukesbank-ear-0:1.0-24.noarchdukesbank-ear-0:1.0-25.noarchdukesbank-ear-0:1.0-26.noarch[root@centos-ami yum.repos.d]# cat ~jenkins/jobs/dukesbank-ear.rpm/workspace/versions.json[ "1.0-14", "1.0-15", "1.0-16", "1.0-18", "1.0-19", "1.0-20", "1.0-21", "1.0-22","1.0-23", "1.0-24", "1.0-25", "1.0-26", "1.0-27", "1.0-28", "1.0-29", "1.0-30","1.0-31", "1.0-32", "1.0-33", "1.0-34" ][root@centos-ami yum.repos.d]#
  19. Deploy - Nodes & roles
  20. Deploy - Node provider[ec2-user@ip-10-36-119-128 ~]$ curl<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN""project.dtd"><project><node name="ip-10-195-11-159.ec2.internal" type="Node" description="ip-10-195-11-159.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="_default,base,chef_server" username="ec2-user" hostname="ip-10-195-11-159.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-195-11-159.ec2.internal/edit"/><node name="ip-10-36-119-128.ec2.internal" type="Node" description="ip-10-36-119-128.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="dev,base,dukesbank_database_master,dukesbank" username="ec2-user" hostname="ip-10-36-119-128.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-36-119-128.ec2.internal/edit"/></project>
  21. Deploy - Application attributes[ec2-user@centos-ami chef-repo]$ cat data_bags/apps/dukesbank.json{ "id": "dukesbank", "server_roles": [ "dukesbank" ], "type": { "dukesbank": [ "jboss", "jboss_webapp" ] }, "database_master_role": [ "dukesbank_database_master" ], "packages": { "dev": { "dukesbank-ear": "1.0-23" } }, "jboss_state": { "dev": "start" },...
  22. Links• DTO Solutions -• DTO Labs -• DTO Labs Github -• Rundeck -• Devops-Toolchain - toolchain