DevOps & BRMS CD
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


DevOps & BRMS CD

Uploaded on

Gene Kim DevOps Presentation & ILOG BRMS Tools + Delivery

Gene Kim DevOps Presentation & ILOG BRMS Tools + Delivery

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • DevOps + BRMS Continuous Delivery.The term “DevOps” typically refers to the emerging professional movement that advocates a collaborative working relationship between Developmentand IT Operations, resulting in the fast flow of planned work (i.e., high deploy rates), while simultaneously increasing the reliability, stability, resilience and security of the production environment.Some DevOps practices you can put into place and some steps that have already been taken.DevOps looks to solve the costs related to IT, this could be the business cost and also human cost associated with IT failures. One of the most monumental challenges as a profession.One of the most gigantic business problems we can solve which could create more economic value than anything we have seen in the past 30 years.
  • Supporting fragile artifactsAlways failingHard to find what has gone wrong, often problems are discovered by the customer.Bad day:Not as prepared for the audit as they thoughtSpending 30% of their time scrambling, generating presentation for auditorsOr an outage, and the developer is adamant that they didn’t make the change – they’re saying, “it must be the security guys – they’re always causing outages”Or, there’s 50 systems behind the load balancer, and six systems are acting funny – what’s different, and who made them differentOr every server is like a snowflake, each having their own personality
  • Always dreaming up new features without any consultation from IT to whether it would be possible.They don’t have the best idea of what technology can or cannot do.
  • Show me a developer who isn’t causing an outage, I’ll show you one who is on vacationOften led into date driven projects because of promises made by the businessFeatures need to be deployed quicklyShort cuts are taken, meaning more fragile code put into productionBecause of due dates nothing is left for operationsThis causes technical debt.Who’s introducing variance? Well, it’s often these guys. Show me a developer who isn’t causing an outage, I’ll show you one who is on vacation.Primary measurement is deploy features quickly – get to market.Bad day: We do 6 weeks of testing, but deployment still fails. Why? QA environment doesn’t match productionOr there’s a failure in testing, and no one can agree whether it’s a code failure or an environment failureOr changes are made in QA, but no one wrote them down, so they didn’t get replicated downstream in production.
  • Technical starts building,and then it compounds. Mostly short cuts to provide new features/application faster.
  • Drowned in technical debt. Something more insidious occurs, deployments start taking longer and longer due to this technical debt. Human nature is to perform painful things less frequentlyThis usually causes the release cycle to be increased by the business reducing the number of deployments. However the inner Jez Humble (Continuous Delivery) would say to shrink batch sizes and deploy frequently and not increase them and deploy every 9-12 months.This causes failures to always occur and new features to just trickle in.
  • When this happens nothing is getting done.More fragile artifacts are being deployed, More outages are being caused And technical debt is continually building.Ever growing tension
  • This is what happens to Non functional requirements Technical debt builds up in other areas as a consequence Ignored:Protection of critical systems and dataSafeguarding organizational commitmentsPrevention of security breaches, help quickly detect and recover from themBad day: no security standardsNo one is complyingYes, we’re 3 years behind. “Whaddyagonna do about it?”Vs. we can become more relevant and add value by infosec by leveraging all the configuration guidance out thereMeasure variance between production and those known good statesTrust and verify that when management says, we’ve trued up the configurations, they’ve actually done itWhy? Now, more than ever, there are an ever increasing amount of regulatory and contractual requirements to protect systems and data
  • Feature flow rate becomes a trickle, Consumed in unplanned work, project work is being starved.Trapped in a system that preordains us to failure. Causing business, dev and operations problems.A study found the rate of burnout in IT operations is one of the highest, at the same level of law enforcement, working in ER wards, or military deployment/tour of duties.It can be seen not only as an IT cost but also a human cost as you can feel helpless and trapped.
  • The downward spiral.
  • How each side Actively impedes the achievement of each other’s goals.
  • Two simultaneous business needs. Development motivated to ship code whatever it takesOperations is motivated to keep the system stable, so could mean no changes ever!!Leads to the chronic deadlock
  • Every company is an IT company regardless of what ever business we may think we are in. Height of delusion to not see it as anything else than a core competency like Maths or English.Some of the most important flows of work for the business is done within IT.
  • Spock up on the bridge with the captain.Scotty never getting invited to the captain meetings until the warp engines are down and everyone is running around in a panic.“I can’t change the laws of physics! I’ve got to have 30 minutes.”
  • The term “DevOps” typically refers to the emerging professional movement that advocates a collaborative working relationship between Developmentand IT Operations, resulting in the fast flow of planned work (i.e., high deploy rates), while simultaneously increasing the reliability, stability, resilience and security of the production environment.CAMS can also include Lean to make CALMS.
  • Statistics based on research with Puppet LabsImplementing infrastructure as code into source control and creating automated deployments of these environments. Graphs show the performance increase that can be gained by implementing this approach for environments and configuration.High performers: Non commissioned Officer in the military, chemical engineers, auditor's (IT). Discipline, rigor as the main values.
  • Statistics based on research with Puppet Labs.
  • The best always get betterNow at Amazon deployments are made to there production environment to support there cloud services on average every 11.6 seconds.
  • But it is not all about deployment rates. DevOps is all about a culture of reliability and consistency through automation of builds, tests and deployments for both applications and environmentsTherefore preventing outages or unwanted results.
  • It’s not just the startups implementing DevOps. Banks, Online Stores, Universities, Government,etc
  • Underpinning principles which you can derive all DevOps patterns from.The first way is all about left to right. Development into operations. And why?
  • Why Dev and Ops, because they lie between the business and the customer.Business requirements turn into Dev work and value is created out of Operations to the customer.The First Way emphasizes the performance of the entire system, as opposed to the performance of a specific silo of work or departmentThe focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.
  • Godfather of DevOps and creator of DevOps Days conference held all over the world (Europe/USA). DevOps philosophy seeks to solve the human and management problem of an organization.Not only is it just Dev and Ops, but it also evrything within and outside of that across the organsaition.
  • Increase flow by minimizing wait times, queue times, theory of constraints thinking, lean thinking.Find the bottlenecks and approve them and remember once improved seek the next bottleneck and continue to improve.
  • Instead of looking at dev or operations as one big pool or buffet the work needs to be visible.Unplanned work can be reduced by completing Internal projects that optimize the delivery of business projects.The number of changes needs to be quantified and managed as they represent commitments.Seek to control WIP so that business projects do not always override other types of work. By identifying the work you can have a fairer discussion with the business.
  • A practice to have environment available for deployment (one step creation of the environment), or built in parallel with the development process. The configuration of the environment should flow from dev into production and tests created against configuration as well as against our development code.This makes sure each environment is configured the same by automation with the same infrastructure and versions, libraries and other aspects.
  • This includes operations in the agile process for development.And reduces problem downstream.
  • Not only a single source of truth, but a communication vehicle in itself.In order to be repeatable, you must first define itRefactor to single step build and deployment automation for both code and environments.Going from a deployment every few months or year down to 10 deploys a day is through application of the first way.
  • The Second Way is about creating the right to left feedback loops. The reverse.The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.
  • Not just the external customer but also the next customer downstream, in terms of the Lean approach you want to optimize for the next area downstream.Ops is a customer of development.Toyota Andon cord, pull the cord to stop the entire line as greater sin is to let the defect continue.Optimizing the parts deployed into production for the people performing the deployment, and this is gone by embedding the knowledge in the system through automation and cross training.
  • GoogleHRR = Hand off readiness review, before the application can be handed to the SRE = Site Reliability EngineerTypes/frequency of pager alertsMaturity of monitoringSystem architecture reviewRelease processDefect counts and severityProduction hygiene
  • Integrate Dev into IT Operations escalation processesHave Dev cross-train IT Operations staffHave Dev improve the environment
  • Understand the affect of the decisions made in developmentMeasurements all the way up and down the application stack
  • Non functional found and fixed fasterStories for operations and information security as part of agile process so we know how long and the resources need.And it brings operation into the development process so communication is better
  • The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failureunderstanding that repetition and practice is the prerequisite to mastery.We need both of these equally. Experimentation and taking risks are what ensure that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.
  • The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failureAnd understanding that repetition and practice is the prerequisite to mastery.Experimentation and taking risks are what ensure that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.
  • Amazon EC2 failure which brought down most online sites affecting all AWS customers. There was only one site that lived through the outage. Netflix….why, because of the Simian army and Chaos Monkey.
  • Netflix designed there systems to expect and tolerate failure and Chaos Monkey did such a thing. Therefore from there learning of how to automatically recover servers if they go down they were able to prevent the outage. “A culture of doing your security/failover tests whenever you want to test that it happens a correctly and the process to handle it is automated.”
  • Before running in production, run in test.
  • Chaos monkey without something else though is just chaos.Marty Cagan inherited product management in 2002 at Ebay when they were experiencing major site failures and performance issues. If you don’t spend 20% of cycles on tech debt, then eventually you will be spending 100% on tech debt.
  • Then you can turn your technical debt, that compounds…
  • Into something manageableCausing fewer issuesmaking deployments easier and allowing people to focus on business needs than management of technical debt and unplanned work.
  • A good way to visualize work is through a Kanban board. You can assign different colors and include your technical debt so everything is visible and you can check that they are being completed also.
  • More than entre economic output of Germany for a year.
  • The outcomes of the Third Way include:allocating time for the improvement of daily work to reduce unplanned workMaking more cycles for planned workintroducing faults into the system to increase resilience
  • Shared ProcessTrustRobust TechnologyDevOps Alignment And helping the business fulfill there needs.
  • Stay ahead of the culture by creating the culture. Anti pattern: Implementing puppet/chef will fix all our problems.
  • DevOps is a cultural change that affects the whole of your organization, not just ITIts is not a single group that manages this change it is not a role one person is given. Act as a agent of change and have everyone in the organization follow the same principals.DevOps requires everyone on board or else it is difficult to implement the cultural change required.
  • team believing they do more or are better.Symmetry of Ignorance, brings more perspectives to help solve a design issue.How a team is viewed from the outside is by behavior while from the inside is usually by intentions. (Sym of Arrogance)Known also as the wicked problem.
  • You don’t become DevOps or have DevOps roles or groups.Devops is a philosophy that spreads across the entire organization and lends well to practices of Agile, Continuous Delivery and the Lean and TOC manufacturing theories.
  • Create a tool chain and not just one tool to control everything.Each tool fulfills a different need as there is usually integration provided from one tool to another through plugins or API’s.
  • DevOps practice to share the tools to allow a single source of truth and to increase collaboration
  • Release build, to promotion, to deployment, to backout
  • One step to build release, including deployment to CI/DEV environment and smoke tests with a single click.
  • Claim Builds if they fail
  • Deployment job screenshot of automated deployment running on promotion. Approve button with users allowed to approve.
  • Deployment, promotion, exactly what has been deployed, aritfacts. Always using the same artifacts.
  • Pipelines showing the flow of artifacts. Not all pipelines need to finish. Every build is seen as a release candidate.
  • Pipelines showing the flow of artifacts. Not all pipelines need to finish.
  • Removing Perl, artifactory, infrastructure as code, combined deployments, bring it all together
  • Automation doesn’t happen overnight“The automated still needs to be innovated.”Fixes/changes still need to occur to the automation processNew projects still need to have automation setup.However, setting up the automation once is easier than having to continually repeat the same processMake sure that the automated process is simple to start and based on pre-defined properties to control settings
  • Pipelines showing the flow of artifacts. Not all pipelines need to finish.
  • Pipelines showing the flow of artifacts. Not all pipelines need to finish.
  • Puppet uses a declarative, model-based approach to IT automation.
  • With continuous delivery we increase our velocity.
  • Automation removes “works on my box” phenomenon. Monitoring of all environments, including dev/test/qaetc, bottlenecks or loss of time in development will affect due date performance.
  • Saying “Show me a developer not causing a production outage and I’ll show you one that’s on Vacation”
  • Continuous Delivery by Jez Humble and David Farley
  • Add Flickr and Yahoo. Link to Flickr 10 deployments a day you tube.


  • 1. DevOps & BRMSContinuous Delivery
  • 2. Act I: IT Ops Fixing Fragile Artifacts
  • 3. The Product Managers
  • 4. Act II: The Developers
  • 5. “It’s not my machines,it’s your code!”
  • 6. “It’s not my code,it’s your machines!”
  • 7. Act III: IT Ops And Dev At War10
  • 8. Nothing Left For Infosec
  • 9. The DownwardSpiral…
  • 10. The IT Core Chronic Conflict Every IT organization is pressured to simultaneously: Respond more quickly to urgent business needs Provide stable, secure and predictable IT serviceSource: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints andauthor of The Goal, has written extensively on the theory and practice of identifying and resolvingcore, chronic conflicts.
  • 11. Every Company Is An IT Company… 95% of all capital projects have an IT component… 50% of all capital spending is technology-relatedWe are here…Where we need tobe…IT is always in theway(again…)
  • 12. Lowering risk of changethrough tools and culture
  • 13. Ops who think like devsDevs who think like ops
  • 14. Culture - Automation - Measurement - Sharing
  • 15.  They’re more agile High performing organizations deploy code 30times more frequently, and 8,000 times fasterthan their peers, deploying multiple times aday, versus an average of once a month. They’re more reliable High performing organizations have double thechange success rate and restore service 12times faster than their peers. Fewer failures andfaster recovery meaning less risk to the businesswhen changes are deployed.High Performing DevOps Teams
  • 16.  They’re more agile High performing organizations deploy code 30times more frequently, and 8,000 times fasterthan their peers, deploying multiple times aday, versus an average of once a month. They’re more reliable High performing organizations have double thechange success rate and restore service 12times faster than their peers. Fewer failures andfaster recovery meaning less risk to the businesswhen changes are deployed.High Performing DevOps Teams
  • 17. The First Way:Systems Thinking
  • 18. (Business) (Customer)The First Way:Systems Thinking
  • 19. “Some people get stuck on the word ‘devops’, thinkingthat it’s just about development and operationsworking together.Systems thinking advises us to optimize the whole;therefore devops must apply to the whole organization,not only the part between development andoperations.”— Patrick Debois
  • 20. The First Way:Systems Thinking (Left To Right) Understand the flow of work Always seek to increase flow Never unconsciously pass defects downstream Never allow local optimization to cause globaldegradation Achieve profound understanding of the system
  • 21. “Annual business planning sessions can be madding. Theythink IT Operations is an „all you can eat buffet.‟”-Ben Rockwood,Director Systems Engineering,Joyent
  • 22. Define The Work and Make It Visible Business projects (e.g., invoice handling features, claimssystems, premium rules) Internal IT projects (e.g., Puppet automation) Changes (e.g., deploys, improve database performance) Unplanned work (e.g., site down, site impaired)
  • 23. Day 2: PMO Meeting
  • 24. Create One Step Environment Creation Process Make environments available early in the Developmentprocess Make sure Dev builds the code and environment at thesame time Create a common Dev, QA and Production environmentcreation process
  • 25. Change the Agile sprint policy:“At the end of each sprint, we must have working code andthe environment it runs in.”
  • 26. The First Way:Outcomes Creating single repository for code and environments Determinism in the release process Consistent Dev, Test and Production environments, allproperly built before deployment begins A continuous delivery pipeline that can be relied uponand daily Dev code commits Decreased cycle time Reduce deployment times from 6 hours to 45 minutes Refactor deployment process that had 1300+ steps spanning 4weeks Faster release cadence
  • 27. The Second Way:Amplify Feedback Loops
  • 28. The Second Way:Amplify Feedback Loops (Right to Left) Understand and respond to the needs of allcustomers, internal and external Shorten and amplify all feedback loops: stop theline when necessary Create quality at the source Create and embed knowledge where we need it
  • 29. Require That Dev Initially Maintain Their OwnServiceSource: Tom Limoncelli, Google
  • 30. Return Fragile Services Back To DevSource: Tom Limoncelli, Google
  • 31. Embed Dev Into IT Ops Embed Dev into IT Ops incident escalation process Invite Dev to post-mortems/root cause analysis meeting Have Dev and Infosec cross-train IT Operations Ensure application monitoring/metrics to aid in Ops andInfosec work (e.g., incident/problem management)
  • 32. The Second Way:Outcomes Defects and security issues getting fixed faster than ever Standardized and reusable Ops and Infosec user storiesnow part of the Agile process All groups communicating and coordinating better Everybody is getting more work done
  • 33. The Third Way:Culture Of Continual Experimentation & Learning
  • 34. The Third Way:Culture Of Continual Experimentation & Learning Foster a culture that rewards: Experimentation (taking risks) and learning from failure Repetition is the prerequisite to mastery Why? You need a culture that keeps pushing into the dangerzone And have the habits that enable you to survive in thedanger zone
  • 35. Break Things Early And Often“Do painful things more frequently, so you can make it lesspainful… We don‟t get pushback from Dev, because theyknow it makes rollouts smoother.”-- Netflix
  • 36. 53
  • 37. Inject Failures Often
  • 38. You Don‟t Choose Chaos Monkey…Chaos Monkey Chooses You
  • 39. Break Things Before Production Enforce consistency in code, environments andconfigurations across the environments Add your ASSERTs to find misconfigurations, enforcehttps, etc. Add static code analysis to automated continuousintegration and testing process
  • 40. Allocate 20% Of Cycles To Technical DebtReduction
  • 41. Recognize Compounding Technical Debt…
  • 42. That Gets Worse…
  • 43. And Fixing It…
  • 44. The Third Way:Outcomes Technical debt is being paid off Continual reduction of unplanned work More cycles for planned work More resilient code and environments Increased experimentation and quicker time to value forthe business
  • 45. The Three Ways: Some Patterns65First Way Second Way Third WayDefine TheWork AndMake It VisibleMakeEnvironmentsAvailableEarlyWake UpDevelopersEmbed Dev IntoIT OperationsBreak ThingsEarly And OftenReserve 20% OfCycles ForTechnical DebtReduction
  • 46. Devops Anti Patterns
  • 47. 67Anti Pattern #1 - Config Mgmt = DevopsToolsProcessCulture
  • 48. Anti-Pattern #2 - Guerrilla DevopsIntegrateinto the processOnly local changesdont have much effect
  • 49. 69Anti-Pattern #3 - Start a separate DevOps group(Yet Another Silo)Act as a change agent
  • 50. 70Anti-Pattern #4 - Silo X takes over70If devs take over, theyhave to (re)-learnproductionSymmetry ofIgnorance/Arrogance
  • 51. 71Anti-Pattern #5 - Sell it as a buzzword
  • 52. ToolsBRMS Continuous Delivery
  • 53. “Tool chain” not “tool”Dev to Prod to DevSourceRepoSourceReposPackageReposBuild System DeploymentEngineConfigManagement / CMDBManifestCreationReleaseTrackingDashboards andMetricsEnvironmentProvisioningTest 1Test ...Test nProdBuildsImagesMonitoringTest Tooling
  • 54. Shared version controlShared build serverShared artifact repositoryShared deployments
  • 55. The past…• September 2012– Retrieval from SCM was hard-coded into PERL scripts.– Build files and property files to start a build were stored onthe Build Slave– Build scripts were not source controlled and had to bemanually copied into the correct location.– Setting up a new BRMS build for a new release could takedays to configure.– Deployments to every environment and Smoke tests wereperformed from the developers local IDE.– Artifacts related to Smoke Tests were re-generated.– The build server being used was outdated.– Full build could not be performed in local IDE.– Build was slow.
  • 56. The present…
  • 57. 1. One step buildand deploy
  • 58. 2. Automated Test Suitesand automatic re-generation
  • 59. 3. One step promotion
  • 60. Who? When? What?
  • 61. 4. Pipelines
  • 62. A Pipeline in the wildCodeCommitPCS BuildPCS BuildManagerDaily ManualDeploymentFortnightlyManualDeploymentQualityAssurance
  • 63. The future…
  • 64. Artifact Repository
  • 65. Artifactory in DevOps Ecosystem
  • 66. Artifact Repositories• Upload a package whenever a build happens• Archive Strategies• Dependencies – Maven, Gradle, Ivy…etc• Statistics• API to get status of the metadata• Use your binaries storage to release• Store third party and local artifacts• Caching & Collaboration
  • 67. Infrastructure as Code
  • 68. ApplicationEnvironmentInfrastructure
  • 69. Puppet is IT automation software that helps systemadministrators manage infrastructurethroughout its lifecycle, fromprovisioning and configurationto patch management and compliance.
  • 70. Don’t make the Version Control Sad
  • 71. Continuous Delivery
  • 72. Why Continuous Delivery• Get measurable changes fast• Reduce the risk of releases• Loosely coupled but coherent teams• Get rid of the “works-on-my-box”phenomenon• Forces you to think where does state fullthings live in your system
  • 73. Book club
  • 74. Velocity 09: John Allspaw and Paul Hammond, “10+ Deploys Per Day” Real Gene Kim, co-author the Phoenix Project Revolution Press (Short) History of DevOps
  • 75. Questions ?!@