Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps - Boldly Go for Distro


Published on

This is a 90 min talk with some exercises and discussion that I gave at the DHS Agile Expo. It places DevOps as a series of feedback loops and emphasizes agile engineering practices being at the core.

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

DevOps - Boldly Go for Distro

  1. 1. DevOps Boldly Go Where You Never Have Gone Before… Paul Boos
  2. 2. DevOps: Episodes A Tale of Perspectives Exploring New Worlds: The DevOps System Feedback & Learning • Inner System • Outer System Achieving Warp Star Trek and its images are owned by Paramount Pictures
  3. 3. Meet Mr. Spock Mr. Spock develops software. He wants to develop software his users will love find fascinating. To him, it’s creative problem solving. (How logical.) He wants to deploy it often so his users can be more productive. He wants tools and techniques to help him accomplish these things.
  4. 4. Meet Mr. Scott Mr. Scott runs systems. He wants the systems to run reliably. Problem-solving almost always means bad news for him. Users usually do not love it when he has to do problem-solving. Fast deployments hopefully don’t mean more work and outages.
  5. 5. When things don’t go well… CAPT Kirk, the CIO, shows more emotion than Mr. Spock and Scotty would like. You’d thing the Klingons had taken his son or something…
  6. 6. So how do we unite these two departments successfully?
  7. 7. Incoming Transmission Captain… DevOps is all about increased and improved feedback: oTechnical and oBusiness Starfleet Out…
  8. 8. Starfleet Academy Exercise #1 Make a list of feedback loops you think you would want supported by DevOps Next to those, list the technique you are currently doing that helps achieve that feedback loop, note how long it takes Pair up, discuss these and see what the other is doing and how long it is taking them 5 min 5 min
  9. 9. Debrief
  10. 10. Set Course for DevOps (Warp Factor 5) Story in Ready State
  11. 11. DevOps System Survey oA planetary system consisting of 10 planets in a variety of feedback loops oInhabited by a civilization w/many sophisticated practices oEach planetary system has a set of advanced technological bases
  12. 12. DevOps System Survey (cont’d) oInner system has a lens on fulfilling what is requested. oThese are focused on agile engineering practices. oOuter system lens is on learning what is actually needed. o Occur as operations involvement increases.
  13. 13. DevOps-1 Planetary Scan: Unit Tests oFeedback loop often developed to ensure one is meeting a story’s acceptance criteria oParticularly useful if performing TDD as one writes code to pass the test oAn assurance on the correct functionality on our local machine.
  14. 14. What did our away team learn? The technical base for unit testing is an appropriate unit test framework: jUnit, nUnit, pyUnit, etc. This is code. It should be treated like any other code and checked into the source code repository. This is your safety net for refactoring or adding features. What is the orbital period of this feedback loop? ≈ minutes to a couple of hours
  15. 15. DevOps-2 Planetary Scan: Sync to ∆’s oBefore we check-in, update to ensure everything works with other people’s changes. oCode hoarding tends to create problems as the long cycles make it more likely a problem was created. oIf done often, this helps prevent broken builds.
  16. 16. What did our away team learn? Blindly checking in code increases the integration work required on the many. Every piece of code not checked in is inventory at risk: unit tests, code meeting unit tests… This tension is what creates success as it helps us focus on small batches of work. What is the orbital period of this feedback loop? ≈ minutes to a couple of hours
  17. 17. DevOps-3 Scan: ✓- in to Build oWhen we check-in our code, we get the immediate feedback of whether a problem was introduced. oWe want to check directly into a trunk; no extended branches & merges except for architectural experiments. oIf the build breaks, fix immediately. (That’s you in case it wasn’t clear.)
  18. 18. What did our away team learn? If code worked locally, but breaks the build, it was OUR code that broke it. Fix it in collaboration with the other team members. Code hoarding (code not checked in) hurts our ability to integrate easily. Again small batches of work help us. We’ll need build scripts (e.g. ant for Java) for the CI server to use. What is the orbital period of this feedback loop? ≈ minutes to a couple of hours
  19. 19. DevOps-4 Planetary Scan: Automated Tests on ✓- in oUpon code check-in (which includes our tests), our next line of feedback comes from the collection of automated tests that have been created. oThese tests act as our safety net; similar to what we know as regression tests. oIf the tests fail, fix immediately. (That’s you in case it wasn’t clear.)
  20. 20. What did our away team learn? If code worked locally, but then fails tests on check-in, it was OUR code that caused the failure. Fix it in collaboration with the other team members. This often happens when we failed to synchronize with the source code before a check-in. Again, code hoarding works against us, small batches help us. Repeat CodeUnit Test(locally)SyncCheck-in cycle. What is the orbital period of this feedback loop? ≈ minutes to a couple of hours
  21. 21. DevOps- 5 Planetary Scan: Automated Acceptance Tests on ✓- in oFollowing automated unit tests is automated acceptance tests (behavior tests). oThese ensure we built what was collectively articulated within the user story and its acceptance criteria. oIf the tests fail, we haven’t finished fulfilling the story.
  22. 22. What did our away team learn? Behavioral code is fulfilling the user story. We build to this need. If we have difficulty, remember the Agile value: customer collaboration > contract negotiation. When these tests pass green, we are basically ready to head to the outer system and exchange with the outside. (There’s one more waypoint…) What is the orbital period of this feedback loop? ≈ generally one hour to a day or so
  23. 23. DevOps- 6 Planetary Scan: Exploratory & Accessibility Testing oExploratory Testing is looking for edge cases and additional cases that perhaps should become acceptance criteria. This manual testing is interleaved to help design additional automated tests. oAccessibility (508) Testing should be automated where possible. oThese test activities can be performed in parallel.
  24. 24. What did our away team learn? Exploratory testing is a means of uncovering more complete and better unit tests. We want skilled test engineers (red shirts). Accessibility (508) testing can be automated, and it will be very different than what is typically seen. Requires excellent test design. Much of the negotiation with the product owner may occur with these. There should be a final check before going to the next [production- like] environment (the outer system). What is the orbital period of this feedback loop? continuously
  25. 25. DevOps (Agile Development) Workflow ✓- Out Code for Work Write Code & Pass Unit Test Sync from Repository ✓ - In & Build Automated Tests Run (Unit Test Suite) Repeat until BDD Tests Pass Automated Behavior-Driven Tests Run (BDD Suite) Repeat until all Unit Tests Pass Exploratory/Accessibility Testing Ready Story to the Outer System Unit Test Loop: Minutes To Hours Behavior Loop: Few Hours ± Day This is continuous! Consider a last waypoint before further journeying.
  26. 26. What ‘tools’ do we need onboard to exchange with external cultures? Before we head to the outer system…
  27. 27. Source Code Repository – distributed system or supporting optimistic locking (Git, Subversion, Vault, etc.) Unit Test Framework (Language Dependent) Behavior Testing Tools (e.g. Cucumber, Lettuce, Fitness, etc.) UI Testing Tools – these need to be able to be integrated into the CI pipeline (e.g. Selenium Web Driver, etc.) Test-Oriented Code Management Documentation ‘Production’ (Language Dependent & Independent – e.g. javadoc, pydoc, etc. + UML reverse- engineering tools) Static Analysis Tools – examples: • Coding ‘standards’ tools • Cyclomatic Complexity Analysis Other Useful Tools Continuous Integration (CI) Server (Jenkins, Bamboo, Microsoft Team Foundation Server, etc.) w/build scripts
  28. 28. Surprise! You’ll be writing code to enable most of these…
  29. 29. What Difficulties Have You Faced to This Point Spock?
  30. 30. DevOps-7 Planetary Scan: Performance & Security Testing oIn preparation for production, often performance and security testing will be used to look for weaknesses. oThese help ensure system integrity when used by actual customers. oThese tests should be as close as possible to the development work (w/in an iteration).
  31. 31. What did our away team learn about performance testing? Performance testing requires an environment that can mimic production. • Architecture, servers, data, etc. Tooling depends on the type of application; for web apps, tools such a Selenium Grid or Jmeter can be used for load testing with Selenium Web Driver. What is the orbital period of this feedback loop? from within iteration to a release hardening period (shorter is better)
  32. 32. DevSecOps Incorporating Security Practices into DevOps
  33. 33. What did our away team learn about security testing? There are many types of security tests: Penetration, vulnerability, etc. Tools are dependent on the type of testing and sometimes language Tools like Chaos Monkey can also be used for cloud-like environments Hardened server images can also be created for use by teams for their deployments; separates concerns. Many security tests can be incorporated into behavior tests through user|dark stories & acceptance criteria. What is the orbital period of this feedback loop? from within iteration to a release hardening period (again shorter is better)
  34. 34. ark Stories • Basically sets of behavioral tests for this persona… • Objective is to have all stories FAIL • Persona is ‘hostile’: internal or external
  35. 35. Integrating Security in a DevOps Fashion Coding Deployment Dev-Test Environment(s) Stage Environment Deployment Production Environment Penetration/Vulnerability Testing
  36. 36. DevOps-8 Planetary Scan: Automated Deployments oAutomated deployments ensure stable, repeatable configurations in an environment. oNeeds to include an automated rollback strategy. oTreat infrastructure as code.
  37. 37. What did our away team learn? Treating infrastructure as code requires versioning; thus some form of repository. • The continuous delivery fans would say these must be binary images. • I’ve had success using a CI server and building through to production. Besides the images, the deployment (and rollback) scripts should also be versioned. It’s usually a good practice to have an automated smoke test following a completed production deployment. Having QA decision points along the entire build/deploy pipeline. What is the orbital period of this feedback loop? when deployment occurs -- often
  38. 38. Comparing Deployment Approaches Deploying Binaries Pluses • Faster • No compilation is necessary • Easily can avoid downtime Minuses • Deployment scripts are different than build scripts • More infrastructure is required Building Through Pluses • Build scripts to every environment are the same • Compile to each server Minuses • Need to clean-up any build tools used in production • Slower to build • More difficult to avoid downtime
  39. 39. DevOps-9 Scan: Product Owner/Business Review oProduct Owner/business ‘approval’ is still desired. oFrequently the Product Owner or business is tugged by the gravitational pulls of other stakeholders. oReview can occur at any point in the process & when in any environment.
  40. 40. What did our away team learn? IT/App Dev does work for the business’ value stream. Report what passes ‘green’ and what FAILed. Reviewing with the Product Owner/business can occur post-Production deployment. How? • Using toggles coded into the configuration. • Toggles turn on/off features around stories, epics, or other groupings of features What is the orbital period of this feedback loop? whenever the business desires
  41. 41. DevOps-10 Planetary Scan: Customer Monitoring oEverything the business says they want on behalf of a customer-user is a hypothesis. oIts important to find out what works (or doesn’t). oMore frequent is better. oThis is where the business learns.
  42. 42. What did our away team learn? Obvious ways to learn from customers: • Customer surveys • Net Promoter Score Less obvious ways: • Periodic focus groups • Flows through the system with dwell times, conversions, and abandoned sessions • Heat maps of where users click What is best is entirely dependent on what the application or system is supposed to do for them. Can cause a rollback!! What is the orbital period of this feedback loop? every release or ‘toggle on’
  43. 43. Load Testing – multiple browsers (& browser engines) simultaneously (Selenium Grid, Jmeter, etc.) Flow/Conversions/Abandoned Sessions (Google Analytics, Matomo, W3Counter, MixPanel, Kissmetrics, etc.) Click Tracking (Open Web Analytics, ClickHeat, etc.) Customer Surveys (Survey Monkey, Foresight, etc.) Customer Monitoring Performance & Security Testing Test Portal/Orchestration (SonarQube, etc.) Binary Source Management (Artifactory, Nexus, etc.) Deployment Configuration (Chef, Puppet, etc.) OS Virtualization/Containers (Docker, Vagrant, Apache Mesos, etc.) Pipeline-Oriented Penetration Testing (Metasploit Framework, etc.) Vulnerability Scanning (Nmap, OpenVAS, etc.)
  44. 44. What tool have I not mentioned?
  45. 45. Starfleet Academy Exercise #2 Get into a group of 4-6 (preferably from the same component); create a mindmap of the types of quality (e.g. reliability, data integrity, performance) Now create a mindmap of the stakeholders that care about quality And add the quality types to stakeholders You can now use this technique (even at a story level) to determine the type of testing to emphasize for development work in your pipeline. • Emphasize by stakeholder importance • Improvement stories can be used to address future needs 5 min 5 min Since automated testing is crucial for DevOps…
  46. 46. Debrief
  47. 47. A glimpse into the future
  48. 48. Captain’s Log Stardate 10.23.2028 Scaling DevOps • Cadenced delivery  review at scale gives way to continuous flow of value • Learn more about scaling with the Manifesto for Scaling Agility Artificial Intelligence/Machine Learning • Helps make decisions on criticality of errors and whether to promote code or not Refocus on the Human-Side of Agile • Source Code Repositories Report who has code already checked out and assist in loading a virtual pairing session • We realize Scrum Masters (team facilitators) can’t be replaced with AI, but is an essential skill all team members should have
  49. 49. Back to Now…
  50. 50. Captain’s Log Stardate 10.23.2018 Review what feedback loops you want to improve Write down the ONE next loop that you immediately want to improve. Also describe how you will collaboratively work with the others to make this happen. 5 min
  51. 51. Red Alert :: Conference Identified 13-14 November AMERICAN UNIVERSITY WASHINGTON COLLEGE OF LAW 4300 Nebraska Ave NW
  52. 52. Red Alert :: 2nd Conference Identified Government Lean-Agile Software & Systems Conference
  53. 53. Thank You! Paul Boos @paul_boos
  54. 54. Any Questions?
  55. 55. Enjoy Your Voyage Paul Boos @paul_boos