Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evolutionary infrastructure agile 2018 - kief morris

10 views

Published on

This talk explores infrastructure design patterns to support continuous change to:

- Reduce the “blast radius” for a given change
- Make it easy to update, upgrade, and refactor systems without requiring massive, organization-wide programmes of change
- Improve security, visibility, auditability, and observability of systems
- Increase the number of people and teams able to work across systems while minimizing coordination overhead

Topics covered include:

- Designing and implementing an effective infrastructure test automation strategy
- Creating change management pipelines that enforce rigorous change control processes while supporting rapid, frequent changes
- Structuring infrastructure codebases to optimize for continuous change

Published in: Software
  • Be the first to comment

  • Be the first to like this

Evolutionary infrastructure agile 2018 - kief morris

  1. 1. Building EVOLUTIONARY infrastructure
  2. 2. kief@thoughtworks.com Infrastructure Technologist@ Twitter: @kief Book: http://oreil.ly/1JKIBVe Site: http://infrastructure-as-code.com Some visuals found from the Vault of the Atomic Space Age http://bit.ly/2KBTbAd
  3. 3. Why? Ongoing pressure to deliver while requirements change
  4. 4. PRODTESTDEV Inconsistent path to production
  5. 5. Snowflake production
  6. 6. PROD Continuous, reliable delivery of changes to all your stuff TESTDEV
  7. 7. "Since we can't avoid change, we need to exploit it" Building Evolutionary Architectures Neal Ford, Rebecca Parsons, Pat Kua
  8. 8. Speed or Quality?
  9. 9. Fast Slow Careless Careful
  10. 10. Move fast and fix things
  11. 11. Fast Slow Careless Careful
  12. 12. FastSlow Careful Careless
  13. 13. FastSlow Careless Prioritize speed over correctness Fragile mess
  14. 14. Slow Careful Careless Prioritize correctness over speed of change Fragile mess
  15. 15. Slow Careful Careless Prioritize correctness over speed of change Fragile mess Time pressures Over- engineering Technical debt Barriers to improvement
  16. 16. FastSlow Careful Careless Agile, Lean, DevOpsBuild the simplest thing Build quality in Optimize for change Fast, frequent feedback
  17. 17. Infrastructure Tools and Technology
  18. 18. Dynamic infrastructure platform Change delivery services Infrastructure management tools Operational management services Application Packaging and Deployment Development Tooling Application runtime platform
  19. 19. Dynamic infrastructure platform Change delivery services Infrastructure management tools Operational management services Application Packaging and Deployment Development Tooling Application runtime platform
  20. 20. Dynamic infrastructure platform Compute StorageNetwork A pool of resources
  21. 21. Provision resources • On demand • Programmatically
  22. 22. Physical Servers Containers Serverless (FaaS) Variations of compute resources Virtual Machines
  23. 23. Dynamic infrastructure platform Change delivery services Infrastructure management tools Operational management services Application Packaging and Deployment Development Tooling Application runtime platform
  24. 24. Stack management tool Stack instances Platform API Stack definition (Terraform, CloudFormation, ...)
  25. 25. Server configuration tool Configuration tool Configuration definitions Server instance (Ansible, Chef, Puppet, Saltstack, ...)
  26. 26. Server image creation Base server image Custom image Server configuration definitions Image definition (Packer, ...)
  27. 27. Concept: Infrastructure Stack
  28. 28. Infrastructure stack A collection of infrastructure resources provisioned and updated as a unit
  29. 29. A stack may include the infrastructure for a single application
  30. 30. Or, multiple applications might run in a single stack
  31. 31. Or the parts of one application could be spread across several stacks
  32. 32. Stack definition A stack definition is code used by a tool to provision stack instances Stack definition Stack instanceStack management tool
  33. 33. Stack instances A single stack definition may be used to manage one or more stack instances Stack definition Stack instances
  34. 34. Multiple environments
  35. 35. Starting small – single stack instance One or two people working on it, ... the system is fairly simple, ... and there are few users, with low stakes
  36. 36. And then ... More people join the team, ... the system becomes more complex, ... and more people rely on the system to be working
  37. 37. PRODUCTION QA TEST Multiple environments
  38. 38. PRODUCTION QA TEST Multiple environments defined in one stack instance our_env/ └── test.tf └── qa.tf └── production.tf
  39. 39. But the blast radius for a change is too wide PRODUCTION QA TEST
  40. 40. our_env/ └── test/ └── servers.tf our_env/ └── qa/ └── servers.tf our_env/ └── production/ └── servers.tf PRODUCTION QA TEST Separate stack definition project for each stack instance
  41. 41. QA PRODUCTION TEST Single definition project template, used to create a separate stack instance for each environment our_env/ └── servers.tf
  42. 42. Pipelines for infrastructure
  43. 43. Automatically test every change before applying it
  44. 44. Promote changes to environments using a pipeline BUILDLOCAL APPLY TO QA APPLY TO PROD Sandbox QA Production APPLY TO TEST Test
  45. 45. Stack tool is only run by pipeline agents BUILDLOCAL APPLY TO QA APPLY TO PROD APPLY TO TEST
  46. 46. Why not apply changes from our laptops? Our working environments may not be consistent Multiple people may clash We may shortcut our process We may neglect to run all appropriate tests
  47. 47. "Local" pre-commit sandboxes VCS Nita's Instance Alex's Instance Freddy's Instance
  48. 48. Parameterizing stacks
  49. 49. server "web" { name = "web-${env}" type = "${type}" } env = "prod" type = "t2.large" env = "qa" type = "t2.large" env = "test" type = "t2.tiny" web-test web-qa web-prod Parameters per stack instance
  50. 50. Parameters in source project folders web-test web-qa web-prod . ├── infra │ ├── web_server.tf │ ├── subnet.tf │ └── vpc.tf └── environments ├── test.tfvars ├── qa.tfvars └── prod.tfvars
  51. 51. env = "test" type = "t2.tiny" env = "qa" type = "t2.large" env = "prod" type = "t2.large" Parameters in pipeline job definitions BUILD APPLY TO QA APPLY TO PROD APPLY TO TEST web-test web-qa web-prod
  52. 52. Parameters from configuration registry web-test web-qa web-prod envs/test/type t2.tiny envs/qa/type t2.large envs/prod/type t2.large
  53. 53. Recommendation Minimize the configurable surface area to reduce variation between environments
  54. 54. Infrastructure testing
  55. 55. An effective testing regime drives a clean, decoupled, evolvable system design
  56. 56. Fitness functions Define the important characteristics of your system, how to measure these, and how to assess progress
  57. 57. Example: our-app stack Infrastructure stack JDK Tomcat OurApp.war Application server
  58. 58. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP Example pipeline for our-app
  59. 59. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP APPLICATION STAGES: Compile, unit tests, container tests, packaging, etc. (run on build agents) JUnit, Rspec, etc.
  60. 60. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI BUILD INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP SERVER PACKAGES: Validate, test individual package configurations in isolation SERVER ROLE: Test aggregated packages Typically run on build agents, perhaps containerized test-kitchen, serverspec, inspec, puppet-rspec, testinfra, etc.
  61. 61. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP SERVER IMAGE: Build and test serverspec, inspec, etc.
  62. 62. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP INFRASTRUCTURE STACK Validate (e.g. syntax, provision non-application specific infrastructure elements awspec, inspec, etc.
  63. 63. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP INTEGRATED STAGES: Test the various components in combination (deployment environments) selenium, etc.
  64. 64. Recommendations Those who write the code write the tests Avoid reflexive testing (AKA testing your tools) Test where there is risk, complexity Test contracts
  65. 65. Splitting stacks into microstacks
  66. 66. Monolithic stack
  67. 67. Coordinating changes Team A could make a change that breaks things for team B
  68. 68. Divide infrastructure into multiple, independently changeable stacks
  69. 69. Each stack has its own pipeline to deliver changes
  70. 70. Strategies for drawing boundaries
  71. 71. Draw boundaries to minimize changes that cross stacks
  72. 72. Infrastructure code boundaries != network boundaries
  73. 73. Infrastructure code boundaries != network boundaries
  74. 74. Infrastructure code boundaries != network boundaries
  75. 75. Group things together when they tend to be changed together
  76. 76. Align team boundaries and change boundaries
  77. 77. Patterns for infrastructure stacks
  78. 78. Shared infrastructure components
  79. 79. Base infrastructure module pipeline Application stack pipelines Components shared across stacks
  80. 80. Challenges with shared components Increases coupling
  81. 81. Shared infrastructure stacks
  82. 82. Shared stack instance Shared network resource stack Multiple application infrastructure stacks
  83. 83. Shared stack pipeline Application infrastructure pipelines
  84. 84. Shared infrastructure stacks The provider stack should not have knowledge of its consumers Becomes a bottleneck as the number of consumers grows Should be kept simple, minimal Split into smaller stacks to avoid monoliths
  85. 85. Shared nothing stacks
  86. 86. Each stack is self-contained
  87. 87. Each stack is independently releasable
  88. 88. Techniques for integrating stacks
  89. 89. Example of integrated stacks Shared infrastructure stack Application infrastructure stack vpc_id = "vpc-abcde" private_subnets = ["sn-12345", "sn-54321"]
  90. 90. Integration at apply-time vpc_id = "vpc-abcde" private_subnets = ["sn-12345", "sn-54321"]
  91. 91. Integration by resource discovery data "aws_vpc" "vpc" { filter { name = "tag:Environment" values = ["${var.env}"] } } data "aws_subnet" "private_subnet" { filter { name = "tag:Environment" values = ["${var.env}"] } }
  92. 92. Integration by configuration registry envs/qa/vpc_id vpc-abcde envs/qa/private_subnets "sn-12345", "sn-54321"
  93. 93. Integration by remote state
  94. 94. Recommendations Stack integration points are contracts Follow good practice for API design when making changes to contracts Use fitness functions (automated tests) to ensure the integrity of contracts
  95. 95. kief@thoughtworks.com Infrastructure Technologist@ Twitter: @kief Book: http://oreil.ly/1JKIBVe Site: http://infrastructure-as-code.com

×