Evolutionary infrastructure agile 2018 - kief morris

1. Building EVOLUTIONARY infrastructure

2. kief@thoughtworks.com Infrastructure Technologist@ Twitter: @kief Book: http://oreil.ly/1JKIBVe Site: http://infrastructure-as-code.com Some visuals found from the Vault of the Atomic Space Age http://bit.ly/2KBTbAd

3. Why? Ongoing pressure to deliver while requirements change

4. PRODTESTDEV Inconsistent path to production

5. Snowflake production

6. PROD Continuous, reliable delivery of changes to all your stuff TESTDEV

7. "Since we can't avoid change, we need to exploit it" Building Evolutionary Architectures Neal Ford, Rebecca Parsons, Pat Kua

8. Speed or Quality?

9. Fast Slow Careless Careful

10. Move fast and fix things

11. Fast Slow Careless Careful

12. FastSlow Careful Careless

13. FastSlow Careless Prioritize speed over correctness Fragile mess

14. Slow Careful Careless Prioritize correctness over speed of change Fragile mess

15. Slow Careful Careless Prioritize correctness over speed of change Fragile mess Time pressures Over- engineering Technical debt Barriers to improvement

16. FastSlow Careful Careless Agile, Lean, DevOpsBuild the simplest thing Build quality in Optimize for change Fast, frequent feedback

17. Infrastructure Tools and Technology

18. Dynamic infrastructure platform Change delivery services Infrastructure management tools Operational management services Application Packaging and Deployment Development Tooling Application runtime platform

20. Dynamic infrastructure platform Compute StorageNetwork A pool of resources

21. Provision resources • On demand • Programmatically

22. Physical Servers Containers Serverless (FaaS) Variations of compute resources Virtual Machines

24. Stack management tool Stack instances Platform API Stack definition (Terraform, CloudFormation, ...)

25. Server configuration tool Configuration tool Configuration definitions Server instance (Ansible, Chef, Puppet, Saltstack, ...)

26. Server image creation Base server image Custom image Server configuration definitions Image definition (Packer, ...)

27. Concept: Infrastructure Stack

28. Infrastructure stack A collection of infrastructure resources provisioned and updated as a unit

29. A stack may include the infrastructure for a single application

30. Or, multiple applications might run in a single stack

31. Or the parts of one application could be spread across several stacks

32. Stack definition A stack definition is code used by a tool to provision stack instances Stack definition Stack instanceStack management tool

33. Stack instances A single stack definition may be used to manage one or more stack instances Stack definition Stack instances

34. Multiple environments

35. Starting small – single stack instance One or two people working on it, ... the system is fairly simple, ... and there are few users, with low stakes

36. And then ... More people join the team, ... the system becomes more complex, ... and more people rely on the system to be working

37. PRODUCTION QA TEST Multiple environments

38. PRODUCTION QA TEST Multiple environments defined in one stack instance our_env/ └── test.tf └── qa.tf └── production.tf

39. But the blast radius for a change is too wide PRODUCTION QA TEST

40. our_env/ └── test/ └── servers.tf our_env/ └── qa/ └── servers.tf our_env/ └── production/ └── servers.tf PRODUCTION QA TEST Separate stack definition project for each stack instance

41. QA PRODUCTION TEST Single definition project template, used to create a separate stack instance for each environment our_env/ └── servers.tf

42. Pipelines for infrastructure

43. Automatically test every change before applying it

44. Promote changes to environments using a pipeline BUILDLOCAL APPLY TO QA APPLY TO PROD Sandbox QA Production APPLY TO TEST Test

45. Stack tool is only run by pipeline agents BUILDLOCAL APPLY TO QA APPLY TO PROD APPLY TO TEST

46. Why not apply changes from our laptops? Our working environments may not be consistent Multiple people may clash We may shortcut our process We may neglect to run all appropriate tests

47. "Local" pre-commit sandboxes VCS Nita's Instance Alex's Instance Freddy's Instance

48. Parameterizing stacks

49. server "web" { name = "web-${env}" type = "${type}" } env = "prod" type = "t2.large" env = "qa" type = "t2.large" env = "test" type = "t2.tiny" web-test web-qa web-prod Parameters per stack instance

50. Parameters in source project folders web-test web-qa web-prod . ├── infra │ ├── web_server.tf │ ├── subnet.tf │ └── vpc.tf └── environments ├── test.tfvars ├── qa.tfvars └── prod.tfvars

51. env = "test" type = "t2.tiny" env = "qa" type = "t2.large" env = "prod" type = "t2.large" Parameters in pipeline job definitions BUILD APPLY TO QA APPLY TO PROD APPLY TO TEST web-test web-qa web-prod

52. Parameters from configuration registry web-test web-qa web-prod envs/test/type t2.tiny envs/qa/type t2.large envs/prod/type t2.large

53. Recommendation Minimize the configurable surface area to reduce variation between environments

54. Infrastructure testing

55. An effective testing regime drives a clean, decoupled, evolvable system design

56. Fitness functions Define the important characteristics of your system, how to measure these, and how to assess progress

57. Example: our-app stack Infrastructure stack JDK Tomcat OurApp.war Application server

58. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP Example pipeline for our-app

59. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP APPLICATION STAGES: Compile, unit tests, container tests, packaging, etc. (run on build agents) JUnit, Rspec, etc.

60. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI BUILD INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP SERVER PACKAGES: Validate, test individual package configurations in isolation SERVER ROLE: Test aggregated packages Typically run on build agents, perhaps containerized test-kitchen, serverspec, inspec, puppet-rspec, testinfra, etc.

61. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP SERVER IMAGE: Build and test serverspec, inspec, etc.

62. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP INFRASTRUCTURE STACK Validate (e.g. syntax, provision non-application specific infrastructure elements awspec, inspec, etc.

63. QA PRODTEST ALL TEST JDK COOKBOOK TEST TOMCAT COOKBOOK TEST APP SERVER ROLE BUILD APP SERVER AMI VALIDATE INFRA STACK BUILD OUR-APP TEST INFRA STACK TEST OUR- APP INTEGRATED STAGES: Test the various components in combination (deployment environments) selenium, etc.

64. Recommendations Those who write the code write the tests Avoid reflexive testing (AKA testing your tools) Test where there is risk, complexity Test contracts

65. Splitting stacks into microstacks

66. Monolithic stack

67. Coordinating changes Team A could make a change that breaks things for team B

68. Divide infrastructure into multiple, independently changeable stacks

69. Each stack has its own pipeline to deliver changes

70. Strategies for drawing boundaries

71. Draw boundaries to minimize changes that cross stacks

72. Infrastructure code boundaries != network boundaries

75. Group things together when they tend to be changed together

76. Align team boundaries and change boundaries

77. Patterns for infrastructure stacks

78. Shared infrastructure components

79. Base infrastructure module pipeline Application stack pipelines Components shared across stacks

80. Challenges with shared components Increases coupling

81. Shared infrastructure stacks

82. Shared stack instance Shared network resource stack Multiple application infrastructure stacks

83. Shared stack pipeline Application infrastructure pipelines

84. Shared infrastructure stacks The provider stack should not have knowledge of its consumers Becomes a bottleneck as the number of consumers grows Should be kept simple, minimal Split into smaller stacks to avoid monoliths

85. Shared nothing stacks

86. Each stack is self-contained

87. Each stack is independently releasable

88. Techniques for integrating stacks

89. Example of integrated stacks Shared infrastructure stack Application infrastructure stack vpc_id = "vpc-abcde" private_subnets = ["sn-12345", "sn-54321"]

90. Integration at apply-time vpc_id = "vpc-abcde" private_subnets = ["sn-12345", "sn-54321"]

91. Integration by resource discovery data "aws_vpc" "vpc" { filter { name = "tag:Environment" values = ["${var.env}"] } } data "aws_subnet" "private_subnet" { filter { name = "tag:Environment" values = ["${var.env}"] } }

92. Integration by configuration registry envs/qa/vpc_id vpc-abcde envs/qa/private_subnets "sn-12345", "sn-54321"

93. Integration by remote state

94. Recommendations Stack integration points are contracts Follow good practice for API design when making changes to contracts Use fitness functions (automated tests) to ensure the integrity of contracts

95. kief@thoughtworks.com Infrastructure Technologist@ Twitter: @kief Book: http://oreil.ly/1JKIBVe Site: http://infrastructure-as-code.com

Evolutionary infrastructure agile 2018 - kief morris

Kief Morris

Evolutionary infrastructure agile 2018 - kief morris