Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to test infrastructure code: automated testing for Terraform, Kubernetes, Docker, Packer and more

5,785 views

Published on

This talk is a step-by-step, live-coding class on how to write automated tests for infrastructure code, including the code you write for use with tools such as Terraform, Kubernetes, Docker, and Packer. Topics covered include unit tests, integration tests, end-to-end tests, test parallelism, retries, error handling, static analysis, and more.

Published in: Software
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

How to test infrastructure code: automated testing for Terraform, Kubernetes, Docker, Packer and more

  1. 1. Automated testing for: ✓ terraform ✓ docker ✓ packer ✓ kubernetes ✓ and more Passed: 5. Failed: 0. Skipped: 0. Test run successful. How to test infrastructure code
  2. 2. The DevOps world is full of Fear
  3. 3. Fear of outages
  4. 4. Fear of security breaches
  5. 5. Fear of data loss
  6. 6. Fear of change
  7. 7. “Fear leads to anger. Anger leads to hate. Hate leads to suffering.” Scrum Master Yoda
  8. 8. And you all know what suffering leads to, right?
  9. 9. Credit: Daniele Polencic
  10. 10. Many DevOps teams deal with this fear in two ways:
  11. 11. 1) Heavy drinking and smoking
  12. 12. 2) Deploying less frequently
  13. 13. Sadly, both of these just make the problem worse!
  14. 14. There’s a better way to deal with this fear:
  15. 15. Automated tests
  16. 16. Automated tests give you the confidence to make changes
  17. 17. Fight fear with confidence
  18. 18. We know how to write automated tests for application code…
  19. 19. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But how do you test your Terraform code deploys infrastructure that works?
  20. 20. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers: - name: hello-world-app image: gruntwork-io/hello-world-app:v1 ports: - containerPort: 8080 How do you test your Kubernetes code configures your services correctly?
  21. 21. This talk is about how to write tests for your infrastructure code.
  22. 22. I’m Yevgeniy Brikman ybrikman.com
  23. 23. Co-founder of Gruntwork gruntwork.io
  24. 24. Author
  25. 25. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  26. 26. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  27. 27. Static analysis: test your code without deploying it.
  28. 28. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  29. 29. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  30. 30. Statically check your code for syntactic and structural issues
  31. 31. Tool Command Terraform terraform validate Packer packer validate <template> Kubernetes kubectl apply -f <file> --dry-run --validate=true Examples:
  32. 32. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  33. 33. Statically validate your code to catch common errors
  34. 34. Tool Linters Terraform 1. conftest 2. terraform_validate 3. tflint Docker 1. dockerfile_lint 2. hadolint 3. dockerfilelint Kubernetes 1. kube-score 2. kube-lint 3. yamllint Examples:
  35. 35. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  36. 36. Partially execute the code and validate the “plan”, but don’t actually deploy
  37. 37. Tool Dry run options Terraform 1. terraform plan 2. HashiCorp Sentinel 3. terraform-compliance Kubernetes kubectl apply -f <file> --server-dry-run Examples:
  38. 38. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  39. 39. Unit tests: test a single “unit” works in isolation.
  40. 40. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  41. 41. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  42. 42. You can’t “unit test” an entire end- to-end architecture
  43. 43. Instead, break your infra code into small modules and unit test those! module module module module module module module module module module module module module module module
  44. 44. With app code, you can test units in isolation from the outside world
  45. 45. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But 99% of infrastructure code is about talking to the outside world…
  46. 46. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } If you try to isolate a unit from the outside world, you’re left with nothing!
  47. 47. So you can only test infra code by deploying to a real environment
  48. 48. Key takeaway: there’s no pure unit testing for infrastructure code.
  49. 49. Therefore, the test strategy is: 1. Deploy real infrastructure 2. Validate it works (e.g., via HTTP requests, API calls, SSH commands, etc.) 3. Undeploy the infrastructure (So it’s really integration testing of a single unit!)
  50. 50. Tool Deploy / Undeploy Validate Works with Terratest Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc. kitchen-terraform Yes Yes Terraform Inspec No Yes Servers, Cloud APIs Serverspec No Yes Servers Goss No Yes Servers Tools that help with this strategy:
  51. 51. Tool Deploy / Undeploy Validate Works with Terratest Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc. kitchen-terraform Yes Yes Terraform Inspec No Yes Servers, Cloud APIs Serverspec No Yes Servers Goss No Yes Servers In this talk, we’ll use Terratest:
  52. 52. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  53. 53. Sample code for this talk is at: github.com/gruntwork-io/infrastructure-as-code-testing-talk
  54. 54. An example of a Terraform module you may want to test:
  55. 55. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ main.tf └ outputs.tf └ variables.tf └ modules └ test └ README.md hello-world-app: deploy a “Hello, World” web service
  56. 56. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } Under the hood, this example runs on top of AWS Lambda & API Gateway
  57. 57. $ terraform apply Outputs: url = ruvvwv3sh1.execute-api.us-east-2.amazonaws.com $ curl ruvvwv3sh1.execute-api.us-east-2.amazonaws.com Hello, World! When you run terraform apply, it deploys and outputs the URL
  58. 58. Let’s write a unit test for hello-world-app with Terratest
  59. 59. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ README.md Create hello_world_app_test.go
  60. 60. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } The basic test structure
  61. 61. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 1. Tell Terratest where your Terraform code lives
  62. 62. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 2. Run terraform init and terraform apply to deploy your module
  63. 63. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 3. Validate the infrastructure works. We’ll come back to this shortly.
  64. 64. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 4. Run terraform destroy at the end of the test to undeploy everything
  65. 65. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate function
  66. 66. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 1. Run terraform output to get the web service URL
  67. 67. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 2. Make HTTP requests to the URL
  68. 68. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 3. Check the response for an expected status and body
  69. 69. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 4. Retry the request up to 10 times, as deployment is asynchronous
  70. 70. Note: since we’re testing a web service, we use HTTP requests to validate it.
  71. 71. Infrastructure Example Validate with… Example Web service Dockerized web app HTTP requests Terratest http_helper package Server EC2 instance SSH commands Terratest ssh package Cloud service SQS Cloud APIs Terratest aws or gcp packages Database MySQL SQL queries MySQL driver for Go Examples of other ways to validate:
  72. 72. $ export AWS_ACCESS_KEY_ID=xxxx $ export AWS_SECRET_ACCESS_KEY=xxxxx To run the test, first authenticate to AWS
  73. 73. $ go test -v -timeout 15m -run TestHelloWorldAppUnit … --- PASS: TestHelloWorldAppUnit (31.57s) Then run go test. You now have a unit test you can run after every commit!
  74. 74. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  75. 75. What about other tools, such as Docker + Kubernetes?
  76. 76. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ Dockerfile └ deployment.yml └ modules └ test └ README.md docker-kubernetes: deploy a “Hello, World” web service to Kubernetes
  77. 77. FROM ubuntu:18.04 EXPOSE 8080 RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y busybox RUN echo 'Hello, World!' > index.html CMD ["busybox", "httpd", "-f", "-p", "8080"] Dockerfile: Dockerize a simple “Hello, World!” web service
  78. 78. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers: - name: hello-world-app image: gruntwork-io/hello-world-app:v1 ports: - containerPort: 8080 deployment.yml: define how to deploy a Docker container in Kubernetes
  79. 79. $ cd examples/docker-kubernetes $ docker build -t gruntwork-io/hello-world-app:v1 . Successfully tagged gruntwork-io/hello-world-app:v1 $ kubectl apply -f deployment.yml deployment.apps/hello-world-app-deployment created service/hello-world-app-service created $ curl localhost:8080 Hello, World! Build the Docker image, deploy to Kubernetes, and check URL
  80. 80. Let’s write a unit test for this code.
  81. 81. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ docker_kubernetes_test.go └ README.md Create docker_kubernetes_test.go
  82. 82. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } The basic test structure
  83. 83. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 1. Build the Docker image. You’ll see the buildDockerImage method shortly.
  84. 84. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 2. Tell Terratest where your Kubernetes deployment is defined
  85. 85. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 3. Configure kubectl options to authenticate to Kubernetes
  86. 86. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 4. Run kubectl apply to deploy the web app to Kubernetes
  87. 87. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 5. Check the app is working. You’ll see the validate method shortly.
  88. 88. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 6. At the end of the test, remove all Kubernetes resources you deployed
  89. 89. func buildDockerImage(t *testing.T) { options := &docker.BuildOptions{ Tags: []string{"gruntwork-io/hello-world-app:v1"}, } path := "../examples/docker-kubernetes" docker.Build(t, path, options) } The buildDockerImage method
  90. 90. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } The validate method
  91. 91. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 1. Wait until the service is deployed
  92. 92. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 2. Make HTTP requests
  93. 93. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 3. Use serviceUrl method to get URL
  94. 94. func serviceUrl(t *testing.T, opts *k8s.KubectlOptions) string { service := k8s.GetService(t, options, "hello-world-app-service") endpoint := k8s.GetServiceEndpoint(t, options, service, 8080) return fmt.Sprintf("http://%s", endpoint) } The serviceUrl method
  95. 95. $ kubectl config set-credentials … To run the test, first authenticate to a Kubernetes cluster.
  96. 96. Note: Kubernetes is now part of Docker Desktop. Test 100% locally!
  97. 97. $ go test -v -timeout 15m -run TestDockerKubernetes … --- PASS: TestDockerKubernetes (5.69s) Run go test. You can validate your config after every commit in seconds!
  98. 98. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  99. 99. Note: tests create and destroy many resources!
  100. 100. Pro tip #1: run tests in completely separate “sandbox” accounts
  101. 101. Tool Clouds Features cloud-nuke AWS (GCP planned) Delete all resources older than a certain date; in a certain region; of a certain type. Janitor Monkey AWS Configurable rules of what to delete. Notify owners of pending deletions. aws-nuke AWS Specify specific AWS accounts and resource types to target. Azure Powershell Azure Includes native commands to delete Resource Groups Pro tip #2: run these tools in cron jobs to clean up left-over resources
  102. 102. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  103. 103. Integration tests: test multiple “units” work together.
  104. 104. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  105. 105. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  106. 106. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md Let’s say you have two Terraform modules you want to test together:
  107. 107. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md proxy-app: an app that acts as an HTTP proxy for other web services.
  108. 108. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md web-service: a web service that you want proxied.
  109. 109. variable "url_to_proxy" { description = "The URL to proxy." type = string } proxy-app takes in the URL to proxy via an input variable
  110. 110. output "url" { value = module.web_service.url } web-service exposes its URL via an output variable
  111. 111. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ docker_kubernetes_test.go └ proxy_app_test.go └ README.md Create proxy_app_test.go
  112. 112. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } The basic test structure
  113. 113. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 1. Configure options for the web service
  114. 114. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 2. Deploy the web service
  115. 115. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 3. Configure options for the proxy app (passing it the web service options)
  116. 116. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 4. Deploy the proxy app
  117. 117. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 5. Validate the proxy app works
  118. 118. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 6. At the end of the test, undeploy the proxy app and the web service
  119. 119. func configWebService(t *testing.T) *terraform.Options { return &terraform.Options{ TerraformDir: "../examples/web-service", } } The configWebService method
  120. 120. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } The configProxyApp method
  121. 121. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 1. Read the url output from the web- service module
  122. 122. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 2. Pass it in as the url_to_proxy input to the proxy-app module
  123. 123. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code `{"text":"Hello, World!"}`, // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate method
  124. 124. $ go test -v -timeout 15m -run TestProxyApp … --- PASS: TestProxyApp (182.44s) Run go test. You’re now testing multiple modules together!
  125. 125. $ go test -v -timeout 15m -run TestProxyApp … --- PASS: TestProxyApp (182.44s) But integration tests can take (many) minutes to run…
  126. 126. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  127. 127. Infrastructure tests can take a long time to run
  128. 128. One way to save time: run tests in parallel
  129. 129. func TestProxyApp(t *testing.T) { t.Parallel() // The rest of the test code } func TestHelloWorldAppUnit(t *testing.T) { t.Parallel() // The rest of the test code } Enable test parallelism in Go by adding t.Parallel() as the 1st line of each test.
  130. 130. $ go test -v -timeout 15m === RUN TestHelloWorldApp === RUN TestDockerKubernetes === RUN TestProxyApp Now, if you run go test, all the tests with t.Parallel() will run in parallel
  131. 131. But there’s a gotcha: resource conflicts
  132. 132. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } Example: module with hard-coded IAM Role and Security Group names
  133. 133. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } If two tests tried to deploy this module in parallel, the names would conflict!
  134. 134. Key takeaway: you must namespace all your resources
  135. 135. resource "aws_iam_role" "role_example" { name = var.name } resource "aws_security_group" "sg_example" { name = var.name } Example: use variables in all resource names…
  136. 136. uniqueId := random.UniqueId() return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "name": fmt.Sprintf("text-proxy-app-%s", uniqueId) }, } At test time, set the variables to a randomized value to avoid conflicts
  137. 137. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  138. 138. Consider the structure of the proxy-app integration test:
  139. 139. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service
  140. 140. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service When iterating locally, you sometimes want to re-run just one of these steps.
  141. 141. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service But as the code is written now, you have to run all steps on each test run.
  142. 142. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service And that can add up to a lot of overhead. (~3 min) (~2 min) (~30 seconds) (~1 min) (~2 min)
  143. 143. Key takeaway: break your tests into independent test stages
  144. 144. webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) The original test structure
  145. 145. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) The test structure with test stages
  146. 146. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 1. RunTestStage is a helper function from Terratest.
  147. 147. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 2. Wrap each stage of your test with a call to RunTestStage
  148. 148. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 3. Define each stage in a function (you’ll see this code shortly).
  149. 149. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 4. Give each stage a unique name
  150. 150. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) Any stage foo can be skipped by setting the env var SKIP_foo=true
  151. 151. $ SKIP_cleanup_web_service=true $ SKIP_cleanup_proxy_app=true Example: on the very first test run, skip the cleanup stages.
  152. 152. $ go test -v -timeout 15m -run TestProxyApp Running stage 'deploy_web_service'… Running stage 'deploy_proxy_app'… Running stage 'validate'… Skipping stage 'cleanup_proxy_app'… Skipping stage 'cleanup_web_service'… --- PASS: TestProxyApp (105.73s) That way, after the test finishes, the infrastructure will still be running.
  153. 153. $ SKIP_deploy_web_service=true $ SKIP_deploy_proxy_app=true Now, on the next several test runs, you can skip the deploy stages too.
  154. 154. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_service’… Skipping stage 'deploy_proxy_app'… Running stage 'validate'… Skipping stage 'cleanup_proxy_app'… Skipping stage 'cleanup_web_service'… --- PASS: TestProxyApp (14.22s) This allows you to iterate on solely the validate stage…
  155. 155. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_service’… Skipping stage 'deploy_proxy_app'… Running stage 'validate'… Skipping stage 'cleanup_proxy_app'… Skipping stage 'cleanup_web_service'… --- PASS: TestProxyApp (14.22s) Which dramatically speeds up your iteration / feedback cycle!
  156. 156. $ SKIP_validate=true $ unset SKIP_cleanup_web_service $ unset SKIP_cleanup_proxy_app When you’re done iterating, skip validate and re-enable cleanup
  157. 157. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_service’… Skipping stage 'deploy_proxy_app’… Skipping stage 'validate’… Running stage 'cleanup_proxy_app’… Running stage 'cleanup_web_service'… --- PASS: TestProxyApp (59.61s) This cleans up everything that was left running.
  158. 158. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } Note: each time you run test stages via go test, it’s a separate OS process.
  159. 159. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } So to pass data between stages, one stage needs to write the data to disk…
  160. 160. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } And the other stages need to read that data from disk.
  161. 161. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  162. 162. Real infrastructure can fail for intermittent reasons (e.g., bad EC2 instance, Apt downtime, Terraform bug)
  163. 163. To avoid “flaky” tests, add retries for known errors.
  164. 164. &terraform.Options{ TerraformDir: "../examples/proxy-app", RetryableTerraformErrors: map[string]string{ "net/http: TLS handshake timeout": "Terraform bug", }, MaxRetries: 3, TimeBetweenRetries: 3*time.Second, } Example: retry up to 3 times on a known TLS error in Terraform.
  165. 165. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  166. 166. End-to-end tests: test your entire infrastructure works together.
  167. 167. How do you test this entire thing?
  168. 168. You could use the same strategy… 1. Deploy all the infrastructure 2. Validate it works (e.g., via HTTP requests, API calls, SSH commands, etc.) 3. Undeploy all the infrastructure
  169. 169. But it’s rare to write end-to- end tests this way. Here’s why:
  170. 170. e2e Tests Test pyramid Integration Tests Unit Tests Static analysis
  171. 171. e2e Tests Integration Tests Unit Tests Static analysis Cost, brittleness, run time
  172. 172. e2e Tests Integration Tests Unit Tests Static analysis 60 – 240+ minutes 5 – 60 minutes 1 – 20 minutes 1 – 60 seconds
  173. 173. e2e Tests Integration Tests Unit Tests Static analysis E2E tests are too slow to be useful 60 – 240+ minutes 5 – 60 minutes 1 – 20 minutes 1 – 60 seconds
  174. 174. Another problem with E2E tests: brittleness.
  175. 175. Let’s do some math:
  176. 176. Assume a single resource (e.g., EC2 instance) has a 1/1000 (0.1%) chance of failure.
  177. 177. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ The more resources your tests deploy, the flakier they will be.
  178. 178. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ You can work around the failure rate for unit & integration tests with retries
  179. 179. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ You can work around the failure rate for unit & integration tests with retries
  180. 180. Key takeaway: E2E tests from scratch are too slow and too brittle to be useful
  181. 181. Instead, you can do incremental E2E testing!
  182. 182. module module module module module module module module module module module module module module module 1. Deploy a persistent test environment and leave it running.
  183. 183. module module module module module module module module module module module module module module module 2. Each time you update a module, deploy & validate just that module
  184. 184. module module module module module module module module module module module module module module module 3. Bonus: test your deployment process is zero-downtime too!
  185. 185. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  186. 186. Testing techniques compared:
  187. 187. Technique Strengths Weaknesses Static analysis 1. Fast 2. Stable 3. No need to deploy real resources 4. Easy to use 1. Very limited in errors you can catch 2. You don’t get much confidence in your code solely from static analysis Unit tests 1. Fast enough (1 – 10 min) 2. Mostly stable (with retry logic) 3. High level of confidence in individual units 1. Need to deploy real resources 2. Requires writing non-trivial code Integration tests 1. Mostly stable (with retry logic) 2. High level of confidence in multiple units working together 1. Need to deploy real resources 2. Requires writing non-trivial code 3. Slow (10 – 30 min) End-to-end tests 1. Build confidence in your entire architecture 1. Need to deploy real resources 2. Requires writing non-trivial code 3. Very slow (60 min – 240+ min)* 4. Can be brittle (even with retry logic)*
  188. 188. So which should you use?
  189. 189. All of them! They all catch different types of bugs.
  190. 190. e2e Tests Keep in mind the test pyramid Integration Tests Unit Tests Static analysis
  191. 191. e2e Tests Lots of unit tests + static analysis Integration Tests Unit Tests Static analysis
  192. 192. e2e Tests Fewer integration tests Integration Tests Unit Tests Static analysis
  193. 193. e2e Tests A handful of high-value e2e tests Integration Tests Unit Tests Static analysis
  194. 194. Infrastructure code without tests is scary
  195. 195. Fight the fear & build confidence in your code with automated tests
  196. 196. Questions? info@gruntwork.io

×