1.
Automated testing for:
✓ terraform
✓ docker
✓ packer
✓ kubernetes
✓ and more
Passed: 5. Failed: 0. Skipped: 0.
Test run successful.
How to
test
infrastructure
code

2.
The DevOps world is full of
Fear

3.
Fear of outages

4.
Fear of security breaches

5.
Fear of data loss

6.
Fear of change

7.
“Fear leads to
anger. Anger
leads to hate.
Hate leads to
suffering.”
Scrum Master Yoda

8.
And you all know what
suffering leads to, right?

9.
Credit: Daniele Polencic

10.
Many DevOps teams deal
with this fear in two ways:

11.
1) Heavy drinking and smoking

12.
2) Deploying less frequently

13.
Sadly, both of these just make
the problem worse!

14.
There’s a better way to deal
with this fear:

15.
Automated tests

16.
Automated tests give you the
confidence to make changes

17.
Fight fear with confidence

18.
We know how to write automated
tests for application code…

19.
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
But how do you test your Terraform code
deploys infrastructure that works?

20.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-app-deployment
spec:
selector:
matchLabels:
app: hello-world-app
replicas: 1
spec:
containers:
- name: hello-world-app
image: gruntwork-io/hello-world-app:v1
ports:
- containerPort: 8080
How do you test your Kubernetes code
configures your services correctly?

21.
This talk is about how to write
tests for your infrastructure code.

22.
I’m
Yevgeniy
Brikman
ybrikman.com

23.
Co-founder of
Gruntwork
gruntwork.io

24.
Author

25.
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline

26.
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline

27.
Static analysis: test your code
without deploying it.

28.
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run

29.
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run

30.
Statically check your code for
syntactic and structural issues

31.
Tool Command
Terraform terraform validate
Packer packer validate <template>
Kubernetes kubectl apply -f <file> --dry-run --validate=true
Examples:

32.
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run

33.
Statically validate your code to
catch common errors

34.
Tool Linters
Terraform
1. conftest
2. terraform_validate
3. tflint
Docker
1. dockerfile_lint
2. hadolint
3. dockerfilelint
Kubernetes
1. kube-score
2. kube-lint
3. yamllint
Examples:

35.
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run

36.
Partially execute the code and
validate the “plan”, but don’t
actually deploy

37.
Tool Dry run options
Terraform
1. terraform plan
2. HashiCorp Sentinel
3. terraform-compliance
Kubernetes kubectl apply -f <file> --server-dry-run
Examples:

38.
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline

39.
Unit tests: test a single “unit”
works in isolation.

40.
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests

41.
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests

42.
You can’t “unit test” an entire end-
to-end architecture

43.
Instead, break your infra code into
small modules and unit test those!
module
module
module
module
module
module
module
module
module
module
module
module
module module
module

44.
With app code, you can test units
in isolation from the outside world

45.
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
But 99% of infrastructure code is about
talking to the outside world…

46.
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
If you try to isolate a unit from the
outside world, you’re left with nothing!

47.
So you can only test infra code by
deploying to a real environment

48.
Key takeaway: there’s no pure
unit testing for infrastructure
code.

49.
Therefore, the test strategy is:
1. Deploy real infrastructure
2. Validate it works
(e.g., via HTTP requests, API calls, SSH commands, etc.)
3. Undeploy the infrastructure
(So it’s really integration testing of a single unit!)

50.
Tool
Deploy /
Undeploy
Validate Works with
Terratest Yes Yes
Terraform, Kubernetes, Packer,
Docker, Servers, Cloud APIs, etc.
kitchen-terraform Yes Yes Terraform
Inspec No Yes Servers, Cloud APIs
Serverspec No Yes Servers
Goss No Yes Servers
Tools that help with this strategy:

51.
Tool
Deploy /
Undeploy
Validate Works with
Terratest Yes Yes
Terraform, Kubernetes, Packer,
Docker, Servers, Cloud APIs, etc.
kitchen-terraform Yes Yes Terraform
Inspec No Yes Servers, Cloud APIs
Serverspec No Yes Servers
Goss No Yes Servers
In this talk, we’ll use Terratest:

52.
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests

53.
Sample code for this talk is at:
github.com/gruntwork-io/infrastructure-as-code-testing-talk

54.
An example of a Terraform
module you may want to test:

55.
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ main.tf
└ outputs.tf
└ variables.tf
└ modules
└ test
└ README.md
hello-world-app: deploy a “Hello,
World” web service

56.
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
Under the hood, this example runs on
top of AWS Lambda & API Gateway

57.
$ terraform apply
Outputs:
url = ruvvwv3sh1.execute-api.us-east-2.amazonaws.com
$ curl ruvvwv3sh1.execute-api.us-east-2.amazonaws.com
Hello, World!
When you run terraform apply, it
deploys and outputs the URL

58.
Let’s write a unit test for
hello-world-app with Terratest

59.
infrastructure-as-code-testing-talk
└ examples
└ modules
└ test
└ hello_world_app_test.go
└ README.md
Create hello_world_app_test.go

60.
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
The basic test structure

61.
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
1. Tell Terratest where your Terraform
code lives

62.
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
2. Run terraform init and terraform
apply to deploy your module

63.
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
3. Validate the infrastructure works.
We’ll come back to this shortly.

64.
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
4. Run terraform destroy at the end of
the test to undeploy everything

65.
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
The validate function

66.
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
1. Run terraform output to get the web
service URL

67.
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
2. Make HTTP requests to the URL

68.
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
3. Check the response for an expected
status and body

69.
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
4. Retry the request up to 10 times, as
deployment is asynchronous

70.
Note: since we’re testing a
web service, we use HTTP
requests to validate it.

71.
Infrastructure Example Validate with… Example
Web service Dockerized web app HTTP requests Terratest http_helper package
Server EC2 instance SSH commands Terratest ssh package
Cloud service SQS Cloud APIs Terratest aws or gcp packages
Database MySQL SQL queries MySQL driver for Go
Examples of other ways to validate:

72.
$ export AWS_ACCESS_KEY_ID=xxxx
$ export AWS_SECRET_ACCESS_KEY=xxxxx
To run the test, first authenticate to
AWS

73.
$ go test -v -timeout 15m -run TestHelloWorldAppUnit
…
--- PASS: TestHelloWorldAppUnit (31.57s)
Then run go test. You now have a unit
test you can run after every commit!

74.
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests

75.
What about other tools, such
as Docker + Kubernetes?

76.
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ Dockerfile
└ deployment.yml
└ modules
└ test
└ README.md
docker-kubernetes: deploy a “Hello,
World” web service to Kubernetes

77.
FROM ubuntu:18.04
EXPOSE 8080
RUN DEBIAN_FRONTEND=noninteractive apt-get update &&
apt-get install -y busybox
RUN echo 'Hello, World!' > index.html
CMD ["busybox", "httpd", "-f", "-p", "8080"]
Dockerfile: Dockerize a simple “Hello,
World!” web service

78.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-app-deployment
spec:
selector:
matchLabels:
app: hello-world-app
replicas: 1
spec:
containers:
- name: hello-world-app
image: gruntwork-io/hello-world-app:v1
ports:
- containerPort: 8080
deployment.yml: define how to deploy a
Docker container in Kubernetes

79.
$ cd examples/docker-kubernetes
$ docker build -t gruntwork-io/hello-world-app:v1 .
Successfully tagged gruntwork-io/hello-world-app:v1
$ kubectl apply -f deployment.yml
deployment.apps/hello-world-app-deployment created
service/hello-world-app-service created
$ curl localhost:8080
Hello, World!
Build the Docker image, deploy to
Kubernetes, and check URL

80.
Let’s write a unit test for this
code.

81.
infrastructure-as-code-testing-talk
└ examples
└ modules
└ test
└ hello_world_app_test.go
└ docker_kubernetes_test.go
└ README.md
Create docker_kubernetes_test.go

82.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
The basic test structure

83.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
1. Build the Docker image. You’ll see
the buildDockerImage method shortly.

84.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
2. Tell Terratest where your Kubernetes
deployment is defined

85.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
3. Configure kubectl options to
authenticate to Kubernetes

86.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
4. Run kubectl apply to deploy the web
app to Kubernetes

87.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
5. Check the app is working. You’ll see
the validate method shortly.

88.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
6. At the end of the test, remove all
Kubernetes resources you deployed

89.
func buildDockerImage(t *testing.T) {
options := &docker.BuildOptions{
Tags: []string{"gruntwork-io/hello-world-app:v1"},
}
path := "../examples/docker-kubernetes"
docker.Build(t, path, options)
}
The buildDockerImage method

90.
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
The validate method

91.
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
1. Wait until the service is deployed

92.
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
2. Make HTTP requests

93.
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
3. Use serviceUrl method to get URL

94.
func serviceUrl(t *testing.T, opts *k8s.KubectlOptions) string {
service := k8s.GetService(t, options, "hello-world-app-service")
endpoint := k8s.GetServiceEndpoint(t, options, service, 8080)
return fmt.Sprintf("http://%s", endpoint)
}
The serviceUrl method

95.
$ kubectl config set-credentials …
To run the test, first authenticate to a
Kubernetes cluster.

96.
Note: Kubernetes is now part of
Docker Desktop. Test 100% locally!

97.
$ go test -v -timeout 15m -run TestDockerKubernetes
…
--- PASS: TestDockerKubernetes (5.69s)
Run go test. You can validate your
config after every commit in seconds!

98.
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests

99.
Note: tests create and destroy
many resources!

100.
Pro tip #1: run tests in completely
separate “sandbox” accounts

101.
Tool Clouds Features
cloud-nuke AWS (GCP planned)
Delete all resources older than a certain
date; in a certain region; of a certain type.
Janitor Monkey AWS
Configurable rules of what to delete.
Notify owners of pending deletions.
aws-nuke AWS
Specify specific AWS accounts and
resource types to target.
Azure Powershell Azure
Includes native commands to delete
Resource Groups
Pro tip #2: run these tools in cron jobs
to clean up left-over resources

102.
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline

103.
Integration tests: test multiple
“units” work together.

104.
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries

105.
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries

106.
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ proxy-app
└ web-service
└ modules
└ test
└ README.md
Let’s say you have two Terraform
modules you want to test together:

107.
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ proxy-app
└ web-service
└ modules
└ test
└ README.md
proxy-app: an app that acts as an HTTP
proxy for other web services.

108.
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ proxy-app
└ web-service
└ modules
└ test
└ README.md
web-service: a web service that you
want proxied.

109.
variable "url_to_proxy" {
description = "The URL to proxy."
type = string
}
proxy-app takes in the URL to proxy via
an input variable

110.
output "url" {
value = module.web_service.url
}
web-service exposes its URL via an
output variable

111.
infrastructure-as-code-testing-talk
└ examples
└ modules
└ test
└ hello_world_app_test.go
└ docker_kubernetes_test.go
└ proxy_app_test.go
└ README.md
Create proxy_app_test.go

112.
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
The basic test structure

113.
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
1. Configure options for the web
service

114.
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
2. Deploy the web service

115.
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
3. Configure options for the proxy app
(passing it the web service options)

116.
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
4. Deploy the proxy app

117.
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
5. Validate the proxy app works

118.
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
6. At the end of the test, undeploy the
proxy app and the web service

119.
func configWebService(t *testing.T) *terraform.Options {
return &terraform.Options{
TerraformDir: "../examples/web-service",
}
}
The configWebService method

120.
func configProxyApp(t *testing.T, webServiceOpts
*terraform.Options) *terraform.Options {
url := terraform.Output(t, webServiceOpts, "url")
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"url_to_proxy": url,
},
}
}
The configProxyApp method

121.
func configProxyApp(t *testing.T, webServiceOpts
*terraform.Options) *terraform.Options {
url := terraform.Output(t, webServiceOpts, "url")
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"url_to_proxy": url,
},
}
}
1. Read the url output from the web-
service module

122.
func configProxyApp(t *testing.T, webServiceOpts
*terraform.Options) *terraform.Options {
url := terraform.Output(t, webServiceOpts, "url")
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"url_to_proxy": url,
},
}
}
2. Pass it in as the url_to_proxy input to
the proxy-app module

123.
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
`{"text":"Hello, World!"}`, // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
The validate method

124.
$ go test -v -timeout 15m -run TestProxyApp
…
--- PASS: TestProxyApp (182.44s)
Run go test. You’re now testing
multiple modules together!

125.
$ go test -v -timeout 15m -run TestProxyApp
…
--- PASS: TestProxyApp (182.44s)
But integration tests can take (many)
minutes to run…

126.
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries

127.
Infrastructure tests can take a
long time to run

128.
One way to save time: run
tests in parallel

129.
func TestProxyApp(t *testing.T) {
t.Parallel()
// The rest of the test code
}
func TestHelloWorldAppUnit(t *testing.T) {
t.Parallel()
// The rest of the test code
}
Enable test parallelism in Go by adding
t.Parallel() as the 1st line of each test.

130.
$ go test -v -timeout 15m
=== RUN TestHelloWorldApp
=== RUN TestDockerKubernetes
=== RUN TestProxyApp
Now, if you run go test, all the tests
with t.Parallel() will run in parallel

131.
But there’s a gotcha:
resource conflicts

132.
resource "aws_iam_role" "role_example" {
name = "example-iam-role"
}
resource "aws_security_group" "sg_example" {
name = "security-group-example"
}
Example: module with hard-coded IAM
Role and Security Group names

133.
resource "aws_iam_role" "role_example" {
name = "example-iam-role"
}
resource "aws_security_group" "sg_example" {
name = "security-group-example"
}
If two tests tried to deploy this module
in parallel, the names would conflict!

134.
Key takeaway: you must
namespace all your resources

135.
resource "aws_iam_role" "role_example" {
name = var.name
}
resource "aws_security_group" "sg_example" {
name = var.name
}
Example: use variables in all resource
names…

136.
uniqueId := random.UniqueId()
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"name": fmt.Sprintf("text-proxy-app-%s", uniqueId)
},
}
At test time, set the variables to a
randomized value to avoid conflicts

137.
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries

138.
Consider the structure of the
proxy-app integration test:

139.
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service

140.
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service
When iterating locally, you sometimes
want to re-run just one of these steps.

141.
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service
But as the code is written now, you
have to run all steps on each test run.

142.
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service
And that can add up to a lot of
overhead.
(~3 min)
(~2 min)
(~30 seconds)
(~1 min)
(~2 min)

143.
Key takeaway: break your
tests into independent test
stages

144.
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
The original test structure

145.
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
The test structure with test stages

146.
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
1. RunTestStage is a helper function
from Terratest.

147.
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
2. Wrap each stage of your test with a
call to RunTestStage

148.
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
3. Define each stage in a function
(you’ll see this code shortly).

149.
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
4. Give each stage a unique name

150.
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
Any stage foo can be skipped by
setting the env var SKIP_foo=true

151.
$ SKIP_cleanup_web_service=true
$ SKIP_cleanup_proxy_app=true
Example: on the very first test run, skip
the cleanup stages.

152.
$ go test -v -timeout 15m -run TestProxyApp
Running stage 'deploy_web_service'…
Running stage 'deploy_proxy_app'…
Running stage 'validate'…
Skipping stage 'cleanup_proxy_app'…
Skipping stage 'cleanup_web_service'…
--- PASS: TestProxyApp (105.73s)
That way, after the test finishes, the
infrastructure will still be running.

153.
$ SKIP_deploy_web_service=true
$ SKIP_deploy_proxy_app=true
Now, on the next several test runs, you
can skip the deploy stages too.

154.
$ go test -v -timeout 15m -run TestProxyApp
Skipping stage 'deploy_web_service’…
Skipping stage 'deploy_proxy_app'…
Running stage 'validate'…
Skipping stage 'cleanup_proxy_app'…
Skipping stage 'cleanup_web_service'…
--- PASS: TestProxyApp (14.22s)
This allows you to iterate on solely the
validate stage…

155.
$ go test -v -timeout 15m -run TestProxyApp
Skipping stage 'deploy_web_service’…
Skipping stage 'deploy_proxy_app'…
Running stage 'validate'…
Skipping stage 'cleanup_proxy_app'…
Skipping stage 'cleanup_web_service'…
--- PASS: TestProxyApp (14.22s)
Which dramatically speeds up your
iteration / feedback cycle!

156.
$ SKIP_validate=true
$ unset SKIP_cleanup_web_service
$ unset SKIP_cleanup_proxy_app
When you’re done iterating, skip
validate and re-enable cleanup

157.
$ go test -v -timeout 15m -run TestProxyApp
Skipping stage 'deploy_web_service’…
Skipping stage 'deploy_proxy_app’…
Skipping stage 'validate’…
Running stage 'cleanup_proxy_app’…
Running stage 'cleanup_web_service'…
--- PASS: TestProxyApp (59.61s)
This cleans up everything that was left
running.

158.
func deployWebService(t *testing.T) {
opts := configWebServiceOpts(t)
test_structure.SaveTerraformOptions(t, "/tmp", opts)
terraform.InitAndApply(t, opts)
}
func cleanupWebService(t *testing.T) {
opts := test_structure.LoadTerraformOptions(t, "/tmp")
terraform.Destroy(t, opts)
}
Note: each time you run test stages via
go test, it’s a separate OS process.

159.
func deployWebService(t *testing.T) {
opts := configWebServiceOpts(t)
test_structure.SaveTerraformOptions(t, "/tmp", opts)
terraform.InitAndApply(t, opts)
}
func cleanupWebService(t *testing.T) {
opts := test_structure.LoadTerraformOptions(t, "/tmp")
terraform.Destroy(t, opts)
}
So to pass data between stages, one
stage needs to write the data to disk…

160.
func deployWebService(t *testing.T) {
opts := configWebServiceOpts(t)
test_structure.SaveTerraformOptions(t, "/tmp", opts)
terraform.InitAndApply(t, opts)
}
func cleanupWebService(t *testing.T) {
opts := test_structure.LoadTerraformOptions(t, "/tmp")
terraform.Destroy(t, opts)
}
And the other stages need to read that
data from disk.

161.
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries

162.
Real infrastructure can fail for
intermittent reasons
(e.g., bad EC2 instance, Apt downtime, Terraform bug)

163.
To avoid “flaky” tests, add
retries for known errors.

164.
&terraform.Options{
TerraformDir: "../examples/proxy-app",
RetryableTerraformErrors: map[string]string{
"net/http: TLS handshake timeout": "Terraform bug",
},
MaxRetries: 3,
TimeBetweenRetries: 3*time.Second,
}
Example: retry up to 3 times on a
known TLS error in Terraform.

165.
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline

166.
End-to-end tests: test your
entire infrastructure works
together.

167.
How do you test this entire thing?

168.
You could use the same strategy…
1. Deploy all the infrastructure
2. Validate it works
(e.g., via HTTP requests, API calls, SSH commands, etc.)
3. Undeploy all the infrastructure

169.
But it’s rare to write end-to-
end tests this way. Here’s why:

170.
e2e
Tests
Test pyramid
Integration Tests
Unit Tests
Static analysis

171.
e2e
Tests
Integration Tests
Unit Tests
Static analysis
Cost,
brittleness,
run time

172.
e2e
Tests
Integration Tests
Unit Tests
Static analysis
60 – 240+
minutes
5 – 60
minutes
1 – 20
minutes
1 – 60
seconds

173.
e2e
Tests
Integration Tests
Unit Tests
Static analysis
E2E tests are too slow to be useful
60 – 240+
minutes
5 – 60
minutes
1 – 20
minutes
1 – 60
seconds

174.
Another problem with E2E
tests: brittleness.

175.
Let’s do some math:

176.
Assume a single resource (e.g.,
EC2 instance) has a 1/1000
(0.1%) chance of failure.

177.
Test type # of resources Chance of failure
Unit tests 10 1%
Integration tests 50 5%
End-to-end tests 500+ 40%+
The more resources your tests deploy,
the flakier they will be.

178.
Test type # of resources Chance of failure
Unit tests 10 1%
Integration tests 50 5%
End-to-end tests 500+ 40%+
You can work around the failure rate
for unit & integration tests with retries

179.
Test type # of resources Chance of failure
Unit tests 10 1%
Integration tests 50 5%
End-to-end tests 500+ 40%+
You can work around the failure rate
for unit & integration tests with retries

180.
Key takeaway: E2E tests from
scratch are too slow and too
brittle to be useful

181.
Instead, you can do
incremental E2E testing!

182.
module
module
module
module
module
module
module
module
module
module
module
module
module module
module
1. Deploy a persistent test
environment and leave it running.

183.
module
module
module
module
module
module
module
module
module
module
module
module
module module
module
2. Each time you update a module,
deploy & validate just that module

184.
module
module
module
module
module
module
module
module
module
module
module
module
module module
module
3. Bonus: test your deployment
process is zero-downtime too!

185.
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline

186.
Testing techniques compared:

187.
Technique Strengths Weaknesses
Static analysis
1. Fast
2. Stable
3. No need to deploy real resources
4. Easy to use
1. Very limited in errors you can catch
2. You don’t get much confidence in your
code solely from static analysis
Unit tests
1. Fast enough (1 – 10 min)
2. Mostly stable (with retry logic)
3. High level of confidence in individual units
1. Need to deploy real resources
2. Requires writing non-trivial code
Integration tests
1. Mostly stable (with retry logic)
2. High level of confidence in multiple units
working together
1. Need to deploy real resources
2. Requires writing non-trivial code
3. Slow (10 – 30 min)
End-to-end tests
1. Build confidence in your entire
architecture
1. Need to deploy real resources
2. Requires writing non-trivial code
3. Very slow (60 min – 240+ min)*
4. Can be brittle (even with retry logic)*

188.
So which should you use?

189.
All of them!
They all catch different types of bugs.

190.
e2e
Tests
Keep in mind the test pyramid
Integration Tests
Unit Tests
Static analysis

191.
e2e
Tests
Lots of unit tests + static analysis
Integration Tests
Unit Tests
Static analysis

192.
e2e
Tests
Fewer integration tests
Integration Tests
Unit Tests
Static analysis

193.
e2e
Tests
A handful of high-value e2e tests
Integration Tests
Unit Tests
Static analysis

194.
Infrastructure code
without tests is scary

195.
Fight the fear & build confidence in
your code with automated tests

196.
Questions?
info@gruntwork.io