SlideShare a Scribd company logo
1 of 200
Download to read offline
Automated testing for:
✓ terraform
✓ docker
✓ packer
✓ kubernetes
✓ and more
Passed: 5. Failed: 0. Skipped: 0.
Test run successful.
How to
test
infrastructure
code
The DevOps world is full of
Fear
Fear of outages
Fear of security breaches
Fear of data loss
Fear of change
“Fear leads to
anger. Anger
leads to hate.
Hate leads to
suffering.”
Scrum Master Yoda
And you all know what
suffering leads to, right?
Credit: Daniele Polencic
Many DevOps teams deal
with this fear in two ways:
1) Heavy drinking and smoking
2) Deploying less frequently
Sadly, both of these just make
the problem worse!
There’s a better way to deal
with this fear:
Automated tests
Automated tests give you the
confidence to make changes
Fight fear with confidence
We know how to write automated
tests for application code…
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
But how do you test your Terraform code
deploys infrastructure that works?
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-app-deployment
spec:
selector:
matchLabels:
app: hello-world-app
replicas: 1
spec:
containers:
- name: hello-world-app
image: gruntwork-io/hello-world-app:v1
ports:
- containerPort: 8080
How do you test your Kubernetes code
configures your services correctly?
This talk is about how to write
tests for your infrastructure code.
I’m
Yevgeniy
Brikman
ybrikman.com
Co-founder of
Gruntwork
gruntwork.io
Author
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline
Static analysis: test your code
without deploying it.
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run
Statically check your code for
syntactic and structural issues
Tool Command
Terraform terraform validate
Packer packer validate <template>
Kubernetes kubectl apply -f <file> --dry-run --validate=true
Examples:
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run
Statically validate your code to
catch common errors
Tool Linters
Terraform
1. conftest
2. terraform_validate
3. tflint
Docker
1. dockerfile_lint
2. hadolint
3. dockerfilelint
Kubernetes
1. kube-score
2. kube-lint
3. yamllint
Examples:
Static analysis
1. Compiler / parser / interpreter
2. Linter
3. Dry run
Partially execute the code and
validate the “plan”, but don’t
actually deploy
Tool Dry run options
Terraform
1. terraform plan
2. HashiCorp Sentinel
3. terraform-compliance
Kubernetes kubectl apply -f <file> --server-dry-run
Examples:
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline
Unit tests: test a single “unit”
works in isolation.
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests
You can’t “unit test” an entire end-
to-end architecture
Instead, break your infra code into
small modules and unit test those!
module
module
module
module
module
module
module
module
module
module
module
module
module module
module
With app code, you can test units
in isolation from the outside world
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
But 99% of infrastructure code is about
talking to the outside world…
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
If you try to isolate a unit from the
outside world, you’re left with nothing!
So you can only test infra code by
deploying to a real environment
Key takeaway: there’s no pure
unit testing for infrastructure
code.
Therefore, the test strategy is:
1. Deploy real infrastructure
2. Validate it works
(e.g., via HTTP requests, API calls, SSH commands, etc.)
3. Undeploy the infrastructure
(So it’s really integration testing of a single unit!)
Tool
Deploy /
Undeploy
Validate Works with
Terratest Yes Yes
Terraform, Kubernetes, Packer,
Docker, Servers, Cloud APIs, etc.
kitchen-terraform Yes Yes Terraform
Inspec No Yes Servers, Cloud APIs
Serverspec No Yes Servers
Goss No Yes Servers
Tools that help with this strategy:
Tool
Deploy /
Undeploy
Validate Works with
Terratest Yes Yes
Terraform, Kubernetes, Packer,
Docker, Servers, Cloud APIs, etc.
kitchen-terraform Yes Yes Terraform
Inspec No Yes Servers, Cloud APIs
Serverspec No Yes Servers
Goss No Yes Servers
In this talk, we’ll use Terratest:
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests
Sample code for this talk is at:
github.com/gruntwork-io/infrastructure-as-code-testing-talk
An example of a Terraform
module you may want to test:
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ main.tf
└ outputs.tf
└ variables.tf
└ modules
└ test
└ README.md
hello-world-app: deploy a “Hello,
World” web service
resource "aws_lambda_function" "web_app" {
function_name = var.name
role = aws_iam_role.lambda.arn
# ...
}
resource "aws_api_gateway_integration" "proxy" {
type = "AWS_PROXY"
uri = aws_lambda_function.web_app.invoke_arn
# ...
}
Under the hood, this example runs on
top of AWS Lambda & API Gateway
$ terraform apply
Outputs:
url = ruvvwv3sh1.execute-api.us-east-2.amazonaws.com
$ curl ruvvwv3sh1.execute-api.us-east-2.amazonaws.com
Hello, World!
When you run terraform apply, it
deploys and outputs the URL
Let’s write a unit test for
hello-world-app with Terratest
infrastructure-as-code-testing-talk
└ examples
└ modules
└ test
└ hello_world_app_test.go
└ README.md
Create hello_world_app_test.go
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
The basic test structure
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
1. Tell Terratest where your Terraform
code lives
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
2. Run terraform init and terraform
apply to deploy your module
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
3. Validate the infrastructure works.
We’ll come back to this shortly.
func TestHelloWorldAppUnit(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/hello-world-app",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
validate(t, terraformOptions)
}
4. Run terraform destroy at the end of
the test to undeploy everything
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
The validate function
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
1. Run terraform output to get the web
service URL
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
2. Make HTTP requests to the URL
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
3. Check the response for an expected
status and body
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
4. Retry the request up to 10 times, as
deployment is asynchronous
Note: since we’re testing a
web service, we use HTTP
requests to validate it.
Infrastructure Example Validate with… Example
Web service Dockerized web app HTTP requests Terratest http_helper package
Server EC2 instance SSH commands Terratest ssh package
Cloud service SQS Cloud APIs Terratest aws or gcp packages
Database MySQL SQL queries MySQL driver for Go
Examples of other ways to validate:
$ export AWS_ACCESS_KEY_ID=xxxx
$ export AWS_SECRET_ACCESS_KEY=xxxxx
To run the test, first authenticate to
AWS
$ go test -v -timeout 15m -run TestHelloWorldAppUnit
…
--- PASS: TestHelloWorldAppUnit (31.57s)
Then run go test. You now have a unit
test you can run after every commit!
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests
What about other tools, such
as Docker + Kubernetes?
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ Dockerfile
└ deployment.yml
└ modules
└ test
└ README.md
docker-kubernetes: deploy a “Hello,
World” web service to Kubernetes
FROM ubuntu:18.04
EXPOSE 8080
RUN DEBIAN_FRONTEND=noninteractive apt-get update && 
apt-get install -y busybox
RUN echo 'Hello, World!' > index.html
CMD ["busybox", "httpd", "-f", "-p", "8080"]
Dockerfile: Dockerize a simple “Hello,
World!” web service
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-app-deployment
spec:
selector:
matchLabels:
app: hello-world-app
replicas: 1
spec:
containers:
- name: hello-world-app
image: gruntwork-io/hello-world-app:v1
ports:
- containerPort: 8080
deployment.yml: define how to deploy a
Docker container in Kubernetes
$ cd examples/docker-kubernetes
$ docker build -t gruntwork-io/hello-world-app:v1 .
Successfully tagged gruntwork-io/hello-world-app:v1
$ kubectl apply -f deployment.yml
deployment.apps/hello-world-app-deployment created
service/hello-world-app-service created
$ curl localhost:8080
Hello, World!
Build the Docker image, deploy to
Kubernetes, and check URL
Let’s write a unit test for this
code.
infrastructure-as-code-testing-talk
└ examples
└ modules
└ test
└ hello_world_app_test.go
└ docker_kubernetes_test.go
└ README.md
Create docker_kubernetes_test.go
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
The basic test structure
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
1. Build the Docker image. You’ll see
the buildDockerImage method shortly.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
2. Tell Terratest where your Kubernetes
deployment is defined
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
3. Configure kubectl options to
authenticate to Kubernetes
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
4. Run kubectl apply to deploy the web
app to Kubernetes
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
5. Check the app is working. You’ll see
the validate method shortly.
func TestDockerKubernetes(t *testing.T) {
buildDockerImage(t)
path := "../examples/docker-kubernetes/deployment.yml"
options := k8s.NewKubectlOptions("", "", "")
defer k8s.KubectlDelete(t, options, path)
k8s.KubectlApply(t, options, path)
validate(t, options)
}
6. At the end of the test, remove all
Kubernetes resources you deployed
func buildDockerImage(t *testing.T) {
options := &docker.BuildOptions{
Tags: []string{"gruntwork-io/hello-world-app:v1"},
}
path := "../examples/docker-kubernetes"
docker.Build(t, path, options)
}
The buildDockerImage method
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
The validate method
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
1. Wait until the service is deployed
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
2. Make HTTP requests
func validate(t *testing.T, opts *k8s.KubectlOptions) {
k8s.WaitUntilServiceAvailable(t, opts, "hello-world-
app-service", 10, 1*time.Second)
http_helper.HttpGetWithRetry(t,
serviceUrl(t, opts), // URL to test
200, // Expected status code
"Hello, World!", // Expected body
10, // Max retries
3*time.Second // Time between retries
)
}
3. Use serviceUrl method to get URL
func serviceUrl(t *testing.T, opts *k8s.KubectlOptions) string {
service := k8s.GetService(t, options, "hello-world-app-service")
endpoint := k8s.GetServiceEndpoint(t, options, service, 8080)
return fmt.Sprintf("http://%s", endpoint)
}
The serviceUrl method
$ kubectl config set-credentials …
To run the test, first authenticate to a
Kubernetes cluster.
Note: Kubernetes is now part of
Docker Desktop. Test 100% locally!
$ go test -v -timeout 15m -run TestDockerKubernetes
…
--- PASS: TestDockerKubernetes (5.69s)
Run go test. You can validate your
config after every commit in seconds!
Unit tests
1. Unit testing basics
2. Example: Terraform unit tests
3. Example: Docker/Kubernetes unit tests
4. Cleaning up after tests
Note: tests create and destroy
many resources!
Pro tip #1: run tests in completely
separate “sandbox” accounts
Tool Clouds Features
cloud-nuke AWS (GCP planned)
Delete all resources older than a certain
date; in a certain region; of a certain type.
Janitor Monkey AWS
Configurable rules of what to delete.
Notify owners of pending deletions.
aws-nuke AWS
Specify specific AWS accounts and
resource types to target.
Azure Powershell Azure
Includes native commands to delete
Resource Groups
Pro tip #2: run these tools in cron jobs
to clean up left-over resources
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline
Integration tests: test multiple
“units” work together.
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ proxy-app
└ web-service
└ modules
└ test
└ README.md
Let’s say you have two Terraform
modules you want to test together:
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ proxy-app
└ web-service
└ modules
└ test
└ README.md
proxy-app: an app that acts as an HTTP
proxy for other web services.
infrastructure-as-code-testing-talk
└ examples
└ hello-world-app
└ docker-kubernetes
└ proxy-app
└ web-service
└ modules
└ test
└ README.md
web-service: a web service that you
want proxied.
variable "url_to_proxy" {
description = "The URL to proxy."
type = string
}
proxy-app takes in the URL to proxy via
an input variable
output "url" {
value = module.web_service.url
}
web-service exposes its URL via an
output variable
infrastructure-as-code-testing-talk
└ examples
└ modules
└ test
└ hello_world_app_test.go
└ docker_kubernetes_test.go
└ proxy_app_test.go
└ README.md
Create proxy_app_test.go
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
The basic test structure
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
1. Configure options for the web
service
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
2. Deploy the web service
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
3. Configure options for the proxy app
(passing it the web service options)
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
4. Deploy the proxy app
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
5. Validate the proxy app works
func TestProxyApp(t *testing.T) {
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
}
6. At the end of the test, undeploy the
proxy app and the web service
func configWebService(t *testing.T) *terraform.Options {
return &terraform.Options{
TerraformDir: "../examples/web-service",
}
}
The configWebService method
func configProxyApp(t *testing.T, webServiceOpts
*terraform.Options) *terraform.Options {
url := terraform.Output(t, webServiceOpts, "url")
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"url_to_proxy": url,
},
}
}
The configProxyApp method
func configProxyApp(t *testing.T, webServiceOpts
*terraform.Options) *terraform.Options {
url := terraform.Output(t, webServiceOpts, "url")
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"url_to_proxy": url,
},
}
}
1. Read the url output from the web-
service module
func configProxyApp(t *testing.T, webServiceOpts
*terraform.Options) *terraform.Options {
url := terraform.Output(t, webServiceOpts, "url")
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"url_to_proxy": url,
},
}
}
2. Pass it in as the url_to_proxy input to
the proxy-app module
func validate(t *testing.T, opts *terraform.Options) {
url := terraform.Output(t, opts, "url")
http_helper.HttpGetWithRetry(t,
url, // URL to test
200, // Expected status code
`{"text":"Hello, World!"}`, // Expected body
10, // Max retries
3 * time.Second // Time between retries
)
}
The validate method
$ go test -v -timeout 15m -run TestProxyApp
…
--- PASS: TestProxyApp (182.44s)
Run go test. You’re now testing
multiple modules together!
$ go test -v -timeout 15m -run TestProxyApp
…
--- PASS: TestProxyApp (182.44s)
But integration tests can take (many)
minutes to run…
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries
Infrastructure tests can take a
long time to run
One way to save time: run
tests in parallel
func TestProxyApp(t *testing.T) {
t.Parallel()
// The rest of the test code
}
func TestHelloWorldAppUnit(t *testing.T) {
t.Parallel()
// The rest of the test code
}
Enable test parallelism in Go by adding
t.Parallel() as the 1st line of each test.
$ go test -v -timeout 15m
=== RUN TestHelloWorldApp
=== RUN TestDockerKubernetes
=== RUN TestProxyApp
Now, if you run go test, all the tests
with t.Parallel() will run in parallel
But there’s a gotcha:
resource conflicts
resource "aws_iam_role" "role_example" {
name = "example-iam-role"
}
resource "aws_security_group" "sg_example" {
name = "security-group-example"
}
Example: module with hard-coded IAM
Role and Security Group names
resource "aws_iam_role" "role_example" {
name = "example-iam-role"
}
resource "aws_security_group" "sg_example" {
name = "security-group-example"
}
If two tests tried to deploy this module
in parallel, the names would conflict!
Key takeaway: you must
namespace all your resources
resource "aws_iam_role" "role_example" {
name = var.name
}
resource "aws_security_group" "sg_example" {
name = var.name
}
Example: use variables in all resource
names…
uniqueId := random.UniqueId()
return &terraform.Options{
TerraformDir: "../examples/proxy-app",
Vars: map[string]interface{}{
"name": fmt.Sprintf("text-proxy-app-%s", uniqueId)
},
}
At test time, set the variables to a
randomized value to avoid conflicts
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries
Consider the structure of the
proxy-app integration test:
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service
When iterating locally, you sometimes
want to re-run just one of these steps.
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service
But as the code is written now, you
have to run all steps on each test run.
1. Deploy web-service
2. Deploy proxy-app
3. Validate proxy-app
4. Undeploy proxy-app
5. Undeploy web-service
And that can add up to a lot of
overhead.
(~3 min)
(~2 min)
(~30 seconds)
(~1 min)
(~2 min)
Key takeaway: break your
tests into independent test
stages
webServiceOpts := configWebService(t)
defer terraform.Destroy(t, webServiceOpts)
terraform.InitAndApply(t, webServiceOpts)
proxyAppOpts := configProxyApp(t, webServiceOpts)
defer terraform.Destroy(t, proxyAppOpts)
terraform.InitAndApply(t, proxyAppOpts)
validate(t, proxyAppOpts)
The original test structure
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
The test structure with test stages
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
1. RunTestStage is a helper function
from Terratest.
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
2. Wrap each stage of your test with a
call to RunTestStage
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
3. Define each stage in a function
(you’ll see this code shortly).
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
4. Give each stage a unique name
stage := test_structure.RunTestStage
defer stage(t, "cleanup_web_service", cleanupWebService)
stage(t, "deploy_web_service", deployWebService)
defer stage(t, "cleanup_proxy_app", cleanupProxyApp)
stage(t, "deploy_proxy_app", deployProxyApp)
stage(t, "validate", validate)
Any stage foo can be skipped by
setting the env var SKIP_foo=true
$ SKIP_cleanup_web_service=true
$ SKIP_cleanup_proxy_app=true
Example: on the very first test run, skip
the cleanup stages.
$ go test -v -timeout 15m -run TestProxyApp
Running stage 'deploy_web_service'…
Running stage 'deploy_proxy_app'…
Running stage 'validate'…
Skipping stage 'cleanup_proxy_app'…
Skipping stage 'cleanup_web_service'…
--- PASS: TestProxyApp (105.73s)
That way, after the test finishes, the
infrastructure will still be running.
$ SKIP_deploy_web_service=true
$ SKIP_deploy_proxy_app=true
Now, on the next several test runs, you
can skip the deploy stages too.
$ go test -v -timeout 15m -run TestProxyApp
Skipping stage 'deploy_web_service’…
Skipping stage 'deploy_proxy_app'…
Running stage 'validate'…
Skipping stage 'cleanup_proxy_app'…
Skipping stage 'cleanup_web_service'…
--- PASS: TestProxyApp (14.22s)
This allows you to iterate on solely the
validate stage…
$ go test -v -timeout 15m -run TestProxyApp
Skipping stage 'deploy_web_service’…
Skipping stage 'deploy_proxy_app'…
Running stage 'validate'…
Skipping stage 'cleanup_proxy_app'…
Skipping stage 'cleanup_web_service'…
--- PASS: TestProxyApp (14.22s)
Which dramatically speeds up your
iteration / feedback cycle!
$ SKIP_validate=true
$ unset SKIP_cleanup_web_service
$ unset SKIP_cleanup_proxy_app
When you’re done iterating, skip
validate and re-enable cleanup
$ go test -v -timeout 15m -run TestProxyApp
Skipping stage 'deploy_web_service’…
Skipping stage 'deploy_proxy_app’…
Skipping stage 'validate’…
Running stage 'cleanup_proxy_app’…
Running stage 'cleanup_web_service'…
--- PASS: TestProxyApp (59.61s)
This cleans up everything that was left
running.
func deployWebService(t *testing.T) {
opts := configWebServiceOpts(t)
test_structure.SaveTerraformOptions(t, "/tmp", opts)
terraform.InitAndApply(t, opts)
}
func cleanupWebService(t *testing.T) {
opts := test_structure.LoadTerraformOptions(t, "/tmp")
terraform.Destroy(t, opts)
}
Note: each time you run test stages via
go test, it’s a separate OS process.
func deployWebService(t *testing.T) {
opts := configWebServiceOpts(t)
test_structure.SaveTerraformOptions(t, "/tmp", opts)
terraform.InitAndApply(t, opts)
}
func cleanupWebService(t *testing.T) {
opts := test_structure.LoadTerraformOptions(t, "/tmp")
terraform.Destroy(t, opts)
}
So to pass data between stages, one
stage needs to write the data to disk…
func deployWebService(t *testing.T) {
opts := configWebServiceOpts(t)
test_structure.SaveTerraformOptions(t, "/tmp", opts)
terraform.InitAndApply(t, opts)
}
func cleanupWebService(t *testing.T) {
opts := test_structure.LoadTerraformOptions(t, "/tmp")
terraform.Destroy(t, opts)
}
And the other stages need to read that
data from disk.
Integration tests
1. Example: Terraform integration tests
2. Test parallelism
3. Test stages
4. Test retries
Real infrastructure can fail for
intermittent reasons
(e.g., bad EC2 instance, Apt downtime, Terraform bug)
To avoid “flaky” tests, add
retries for known errors.
&terraform.Options{
TerraformDir: "../examples/proxy-app",
RetryableTerraformErrors: map[string]string{
"net/http: TLS handshake timeout": "Terraform bug",
},
MaxRetries: 3,
TimeBetweenRetries: 3*time.Second,
}
Example: retry up to 3 times on a
known TLS error in Terraform.
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline
End-to-end tests: test your
entire infrastructure works
together.
How do you test this entire thing?
You could use the same strategy…
1. Deploy all the infrastructure
2. Validate it works
(e.g., via HTTP requests, API calls, SSH commands, etc.)
3. Undeploy all the infrastructure
But it’s rare to write end-to-
end tests this way. Here’s why:
e2e
Tests
Test pyramid
Integration Tests
Unit Tests
Static analysis
e2e
Tests
Integration Tests
Unit Tests
Static analysis
Cost,
brittleness,
run time
e2e
Tests
Integration Tests
Unit Tests
Static analysis
60 – 240+
minutes
5 – 60
minutes
1 – 20
minutes
1 – 60
seconds
e2e
Tests
Integration Tests
Unit Tests
Static analysis
E2E tests are too slow to be useful
60 – 240+
minutes
5 – 60
minutes
1 – 20
minutes
1 – 60
seconds
Another problem with E2E
tests: brittleness.
Let’s do some math:
Assume a single resource (e.g.,
EC2 instance) has a 1/1000
(0.1%) chance of failure.
Test type # of resources Chance of failure
Unit tests 10 1%
Integration tests 50 5%
End-to-end tests 500+ 40%+
The more resources your tests deploy,
the flakier they will be.
Test type # of resources Chance of failure
Unit tests 10 1%
Integration tests 50 5%
End-to-end tests 500+ 40%+
You can work around the failure rate
for unit & integration tests with retries
Test type # of resources Chance of failure
Unit tests 10 1%
Integration tests 50 5%
End-to-end tests 500+ 40%+
You can work around the failure rate
for unit & integration tests with retries
Key takeaway: E2E tests from
scratch are too slow and too
brittle to be useful
Instead, you can do
incremental E2E testing!
module
module
module
module
module
module
module
module
module
module
module
module
module module
module
1. Deploy a persistent test
environment and leave it running.
module
module
module
module
module
module
module
module
module
module
module
module
module module
module
2. Each time you update a module,
deploy & validate just that module
module
module
module
module
module
module
module
module
module
module
module
module
module module
module
3. Bonus: test your deployment
process is zero-downtime too!
1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion
Outline
Testing techniques compared:
Technique Strengths Weaknesses
Static analysis
1. Fast
2. Stable
3. No need to deploy real resources
4. Easy to use
1. Very limited in errors you can catch
2. You don’t get much confidence in your
code solely from static analysis
Unit tests
1. Fast enough (1 – 10 min)
2. Mostly stable (with retry logic)
3. High level of confidence in individual units
1. Need to deploy real resources
2. Requires writing non-trivial code
Integration tests
1. Mostly stable (with retry logic)
2. High level of confidence in multiple units
working together
1. Need to deploy real resources
2. Requires writing non-trivial code
3. Slow (10 – 30 min)
End-to-end tests
1. Build confidence in your entire
architecture
1. Need to deploy real resources
2. Requires writing non-trivial code
3. Very slow (60 min – 240+ min)*
4. Can be brittle (even with retry logic)*
So which should you use?
All of them!
They all catch different types of bugs.
e2e
Tests
Keep in mind the test pyramid
Integration Tests
Unit Tests
Static analysis
e2e
Tests
Lots of unit tests + static analysis
Integration Tests
Unit Tests
Static analysis
e2e
Tests
Fewer integration tests
Integration Tests
Unit Tests
Static analysis
e2e
Tests
A handful of high-value e2e tests
Integration Tests
Unit Tests
Static analysis
Infrastructure code
without tests is scary
Fight the fear & build confidence in
your code with automated tests
Questions?
info@gruntwork.io

More Related Content

What's hot

Comprehensive Terraform Training
Comprehensive Terraform TrainingComprehensive Terraform Training
Comprehensive Terraform TrainingYevgeniy Brikman
 
Terraform introduction
Terraform introductionTerraform introduction
Terraform introductionJason Vance
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusGrafana Labs
 
Terraform -- Infrastructure as Code
Terraform -- Infrastructure as CodeTerraform -- Infrastructure as Code
Terraform -- Infrastructure as CodeMartin Schütte
 
Terraform in deployment pipeline
Terraform in deployment pipelineTerraform in deployment pipeline
Terraform in deployment pipelineAnton Babenko
 
Introduction to kubernetes
Introduction to kubernetesIntroduction to kubernetes
Introduction to kubernetesRishabh Indoria
 
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark OperatorApache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark OperatorDatabricks
 
Design patterns for microservice architecture
Design patterns for microservice architectureDesign patterns for microservice architecture
Design patterns for microservice architectureThe Software House
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to PrometheusJulien Pivotto
 
Terraform: An Overview & Introduction
Terraform: An Overview & IntroductionTerraform: An Overview & Introduction
Terraform: An Overview & IntroductionLee Trout
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaArvind Kumar G.S
 
Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021TomStraub5
 
MySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & GrafanaMySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & GrafanaYoungHeon (Roy) Kim
 
Introduction to Nexus Repository Manager.pdf
Introduction to Nexus Repository Manager.pdfIntroduction to Nexus Repository Manager.pdf
Introduction to Nexus Repository Manager.pdfKnoldus Inc.
 

What's hot (20)

Comprehensive Terraform Training
Comprehensive Terraform TrainingComprehensive Terraform Training
Comprehensive Terraform Training
 
Terraform introduction
Terraform introductionTerraform introduction
Terraform introduction
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
 
Terraform
TerraformTerraform
Terraform
 
Terraform -- Infrastructure as Code
Terraform -- Infrastructure as CodeTerraform -- Infrastructure as Code
Terraform -- Infrastructure as Code
 
Terraform in deployment pipeline
Terraform in deployment pipelineTerraform in deployment pipeline
Terraform in deployment pipeline
 
Terraform Basics
Terraform BasicsTerraform Basics
Terraform Basics
 
infrastructure as code
infrastructure as codeinfrastructure as code
infrastructure as code
 
Terraform on Azure
Terraform on AzureTerraform on Azure
Terraform on Azure
 
Introduction to kubernetes
Introduction to kubernetesIntroduction to kubernetes
Introduction to kubernetes
 
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark OperatorApache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
 
Design patterns for microservice architecture
Design patterns for microservice architectureDesign patterns for microservice architecture
Design patterns for microservice architecture
 
Terraform
TerraformTerraform
Terraform
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to Prometheus
 
Terraform 101
Terraform 101Terraform 101
Terraform 101
 
Terraform: An Overview & Introduction
Terraform: An Overview & IntroductionTerraform: An Overview & Introduction
Terraform: An Overview & Introduction
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
 
Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021
 
MySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & GrafanaMySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & Grafana
 
Introduction to Nexus Repository Manager.pdf
Introduction to Nexus Repository Manager.pdfIntroduction to Nexus Repository Manager.pdf
Introduction to Nexus Repository Manager.pdf
 

Similar to How to test infrastructure code: automated testing for Terraform, Kubernetes, Docker, Packer and more

[xp2013] Narrow Down What to Test
[xp2013] Narrow Down What to Test[xp2013] Narrow Down What to Test
[xp2013] Narrow Down What to TestZsolt Fabok
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
 
Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016Thomas Shaw
 
RichFaces - Testing on Mobile Devices
RichFaces - Testing on Mobile DevicesRichFaces - Testing on Mobile Devices
RichFaces - Testing on Mobile DevicesPavol Pitoňák
 
Stopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestStopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestSeb Rose
 
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...Nagios
 
OpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingOpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingopenstackindia
 
Practical Chaos Engineering
Practical Chaos EngineeringPractical Chaos Engineering
Practical Chaos EngineeringSIGHUP
 
Introduction to PowerShell
Introduction to PowerShellIntroduction to PowerShell
Introduction to PowerShellBoulos Dib
 
Testing NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoverageTesting NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoveragemlilley
 
Server(less) Swift at SwiftCloudWorkshop 3
Server(less) Swift at SwiftCloudWorkshop 3Server(less) Swift at SwiftCloudWorkshop 3
Server(less) Swift at SwiftCloudWorkshop 3kognate
 
Release with confidence
Release with confidenceRelease with confidence
Release with confidenceJohn Congdon
 
Into The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsInto The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsOrtus Solutions, Corp
 
PuppetDB: Sneaking Clojure into Operations
PuppetDB: Sneaking Clojure into OperationsPuppetDB: Sneaking Clojure into Operations
PuppetDB: Sneaking Clojure into Operationsgrim_radical
 
Pythonic Deployment with Fabric 0.9
Pythonic Deployment with Fabric 0.9Pythonic Deployment with Fabric 0.9
Pythonic Deployment with Fabric 0.9Corey Oordt
 
AWS Lambda from the trenches
AWS Lambda from the trenchesAWS Lambda from the trenches
AWS Lambda from the trenchesYan Cui
 
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017Ortus Solutions, Corp
 
AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...
AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...
AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...Amazon Web Services
 
Functional Testing Swing Applications with Frankenstein
Functional Testing Swing Applications with FrankensteinFunctional Testing Swing Applications with Frankenstein
Functional Testing Swing Applications with Frankensteinvivek_prahlad
 

Similar to How to test infrastructure code: automated testing for Terraform, Kubernetes, Docker, Packer and more (20)

[xp2013] Narrow Down What to Test
[xp2013] Narrow Down What to Test[xp2013] Narrow Down What to Test
[xp2013] Narrow Down What to Test
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 
Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016
 
RichFaces - Testing on Mobile Devices
RichFaces - Testing on Mobile DevicesRichFaces - Testing on Mobile Devices
RichFaces - Testing on Mobile Devices
 
Stopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestStopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under Test
 
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
 
OpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingOpenStack Tempest and REST API testing
OpenStack Tempest and REST API testing
 
Practical Chaos Engineering
Practical Chaos EngineeringPractical Chaos Engineering
Practical Chaos Engineering
 
Introduction to PowerShell
Introduction to PowerShellIntroduction to PowerShell
Introduction to PowerShell
 
Testing NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoverageTesting NodeJS with Mocha, Should, Sinon, and JSCoverage
Testing NodeJS with Mocha, Should, Sinon, and JSCoverage
 
Unit testing
Unit testingUnit testing
Unit testing
 
Server(less) Swift at SwiftCloudWorkshop 3
Server(less) Swift at SwiftCloudWorkshop 3Server(less) Swift at SwiftCloudWorkshop 3
Server(less) Swift at SwiftCloudWorkshop 3
 
Release with confidence
Release with confidenceRelease with confidence
Release with confidence
 
Into The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsInto The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applications
 
PuppetDB: Sneaking Clojure into Operations
PuppetDB: Sneaking Clojure into OperationsPuppetDB: Sneaking Clojure into Operations
PuppetDB: Sneaking Clojure into Operations
 
Pythonic Deployment with Fabric 0.9
Pythonic Deployment with Fabric 0.9Pythonic Deployment with Fabric 0.9
Pythonic Deployment with Fabric 0.9
 
AWS Lambda from the trenches
AWS Lambda from the trenchesAWS Lambda from the trenches
AWS Lambda from the trenches
 
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
 
AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...
AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...
AWS December 2015 Webinar Series - Continuous Delivery to Amazon EC2 Containe...
 
Functional Testing Swing Applications with Frankenstein
Functional Testing Swing Applications with FrankensteinFunctional Testing Swing Applications with Frankenstein
Functional Testing Swing Applications with Frankenstein
 

More from Yevgeniy Brikman

Cloud adoption fails - 5 ways deployments go wrong and 5 solutions
Cloud adoption fails - 5 ways deployments go wrong and 5 solutionsCloud adoption fails - 5 ways deployments go wrong and 5 solutions
Cloud adoption fails - 5 ways deployments go wrong and 5 solutionsYevgeniy Brikman
 
Lessons learned from writing over 300,000 lines of infrastructure code
Lessons learned from writing over 300,000 lines of infrastructure codeLessons learned from writing over 300,000 lines of infrastructure code
Lessons learned from writing over 300,000 lines of infrastructure codeYevgeniy Brikman
 
Gruntwork Executive Summary
Gruntwork Executive SummaryGruntwork Executive Summary
Gruntwork Executive SummaryYevgeniy Brikman
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesYevgeniy Brikman
 
The Truth About Startups: What I wish someone had told me about entrepreneurs...
The Truth About Startups: What I wish someone had told me about entrepreneurs...The Truth About Startups: What I wish someone had told me about entrepreneurs...
The Truth About Startups: What I wish someone had told me about entrepreneurs...Yevgeniy Brikman
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSYevgeniy Brikman
 
Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...Yevgeniy Brikman
 
Startup Ideas and Validation
Startup Ideas and ValidationStartup Ideas and Validation
Startup Ideas and ValidationYevgeniy Brikman
 
A Guide to Hiring for your Startup
A Guide to Hiring for your StartupA Guide to Hiring for your Startup
A Guide to Hiring for your StartupYevgeniy Brikman
 
Node.js vs Play Framework (with Japanese subtitles)
Node.js vs Play Framework (with Japanese subtitles)Node.js vs Play Framework (with Japanese subtitles)
Node.js vs Play Framework (with Japanese subtitles)Yevgeniy Brikman
 
Composable and streamable Play apps
Composable and streamable Play appsComposable and streamable Play apps
Composable and streamable Play appsYevgeniy Brikman
 
Play Framework: async I/O with Java and Scala
Play Framework: async I/O with Java and ScalaPlay Framework: async I/O with Java and Scala
Play Framework: async I/O with Java and ScalaYevgeniy Brikman
 
The Play Framework at LinkedIn
The Play Framework at LinkedInThe Play Framework at LinkedIn
The Play Framework at LinkedInYevgeniy Brikman
 
Startup DNA: the formula behind successful startups in Silicon Valley (update...
Startup DNA: the formula behind successful startups in Silicon Valley (update...Startup DNA: the formula behind successful startups in Silicon Valley (update...
Startup DNA: the formula behind successful startups in Silicon Valley (update...Yevgeniy Brikman
 

More from Yevgeniy Brikman (20)

Cloud adoption fails - 5 ways deployments go wrong and 5 solutions
Cloud adoption fails - 5 ways deployments go wrong and 5 solutionsCloud adoption fails - 5 ways deployments go wrong and 5 solutions
Cloud adoption fails - 5 ways deployments go wrong and 5 solutions
 
Lessons learned from writing over 300,000 lines of infrastructure code
Lessons learned from writing over 300,000 lines of infrastructure codeLessons learned from writing over 300,000 lines of infrastructure code
Lessons learned from writing over 300,000 lines of infrastructure code
 
Gruntwork Executive Summary
Gruntwork Executive SummaryGruntwork Executive Summary
Gruntwork Executive Summary
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modules
 
The Truth About Startups: What I wish someone had told me about entrepreneurs...
The Truth About Startups: What I wish someone had told me about entrepreneurs...The Truth About Startups: What I wish someone had told me about entrepreneurs...
The Truth About Startups: What I wish someone had told me about entrepreneurs...
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECS
 
Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...
 
Agility Requires Safety
Agility Requires SafetyAgility Requires Safety
Agility Requires Safety
 
Startup Ideas and Validation
Startup Ideas and ValidationStartup Ideas and Validation
Startup Ideas and Validation
 
A Guide to Hiring for your Startup
A Guide to Hiring for your StartupA Guide to Hiring for your Startup
A Guide to Hiring for your Startup
 
Startup DNA: Speed Wins
Startup DNA: Speed WinsStartup DNA: Speed Wins
Startup DNA: Speed Wins
 
Node.js vs Play Framework (with Japanese subtitles)
Node.js vs Play Framework (with Japanese subtitles)Node.js vs Play Framework (with Japanese subtitles)
Node.js vs Play Framework (with Japanese subtitles)
 
Node.js vs Play Framework
Node.js vs Play FrameworkNode.js vs Play Framework
Node.js vs Play Framework
 
Rapid prototyping
Rapid prototypingRapid prototyping
Rapid prototyping
 
Composable and streamable Play apps
Composable and streamable Play appsComposable and streamable Play apps
Composable and streamable Play apps
 
Play Framework: async I/O with Java and Scala
Play Framework: async I/O with Java and ScalaPlay Framework: async I/O with Java and Scala
Play Framework: async I/O with Java and Scala
 
The Play Framework at LinkedIn
The Play Framework at LinkedInThe Play Framework at LinkedIn
The Play Framework at LinkedIn
 
Kings of Code Hack Battle
Kings of Code Hack BattleKings of Code Hack Battle
Kings of Code Hack Battle
 
Hackdays and [in]cubator
Hackdays and [in]cubatorHackdays and [in]cubator
Hackdays and [in]cubator
 
Startup DNA: the formula behind successful startups in Silicon Valley (update...
Startup DNA: the formula behind successful startups in Silicon Valley (update...Startup DNA: the formula behind successful startups in Silicon Valley (update...
Startup DNA: the formula behind successful startups in Silicon Valley (update...
 

Recently uploaded

UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfkalichargn70th171
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingOpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingShane Coughlan
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfMarharyta Nedzelska
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...OnePlan Solutions
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Cizo Technology Services
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Anthony Dahanne
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Developmentvyaparkranti
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf31events.com
 
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxReal-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxRTS corp
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...Technogeeks
 

Recently uploaded (20)

UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingOpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdf
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf
 
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxReal-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...
 

How to test infrastructure code: automated testing for Terraform, Kubernetes, Docker, Packer and more

  • 1. Automated testing for: ✓ terraform ✓ docker ✓ packer ✓ kubernetes ✓ and more Passed: 5. Failed: 0. Skipped: 0. Test run successful. How to test infrastructure code
  • 2. The DevOps world is full of Fear
  • 4. Fear of security breaches
  • 7. “Fear leads to anger. Anger leads to hate. Hate leads to suffering.” Scrum Master Yoda
  • 8. And you all know what suffering leads to, right?
  • 9.
  • 11.
  • 12.
  • 13. Many DevOps teams deal with this fear in two ways:
  • 14. 1) Heavy drinking and smoking
  • 15. 2) Deploying less frequently
  • 16. Sadly, both of these just make the problem worse!
  • 17.
  • 18. There’s a better way to deal with this fear:
  • 20. Automated tests give you the confidence to make changes
  • 21. Fight fear with confidence
  • 22. We know how to write automated tests for application code…
  • 23. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But how do you test your Terraform code deploys infrastructure that works?
  • 24. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers: - name: hello-world-app image: gruntwork-io/hello-world-app:v1 ports: - containerPort: 8080 How do you test your Kubernetes code configures your services correctly?
  • 25. This talk is about how to write tests for your infrastructure code.
  • 29. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  • 30. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  • 31. Static analysis: test your code without deploying it.
  • 32. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  • 33. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  • 34. Statically check your code for syntactic and structural issues
  • 35. Tool Command Terraform terraform validate Packer packer validate <template> Kubernetes kubectl apply -f <file> --dry-run --validate=true Examples:
  • 36. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  • 37. Statically validate your code to catch common errors
  • 38. Tool Linters Terraform 1. conftest 2. terraform_validate 3. tflint Docker 1. dockerfile_lint 2. hadolint 3. dockerfilelint Kubernetes 1. kube-score 2. kube-lint 3. yamllint Examples:
  • 39. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run
  • 40. Partially execute the code and validate the “plan”, but don’t actually deploy
  • 41. Tool Dry run options Terraform 1. terraform plan 2. HashiCorp Sentinel 3. terraform-compliance Kubernetes kubectl apply -f <file> --server-dry-run Examples:
  • 42. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  • 43. Unit tests: test a single “unit” works in isolation.
  • 44. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  • 45. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  • 46. You can’t “unit test” an entire end- to-end architecture
  • 47. Instead, break your infra code into small modules and unit test those! module module module module module module module module module module module module module module module
  • 48. With app code, you can test units in isolation from the outside world
  • 49. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But 99% of infrastructure code is about talking to the outside world…
  • 50. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } If you try to isolate a unit from the outside world, you’re left with nothing!
  • 51. So you can only test infra code by deploying to a real environment
  • 52. Key takeaway: there’s no pure unit testing for infrastructure code.
  • 53. Therefore, the test strategy is: 1. Deploy real infrastructure 2. Validate it works (e.g., via HTTP requests, API calls, SSH commands, etc.) 3. Undeploy the infrastructure (So it’s really integration testing of a single unit!)
  • 54. Tool Deploy / Undeploy Validate Works with Terratest Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc. kitchen-terraform Yes Yes Terraform Inspec No Yes Servers, Cloud APIs Serverspec No Yes Servers Goss No Yes Servers Tools that help with this strategy:
  • 55. Tool Deploy / Undeploy Validate Works with Terratest Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc. kitchen-terraform Yes Yes Terraform Inspec No Yes Servers, Cloud APIs Serverspec No Yes Servers Goss No Yes Servers In this talk, we’ll use Terratest:
  • 56. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  • 57. Sample code for this talk is at: github.com/gruntwork-io/infrastructure-as-code-testing-talk
  • 58. An example of a Terraform module you may want to test:
  • 59. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ main.tf └ outputs.tf └ variables.tf └ modules └ test └ README.md hello-world-app: deploy a “Hello, World” web service
  • 60. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } Under the hood, this example runs on top of AWS Lambda & API Gateway
  • 61. $ terraform apply Outputs: url = ruvvwv3sh1.execute-api.us-east-2.amazonaws.com $ curl ruvvwv3sh1.execute-api.us-east-2.amazonaws.com Hello, World! When you run terraform apply, it deploys and outputs the URL
  • 62. Let’s write a unit test for hello-world-app with Terratest
  • 63. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ README.md Create hello_world_app_test.go
  • 64. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } The basic test structure
  • 65. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 1. Tell Terratest where your Terraform code lives
  • 66. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 2. Run terraform init and terraform apply to deploy your module
  • 67. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 3. Validate the infrastructure works. We’ll come back to this shortly.
  • 68. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 4. Run terraform destroy at the end of the test to undeploy everything
  • 69. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate function
  • 70. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 1. Run terraform output to get the web service URL
  • 71. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 2. Make HTTP requests to the URL
  • 72. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 3. Check the response for an expected status and body
  • 73. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 4. Retry the request up to 10 times, as deployment is asynchronous
  • 74. Note: since we’re testing a web service, we use HTTP requests to validate it.
  • 75. Infrastructure Example Validate with… Example Web service Dockerized web app HTTP requests Terratest http_helper package Server EC2 instance SSH commands Terratest ssh package Cloud service SQS Cloud APIs Terratest aws or gcp packages Database MySQL SQL queries MySQL driver for Go Examples of other ways to validate:
  • 76. $ export AWS_ACCESS_KEY_ID=xxxx $ export AWS_SECRET_ACCESS_KEY=xxxxx To run the test, first authenticate to AWS
  • 77. $ go test -v -timeout 15m -run TestHelloWorldAppUnit … --- PASS: TestHelloWorldAppUnit (31.57s) Then run go test. You now have a unit test you can run after every commit!
  • 78. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  • 79. What about other tools, such as Docker + Kubernetes?
  • 80. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ Dockerfile └ deployment.yml └ modules └ test └ README.md docker-kubernetes: deploy a “Hello, World” web service to Kubernetes
  • 81. FROM ubuntu:18.04 EXPOSE 8080 RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y busybox RUN echo 'Hello, World!' > index.html CMD ["busybox", "httpd", "-f", "-p", "8080"] Dockerfile: Dockerize a simple “Hello, World!” web service
  • 82. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers: - name: hello-world-app image: gruntwork-io/hello-world-app:v1 ports: - containerPort: 8080 deployment.yml: define how to deploy a Docker container in Kubernetes
  • 83. $ cd examples/docker-kubernetes $ docker build -t gruntwork-io/hello-world-app:v1 . Successfully tagged gruntwork-io/hello-world-app:v1 $ kubectl apply -f deployment.yml deployment.apps/hello-world-app-deployment created service/hello-world-app-service created $ curl localhost:8080 Hello, World! Build the Docker image, deploy to Kubernetes, and check URL
  • 84. Let’s write a unit test for this code.
  • 85. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ docker_kubernetes_test.go └ README.md Create docker_kubernetes_test.go
  • 86. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } The basic test structure
  • 87. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 1. Build the Docker image. You’ll see the buildDockerImage method shortly.
  • 88. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 2. Tell Terratest where your Kubernetes deployment is defined
  • 89. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 3. Configure kubectl options to authenticate to Kubernetes
  • 90. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 4. Run kubectl apply to deploy the web app to Kubernetes
  • 91. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 5. Check the app is working. You’ll see the validate method shortly.
  • 92. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 6. At the end of the test, remove all Kubernetes resources you deployed
  • 93. func buildDockerImage(t *testing.T) { options := &docker.BuildOptions{ Tags: []string{"gruntwork-io/hello-world-app:v1"}, } path := "../examples/docker-kubernetes" docker.Build(t, path, options) } The buildDockerImage method
  • 94. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } The validate method
  • 95. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 1. Wait until the service is deployed
  • 96. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 2. Make HTTP requests
  • 97. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 3. Use serviceUrl method to get URL
  • 98. func serviceUrl(t *testing.T, opts *k8s.KubectlOptions) string { service := k8s.GetService(t, options, "hello-world-app-service") endpoint := k8s.GetServiceEndpoint(t, options, service, 8080) return fmt.Sprintf("http://%s", endpoint) } The serviceUrl method
  • 99. $ kubectl config set-credentials … To run the test, first authenticate to a Kubernetes cluster.
  • 100. Note: Kubernetes is now part of Docker Desktop. Test 100% locally!
  • 101. $ go test -v -timeout 15m -run TestDockerKubernetes … --- PASS: TestDockerKubernetes (5.69s) Run go test. You can validate your config after every commit in seconds!
  • 102. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests
  • 103. Note: tests create and destroy many resources!
  • 104. Pro tip #1: run tests in completely separate “sandbox” accounts
  • 105. Tool Clouds Features cloud-nuke AWS (GCP planned) Delete all resources older than a certain date; in a certain region; of a certain type. Janitor Monkey AWS Configurable rules of what to delete. Notify owners of pending deletions. aws-nuke AWS Specify specific AWS accounts and resource types to target. Azure Powershell Azure Includes native commands to delete Resource Groups Pro tip #2: run these tools in cron jobs to clean up left-over resources
  • 106. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  • 107. Integration tests: test multiple “units” work together.
  • 108. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  • 109. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  • 110. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md Let’s say you have two Terraform modules you want to test together:
  • 111. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md proxy-app: an app that acts as an HTTP proxy for other web services.
  • 112. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md web-service: a web service that you want proxied.
  • 113. variable "url_to_proxy" { description = "The URL to proxy." type = string } proxy-app takes in the URL to proxy via an input variable
  • 114. output "url" { value = module.web_service.url } web-service exposes its URL via an output variable
  • 115. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ docker_kubernetes_test.go └ proxy_app_test.go └ README.md Create proxy_app_test.go
  • 116. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } The basic test structure
  • 117. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 1. Configure options for the web service
  • 118. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 2. Deploy the web service
  • 119. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 3. Configure options for the proxy app (passing it the web service options)
  • 120. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 4. Deploy the proxy app
  • 121. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 5. Validate the proxy app works
  • 122. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 6. At the end of the test, undeploy the proxy app and the web service
  • 123. func configWebService(t *testing.T) *terraform.Options { return &terraform.Options{ TerraformDir: "../examples/web-service", } } The configWebService method
  • 124. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } The configProxyApp method
  • 125. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 1. Read the url output from the web- service module
  • 126. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 2. Pass it in as the url_to_proxy input to the proxy-app module
  • 127. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code `{"text":"Hello, World!"}`, // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate method
  • 128. $ go test -v -timeout 15m -run TestProxyApp … --- PASS: TestProxyApp (182.44s) Run go test. You’re now testing multiple modules together!
  • 129. $ go test -v -timeout 15m -run TestProxyApp … --- PASS: TestProxyApp (182.44s) But integration tests can take (many) minutes to run…
  • 130. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  • 131. Infrastructure tests can take a long time to run
  • 132. One way to save time: run tests in parallel
  • 133. func TestProxyApp(t *testing.T) { t.Parallel() // The rest of the test code } func TestHelloWorldAppUnit(t *testing.T) { t.Parallel() // The rest of the test code } Enable test parallelism in Go by adding t.Parallel() as the 1st line of each test.
  • 134. $ go test -v -timeout 15m === RUN TestHelloWorldApp === RUN TestDockerKubernetes === RUN TestProxyApp Now, if you run go test, all the tests with t.Parallel() will run in parallel
  • 135. But there’s a gotcha: resource conflicts
  • 136. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } Example: module with hard-coded IAM Role and Security Group names
  • 137. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } If two tests tried to deploy this module in parallel, the names would conflict!
  • 138. Key takeaway: you must namespace all your resources
  • 139. resource "aws_iam_role" "role_example" { name = var.name } resource "aws_security_group" "sg_example" { name = var.name } Example: use variables in all resource names…
  • 140. uniqueId := random.UniqueId() return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "name": fmt.Sprintf("text-proxy-app-%s", uniqueId) }, } At test time, set the variables to a randomized value to avoid conflicts
  • 141. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  • 142. Consider the structure of the proxy-app integration test:
  • 143. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service
  • 144. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service When iterating locally, you sometimes want to re-run just one of these steps.
  • 145. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service But as the code is written now, you have to run all steps on each test run.
  • 146. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service And that can add up to a lot of overhead. (~3 min) (~2 min) (~30 seconds) (~1 min) (~2 min)
  • 147. Key takeaway: break your tests into independent test stages
  • 148. webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) The original test structure
  • 149. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) The test structure with test stages
  • 150. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 1. RunTestStage is a helper function from Terratest.
  • 151. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 2. Wrap each stage of your test with a call to RunTestStage
  • 152. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 3. Define each stage in a function (you’ll see this code shortly).
  • 153. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 4. Give each stage a unique name
  • 154. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) Any stage foo can be skipped by setting the env var SKIP_foo=true
  • 155. $ SKIP_cleanup_web_service=true $ SKIP_cleanup_proxy_app=true Example: on the very first test run, skip the cleanup stages.
  • 156. $ go test -v -timeout 15m -run TestProxyApp Running stage 'deploy_web_service'… Running stage 'deploy_proxy_app'… Running stage 'validate'… Skipping stage 'cleanup_proxy_app'… Skipping stage 'cleanup_web_service'… --- PASS: TestProxyApp (105.73s) That way, after the test finishes, the infrastructure will still be running.
  • 157. $ SKIP_deploy_web_service=true $ SKIP_deploy_proxy_app=true Now, on the next several test runs, you can skip the deploy stages too.
  • 158. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_service’… Skipping stage 'deploy_proxy_app'… Running stage 'validate'… Skipping stage 'cleanup_proxy_app'… Skipping stage 'cleanup_web_service'… --- PASS: TestProxyApp (14.22s) This allows you to iterate on solely the validate stage…
  • 159. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_service’… Skipping stage 'deploy_proxy_app'… Running stage 'validate'… Skipping stage 'cleanup_proxy_app'… Skipping stage 'cleanup_web_service'… --- PASS: TestProxyApp (14.22s) Which dramatically speeds up your iteration / feedback cycle!
  • 160. $ SKIP_validate=true $ unset SKIP_cleanup_web_service $ unset SKIP_cleanup_proxy_app When you’re done iterating, skip validate and re-enable cleanup
  • 161. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_service’… Skipping stage 'deploy_proxy_app’… Skipping stage 'validate’… Running stage 'cleanup_proxy_app’… Running stage 'cleanup_web_service'… --- PASS: TestProxyApp (59.61s) This cleans up everything that was left running.
  • 162. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } Note: each time you run test stages via go test, it’s a separate OS process.
  • 163. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } So to pass data between stages, one stage needs to write the data to disk…
  • 164. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } And the other stages need to read that data from disk.
  • 165. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries
  • 166. Real infrastructure can fail for intermittent reasons (e.g., bad EC2 instance, Apt downtime, Terraform bug)
  • 167. To avoid “flaky” tests, add retries for known errors.
  • 168. &terraform.Options{ TerraformDir: "../examples/proxy-app", RetryableTerraformErrors: map[string]string{ "net/http: TLS handshake timeout": "Terraform bug", }, MaxRetries: 3, TimeBetweenRetries: 3*time.Second, } Example: retry up to 3 times on a known TLS error in Terraform.
  • 169. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  • 170. End-to-end tests: test your entire infrastructure works together.
  • 171. How do you test this entire thing?
  • 172. You could use the same strategy… 1. Deploy all the infrastructure 2. Validate it works (e.g., via HTTP requests, API calls, SSH commands, etc.) 3. Undeploy all the infrastructure
  • 173. But it’s rare to write end-to- end tests this way. Here’s why:
  • 175. e2e Tests Integration Tests Unit Tests Static analysis Cost, brittleness, run time
  • 176. e2e Tests Integration Tests Unit Tests Static analysis 60 – 240+ minutes 5 – 60 minutes 1 – 20 minutes 1 – 60 seconds
  • 177. e2e Tests Integration Tests Unit Tests Static analysis E2E tests are too slow to be useful 60 – 240+ minutes 5 – 60 minutes 1 – 20 minutes 1 – 60 seconds
  • 178. Another problem with E2E tests: brittleness.
  • 179. Let’s do some math:
  • 180. Assume a single resource (e.g., EC2 instance) has a 1/1000 (0.1%) chance of failure.
  • 181. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ The more resources your tests deploy, the flakier they will be.
  • 182. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ You can work around the failure rate for unit & integration tests with retries
  • 183. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ You can work around the failure rate for unit & integration tests with retries
  • 184. Key takeaway: E2E tests from scratch are too slow and too brittle to be useful
  • 185. Instead, you can do incremental E2E testing!
  • 189. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline
  • 191. Technique Strengths Weaknesses Static analysis 1. Fast 2. Stable 3. No need to deploy real resources 4. Easy to use 1. Very limited in errors you can catch 2. You don’t get much confidence in your code solely from static analysis Unit tests 1. Fast enough (1 – 10 min) 2. Mostly stable (with retry logic) 3. High level of confidence in individual units 1. Need to deploy real resources 2. Requires writing non-trivial code Integration tests 1. Mostly stable (with retry logic) 2. High level of confidence in multiple units working together 1. Need to deploy real resources 2. Requires writing non-trivial code 3. Slow (10 – 30 min) End-to-end tests 1. Build confidence in your entire architecture 1. Need to deploy real resources 2. Requires writing non-trivial code 3. Very slow (60 min – 240+ min)* 4. Can be brittle (even with retry logic)*
  • 192. So which should you use?
  • 193. All of them! They all catch different types of bugs.
  • 194. e2e Tests Keep in mind the test pyramid Integration Tests Unit Tests Static analysis
  • 195. e2e Tests Lots of unit tests + static analysis Integration Tests Unit Tests Static analysis
  • 196. e2e Tests Fewer integration tests Integration Tests Unit Tests Static analysis
  • 197. e2e Tests A handful of high-value e2e tests Integration Tests Unit Tests Static analysis
  • 199. Fight the fear & build confidence in your code with automated tests