Terraform: Tales from the Trenches

Terraform: Tales from the Trenches
Santa Barbara DevOps
Copyright © HG Data 2019

Presenters
Rob Fox
CTO
Sam Chapin
Chief Architect
Brendan Keane
DevOps Engineer
Chris Deutsch
DevOps Architect

Background
Product: https://discovery.hgdata.com
(SaaS Platform and related services)
Issues Includes:
● Previous iteration of infrastructure as code fell way short
○ Rigid configuration
○ Ansible+bash hell
○ No automation
● No ability to validate
● No visibility as to what was actually running and where
● Holes around secrets management
● Had to run on a specific version of Ubuntu

Why Terraform @ HG Data?
● Infrastructure as Code
● Immutable Infrastructure
● Declarative
● Client Only
● Solves a lot of our current issues

Success so far...
● Customer facing platform completely refactored using Terraform,
Kubernetes, Vault
● Blue/Green Deployment tied into CI/CD
● Many shared modules and libraries in github created and distributed to
rest of engineering
● Platform rock solid (easy to deploy, easy to manage and monitor)
● Big Data team now adopting Terraform and related tooling to manage
large-scaled Spark clusters and pipeline automation/orchestration
Engineers much happier and way more productive!!

What we learned along the way...
...A lot!!
What follows are three mini-presentations based on our learnings and
experiences to help you be successful
Enjoy!

Evolution of a Terraform project
Sam Chapin

Step 1. Experimentation (Someone allowed me to play with cool new toys)
The starting state of the tf project:
- Single developer
- Toy deployment
- No "environment"
- No modularization
- No shared state

Step 1. Experimentation (The halcyon days)
main.tf:Hierarchy:
terraform apply -auto-approve
Example usage:

Step 1. Experimentation - What Hurts?
- no sandbox
- isn't parameterizable
- hard to test
- non-DRY
- can't execute anywhere

Step 2. Reality sets in (My boss tells me I need to actually deploy it)
The OLD State of the tf project:
- Single developer
- Toy deployment
- No "environment"
- No modularization
- No shared state
The NEW State of the tf project:
- Single developer
- Toy deployment Prod/Stage/Dev deployment
- No "environment" Environment management
- No modularization
- No shared state

Step 2. Reality sets in - Environment via configuration
prod.tfvars:Hierarchy:
terraform apply
-var-file=prod.tfvars
Example usage:

Step 2. Reality sets in - Environment via symlinks
conf.auto.tfvars:Hierarchy:
cd prod
terraform apply
Example usage:

Step 2. Reality sets in - Environment via workspace
conf.auto.tfvars:Hierarchy:
terraform workspace new prod
terraform apply
Example usage:

Step 2. Reality sets in - Environment via workspace & consul
If you’re lucky enough to already have consul...

Step 2. Reality sets in - What Hurts?
- no sandbox
- hard to test
- non-DRY

Step 3. Refactor (I panic at how much I've been copy-pasting)
- Single developer
- Prod/Stage/Dev deployment
- Environment management (confs, copies, or workspaces)
- No modularization
- No shared state
- Single developer
- No modularization Modularization (inner modules / module library)
- No shared state

Step 3. Refactor - Modules
Hierarchy: modules/instance/main.tf:

Hierarchy: ./main.tf:

Hierarchy:
More on this from Brendan!

Step 3. Refactor - External Modules
What if we want them to be external libs?

Step 3. Refactor - What Hurts?
- no sandbox
- hard to test
- non-DRY

Step 4. Make it easy (Someone is going to see the mess I made)
- Single developer
- Modularization (inner modules or reusable module library)
- No shared state
- Single developer Multiple developers
- Modularization (inner modules or reusable module library)
- No shared state Shared state

Step 4. Make it easy - Shared state across environments
Hierarchy: ./main.tf:
Modularize this!

Step 4. Make it easy - State locking
Lock your damn state.
All the standard backends support state locking.

Backend
Step 4. Make it easy - State locking - A tale of two
terraformers
.tfstate
ApplyApply

Step 4. Make it easy - What Hurts?
- no sandbox
- hard to test
- non-DRY
- Collaboration with others (PR plan workflow)
- CI/CD locking your state all the time (Build queue)
Check out https://www.runatlantis.io/
Okay, okay... What Still Hurts?

In summary...
1. Experiment and get familiar with your new declarative friend
1. Solve your environment problem with convention & opinions
1. Make your codebase small and testable with modules
1. Ease collaboration & FUD via state sharing & locking

Terraform, Behave!
Brendan Keane

What we needed
● Shared modules
● Certainty on shared code.
● Ability to build on top of shared code, we need a high level of
certainty

What would make us more certain?
Tests!
But what kind of tests in this scenario?

Do what I say not what I mean
- Terraform does this for you with every action.
resource “aws_instance” “my_server” {
ami = “ami-0c32356aac847d7e8”
}
resource “aws_rds_instance” “my_database” {
ami = “ami-0c223918e1”
}

Do what I mean not what I say
- Terraform does not test for what you mean.
subject { “my_server” }
it “should be able to reach the database” do
# db connection test
end

Do what I say not what I mean
Great for auditing known good configuration
Do what I mean not what I say
Great for iterating towards a good configuration

Base Case:
Assert: A can ping B
BA
● Tests more than at first sight: security groups, subnets, routing, dns…
● Terraform tells me everything is as I declared it.
● Terraform does not tell me if it does what I want it to do.

Building Blocks - Bats!
https://github.com/bats-core/bats-core
● Rspec-ish.No ‘shelling out’, can run where infra is stood up
● Good helpers for stdout, stderr, and exit status.
● Before and after hooks if needed.

Building Blocks - Templating
https://www.terraform.io/docs/providers/template/d/file.html
● ssh driven: this is a plus and minus.
● Many approaches: scripts, inline, powerful when combined with templating.

Building Blocks - Remote Exec
https://www.terraform.io/docs/provisioners/null_resource.html

Building Blocks - Structure

Spec Run
Host A
- Install bats
- Run bats
Dev Machine
- Terraform apply
1. Spin up
3. Runs tests in cloud context
2. Remote-exe Bats
Host B

Proof Pudding (Demo Time)

Concluding Thoughts
● Keep the tests simple, too many null_providers == bad
● Positive assertions are usually functionality: A can ping B
● Negative assertions are usually security: B cannot ping A
Future: dockerize, dashboard of health tests orthogonal from application
health checks.

Terraforming Blue/Green
Deployments
Chris Deutsch

The problem...
“One of the challenges with automating deployment is the cut-over itself,
taking software from the final stage of testing to live production.”
https://www.martinfowler.com/bliki/BlueGreenDeployment.html

Example basic setup

Blue and green deployments

Users are routed to green (live)

Dev/QA are routed to blue (next)

Bringing a new release live

Oh no!

Recovery: flip live back to green

Considerations
While the idea is relatively simple, here are three important things to
keep in mind.

1. Infrastructure
● We have two clusters: blue and green
● Each cluster should be configured in the same way
● Each cluster should operate independently and be easy to maintain
In addition, the more we split things up, the more value we can get out
of blue/green deploys for our infrastructure. But we also increase cost
and complexity.

2. Configuration Management
Applications should have the same configuration regardless of whether
they are blue or green.
Applications should have the same configuration regardless whether
they are the live version or the next version.
(Well… as much as is reasonable.)

3. Application State
“Twelve-factor processes are stateless and share-nothing. Any data that
needs to persist must be stored in a stateful backing service, typically a
database.”
https://12factor.net/processes

Terraform + ELB + Kube + RDS + ElastiCache

Module 1: Terraforming a Load Balancer

Starting the app_load_balancer module

Add an aws_lb_target_group...

And an aws_lb_listener_rule...

Okay! Time to use it

Starting the app_cluster module

Attaching the Instance to a Target Group
my_arns = {
next = “some_next_arn”
live = “some_live_arn”
}
lookup(var.my_arns, “next”)
=> “some_next_arn”

Excellent! Time to use it

Module 3: the green deployment

(It’s actually almost the same as blue)

(It’s actually almost the same as blue)Color flipping, Terraform style

1. Change Green From Next to Live

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes

Now both are live!

Verify With Curl
$ curl http://live.example.com/
I'm an app running in the production environment on
green
I'm an app running in the production environment on blue

2. Move Blue From Live To Next

Now green is live, but blue is next

Verify With Curl
I'm an app running in the production environment on
green
$ curl http://next.example.com/
503 Service Temporarily Unavailable
# Wait a couple of minutes...
$ curl http://next.example.com/
I'm an app running in the production environment on blue

Deploy Complete!

Q&A
Thank you!
Follow us online @ https://www.hgdata.com

Terraform: Tales from the Trenches

Recommended

Recommended

More Related Content

Similar to Terraform: Tales from the Trenches

Similar to Terraform: Tales from the Trenches (20)

Recently uploaded

Recently uploaded (20)

Terraform: Tales from the Trenches