Terraform your Infrastructure with Modules, State, and Best Practices

The Hitchhiker's Guide to
Terraform your Infrastructure
Fernanda Martins - Tweakers Developers Summit 2020

Bio
github.com/flmmartins
linkedin.com/flmmartins
@flmmartins
Work @ L1NDA
Smart tools for workers and
businesses in the hospitality
industry.
DevOps Minion
Infrastructure as code lover
CI/CD enthusiast
Open Source addict
Gamer
Pianist wanna-be
Dutch learner

Agenda
● What is terraform
● Terraform 0.12
Insights on:
● Terraform Modules
● Terraform State
● Naming & Tagging
● Data
● Null Resources
● Local Files & Templating
● Kubernetes with Terraform
● Running Terraform & Pipelines

What is Terraform?
Provider: Use AWS
Resource: Create role
Resource:
Attach other
policies
Data: Build policy
https://learn.hashicorp.com/
terraform
https://events.hashicorp.co
m/workshops/amsterdam/

Terraform 0.12
● If you are starting to use terraform, use terraform 0.12
● All of the code in this presentation is 0.11
● All best practices in this presentation are terraform 0.12 compatible
Tip #1 - Migrating to 0.12
Use tfswitch to have multiple terraform
versions

I used to write my own modules all
the time…
Module Writing

Modules
Tip #2 - Open Source Modules
Use Open Source modules! Help the community!
https://github.com/terraform-aws-modules
https://github.com/cloudposse
Tip #3 - Build modules & divide to conquer
Use Open Source modules + Your business rules

Module Source in Terraform
Repo Module Folder
Version
Local Path Module

Modules - Git Repository
● Allow re-use across diﬀerent
terraform repositories
● Allow versioning
● Versioning allow better
development
Versioning will make your workflow
more complex

Modules - One repository for each module
Complexity will increase even more
because you will have thousands of
repositories to manage version & etc

Modules - Git Repository
Tip #4 - Use separate repository for modules
Use separate Git Repository for your modules if they are used across
multiple repositories or if you need versioning
Tip #5 - Avoid creating one repository for module
Use one repository with all your modules instead!

Modules - Securing Inputs & Outputs
Tip #6 - Secure Variables: Use CLI for passwords
Use var_file or TF_VAR
Tip #7 - Secure Outputs
If Terraform generates password use the sensitive option in your
outputs or a secret manager (Vault, etc…)

Document your modules with terraform-docs
Tip #8 - Tool to document your modules
It will generate documentation based in the description attribute
from inputs and outputs
https://github.com/segmentio/terraform-docs

Module Order & Dependency
● Terraform executes everything in parallel
● Order execution is done by using one module as input to other module.
This will tell Terraform to first run
run the bucket principal and then
run hosting_bucket

Dependency Drama
Troubled Dev: What if there’s no way to link 2
modules? What if only order does the trick?
https://github.com/hashicorp/terraform/issues/10462

Dependency Problem
Tip #9 - Use terragrunt to manage dependencies
You will see terragrunt be used to solve issues that terraform does
not during this presentation. More spoilers on next slides!
https://github.com/gruntwork-io/terragrunt
Before running this, run
the kms_role

Module Recap
● Avoid hassle: Use Open Source Modules
https://github.com/terraform-aws-modules
https://github.com/cloudposse
● Make your code in modules & divide to conquer
● Use a separate git repository for your modules to have versioning and more control
● Avoid creating one repository to each module to help your sanity
● Document your modules with https://github.com/segmentio/terraform-docs
● Use Terragrunt for module dependency control (more in next slides)

What is a Terraform State?
Terraform must store state. This state is used by Terraform to map real world resources
to your configuration, keep track of metadata, and to improve performance for large
infrastructures.
Terraform state is a JSON that represents your infrastructure

Important State Requirements
● Keep your state secure
● Remote
● Locking Mechanism
● State per environments
● No manual creation of state

Terraform Workspace, Right?
https://www.reddit.com/r/Terraform/comments/8qj3r2/when_to_use_workspaces/

Terraform State
Tip #10 - Use terragrunt to manage your state for you
https://github.com/gruntwork-io/terragrunt
What is terragrunt?
Terragrunt is a thin wrapper for Terraform that provides extra
tools for keeping your Terraform configurations DRY, working with
multiple Terraform modules, and managing remote state.

Terragrunt - A Terraform Wrapper
Terraform Code
Terragrunt Folder Structure

Terragrunt Environments
Environment Folder
State for Environment

Terragrunt State File development/terraform.tfvars
Remotely in S3
Secure State
Locking
Environment Variables!

Terragrunt - Terraform Call
development/app_storage/terraform.tfvars

Terraform - Don’t forget to use the state!
../../modules/app_storage/main
Required

Naming your infra
${var.product}.${var.environment}.name.service
Example: webapp.demo.user-data.bucket.
PS: If you use more than one region be sure to add it in the name as well
Tip #11 - Use naming conventions

Tagging your infra - Why it is important?
● Distinguish your infrastructure (footprint)
● Monitor your infrastructure
● Group resources.
● Cost Separation

Tag in AWS
myid.label = value
Myid is your identifier and will diﬀerentiate your labels from external automation
I also use this format for Kubernetes Labels

Tag in AWS / Use labels in Kubernetes
Label can be:
1. Name - Name is not necessary a tag but I always add it for AWS
2. product - Name of your product
3. environment - your environment
4. component - categorization of asset. AWS: This can be your service like bucket. For
Kubernetes it could be deployment, ingress…
5. part-of - Is this component part of a bigger plan? What? Example: Monitoring,
Cluster-XYZ, CI-server
6. terraform.repository
7. terraform.module = name of your module
8. terraform.module.version = version (it can also be branch name)

Label example
Environment tags
Tip #12 - Standarise Tags/Labels by using Terraform
Module tags

Data Common Mistakes
● Data can make a resource change on every execution if you use as input.
Examples in aws_security_group:
vpc_id = “${data.aws_vpc.cluster_vpc.id}”
Tip #13 - If data gives trouble, use state data
only as last resort
If you use data from state you might suﬀer
compilation errors during plan/apply terraform
Only use data from state if absolute necessary

What is Data from the state?
When you use data you are making a request to your current infra for data
When you use data from state you are actually fetching from the actual terraform state
based on what you built. That’s why you should use output in your modules.
data "terraform_remote_state" "vpc" {}

null-resource
The Rebel Terraform Resource
Tip #14 - Avoid using null resources
Check if a terraform provider does what needs to be done instead of shell scripting
You will have less work, less coding

null-resources
If trigger change the command will run
Tip #15 - Use null resources with triggers & dependencies (explicit or not)
To ensure everything will run when it has to run
Use cases: Run Ansible & restarts pods in Kubernetes after infra change

Null-resources need to be programed to
destroy

BE CAREFUL with destroys in null_resources
Tip #16 - Add Destroy Conditional to
null-resources
Use with caution

Local File
Tip #17 - Avoid using local_files
Because on every CI/laptop terraform consider local file creation as a
change

Coding Recap
● Use naming conventions and use terraform to enforce it
● Use tags/label conventions and use terraform to enforce it
● If data is forcing your resource to change an alternative it is to get data from the
state
● Avoid use of null provider
● Null Provider: always use with triggers & dependency (explicit or implicit)
● Null Provider: add a destroy conditional when is safe
● Avoid using Local File

Using Kubernetes with
Terraform

Motivation
● One Tool execution
● Seize the templating from terraform instead of HELM or
another templating library
● I need to alternate between terraform & kubernetes
commands.
Example:
○ Run Kubernetes to create load balancer
○ Run terraform to create dns record

Using null_resource provider
Tip #14 - AGAIN - Avoid using null resources

Using Kubernetes Provider - The issue

Using Kubernetes Provider - The issue
Tip #18 - For simple low maintenance kubernetes resources use terraform
Tip #19 - For sensitive/high maintenance kubernetes resources use other tools
Example: Your application

Terragrunt plan-all/apply-all
Tip #20 - Use Terragrunt plan/apply-all to make sure your infra is always
matching code
● Good for testing if all modules are fine after adding another one
● Module dependency control
● Support for multiple aws-accounts
● And much more...

Terraform plan/apply for 1 module
Why? Plan-All output can be really bad and if you are testing/coding or if when the
output is looking horrible.
To inspect your code better I would use plan with terraform-landscape and during initial
testing I would only apply in one module.
Tip #21 - To better see the results of your plan use landscape CLI

Why use a Pipeline?
● No more hassle of setting your AWS account every run
● My CLI secret variables are now in the CI and I don’t have to
worry about them in my laptop
● Everyone can run terraform
Tip #22 - Build a pipeline to run terraform for you

Plan/Apply
You can choose a module. Ex: app_storage
Environment

Tools still to explore
● Terratest - Testing
● Atlantis - CI for terraform
● Helm with Terraform for Kubernetes
● Dependabot - Keep your external modules up to date
● SecretHub - Provision Passwords & Keys for applications
● Will Pulumi beat Terraform? (https://www.pulumi.com/)
○ No more HCLs and Infra Languages: just use pure
Python/JS to build your infra

Thanks!
github.com/flmmartins
linkedin.com/flmmartins
@flmmartins
Questions?
I know the answer is 42

Terraform your Infrastructure with Modules, State, and Best Practices

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Terraform your Infrastructure with Modules, State, and Best Practices

Similar to Terraform your Infrastructure with Modules, State, and Best Practices (20)

Recently uploaded

Recently uploaded (20)

Terraform your Infrastructure with Modules, State, and Best Practices