A demonstration on how they use Terraform Modules, Packer and Jenkins to build a continuous delivery pipeline to AWS.

1. zz
Terraform Modules and
Continuous Deployments
San Francisco Infrastructure as Code Meetup

2. zz
Your speakers…
• Zane Williamson
• Trulia Sr. DevOps
• Twitter @zane_williamson
• Github @sepulworld
• Marius Ducea
• AWS Consultant
• Twitter @mariusducea
• Github @mdxp

3. zz
Overview
1. The Challenge
2. System overview
3. Why we chose Terraform and Packer for Continuous
Deployment
4. Terraform modules and how we are using them
5. Demo a continuous deployment with Terraform and Packer

4. zz
The Challenge
• Deploying an array of micro services
• Multiple teams
• Repeatable Cloud configuration
• Peer reviewed infrastructure changes
• Tooling flexibility
• As simple as possible for non experts

5. zz
System Overview
• Cloud Provider is AWS
• Micro-services APIs with ELB
endpoints
• Micro-services acting as worker
clusters without ELB endpoints
• A common shared backend

6. zz
System Overview
Terraform Managed
• Kafka cluster
• SQS
• s3
• RDS
• EMR (Hbase)
• VPC, subnets, routes, etc

7. zz
Why We Chose Packer to Build Images
1. Dev and ops personnel were already familiar with Packer
2. Allowed us to reuse Puppet modules
3. Multiple outputs (VirtualBox, Docker, AMI, etc)
4. Able to embed in the micro-service code base
5. Easy to call from Jenkins server

8. zz
Why we chose Terraform to Manage
Infrastructure
1. CloudFormation was limited to AWS services
2. Libraries like Boto and Fog required programming skills and don’t
manage state as well
3. Some developers and operations personnel were already familiar with
Terraform
4. Easy to integrate into Jenkins which is the preferred build and
deployment platform of most of our teams
5. Ability to put the Terraform directly into the code base for each micro-
service
6. Leverage Terraform Modules and the capability to use a Git repo as a
source

9. zz
Challenges we faced early on
Multiple teams working with terraform
• code duplication | code drift
• management & maintenance
• versioning
• testing

10. zz
Terraform modules
The solution to all our problems?

11. zz
Terraform modules basics
• Modules are used to create reusable components in
Terraform as well as for basic code organization.
• Modules are very easy to create and use.
• A module is technically just a folder with some
terraform templates.

12. zz
Terraform modules - example ELB module
elb
├── CHANGELOG.md
├── README.md
├── main.tf
├── outputs.tf
├── test
│ ├── aws.tf
│ └── test.tf
└── variables.tf

13. zz
variables.tf
variable "name" {
default = "dev-elb"
}
variable "subnet_ids" {
description = "comma separated list of subnet IDs"
}
variable "security_groups" {
description = "comma separated list of security group IDs"
}
variable "port" {
description = "Instance port"
default = 80
}

14. zz
main.tf
resource "aws_elb" "main" {
name = "${var.name}"
cross_zone_load_balancing = true
subnets = ["${split(",", var.subnet_ids)}"]
security_groups = ["${split(",",var.security_groups)}"]
listener {
lb_port = 80
lb_protocol = "http"
instance_port = "${var.port}"
instance_protocol = "http"
}
. . .

15. zz
outputs.tf
output "elb_name" {
value = "${aws_elb.main.name}"
}
output "elb_id" {
value = "${aws_elb.main.id}"
}
output "elb_dns_name" {
value = "${aws_elb.main.dns_name}"
}
output "elb_zone_id" {
value = "${aws_elb.main.zone_id}"
}

16. zz
Module usage
Using modules in Terraform is very similar to built-in
resources:
module "elb" {
source = “./tf_elb”
name = "myelb"
port = "80"
health_check_url = "HTTP:80/"
}

17. zz
Module usage - remote git source
The module can live in the same place with the code
using it (subfolder) or it can be in a separate repo
(recommended).
module "elb" {
source =
"github.com/sepulworld/tf_elb.git?ref=v0.0.1"
name = "myelb"
port = "80"
health_check_url = "HTTP:80/"
}

18. zz
Module usage - multiple instances
We can instantiate a module multiple times:
module "elb-frontend" {
source = “./tf_elb”
name = "frontend"
port = "80"
health_check_url = "HTTP:80/"
}
module "elb-internal" {
source = “./tf_elb”
name = "internal"
port = "8080"
health_check_url = "HTTP:8080/health"
}

19. zz
Module usage - get sources
Finally, before using the module we need to first
download it from the source, using terraform get:
terraform get -update
Get: git::https://github.com/sepulworld/tf_asg.git?ref=v0.0.3
(update)
Get: git::https://github.com/sepulworld/tf_elb.git?ref=v0.0.1
(update)

20. zz
So are modules helping solve our challenges?
Let’s see…
✓ code duplication | code drift
✓ management & maintenance
✓ versioning
✓ testing

21. zz
Lessons learned
• Use variables for everything; have sane defaults.
• Document your modules; we use a changelog to
have a history of all bug fixes and new features.
• Use separate git repositories for your modules and
use tags to release new versions of the module.
• Test your modules (ideally automatically); we use
terraform validate on all commits and a test run on
new releases.
• Be aware that different versions of terraform might
behave differently.

22. zz
Lessons learned
• Separate your terraform code to minimize the
impact of a failure; we use something like:
global (global resources like IAM, cloudtrail, s3, etc.)
└ users
development (dev environment)
└ core (base resources like vpc, sg, etc.)
└ db (persistent storage, rds, etc.)
└ microservice1 (resources for services like asg, elb, route53, etc.)
production (prod environment)
└ core (base resources like vpc, sg, etc.)
└ db (persistent storage, rds, etc.)
└ microservice2 (resources for services like asg, elb, route53, etc.)

23. zz
Lessons learned
• Use terraform_remote_state to share the state
between different terraform runs.
data "terraform_remote_state" "vcp" {
backend = "s3"
config {
bucket = “terraform-state-s3-bucket"
key = "dev-vpc-us-west-2"
}
}
resource "aws_instance" "foo" {
# ...
subnet_id = "${data.terraform_remote_state.vpc.subnet_id}"
}

24. zz
Lessons learned
• Terraform new features and improvements added
by a new version might break the run on an older
version. Always tag new releases of a module that
might break older version runs.
• If a new resource or a new argument to an existing
one was introduced that will be seen as an error on
older versions.
resource "aws_elasticsearch_domain" "es" {
elasticsearch_version = “${var.es_version}” # added in 0.7.1
domain_name = "${var.es_domain_name}"

25. zz
Terraform version manager
Use different versions of terraform (tfenv):
https://github.com/kamatama41/tfenv
tfenv install 0.7.7
install Terraform 0.7.7
get archive from
https://releases.hashicorp.com/terraform/0.7.7/terraform_0.7.7_darw
in_amd64.zip
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 16.8M 100 16.8M 0 0 7163k 0 0:00:02 0:00:02 --
:--:-- 7167k
Archive: /tmp/terraform_0.7.7_darwin_amd64.zip
inflating: /Users/marius/bin/tfenv/versions/0.7.7/terraform
the installation 0.7.7 was successful!!!

26. zz
Future improvements
• Conditionals, conditionals, conditionals…
• Terraform language is very limited and this prevents us
from writing ‘real’ reusable modules; at this time we are
using all sort of count related hacks to overcome
conditional limitations but hopefully we’ll have better
solutions in the future.
• Terraform state locking
• Better handling of multiple versions of Terraform
• Testing improvements

27. zz
Demo
Infrastructure Timeline

28. zz
Challenges and No Silver Bullets
1. Doesn’t work with systems that require code to be in
consistent state. For this a Blue/Green type of deployment is
required.
2. Clean up of old AMIs needs to be handled external to
Packer and Terraform
3. Requires application to be engineered for the Cloud
4. Autoscaling Group failure scenarios don’t automatically
cleanup

29. zz
https://github.com/sepulworld/tf_elb
https://github.com/sepulworld/tf_asg
https://github.com/sepulworld/terraform-examples
Demo links:
Questions!