Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Reusable, composable, battle-tested
TERRAFORM
MODULES
gruntwork.io
Your project is code complete
and it’s time to deploy it!
I know, I’ll use AWS!
You login to the AWS console
(Spend hours reading docs)
OK, I have a server running!
What else do I need?
Well, you probably want more than
one server for high availability
And a load balancer to distribute
traffic across those servers
And EBS Volumes and RDS databases
to store all of your data
You’ll need an S3 bucket for files
CloudWatch for monitoring
Don’t forget the VPC, subnets, route
tables, NAT gateways
Route 53 for DNS
ACM for SSL/TLS certs and KMS to
encrypt / decrypt secrets
And you need all of that in separate
environments for stage and prod
stage prod
Plus DevOps tooling to manage it all
stage prod
And a CI server to test it all
stage prod
Plus alerts and on-call rotation to
notify you when it all breaks
stage prod
stage prod
And also…
And you have to maintain it all.
Forever.
AWS: 1,000 new releases in 2016
Terraform: release every ~2 weeks
Security vulnerabilities: daily
There’s a better way to deploy
and manage infrastructure:
TERRAFORM
MODULES
TERRAFORM
MODULES
Reusable, composable, battle-tested
infrastructure code
In this talk, I’ll show you how
Terraform Modules work
stage prod
And how they will allow you to do all
of this…
> terraform init <…>
> terraform apply
In just a few simple commands
I’m
Yevgeniy
Brikman
ybrikman.com
Author
Co-founder of
Gruntwork
gruntwork.io
Your entire AWS
infrastructure.
Defined as code.
In about a day.
gruntwork.io
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Outline
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Outline
The two primary types of
infrastructure providers:
Infrastructure as a Service (IaaS): e.g.,
AWS, Azure, Google Cloud
CPU Memory Disk Drive Network Server DB
They offer many primitives and it’s up
to you to put them together
CPU Memory Disk Drive Network Server DB
Platform as a Service (PaaS): e.g.,
Heroku, Docker Cloud, Engine Yard
Rails MySQL GitHub
CPU Memory Disk Drive Network Ser...
They hide all the lower-level details so
you can focus on your apps
Rails MySQL GitHub
CPU Memory Disk Drive Network Serve...
> cd my-nodejs-app
> heroku create my-nodejs-app
> git push heroku master
Getting started with a PaaS is
easy!
Heroku limitations
1. Can only use supported runtimes & versions (e.g., python-3.6.2 or python-2.7.13)
2. Can only use sup...
For most software companies, IaaS is
the only way to grow
CPU Memory Disk Drive Network Server DB
stage prod
But that requires dealing with this…
We are developers.
We know how to fix this.
Use code!
Terraform is a tool for defining and
managing infrastructure as code
provider "aws" {
region = "us-east-1"
}
resource "aws_instance" "example" {
ami = "ami-408c7f28"
instance_type = "t2.micro...
> terraform apply
aws_instance.example: Creating...
ami: "" => "ami-408c7f28"
instance_type: "" => "t2.micro"
key_name: ""...
Terraform supports modules
They are like reusable blueprints
for your infrastructure
resource "aws_autoscaling_group" "example" {
name = "${var.name}-service"
min_size = "${var.num_instances}"
max_size = "${...
module "service_foo" {
source = "/modules/microservice"
image_id = "ami-123asd1"
num_instances = 3
}
Now you can use the m...
module "service_foo" {
source = "/modules/microservice"
image_id = "ami-123asd1"
num_instances = 3
}
module "service_bar" ...
Modules allow you to use your
favorite IaaS provider…
CPU Memory Disk Drive Network Server DB
With easy-to-use, high-level
abstractions, like a PaaS
CPU Memory Disk Drive Network Server DB
Rails MySQL GitHub
But since you have all the code, you
still have full control!
CPU Memory Disk Drive Network Server DB
Rails MySQL GitHub
Best of all: code can be shared
and re-used!
The Terraform Module Registry
A collection of reusable, verified,
supported Modules
Example: Vault Module for AWS
Example: Consul for Azure
Example: Nomad for GCP
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Outline
Imagine you wanted to deploy Vault
The old way:
1. Open up the Vault docs
2. Deploy a few servers
3. Install Vault
4. Install supervisord
5. Configure mlock
6. Generate s...
The new way:
> terraform init hashicorp/vault/aws
> terraform apply
And now you have this deployed
As simple as a PaaS!
> tree
.
├── README.md
├── main.tf
├── outputs.tf
├── packer
├── user-data
└── variables.tf
But you also have all the code...
module "vault_cluster" {
source = "hashicorp/vault/aws"
cluster_name = "example-vault-cluster"
cluster_size = 3
vpc_id = "...
> terraform apply
Run apply when you’re done!
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Outline
def add(x, y):
return x + y
Most programming languages
support functions
def add(x, y):
return x + y
The function has a name
def add(x, y):
return x + y
It can take in inputs
def add(x, y):
return x + y
And it can return outputs
def add(x, y):
return x + y
add(3, 5)
add(10, 35)
add(-45, 6)
Key idea: code reuse
def add(x, y):
return x + y
assert add(3, 5) == 8
assert add(10, 35) == 45
assert add(-45, 6) == -39
Key idea: testing
def add(x, y):
return x + y
def sub(x, y):
return x - y
sub(add(5, 3), add(4, 7))
Key idea: composition
def run_classifier(data_set):
X_pca = PCA(n_components=2).fit_transform(X_train)
clusters = clf.fit_predict(X_train)
fig, ...
def run_classifier(data_set):
X_pca = PCA(n_components=2).fit_transform(X_train)
clusters = clf.fit_predict(X_train)
fig, ...
def run_classifier(data_set):
X_pca = PCA(n_components=2).fit_transform(X_train)
clusters = clf.fit_predict(X_train)
fig, ...
Modules are Terraform’s
equivalent of functions
A simple module:
> tree minimal-module
.
├── main.tf
├── outputs.tf
├── variables.tf
└── README.md
It’s just Terraform code in a folder!
variable "name" {
description = "The name of the EC2 instance"
}
variable "image_id" {
description = "The ID of the AMI to...
output "url" {
value = "http://${aws_instance.example.ip}:${var.port}"
}
The outputs are in outputs.tf
resource "aws_autoscaling_group" "example" {
name = "${var.name}-service"
min_size = "${var.num_instances}"
max_size = "${...
# Foo Module for AWS
This is a Terraform Module to deploy
[Foo](http://www.example.com) on AWS, including:
* foo
* bar
* b...
A more complicated module:
> tree complete-module
.
├── main.tf
├── outputs.tf
├── variables.tf
├── README.MD
├── modules
├── examples
└── test
We ad...
> tree complete-module/modules
modules/
├── submodule-bar
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
└── submodule-...
> tree complete-module/modules
modules/
├── submodule-bar
│ ├── install-vault.sh
│ └── run-vault.sh
└── submodule-foo
└── ...
For example, one submodule can be
used to install Vault in an AMI
Another to create self-signed TLS
certificates
Another to deploy the AMI across an
Auto Scaling Group (ASG)
Another to create an S3 bucket and
IAM policies as a storage backend
Another to configure the Security
Group settings
And one more to deploy a load
balancer (ELB)
You can use all the submodules
Or pick the ones you want and swap
in your own for the rest
> tree complete-module/examples
examples/
├── example-foo
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
└── example-ba...
Note: the code in the root is
usually a “canonical” example
> tree complete-module
.
├── main.tf
├── outputs.tf
├── variab...
It’s typically an opinionated way to
use all the submodules together
> tree complete-module/examples
examples/
├── example-foo
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
└── example-ba...
E.g., How to use just one or two of the
submodules together
Or how to combine with other
modules (e.g., Vault + Consul)
This is like function composition!
> tree complete-module/test
test/
├── example_foo_test.go
└── example_bar_test.go
The test folder contains
automated tests
> tree complete-module/test
test/
├── example_foo_test.go
└── example_bar_test.go
The tests are typically “integration
tes...
func vaultTest(t *testing.T, options *terratest.Options) {
tlsCert := generateSelfSignedTlsCert(t)
defer cleanupSelfSigned...
func vaultTest(t *testing.T, options *terratest.Options) {
tlsCert := generateSelfSignedTlsCert(t)
defer cleanupSelfSigned...
func vaultTest(t *testing.T, options *terratest.Options) {
tlsCert := generateSelfSignedTlsCert(t)
defer cleanupSelfSigned...
func vaultTest(t *testing.T, options *terratest.Options) {
tlsCert := generateSelfSignedTlsCert(t)
defer cleanupSelfSigned...
func vaultTest(t *testing.T, options *terratest.Options) {
tlsCert := generateSelfSignedTlsCert(t)
defer cleanupSelfSigned...
Using a module:
module "service_foo" {
source = "./minimal-module"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}
For simple Modules ...
module "submodule_foo" {
source = "./complete-module/modules/submodule-foo"
param_foo = "foo"
param_bar = 8080
}
module "s...
module "service_foo" {
source = "./minimal-module"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}
Abstraction: simple...
module "service_foo" {
source = "./minimal-module"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}
module "service_bar...
Versioning:
module "service_foo" {
source = "./foo"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}
You can set source to point to...
module "service_foo" {
source = "hashicorp/vault/aws"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}
Alternatively, y...
module "service_foo" {
source = "git::git@github.com:foo/bar.git"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}
Or a...
module "service_foo" {
source = "git::git@github.com:foo/bar.git?ref=v1.0.0"
name = "Foo"
image_id = "ami-123asd1"
port = ...
module "service_foo" {
source = "git::git@github.com:foo/bar.git?ref=v1.0.0"
name = "Foo"
image_id = "ami-123asd1"
port = ...
module "service_foo" {
source = "git::git@github.com:foo/bar.git?ref=v2.0.0"
name = "Foo"
image_id = "ami-123asd1"
port = ...
Promote immutable, versioned
infrastructure across environments
qa
stage
prod
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Outline
“You are not special.
Your infrastructure is
not a beautiful and
unique snowflake. You
have the same tech debt
as everyone...
stage prod
You need this
stage prod
And so does everyone else
Stop reinventing the wheel
Start building on top of
battle-tested code
Start building on top of
commercially-supported code
Start building on top of
code
Advantages of code
1. Reuse
2. Compose
3. Configure
4. Customize
5. Debug
6. Test
7. Version
8. Document
At Gruntwork, we’ve
been building
Modules for years
gruntwork.io
Many companies, all
running on the same
infrastructure code
gruntwork.io
stage prod
Modules allow us to turn this…
> terraform init <…>
> terraform apply
… into this
With some help from this
Questions?
modules@gruntwork.io
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modules
Upcoming SlideShare
Loading in …5
×

Reusable, composable, battle-tested Terraform modules

14,713 views

Published on

Listen up, developers. You are not special. Your infrastructure is not a beautiful and unique snowflake. You have the same tech debt as everyone else. This is a talk about a better way to build and manage infrastructure: Terraform Modules. It goes over how to build infrastructure as code, package that code into reusable modules, design clean and flexible APIs for those modules, write automated tests for the modules, and combine multiple modules into an end-to-end techs tack in minutes.

You can find the video here: https://www.youtube.com/watch?v=LVgP63BkhKQ

Published in: Technology

Reusable, composable, battle-tested Terraform modules

  1. 1. Reusable, composable, battle-tested TERRAFORM MODULES gruntwork.io
  2. 2. Your project is code complete and it’s time to deploy it!
  3. 3. I know, I’ll use AWS!
  4. 4. You login to the AWS console
  5. 5. (Spend hours reading docs)
  6. 6. OK, I have a server running!
  7. 7. What else do I need?
  8. 8. Well, you probably want more than one server for high availability
  9. 9. And a load balancer to distribute traffic across those servers
  10. 10. And EBS Volumes and RDS databases to store all of your data
  11. 11. You’ll need an S3 bucket for files
  12. 12. CloudWatch for monitoring
  13. 13. Don’t forget the VPC, subnets, route tables, NAT gateways
  14. 14. Route 53 for DNS
  15. 15. ACM for SSL/TLS certs and KMS to encrypt / decrypt secrets
  16. 16. And you need all of that in separate environments for stage and prod stage prod
  17. 17. Plus DevOps tooling to manage it all stage prod
  18. 18. And a CI server to test it all stage prod
  19. 19. Plus alerts and on-call rotation to notify you when it all breaks stage prod
  20. 20. stage prod And also…
  21. 21. And you have to maintain it all. Forever.
  22. 22. AWS: 1,000 new releases in 2016
  23. 23. Terraform: release every ~2 weeks
  24. 24. Security vulnerabilities: daily
  25. 25. There’s a better way to deploy and manage infrastructure:
  26. 26. TERRAFORM MODULES
  27. 27. TERRAFORM MODULES Reusable, composable, battle-tested infrastructure code
  28. 28. In this talk, I’ll show you how Terraform Modules work
  29. 29. stage prod And how they will allow you to do all of this…
  30. 30. > terraform init <…> > terraform apply In just a few simple commands
  31. 31. I’m Yevgeniy Brikman ybrikman.com
  32. 32. Author
  33. 33. Co-founder of Gruntwork gruntwork.io
  34. 34. Your entire AWS infrastructure. Defined as code. In about a day. gruntwork.io
  35. 35. 1. What’s a Module 2. How to use a Module 3. How Modules work 4. The future of Modules Outline
  36. 36. 1. What’s a Module 2. How to use a Module 3. How Modules work 4. The future of Modules Outline
  37. 37. The two primary types of infrastructure providers:
  38. 38. Infrastructure as a Service (IaaS): e.g., AWS, Azure, Google Cloud CPU Memory Disk Drive Network Server DB
  39. 39. They offer many primitives and it’s up to you to put them together CPU Memory Disk Drive Network Server DB
  40. 40. Platform as a Service (PaaS): e.g., Heroku, Docker Cloud, Engine Yard Rails MySQL GitHub CPU Memory Disk Drive Network Server DB
  41. 41. They hide all the lower-level details so you can focus on your apps Rails MySQL GitHub CPU Memory Disk Drive Network Server DB
  42. 42. > cd my-nodejs-app > heroku create my-nodejs-app > git push heroku master Getting started with a PaaS is easy!
  43. 43. Heroku limitations 1. Can only use supported runtimes & versions (e.g., python-3.6.2 or python-2.7.13) 2. Can only use supported system software & libraries 3. Can only run web services (data stores and other services available only via paid add-ons) 4. Apps can’t access the shell 5. Devs can’t access servers via SSH 6. Local disk is read-only 7. Load balancing is HTTP/HTTPS only 8. Requests are limited to 30 seconds 9. Limited to one AWS region 10. App must boot in 60 seconds or less 11. Apps can be at most 100MB 12. Build must take less than 15 min 13. Logs are limited to 1500 lines unless you use supported (paid) add-ons 14. Manual scaling only 15. Pricing gets very steep as you scale up 16. Support only available on PST time zone 17. Limited control over security settings However, customizing, debugging, and scaling is not.
  44. 44. For most software companies, IaaS is the only way to grow CPU Memory Disk Drive Network Server DB
  45. 45. stage prod But that requires dealing with this…
  46. 46. We are developers. We know how to fix this.
  47. 47. Use code!
  48. 48. Terraform is a tool for defining and managing infrastructure as code
  49. 49. provider "aws" { region = "us-east-1" } resource "aws_instance" "example" { ami = "ami-408c7f28" instance_type = "t2.micro" } Example: Terraform code to deploy a server in AWS
  50. 50. > terraform apply aws_instance.example: Creating... ami: "" => "ami-408c7f28" instance_type: "" => "t2.micro" key_name: "" => "<computed>" private_ip: "" => "<computed>" public_ip: "" => "<computed>” aws_instance.example: Creation complete Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Run terraform apply to deploy the server
  51. 51. Terraform supports modules
  52. 52. They are like reusable blueprints for your infrastructure
  53. 53. resource "aws_autoscaling_group" "example" { name = "${var.name}-service" min_size = "${var.num_instances}" max_size = "${var.num_instances}" } resource "aws_launch_configuration" "example" { image_id = "${var.image_id}" instance_type = "${var.instance_type}" root_block_device { volume_type = "gp2" volume_size = 200 } } Example: create a module to deploy a microservice
  54. 54. module "service_foo" { source = "/modules/microservice" image_id = "ami-123asd1" num_instances = 3 } Now you can use the module to deploy one microservice
  55. 55. module "service_foo" { source = "/modules/microservice" image_id = "ami-123asd1" num_instances = 3 } module "service_bar" { source = "/modules/microservice" image_id = "ami-f2bb05ln" num_instances = 6 } module "service_baz" { source = "/modules/microservice" image_id = "ami-ny6v24xa" num_instances = 3 } Or multiple microservices
  56. 56. Modules allow you to use your favorite IaaS provider… CPU Memory Disk Drive Network Server DB
  57. 57. With easy-to-use, high-level abstractions, like a PaaS CPU Memory Disk Drive Network Server DB Rails MySQL GitHub
  58. 58. But since you have all the code, you still have full control! CPU Memory Disk Drive Network Server DB Rails MySQL GitHub
  59. 59. Best of all: code can be shared and re-used!
  60. 60. The Terraform Module Registry
  61. 61. A collection of reusable, verified, supported Modules
  62. 62. Example: Vault Module for AWS
  63. 63. Example: Consul for Azure
  64. 64. Example: Nomad for GCP
  65. 65. 1. What’s a Module 2. How to use a Module 3. How Modules work 4. The future of Modules Outline
  66. 66. Imagine you wanted to deploy Vault
  67. 67. The old way:
  68. 68. 1. Open up the Vault docs 2. Deploy a few servers 3. Install Vault 4. Install supervisord 5. Configure mlock 6. Generate self-signed TLS cert 7. Create Vault config file 8. Create an S3 bucket as storage backend 9. Figure out IAM policies for the S3 bucket 10. Tinker with security group rules 11. Figure out IP addresses to bind to and advertise 12. Fight with Vault for hours because it won’t accept your TLS cert 13. Regenerate cert with RSA encryption 14. Update OS certificate store to accept self-signed certs 15. Realize you need to deploy a Consul cluster for high availability 16. Open up the Consul docs…
  69. 69. The new way:
  70. 70. > terraform init hashicorp/vault/aws > terraform apply
  71. 71. And now you have this deployed
  72. 72. As simple as a PaaS!
  73. 73. > tree . ├── README.md ├── main.tf ├── outputs.tf ├── packer ├── user-data └── variables.tf But you also have all the code. Feel free to edit it!
  74. 74. module "vault_cluster" { source = "hashicorp/vault/aws" cluster_name = "example-vault-cluster" cluster_size = 3 vpc_id = "${data.aws_vpc.default.id}" subnet_ids = "${data.aws_subnets.default.ids}" } module "consul_cluster" { source = "hashicorp/consul/aws" cluster_name = "example-consul-cluster" cluster_size = 3 vpc_id = "${data.aws_vpc.default.id}" subnet_ids = "${data.aws_subnets.default.ids}" }Example: modify main.tf
  75. 75. > terraform apply Run apply when you’re done!
  76. 76. 1. What’s a Module 2. How to use a Module 3. How Modules work 4. The future of Modules Outline
  77. 77. def add(x, y): return x + y Most programming languages support functions
  78. 78. def add(x, y): return x + y The function has a name
  79. 79. def add(x, y): return x + y It can take in inputs
  80. 80. def add(x, y): return x + y And it can return outputs
  81. 81. def add(x, y): return x + y add(3, 5) add(10, 35) add(-45, 6) Key idea: code reuse
  82. 82. def add(x, y): return x + y assert add(3, 5) == 8 assert add(10, 35) == 45 assert add(-45, 6) == -39 Key idea: testing
  83. 83. def add(x, y): return x + y def sub(x, y): return x - y sub(add(5, 3), add(4, 7)) Key idea: composition
  84. 84. def run_classifier(data_set): X_pca = PCA(n_components=2).fit_transform(X_train) clusters = clf.fit_predict(X_train) fig, ax = plt.subplots(1, 2, figsize=(8, 4)) fig.subplots_adjust(top=0.85) predicted = svc_model.predict(X_test) images_and_predictions = list(zip(images_test, predicted)) ax[0].scatter(X_pca[:, 0], X_pca[:, 1], c=clusters) ax[0].set_title('Predicted Training Labels') ax[1].scatter(X_pca[:, 0], X_pca[:, 1], c=y_train) ax[1].set_title('Actual Training Labels') Key idea: abstraction
  85. 85. def run_classifier(data_set): X_pca = PCA(n_components=2).fit_transform(X_train) clusters = clf.fit_predict(X_train) fig, ax = plt.subplots(1, 2, figsize=(8, 4)) fig.subplots_adjust(top=0.85) predicted = svc_model.predict(X_test) images_and_predictions = list(zip(images_test, predicted)) ax[0].scatter(X_pca[:, 0], X_pca[:, 1], c=clusters) ax[0].set_title('Predicted Training Labels') ax[1].scatter(X_pca[:, 0], X_pca[:, 1], c=y_train) ax[1].set_title('Actual Training Labels')You want to hide a large “volume”…
  86. 86. def run_classifier(data_set): X_pca = PCA(n_components=2).fit_transform(X_train) clusters = clf.fit_predict(X_train) fig, ax = plt.subplots(1, 2, figsize=(8, 4)) fig.subplots_adjust(top=0.85) predicted = svc_model.predict(X_test) images_and_predictions = list(zip(images_test, predicted)) ax[0].scatter(X_pca[:, 0], X_pca[:, 1], c=clusters) ax[0].set_title('Predicted Training Labels') ax[1].scatter(X_pca[:, 0], X_pca[:, 1], c=y_train) ax[1].set_title('Actual Training Labels') Behind a small “surface area”
  87. 87. Modules are Terraform’s equivalent of functions
  88. 88. A simple module:
  89. 89. > tree minimal-module . ├── main.tf ├── outputs.tf ├── variables.tf └── README.md It’s just Terraform code in a folder!
  90. 90. variable "name" { description = "The name of the EC2 instance" } variable "image_id" { description = "The ID of the AMI to run" } variable "port" { description = "The port to listen on for HTTP requests" } The inputs are in variables.tf
  91. 91. output "url" { value = "http://${aws_instance.example.ip}:${var.port}" } The outputs are in outputs.tf
  92. 92. resource "aws_autoscaling_group" "example" { name = "${var.name}-service" min_size = "${var.num_instances}" max_size = "${var.num_instances}" } resource "aws_launch_configuration" "example" { image_id = "${var.image_id}" instance_type = "${var.instance_type}" root_block_device { volume_type = "gp2" volume_size = 200 } }The resources are in main.tf
  93. 93. # Foo Module for AWS This is a Terraform Module to deploy [Foo](http://www.example.com) on AWS, including: * foo * bar * baz Documentation is in README.md
  94. 94. A more complicated module:
  95. 95. > tree complete-module . ├── main.tf ├── outputs.tf ├── variables.tf ├── README.MD ├── modules ├── examples └── test We add three new folders: modules, examples, test
  96. 96. > tree complete-module/modules modules/ ├── submodule-bar │ ├── main.tf │ ├── outputs.tf │ └── variables.tf └── submodule-foo ├── main.tf ├── outputs.tf └── variables.tf The modules folder contains standalone “submodules”
  97. 97. > tree complete-module/modules modules/ ├── submodule-bar │ ├── install-vault.sh │ └── run-vault.sh └── submodule-foo └── main.go Some of the submodules may not even be Terraform code
  98. 98. For example, one submodule can be used to install Vault in an AMI
  99. 99. Another to create self-signed TLS certificates
  100. 100. Another to deploy the AMI across an Auto Scaling Group (ASG)
  101. 101. Another to create an S3 bucket and IAM policies as a storage backend
  102. 102. Another to configure the Security Group settings
  103. 103. And one more to deploy a load balancer (ELB)
  104. 104. You can use all the submodules
  105. 105. Or pick the ones you want and swap in your own for the rest
  106. 106. > tree complete-module/examples examples/ ├── example-foo │ ├── main.tf │ ├── outputs.tf │ └── variables.tf └── example-bar ├── main.tf ├── outputs.tf └── variables.tf The examples folder shows how to use the submodules
  107. 107. Note: the code in the root is usually a “canonical” example > tree complete-module . ├── main.tf ├── outputs.tf ├── variables.tf ├── README.MD ├── modules ├── examples └── test
  108. 108. It’s typically an opinionated way to use all the submodules together
  109. 109. > tree complete-module/examples examples/ ├── example-foo │ ├── main.tf │ ├── outputs.tf │ └── variables.tf └── example-bar ├── main.tf ├── outputs.tf └── variables.tf The code in examples shows other possible permutations
  110. 110. E.g., How to use just one or two of the submodules together
  111. 111. Or how to combine with other modules (e.g., Vault + Consul)
  112. 112. This is like function composition!
  113. 113. > tree complete-module/test test/ ├── example_foo_test.go └── example_bar_test.go The test folder contains automated tests
  114. 114. > tree complete-module/test test/ ├── example_foo_test.go └── example_bar_test.go The tests are typically “integration tests”
  115. 115. func vaultTest(t *testing.T, options *terratest.Options) { tlsCert := generateSelfSignedTlsCert(t) defer cleanupSelfSignedTlsCert(t, tlsCert) amiId := buildVaultAmi(t) defer cleanupAmi(t, amiId) terratest.Apply(options) defer terratest.Destroy(options) assertCanInitializeAndUnsealVault(t, options) } Example test case for Vault
  116. 116. func vaultTest(t *testing.T, options *terratest.Options) { tlsCert := generateSelfSignedTlsCert(t) defer cleanupSelfSignedTlsCert(t, tlsCert) amiId := buildVaultAmi(t) defer cleanupAmi(t, amiId) terratest.Apply(options) defer terratest.Destroy(options) assertCanInitializeAndUnsealVault(t, options) } Create test-time resources
  117. 117. func vaultTest(t *testing.T, options *terratest.Options) { tlsCert := generateSelfSignedTlsCert(t) defer cleanupSelfSignedTlsCert(t, tlsCert) amiId := buildVaultAmi(t) defer cleanupAmi(t, amiId) terratest.Apply(options) defer terratest.Destroy(options) assertCanInitializeAndUnsealVault(t, options) } Run terraform apply
  118. 118. func vaultTest(t *testing.T, options *terratest.Options) { tlsCert := generateSelfSignedTlsCert(t) defer cleanupSelfSignedTlsCert(t, tlsCert) amiId := buildVaultAmi(t) defer cleanupAmi(t, amiId) terratest.Apply(options) defer terratest.Destroy(options) assertCanInitializeAndUnsealVault(t, options) } Run terraform destroy at the end
  119. 119. func vaultTest(t *testing.T, options *terratest.Options) { tlsCert := generateSelfSignedTlsCert(t) defer cleanupSelfSignedTlsCert(t, tlsCert) amiId := buildVaultAmi(t) defer cleanupAmi(t, amiId) terratest.Apply(options) defer terratest.Destroy(options) assertCanInitializeAndUnsealVault(t, options) } Check the Vault cluster works!
  120. 120. Using a module:
  121. 121. module "service_foo" { source = "./minimal-module" name = "Foo" image_id = "ami-123asd1" port = 8080 } For simple Modules and learning, deploy the root
  122. 122. module "submodule_foo" { source = "./complete-module/modules/submodule-foo" param_foo = "foo" param_bar = 8080 } module "submodule_bar" { source = "./complete-module/modules/submodule-bar" param_foo = "abcdef" param_bar = 9091 } For more complicated use-cases, use the submodules
  123. 123. module "service_foo" { source = "./minimal-module" name = "Foo" image_id = "ami-123asd1" port = 8080 } Abstraction: simple Module API for complicated infrastructure
  124. 124. module "service_foo" { source = "./minimal-module" name = "Foo" image_id = "ami-123asd1" port = 8080 } module "service_bar" { source = "./minimal-module" name = "Bar" image_id = "ami-abcd1234" port = 9091 } Re-use: create a Module once, deploy it many times
  125. 125. Versioning:
  126. 126. module "service_foo" { source = "./foo" name = "Foo" image_id = "ami-123asd1" port = 8080 } You can set source to point to modules at local file paths
  127. 127. module "service_foo" { source = "hashicorp/vault/aws" name = "Foo" image_id = "ami-123asd1" port = 8080 } Alternatively, you can use Terraform registry URLs
  128. 128. module "service_foo" { source = "git::git@github.com:foo/bar.git" name = "Foo" image_id = "ami-123asd1" port = 8080 } Or arbitrary Git URLs
  129. 129. module "service_foo" { source = "git::git@github.com:foo/bar.git?ref=v1.0.0" name = "Foo" image_id = "ami-123asd1" port = 8080 } You can even link to a specific Git tag (recommended!)
  130. 130. module "service_foo" { source = "git::git@github.com:foo/bar.git?ref=v1.0.0" name = "Foo" image_id = "ami-123asd1" port = 8080 } Modules use semantic versioning
  131. 131. module "service_foo" { source = "git::git@github.com:foo/bar.git?ref=v2.0.0" name = "Foo" image_id = "ami-123asd1" port = 8080 } So upgrading infrastructure is just a version number bump
  132. 132. Promote immutable, versioned infrastructure across environments qa stage prod
  133. 133. 1. What’s a Module 2. How to use a Module 3. How Modules work 4. The future of Modules Outline
  134. 134. “You are not special. Your infrastructure is not a beautiful and unique snowflake. You have the same tech debt as everyone else.” — your sysadmin, probably
  135. 135. stage prod You need this
  136. 136. stage prod And so does everyone else
  137. 137. Stop reinventing the wheel
  138. 138. Start building on top of battle-tested code
  139. 139. Start building on top of commercially-supported code
  140. 140. Start building on top of code
  141. 141. Advantages of code 1. Reuse 2. Compose 3. Configure 4. Customize 5. Debug 6. Test 7. Version 8. Document
  142. 142. At Gruntwork, we’ve been building Modules for years gruntwork.io
  143. 143. Many companies, all running on the same infrastructure code gruntwork.io
  144. 144. stage prod Modules allow us to turn this…
  145. 145. > terraform init <…> > terraform apply … into this
  146. 146. With some help from this
  147. 147. Questions? modules@gruntwork.io

×