https://www.youtube.com/watch?v=IeweKUdHJc4 My presentation from Hashiconf 2017, discussing our use of Terraform, and our techniques to help make it safe and accessible.

1. Terraform abstractions
for safety and power
Calvin French-Owen, Co-founder Segment
Hashiconf, Sept 2017
@calvinfo

2. Terraform at Segment
- Analytics API for 1000s of
online businesses
- 349 services
- 14k containers peak
- 90B msg/month
- 100k rps
- All AWS
- ECS
(# containers running)

3. - 2.5 years of Terraform
(since v0.4!)
- ~30 developers interacting
with Terraform weekly
- 30-50 ‘applies’ per day
- Tens of thousands of AWS
resources
Terraform at Segment

4. This Talk
- Why is safety such a big deal?
- Some Terraform ‘nouns’
- Safety with your state
- Safety with your modules
- Safety elsewhere

5. Why is ‘safety’ such a
big deal?

7. Developers avoid selecting tools if the … effect
of the tools is unknown, and the tools have
some risks.
To promote development support tools, we
have to suppress the risk of the tools.
- Analyzing the Decision Criteria of Software Based on Prospect Theory

8. Safety → Adoption

10. Terraform can be
scary…
But doesn’t have to be

11. Terraform ‘nouns’

12. Infrastructure == Code

14. HCL: most examples configure terraform via
HCL (hashicorp configuration language), a
relative of JSON

15. variable: a dynamically configured input

17. resource: configuration for a given cloud entity
(instance, load balancer, image, etc)

18. resources take inputs as configuration, and
can produce outputs once they are created in
your infrastructure. they may interpolate
variables

20. resource “<type>” “<id>”

21. resource “<type>” “<id>”
inputs

23. module: a reusable collection of resources that
can be passed its own inputs and produce
outputs

25. plan: the diff between what exists in your
infrastructure, and the changes you would like
to apply

26. How does it know?

27. How does it know?
.tfstate

30. Terraform Workflow
$ terraform plan
$ terraform apply

31. Terraform Workflow
1. load the desired configuration
2. load the stored .tfstate file
3. calculate the diff between the current and desired states
4. use CRUD APIs to update the current state to match the
desired state
5. update the state file

32. Terraform applies diffs in your
configuration to manage your
infrastructure

33. Part I: State

34. Terraform wants to manage
everything

35. Terraform wants to manage
everything
How do I keep it from
destroying existing
infrastructure?

36. AWS accounts
dev stage prod old prodVPC peering

37. AWS accounts
dev stage prod old prod
terraform managed
VPC peering
left to rot and die
sunset

38. The advantage of states
per environment?

39. The advantage of states
per environment?
safety

41. Basic State
Management

42. The one golden rule:
Always use remote state
or backends

47. Terraform Enterprise S3

48. - Price? S3 or Consul
- Custom configuration? S3 or Consul
- Out-of-the-box dashboard + changelog? TFE
- Remote applies? TFE
- CI Integration? TFE
- Versioning? Either (with tweaks)
- Locking? Either!
What remote state provider should I use?

49. - Price? S3 or Consul
- Custom configuration? S3 or Consul
- Out-of-the-box dashboard + changelog? TFE
- Remote applies? TFE
- CI Integration? TFE
- Versioning? Either (with tweaks)
- Locking? Either!
(at Segment, we’ve used S3 but moved to TFE)
What remote state provider should I use?

50. More advanced state
management

51. States per service

52. →readonly→

53. States per team

55. Many ways to manage
read-only state

56. 1. terraform_remote_state

58. read only
reference

59. 2. data sources

62. 3. shared outputs

64. module outputs

65. State Safety
- Separate AWS (or GCP) accounts
- A state per environment
- Consider states per service or per team
- We use per-team states
- Use a remote state manager like TFE or S3
- Limit your blast radius
- Use some sort of ‘read-only’ state
- We use a combination of data sources and shared outputs

66. Part II: Modules

68. Modules at Segment

69. terracode
terracode-integrations
terracode-warehouses
terracode-ops
terracode-site
Shared modules + variables
per-team
repositories
source = “github.com/segmentio/terracode/modules/…”

70. We use and re-use modules liberally

71. Our worker module

72. Worker: Readme.md

73. Readme.md
generated via
github.com/segmentio/terraform-docs

74. Worker: input.tf

75. Worker: output.tf

76. Worker: output.tf - render JSON

77. Worker – understandable defaults

78. Worker – var.enable

79. Worker – heredoc task definition

80. Worker – autoscaling rules

81. Worker – IAM role

82. Worker – in practice

83. Worker – in practice
source

84. Worker – in practice
image

85. Worker – in practice
cluster to run on

86. Worker – in practice
‘T-shirt size’ resources

87. Worker – in practice
auto-scaling
tweaks

88. Worker – in practice
specific configuration

89. A last step: adding IAM
permissions

90. Worker - adding IAM permissions

91. S3 bucket permissions under the hood

92. S3 bucket permissions under the hood

93. Worker - hiding the complexity

94. Default IAM permissions

96. - Modules for logical ‘units’ of resources
- Simple defaults to hide complexity
- Variable all the things
- If you write it more than twice, make it a module
- Modules can reference across repos, share them
- github.com/segmentio/terraform-docs
- github.com/segmentio/stack
Safety with modules

97. Safety elsewhere

98. Safety → Adoption

99. What adoption can buy
you

100. What if you have a
common substrate for
defining infrastructure
and cloud services?

101. Alerting

103. three-line change; zero-cost alerting

104. Cost management

105. Cost management

106. We need to tie parts of
our infrastructure to
business lines...

107. Cost management

108. Key/Secrets Management

109. One last thought...

110. Terraform as the common substrate

112. When Terraform is
your substrate, any
cloud is your cloud

113. When Terraform is
your substrate, any
cloud product is your
cloud product

114. Fin
@calvinfo