The document discusses the "Data, Transformations, Resources" pattern in Terraform for organizing code. It introduces new data types and features in Terraform versions 12 and 13 for working with structured data. These include lists, maps, objects, and for expressions for mapping data into resources. The pattern advocates separating data collection and normalization, data transformations using the new functions, and defining each resource once to make the code understandable, easy to change, and efficient.

1. Lead DevOps Engineer - Salesforce
Robert Blumen
The 'Data,
Transformations,
Resources' Pattern in
Terraform

2. Outline
● Terraform anti-patterns
● New features in versions 12 and 13
● The "Data + Transforms + Resources" Pattern

3. What Terraform 11 Can Not Do
● Looping
● Conditional
● Nested loops
● Data structures

4. Data Types in Terraform
12 & 13

5. structured data types
● list
● map
● set
● tuple
● object

6. list(string)
list(tuple[string,number,bool])
map(string)
map(object({ id=string, cidr_block=string }))
composition of data types

7. variable group_members {
type = list(string)
description = "list of emails of the group members"
}
example

8. variable people {
type = list(object({ name=string, age=number }))
default = [
{
name = "John"
age = 32
}
]
}
type checking (continued)

9. ~/Devel/devops-tools (rblumen) $ /opt/hashi/terraform-0.12.20
apply -var 'people=[ { name="Job", age=71 } ]'
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
~/Devel/devops-tools (rblumen) $ /opt/hashi/terraform-0.12.20
apply -var 'people=[ { name="Job", age=true } ]'
Error: Invalid value for input variable
The argument -var="people=..." does not contain a valid value for variable
"people": element 0: attribute "name": string required.

10. groups = [
{
"key" : "value"
},
{
"key-2": "value-2"
}
]
all_groups = merge(local.groups...)
###
{
"key": "value",
"key-2": "value"
}
python-style splatting

11. Collections and
Iterations

12. terraform 11 - indexed lists
resource providerX web_thing {
count = var.num_servers
name = element(var.server_names,count.index)
}
#####
web_thing[0]
web_thing[1]
web_thing[2]
web_thing[3]

13. locals {
server_names = [ "db", "front-end", "back-end" ]
}
terraform 12+ - maps
web_srv["db"]
web_srv["front-end"]
web_srv["back-end"]
resource providerY web_srv {
for_each = toset(var.server_names)
name = each.value
}

14. old - (count based):
aws_iam_policy.aws_admin_access[3]
new - (for_each):
aws_iam_policy.aws_admin_access["AmazonS3FullAccess"]
indexing

15. locals {
administrator_policies = [
"AmazonEC2FullAccess",
"AmazonS3FullAccess",
"AWSKeyManagementServicePowerUser",
"AWSLambdaFullAccess",
]
}
data aws_iam_policy aws_admin_access {
for_each = toset(local.administrator_policies)
arn = "arn:aws:iam::aws:policy/${each.value}"
}
for_each - iterating over a list

16. resource heroku_app_feature app_feature {
for_each = {
for f, e in {
spaces-tls-legacy : false,
spaces-tls-modern : true
spaces-strict-tls : false
} :
f => e
}
app = heroku_app.app.name
name = each.key
enabled = each.value
}
for_each - iterating over a map

17. data aws_iam_policy_document access {
for_each = toset(local.groups)
statement {
effect = "Allow"
actions = ["s3:PutObject", ]
resources = data.aws_s3_bucket.devel.arn
}
}
chaining iteration
resource aws_iam_policy access {
for_each = data.aws_iam_policy_document.access
name = "${each.key}Access"
policy = each.value.json
}
resource aws_iam_group_policy_attachment group_access {
for_each = aws_iam_policy.access
group = each.key
policy_arn = each.value.arn
}

18. ● 12: for_each does not work with modules
● 13: for_each supports modules
terraform 12 versus 13

19. set operations

20. functions on sets
● setintersection
● setproduct
● setsubtract
● setunion

21. locals {
all_groups = toset([ "dev", "core", "admin", "ops", "chat" ])
some_groups = toset(["dev", "voice"])
other_groups = setsubtract(all_groups, some_groups) # in 12.21
}
resource aws_iam_group groups {
for_each = local.all_groups
name = each.value
}
resource aws_thing thing_one {
for_each = local.some_groups
group = aws_iam_group.groups[each.value]
}
resource aws_thing thing_two {
for_each = local.other_groups
group = aws_iam_group.groups[each.value]
}
implementing conditionals with sets

22. locals {
test_envs = [ "test1", "test2" ]
}
resource saas_thing_one test_only {
foreach = setintersection(toset([var.environment]), local.test_envs)
…
}
conditionals (2)

23. for expressions

24. for expressions
users = [ "achman", "aziss", "bwong", "cshah", ]
# list
emails = [
for u in local.users : "${replace(u, "-", ".")}@pagerduty.com"
]
# map
emails_by_user = {
for u in local.users :
u => "${replace(u, "-", ".")}@pagerduty.com"
}

25. [for s in var.list :
upper(s)
if s != ""]
for expressions - conditionals

26. var user_email_assn {
type = map(string)
default = { ... }
}
local {
labels = [ for user, email in var.user_email_assn: "${user}:${email} ]
}
for expressions - map

27. {
for s in [ "abc", "def", "aef", "dps" ] :
substr(s, 0, 1) => s...
}
{
"a" = [
"abc",
"aef",
]
"d" = [
"def",
"dps",
]
}
for expressions - group by

28. putting it together

29. resource backend_foo left_hand_thing {
for_each = ...
}
resource backend_bar right_hand_thing {
for_each = ..
}
resource backend_foo_bar left_right_assn {
for_each = ??
left_hand_thing = each.value.left
right_hand_thing = each.value.right
}
many-to-many

30. # groups have multiple members
# user may belong to more than one group
resource pagerduty_user { … }
resource pagerduty_team { … }
resource pagerduty_team_membership {
for_each = ??
user_id = ??
team_id = ??
}
example: PagerDuty

31. for g in groups {
create group
for u in groups.users {
create user
create create user_group_membership
}
}
conventional imperative language

32. locals {
on_call_teams = {
tier_1 = [ "abe", "jaxel", "beldon" ]
tier_2 = [ "abe", "janpax", "fanlo" ]
escalation = [ "adam", "shefty", "fanlo" ]
}
}
terraform solution

33. resource pagerduty_team teams {
for_each = keys(local.on_call_teams)
name = each.value
}
resource pagerduty_user users {
for_each = distinct(flatten(values(local.on_call_teams)))
name = each.value
email = "${each.value}@companyx.com"
}

34. locals {
team_user_pairs = [
for team, users in local.on_call_teams: [
for for user in users: {
user: user
team: team
}
]
]
team_users = {
for pair in flatten(local.team_user_pairs):
"${pair.team}-${pair.user}" => {
team_id: pagerduty_team[pair.team].id,
user_id: pagerduty_user[pair.user].id
}
}
}
resource pagerduty_team_membership membership {
for_each = local.team_users
team_id = each.value.team_id
user_id = each.value.user_id
}

35. locals {
on_call_teams = {
tier_1 = [ "abe", "jaxel", "beldon" ]
tier_2 = [ "abe", "janpax", "fanlo" ]
escalation = [ "adam", "shefty", "fanlo" ]
}
}
###
resource saas_team_membership membership {
for_each = ??
user = each.key
groups = each.value
}
one:many inverting

36. user_group_pairs = flatten([
for group, members in local.group_memberships : [
for user in members : {
user = user
group = group
}
]
])
groups_by_user = {
for pair in local.user_group_pairs :
pair.user => pair.group...
}
}

37. resource saas_team_membership membership {
for_each = local.groups_by_user
user = each.key
groups = each.value
}

38. The
"Data
+ Transformations
+ Resources"
Pattern

39. objectives
● understandable
● easy to change
● economical (DRY)

40. ● data
○ structure according to your domain
○ use data structures
○ normalize/minimize duplication
○ optimize for frequent changes
● transform
○ use for, if and set functions
○ inputs: your data
○ outputs: what your provider's resources need
● resources
○ create each resource once
○ chain resources
pattern for code organization

41. Thank You