Back in the day in a IT company long ago, where the BOFHs roamed and the ITIL was strong. We used to keep long lists of CIs that used to enviably and hopelessly out of date. Because we either didnt care, know or bother keeping up to date. That was totally fine in a relatively static environment the IT company of long ago. We would have our yearly inventory day and forget about it again. Of course we all use some form of infrastructure as code right now. Some of us might go as far that "if it isn't in code it doesnt exist", but can we truly say that whatever is in the OpenTofu state really is the only thing running? What about that recurring 1$ in that dormant AWS account, where is that coming from? How about the playground projects the CEO likes to play around with in his sparetime? or that one time the opentofu destroy didnt exit cleanly and some resources weren't cleaned during that timeout, did we really manually cleanup all resources?

1. Cost reconciliation
in a post CMDB world
Bram Vogelaar
@attachmentgenie

2. Confidential and Proprietary
~ ❯ whoami
• Used to be a Molecular Biologist
• Then became a Dev, now an Ops
• Currently Cloud Engineer @ Seaplane.io

3. Confidential and Proprietary
Who has said this before?
“If it isn’t in code (*),
it doesn’t exist!”
(*) puppet, chef, ansible, opentofu…..etc

4. Confidential and Proprietary
And also said…
“Just ignore that $17.63 on
on the AWS bill”

5. Confidential and Proprietary
And followed by……
“We don’t know
what that is for
….
but it has been
there since forever”

6. Confidential and Proprietary
And than said………
“ ¯_(ツ)_/¯ ”

7. Confidential and Proprietary
And while the CISO said…

8. Confidential and Proprietary
Because nobody told to the infra team(*)
“Trust but verify”
(*) {ops, sysadmin, infra, devops, sre, platform}

9. Confidential and Proprietary
To understand why, we need to understand how

10. Confidential and Proprietary
Let’s pretend it is data science

11. Confidential and Proprietary
Count the # resources in your state file/code
terraform show -json | jq -r '.values[].resources[]
grep -nr "resource "google_compute_instance""

12. Confidential and Proprietary
Count the # in your single source of truth

13. Confidential and Proprietary
Count the # in your 011y tools
sum (up{env="production",job="node_exporter"})

14. Confidential and Proprietary
Oh…Crap….

15. Confidential and Proprietary
Check your DNS/IPAM entries

16. Confidential and Proprietary
Count your CICD deploy targets

17. Confidential and Proprietary
Observe vendor billing

18. Confidential and Proprietary
The lack of signal is a singal

19. Confidential and Proprietary
You ll need to become good friends with UIs

20. Confidential and Proprietary
Compare your single source truth vs vendor
billing
SKUs applied === SKUs paid for
cloud regions defined in .tf files === regions with
resources

21. Confidential and Proprietary
Observe volume metrics

22. Confidential and Proprietary
Should I go crazy automating this?

23. Confidential and Proprietary
Vendors as a SQL Schema
SELECT
*
FROM
aws_elbv2_load_balancers
WHERE
scheme = 'internet-facing';
https://www.cloudquery.io/

24. Confidential and Proprietary
But what about…
Ways to
fix it
keep up with life

25. Confidential and Proprietary
“Make doing the right thing,
the simplest thing to do”

26. Confidential and Proprietary
Lock Vendor Regions Down
data "aws_iam_policy_document" "eu-central-1-only" {
statement {
actions = [
"ec2:RunInstances",
]
….
condition {
test = "StringEquals"
variable = "aws:RequestedRegion"
values = [
"eu-central-1",
]
}
}
}

27. Confidential and Proprietary
Lock Vendor Regions Down
resource "aws_organizations_policy" "eu-central-1-only" {
name = "eu-central-1-only"
content = data.aws_iam_policy_document.eu-central-1-only.json
}

28. Confidential and Proprietary
Tag everything

29. Confidential and Proprietary
Tag everything
provider "aws" {
profile = "default"
region = "eu-central-1"
default_tags {
tags = {
Environment = "Test"
Service = "Example"
terraform = true
}
}
}

30. Confidential and Proprietary
Forcing labels / Decorating metrics
global:
scrape_interval: 10s
external_labels:
datacenter: "%{::trusted.extensions.pp_datacenter}"
region: "%{::trusted.extensions.pp_region}" <- use ${IATA_CODES}
env: "%{::trusted.extensions.pp_environment}"

31. Confidential and Proprietary
But my system is Cloud native
Alert on $THING
that costs money

32. Confidential and Proprietary
psst………
“You really should use a
CMDB”

33. Confidential and Proprietary
e.g Netbox
https://netbox.dev/

34. Confidential and Proprietary
IPAM made easy
resource "netbox_prefix" "supplier_site" {
status = "active"
prefix = "${local.base_ip}/${local.base_bits}"
tags = [
"${local.supplier}-${var.supplier_site}"
]
}
data "netbox_prefix" "supplier_site" {
tag = "${local.supplier}-${var.supplier_site}"
}
resource "netbox_available_ip_address" "nebula_ip" {
prefix_id = data.netbox_prefix.supplier_site.id
dns_name = "${var.hostname}.${var.fqdn_suffix}"
}

35. Confidential and Proprietary
But that didn’t sounds very easy
resource "supplier_x_server" "vm" {
image_id = “Ubuntu 22.04”
hostname = "${var.hostname}.${var.fqdn_suffix}"
plan = var.supplier_type
location = var.supplier_site
user_data = module.cloud-init-vm.user_data
}
module "netbox_vm" {
source = "your.registry/record_node/supplier_x"
version = "1.0.0"
fqdn_suffix = var.fqdn_suffix
tags = [local.netbox_tag]
node_module = module.cloud-init-vm.user_data
supplier_x_instance = supplier_x_server.vm
}

36. Questions Before Takeoff?
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie