Given at HashiTalks, 17th Feb 2022 Code for the demo: https://github.com/glynnforrest/talk-demos

1. ‘Static Typing’ in Vault

2. Big corp tales
Episode 1: Sharing unstructured Vault secrets

3. Alice wants to deploy a new service
● Selfie service
● Takes a picture, uploads to S3
● Tested locally
● Ready to deploy to Nomad
● Time to get an S3 bucket and
credentials

4. Bob is the gatekeeper to S3
● Part of the infrastructure team
● Provisions AWS with Terraform
(hopefully)
● Works with purchasing to ensure
budget goals are being met
● “File a ticket, we can probably get it
done this quarter”

5. Alice files a ticket
Dear Bob,
I hope your well.
Please create an S3 bucket for my selfie
service and store the credentials in Vault at:
/secret/prod/selfie/s3-creds
Regards, Alice

6. Bob creates the bucket
Dear Alice,
Bucket created as requested and
credentials added to Vault. I’m going to
close this ticket now.
Robert

7. Ship it
job "selfie" {
type = "service"
vault { policies = ["application"] }
group "server" {
task "server" {
template {
data = <<-EOF
{{with secret "secret/prod/selfie/s3-creds"}}
AWS_KEY_ID={{.Data.access_key_id}}
AWS_KEY_SECRET={{.Data.access_key_secret}}
S3_BUCKET={{.Data.bucket}}
S3_REGION={{.Data.region}}
{{end}}
EOF
destination = "local/env"
env = true
}
}
}
}

8. Ship it
nomad job run selfie.nomad

9. Ship it (again)
job "selfie" {
type = "service"
vault { policies = ["application"] }
group "server" {
task "server" {
template {
data = <<-EOF
{{with secret "secret/production/selfie/s3-creds"}}
AWS_KEY_ID={{.Data.access_key_id}}
AWS_KEY_SECRET={{.Data.access_key_secret}}
S3_BUCKET={{.Data.bucket}}
S3_REGION={{.Data.region}}
{{end}}
EOF
destination = "local/env"
env = true
}
}
}
}

10. Ship it (again)
nomad job run selfie.nomad

11. Independent Software Consultant
Backbeat Software
glynn@backbeat.tech
github.com/glynnforrest
twitter.com/glynn_forrest
Glynn
Forrest

12. Agenda
● The problem with unstructured secrets
● Three ways to fix it
● Challenges
● Next steps
● What you can do

13. If they are managed correctly
Shared secret
stores are great

14. What you want secrets to look like

15. What they actually look like

16. Alice and Bob
Symptom of a larger problem

17. Goals
Secret location is clear
Alice thought it was in /secret/prod
Bob wrote to /secret/production
Secret structure is clear
Alice thought that the keys were access_key_id and access_key_secret
Bob actually wrote access_key_id and secret_access_key
✅
✅

18. An aside
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_ACCESS_KEY_SECRET="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
VS

19. 1. Let’s fix this
with documentation

20. Fix 1: Documentation
● Add Vault secret conventions to company wiki
● Simplest way to establish order
● No code required
● Sometimes ignored or not known about
● Doesn’t prevent errors or typos
● Pro tip: bully your colleagues who haven’t read it and create invalid
secrets

21. Scriptable ❌
Goals
Secret location is clear
Alice thought it was in /secret/prod
Bob wrote to /secret/production
Secret structure is clear
Alice thought that the keys were access_key_id and access_key_secret
Bob actually wrote access_key_id and secret_access_key
✅
✅

22. 2. Let’s fix this
with a script

23. Fix 2: Bob’s Vault S3 enforcer script
data = vault_client.get(sys.argv[1])
for key in ['access_key_id', 'secret_access_key', 'bucket', 'region']:
if key not in data:
print('ERROR: Missing key ' + key)
sys.exit(1)
VAULT_TOKEN=s.abc123 s3-enforcer.py secret/production/selfie/s3-creds

24. Scriptable ✅
Goals
Secret location is clear
Alice thought it was in /secret/prod
Bob wrote to /secret/production
Secret structure is clear
Alice thought that the keys were access_key_id and access_key_secret
Bob actually wrote access_key_id and secret_access_key
✅
✅
Supports all secret types ❌

25. 3. Let’s fix this
with static typing

26. Static typing?
A statically-typed language is a language (such as Java, C, or C++)
where variable types are known at compile time.
- MDN Web Docs Glossary

27. Vault Static typing
A statically-typed Vault is a Vault where the structure and location
of secrets are known before they are requested.
- Us

28. Fix 3: Generic tool to add static typing
● Script to enforce the location and structure of all secrets in a
KV backend
● Command to check that secrets in Vault match the structure
● Command to write secrets to Vault that match the structure

29. Certain engines already have this
$ vault read database/creds/my-database
Key Value
--- -----
lease_id database/creds/my-database/CiJG2YV6W4s7QIhj8hZPUAmG
lease_duration 768h
lease_renewable true
password qvL-78da6zg28feg6dp
username v-root-postgres-mD1tvThoN364PudYh3Ud-1644872138

30. Certain engines already have this
$ vault read --format json database/creds/my-database
{
"request_id": "f0463f7a-9d98-e689-83a3-a80617b5a0bb",
"lease_id": "database/creds/my-database/CiJG2YV6W4s7QIhj8hZPUAmG",
"lease_duration": 2764800,
"renewable": true,
"data": {
"password": "qvL-78da6zg28feg6dp",
"username": "v-root-postgres-mD1tvThoN364PudYh3Ud-1644872138"
},
"warnings": null
}

31. What does a KV response look like?
$ vault kv get secret/simple
====== Data ======
Key Value
--- -----
password hunter2
user admin
$ vault kv get --format json
secret/simple
{
"request_id":
"d9c8ec0b-9312-3c5a-86ca-4223ffccca7e",
"lease_id": "",
"lease_duration": 2764800,
"renewable": false,
"data": {
"password": "hunter2",
"user": "admin"
},
"warnings": null
}

32. What does a KV response look like?
$ vault kv get secret/typed
======= Data =======
Key Value
--- -----
is_special true
number 100
secret s3cr3t_v@lue
$ vault kv get --format json
secret/typed
{
"request_id":
"a8fa5a7d-e67e-da49-26ce-ca4f07d84150",
"lease_id": "",
"lease_duration": 2764800,
"renewable": false,
"data": {
"is_special": true,
"number": 100,
"secret": "s3cr3t_v@lue"
},
"warnings": null
}

33. What does a KV response look like?
$ vault kv get --format json secret/complex
{
"request_id": "2d9487d8-093d-1d86-4433-65bb228b149e",
"lease_id": "",
"lease_duration": 2764800,
"renewable": false,
"data": {
"users": [
{
"access_level": 3,
"name": "Alice"
},
{
"access_level": 5,
"name": "Bob"
}
]
},
"warnings": null
}

34. What does a KV response look like?
$ vault kv get secret/complex
==== Data ====
Key Value
--- -----
users [map[access_level:3 name:Alice] map[access_level:5 name:Bob]]

35. What does a KV response look like?

36. What does a KV response look like?

37. We need a structure for arbitrary JSON
"data": {
"access_key_id": "AKIAIOSFODNN7EXAMPLE",
"secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"bucket": "big-corp-123-selfie",
"region": "us-east-1"
}
"data": {
"key": "uRnkVRrYOTY0O3B5u0Yh3NUBbdLi7EApXaCZW391pChgeoFNq5EjzSAN5anG",
"rounds": 400
}
string
string, must be valid region
integer between 300 and 600
60 character alphanumeric string

38. We need a structure for arbitrary JSON
● Structure of data, also known as a schema
● A schema for JSON
“JSON Schema is a vocabulary that allows you to annotate
and validate JSON documents.”
- json-schema.org

39. We need a structure for arbitrary JSON
{
"type": "object",
"properties": {
"access_key_id": {
"type": "string"
},
"secret_access_key": {
"type": "string"
},
"bucket": {
"type": "string"
},
"region": {
"type": "string",
"pattern": "^(us|eu)-[a-z]+-[0-9]$"
}
},
"additionalProperties": false,
"required": ["access_key_id", "secret_access_key", "bucket", "region"]
}
string
string, must be valid region

40. We need a structure for arbitrary JSON
{
"type": "object",
"properties": {
"key": {
"type": "string",
"pattern": "^[a-zA-Z0-9]{60}$"
},
"rounds": {
"type": "integer",
"minimum": 300,
"maximum": 600
},
"audit": {
"type": "boolean"
}
},
"additionalProperties": false,
"required": [ "key", "rounds"]
}
integer between 300 and 600
60 character alphanumeric string

41. Fix 3: Generic tool to add static typing
● A script to enforce the location and schema of all secrets in a
KV backend
● Command to check that secrets in Vault match the schema
● Command to write secrets to Vault that match the schema
● Load these schemas from a configuration file
● Describe a particular schema without connecting to Vault
(great for Alice)

42. vault-helper.py
● General purpose tool for common Vault tasks
● Check a secret
vault-helper.py validate --rules-file rules.json
secret/production/selfie/s3-creds
● Check a list of secrets
vault-helper.py validate-path --rules-file rules.json
secret/production
● Write a secret
vault-helper.py write --rules-file rules.json
secret/production/selfie/s3-creds
● Describe a schema
vault-helper.py describe-path --rules-file rules.json
secret/production/selfie/s3-creds
vault-helper.py describe --rules-file rules.json s3-creds

43. Secret ‘rules’
{
"name": "s3-creds",
"description": "Credentials for an AWS S3 bucket",
"pattern": "secret/(production|staging)/.+/s3-creds$",
"schema": {
"type": "object",
"properties": {
"access_key_id": {"type": "string"},
"secret_access_key": {"type": "string"},
"bucket": {"type": "string"},
"region": {
"type": "string",
"pattern": "^(us|eu)-[a-z]+-[0-9]$"
}
},
"additionalProperties": false,
"required": ["access_key_id", "secret_access_key", "bucket", "region"]
}
}

44. Shorthand
{
"name": "encryption-key",
"description": "Super secret encryption key",
"pattern": "secret/.+/encryption-key$",
"keys": {
"key": "string:^[a-zA-Z0-9]{60}$",
"rounds": "integer:300-600",
"audit": "optional_boolean"
}
}
{
"name": "encryption-key",
"description": "Super secret encryption key",
"pattern": "secret/.+/encryption-key$",
"schema": {
"type": "object",
"properties": {
"key": {
"type": "string",
"pattern": "^[a-zA-Z0-9]{60}$"
},
"rounds": {
"type": "integer",
"minimum": 300,
"maximum": 600
},
"audit": {"type": "boolean"}
},
"additionalProperties": false,
"required": ["key", "rounds"]
}
}

45. Demo

46. Check continuously in CI
● Because team members can’t be trusted
● vault-helper.py validate secret/production/selfie/s3-creds
● vault-helper.py validate-path secret
● vault-helper.py validate-path secret --ignore-missing-rule
● Non-zero exit code on failure

47. Scriptable ✅
Goals
Secret location is clear
Alice thought it was in /secret/prod
Bob wrote to /secret/production
Secret structure is clear
Alice thought that the keys were access_key_id and access_key_secret
Bob actually wrote access_key_id and secret_access_key
✅
✅
Supports all secret types ✅

48. Challenges

49. Technical
● Needs a Vault token that can read many secrets
● Migrate existing secrets that don’t fit a schema
○ Use --ignore-missing-rule
○ Make deprecated key names optional properties in the
schema
● Find usages in Nomad jobs, Terraform, other configuration
management tools

50. People
● Needs cooperation from different teams
● Uncovers KV usage in parts of the organization
● Might be considered too much of a risk
○ Another process reading secrets from Vault
○ Prints the keys of secrets in error messages
'key' does not match '^[a-zA-Z0-9]{60}$'.
'region' does not match '^(us|eu)-[a-z]+-[0-9]$'.

51. Next steps

52. Next steps
● Cleanup vault-helper.py and release a standalone tool
● Deploy as a single binary
● Support KV version 2
● Validate Vault secret usage in a Nomad job
● Consul support?
● Looking for name suggestions

53. What you can do

54. What you can do
● Check out vault-helper.py and the demo at
github.com/glynnforrest/talk-demos
● Try writing your own schemas
● Run it against your own Vault
● Feedback
○ Where does it fail?
○ What could be improved?
● Help me decide on a name for a Vault / Consul / Nomad
‘static typing’ tool

55. Recap
● The problem with unstructured secrets - no structure, hard to scale with
multiple people
● Three ways to fix it - documentation, single-purpose script, generic tool
● Challenges - technical and people
● Next steps - turn vault-helper into a HashiStack ‘static typing’ tool
● What you can do - try it out, help me come up with a name

56. Big corp tales
Episode 2: Sharing structured Vault secrets

57. Alice files a ticket
Dear Bob,
I hope your well.
Please create a production S3 bucket for my
selfie service and store the credentials in
Vault.
Regards, Alice

58. Bob creates the bucket
Dear Alice,
Bucket created as requested and credentials
added to Vault.
You can see the validated secret path here:
https://ci.bigcorp.com/builds/vault-prod/33
I’m going to close this ticket now.
Robert

59. Ship it
job "selfie" {
type = "service"
vault { policies = ["application"] }
group "server" {
task "server" {
template {
data = <<-EOF
{{with secret "secret/production/selfie/s3-creds"}}
AWS_KEY_ID={{.Data.access_key_id}}
AWS_KEY_SECRET={{.Data.secret_access_key}}
S3_BUCKET={{.Data.bucket}}
S3_REGION={{.Data.region}}
{{end}}
EOF
destination = "local/env"
env = true
}
}
}
}

60. Ship it (again)
nomad job run selfie.nomad

61. Success!
Our first user

62. Thank You
Photo credits
Google slides stock imagery
@nci - unsplash.com
@geriforsaith - unsplash.com
Cat chemistry meme
That monkey that took a selfie
Glynn Forrest
Backbeat Software
glynn@backbeat.tech
github.com/glynnforrest/talk-demos
twitter.com/glynn_forrest