The document discusses a secure approach to managing application secrets using AWS KMS, focusing on immutable servers and automated deployment processes. It addresses challenges such as ensuring developers do not access sensitive production data while allowing necessary troubleshooting access, as well as how to handle encryption keys and IAM roles. The proposed solution involves creating a KMS master key, implementing strict IAM policies, and utilizing a 'briefcase' cookbook for secret decryption during deployment.

1. 1Confidential
Application Secret Management with KMS
Andrey Utis
Sr Manager – Software Engineering

2. 2Confidential
What’s the problem?
• Immutable servers
– Can’t manually log in and deposit secrets, they will be lost next release
• Fully automated build/deploy process from github to Prod
– Secrets can also change as code changes, so ideally should be versioned in github
with code
• Developer access to Prod
– Developers support application in prod, so they need some access to
research/troubleshoot issues
– We don’t want developers to access prod data or invoke prod APIs, unless explicitly
approved for a specific reason
• => developers should not know the secret values

3. 3Confidential
Chicken and Egg Problem
• If we source control encrypted secret, how do we get the key to the instance?
• Can’t source control the key – then we need to secure the key too
• Most standard solutions don’t address this (Chef, etc) – though Vault is close to solving this in a similar
way

4. 4Confidential
How do we trust a server?
• In AWS world, we give instances IAM roles that allow it access to other
AWS resources
• If we trust IAM role, can we deliver secrets only to servers with that role?

5. 5Confidential
KMS Policy and Context
• KMS Context allows to add “salt” to the encryption
• Can decrypt only with same “salt”
• KMS Key Policy restricts which IAM role can decrypt with the key/salt
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123412341234:role/MyIAMRole"
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:EncryptionContext:AppName": ”MyContext"
}
}
}

6. 6Confidential
Protocol
• Create a KMS master key (can be
shared by multiple apps)
• Create an IAM role for your server
• Add KMS key policy that allows
decrypt to your IAM role for a specific
context
• Encrypt your secret with the key and
context and store the value in github
• At deployment or runtime, invoke KMS
API to decrypt
• Thanks xkcd

7. 7Confidential
What about prod access for developers?
• Developers, even with “readonly” access to the instance, could call KMS
API to decrypt the secret
• This is a broader issue with IAM roles – credentials are generated by
calling the “magic” metadata IP 169.254.169.254
– Anyone who can call to that IP from the instance can access anything IAM role allows
• Developers should not be able to generate prod IAM credentials at all – so
block the magic IP
iptables -F INPUT
iptables -A OUTPUT -d 169.254.169.254 -m owner --gid-owner whitelist_group -j ACCEPT
iptables -A OUTPUT -d 169.254.169.254 -j DROP
service iptables save

8. 8Confidential
Automating it all
• “briefcase” cookbook to abstract decryption of secrets during Chef run in
an interface similar to databag
• iptables cookbook to block metadata IP for all except a whitelist of user
groups (root and any application specific group that makes AWS API
calls)

9. 9Confidential
Potential Alternatives
• Vault may have a viable solution soon
– https://www.vaultproject.io/docs/auth/aws-ec2.html
• The idea is essentially same – use signed EC2 Identity Document to verify
the caller. This ID document is generated by calling the “magic” metadata
IP
• Downsides:
– Currently only supports authorization by AMI ID, but should support more soon
– Secrets are not source controlled/versioned… this is not ideal for “configuration
secrets” such as DB passwords, etc

10. 10Confidential

11. 11Confidential

12. 12Confidential
Questions?