We are building an application platform that is made up of ~20 microservices. Different microservices have the need to use different types of secrets (passwords, API keys, license keys, RSA keys, etc.) and these secrets are given to us by different internal/external teams. The presentation covered the need to manage secrets as code and the challenges that comes with it. One approach of solving this using Git, AWS KMS, Mozilla SOPS, Kubernetes Secrets and Helm Charts was presented.

1. Secrets Management and Delivery to
Kubernetes Pods
Satish Devarapalli

2. $whoami
Satish Devarapalli
Cloud Platform Architect
@devasat
https://www.linkedin.com/in/satish-devarapalli-48a82a5/

3. Agenda
● Project Intro
● Secrets Delivery Pipeline View
● Mozilla SOPS (Secret OPerationS)
● Questions

4. Project Intro
Product
auth 𝞵 svc𝞵 svc ...
Application Secrets
● Database Credentials
● AWS Credentials
● API Keys
● License Keys
● Encryption Keys
CI/CD Secrets
● AWS Credentials
● Kube Config Files
● Encryption Keys

5. ● Manage secrets as code → check-in, build and deploy
○ Secrets should be encrypted before check-in
● Encryption keys management
○ Different encryption keys for environments
○ Centralized access → do not store keys locally
○ Audit usage
○ Limit key access to individual members based on environment
○ Grant and revoke access easily
● Jenkins → minimal configuration on Jenkins nodes
○ Do not store encryption keys on Jenkins master or nodes
○ Secrets should not be left in plain text format on build success or failure
○ Store encrypted kube config files and AWS credentials in Git
● Limit secrets access to microservices that use it
Secrets Management Requirements

6. Agenda
● Project Intro
● Secrets Delivery Pipeline View
● Mozilla SOPS (Secret OPerationS)
● Questions

7. Secrets Delivery Pipeline
AWS KMSdec enc
Mozilla SOPS
+
Dev
QA
Prod
AWS KMS decenc
Mozilla SOPS
+
Jenkins job runs inside a container
Helm
install
Helm
install
Helm install
Helm chart versions:
env-config-1.0.0-101-af837dh
env-config-1.0.0-102-x7jwy62

8. Agenda
● Project Intro
● Secrets Delivery Pipeline View
● Mozilla SOPS (Secret OPerationS)
● Questions

9. AWS KMS - Envelope Encryption
Picture source: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/how-it-works.html

10. Mozilla Secret OPerationS (SOPS)
~ > export SOPS_KMS_ARN="arn:aws:kms:us-east-1:212121:key/3434-c5f1-4040
~ > sops -e secret-values.dec.yaml > secret-values.enc.yaml
Picture source: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/how-it-works.html
SOPS Generates
256-bit data key
secret-values
.dec.yaml
SOPS_KMS_ARN
secret-values
.enc.yaml
AWS Boundary

11. Mozilla Secret OPerationS (SOPS)
secret-values.dec.yaml secret-values.enc.yaml

12. Mozilla Secret OPerationS (SOPS)
secret-values.enc.yaml = data + encryption information
Data
Encryption Key ID
Base64 encoded
value of Encrypted
Data key
Message
Authentication Code
Prevents File
Tampering
JSON and YAML keys
are not encrypted

13. Mozilla SOPS - Key Providers
Microsoft Azure Key Vault
SOPS_AZURE_KEYVAULT_URL
Google Cloud KMS
SOPS_GCP_KMS_IDS
AWS KMS
SOPS_KMS_ARN
PGP
SOPS_PGP_FP

14. ● Supported types: JSON, YAML and Binary
● Special support for JSON and YAML files
○ Encrypts only the values
○ File extension is used to determine the type
■ secret-values.yaml.enc → binary file
■ Secret-values.enc.yaml → yaml file
○ Some YAML types (anchors, streams, top-level arrays) are not supported
Mozilla SOPS - File Types
Input file is treated as
blob
Key fingerprint
Encrypted data key

15. Mozilla SOPS - Multiple Master Keys
Shared Services
Global Master
Key
DEV QA PROD
DEV QA PROD
Business Unit 1
Business Unit 2Multiple Master Key Uses:
- Encryption key in master AWS account
- Encryption keys in different regions
- Encryption keys in different key providers (hybrid cloud)
- PGP key (stored offline)

16. Mozilla SOPS - .sops.yaml
Filename path passed to sops is
matched against this regex
dev/secret-values.enc.yaml ✔
dev/secret-values.dec.yaml ✔
dev/aws-credentials.dec ✔
dev/aws-credentials ❌

17. Mozilla SOPS - .sops.yaml
At root directory
~> sops -i -e dev/secret-values.enc.yaml ✔
~> cd dev && sops -i -e secret-values.enc.yaml ❌

18. Option 1: in-place edit
● Data key doesn’t change
● Only the key value that was
modified appears in the git diff
Mozilla SOPS - Edit Files
~ > export EDITOR=vi
~ > sops -i secret-values.enc.yaml

19. Option 2: decrypt and encrypt
● Data key changes
● All key values appears to have
changed in git diff
Mozilla SOPS - Edit Files
~ > #Decrypt
~ > sops -d secret-values.enc.yaml >
secret-values.dec.yaml
~ > #Encrypt
~ > sops -e secret-values.dec.yaml >
secret-values.enc.yaml

20. Mozilla SOPS - Diffs in Cleartext in Git
~ > cat ~/.gitattributes
*.enc diff=sopsdiffer
*.enc.yaml diff=sopsdiffer
*.enc.json diff=sopsdiffer
.gitattributes
~ > grep -A1 sops ~/.gitconfig
[diff "sopsdiffer"]
textconv = "sops -d"
.gitconfig
In my testing, this worked if
.sops.yaml is at the root directory
of the git project and regex expression
is based on the filename but not
directory name
dev/secret-values.enc.yaml →
dev-secret-values.enc.yaml

21. Mozilla SOPS - Demo
● Install SOPS
● Create and export GPG keys
● Set up .sops.yaml, folder per environment and encrypt input file
● In-place edit

22. ● Manage secrets as code → check-in, build and deploy
○ Secrets should be encrypted before check-in ✔
● Encryption keys management
○ Different encryption keys for environments ✔
○ Centralized access → do not store key locally ✔
○ Audit usage ✔
○ Limit key access to individual members based on environment ✔
○ Grant and revoke access easily ✔
● Jenkins → minimal configuration on Jenkins nodes
○ Do not store encryption keys on Jenkins master or nodes
○ Secrets should not be left in plain text format on build success or failure
○ Store encrypted kube config files and AWS credentials in Git
● Limit secrets access to microservices that use it
Secrets Management Requirements

23. Jenkins
/jenkins/job1/ws
Memory
Jenkins job in a container
/build
/ws
tmpfs
bind
● Jenkins → minimal configuration on Jenkins nodes
○ Do not store encryption keys on Jenkins master or nodes → AWS KMS
○ Secrets should not be left in plain text format on build success or failure → Docker tmpfs
○ Store encrypted kube config files and AWS credentials in Git → AWS KMS
Decrypted files are
stored only in /build
directory
Jenkins Access:
- ECR
- KMS
- S3

24. ● Manage secrets as code → check-in, build and deploy
○ Secrets should be encrypted before check-in ✔
● Encryption keys management
○ Different encryption keys for environments ✔
○ Centralized access → do not store key locally ✔
○ Audit usage ✔
○ Limit key access to individual members based on environment ✔
○ Grant and revoke access easily ✔
● Jenkins → minimal configuration on Jenkins nodes
○ Do not store encryption keys on Jenkins master or nodes ✔
○ Secrets should not be left in plain text format on build success or failure ✔
○ Store encrypted kube config files and AWS credentials in Git ✔
● Limit secrets access to microservices that use it
Secrets Management Requirements

25. Limit Secrets Access to Microservices
Separate Secret
object for each
credential
+
{{ if .Values.secret1Enabled }}
- name: {{ .Values.secret.secret1 }}
secret:
secretName: {{ .Values.secret.secret1 }}
{{ end }}
+secret1Enabled: true
secret2Enabled: false
Helm’s
values.yaml
Helm’s
Deployment.yaml

26. ● Manage secrets as code → check-in, build and deploy
○ Secrets should be encrypted before check-in ✔
● Encryption keys management
○ Different encryption keys for environments ✔
○ Centralized access → do not store key locally ✔
○ Audit usage ✔
○ Limit key access to individual members based on environment ✔
○ Grant and revoke access easily ✔
● Jenkins → minimal configuration on Jenkins nodes
○ Do not store encryption keys on Jenkins master or nodes ✔
○ Secrets should not be left in plain text format on build success or failure ✔
○ Store encrypted kube config files and AWS credentials in Git ✔
● Limit secrets access to microservices that use it ✔
Secrets Management Requirements

27. Questions
?