Oded Harevern, CEO & co-founder of Akeyless discusses how Kubernetes secrets management is done today and how to do secrets management better. Learn more about Akeyless Vault Platform for secrets management: https://www.akeyless.io/product-secrets-management/ Watch the video here: https://www.youtube.com/watch?v=hvUuYWXGSJM

1. The Good, The Bad and The Ugly
Kubernetes Secrets
Steve Giguere

2. Akeyless.io
● Background
● How Does ‘K8s Secrets’ Work?
● Security Concerns & Mitigation
● What ‘K8s Secrets’ May Never Solve
● Get to know: Just-in-time Secrets
● Q&A
Agenda
1966, Sergio Leone

3. Poll #1

4. Akeyless.io
Pod X
What happens when we’re not using K8s Secrets
Container
X
Container
Y
Container
Z
● Secret types: API-Keys,
Passwords, Tokens etc.
● Secrets may be hard-coded
inside the application-code
● Secrets are likely to show up on
the application’s config file
○ Who has access to the
code-repository / container images?
○ What if a password requires rotation
or token to be renewed?
○ Developers are exposed to
Production access credentials?

5. Akeyless.io
• Always.
• K8s Secrets provides
• (1) Decoupling your secrets from your application
• (2) Reuse secrets across multiple apps
That said, let’s talk about securing it….
When should we use K8s Secrets?

6. Akeyless.io
Node X
Pod X
How does K8s Secrets work?
Container
X
Container
Y
Container
Z
Pod Y
Container
X
Container
Y
Container
Z
K8s Control / EKS / AKS ...
[Etcd k/v DB]
K8s
Kubelet
● Secrets are stored in Etcd
(k/v store) within the K8s
management
● On pod creation, secrets are
created as virtual files /
environment variables
per pod
● Eventually, the secrets are
placed outside of the container /
application

7. Akeyless.io
Etcd stores Secrets in a non-encrypted form
● base64 is an encoding method, not encryption
Security Concerns & Mitigation

8. Akeyless.io
Etcd stores Secrets in a non-encrypted form
● base64 is an encoding method, not encryption
● AI #1 - Enable Encryption-at-rest*
● AI #2 - Limit access to etcd for admin users only
● AI #3 - Enable TLS/SSL between Etcd and your pods
● AI #4 - Wipe disks
● AI #5 - YAML / JSON files contain base64 secrets. Don’t share or check-it into a repo.
* How to’s: encrypt secret data etcd, configure-etcd
* Don’t place the Encryption key in ‘EncryptionConfig’. Use an external Key Store provider
Security Concerns & Mitigation

9. Akeyless.io
Unfortunately, your Secrets are still exposed.
● Secrets are readable (decrypted) within any container on a particular pod
● Mitigation Options
⇒ (Option 1) Avoid running several containers within a pod - see K8s workload concepts
⇒ (Option 2) Consider using Container Security Solutions that supports container isolation
⇒ (Option 3) Consider using Secrets Management that supports container isolation
Security Concerns & Mitigation

10. Akeyless.io
For further security:
● K8s admin/root can easily access any decrypted secret, on any pod
⇒ (AI #1) Make sure admins are who they are by enforcing MFA to Kubectl
⇒ (AI #2) Consider Auditing Kubectl commands
Security Concerns & Mitigation

11. Akeyless.io
● Applications tend to expose Secrets in audit logs and monitoring systems
What K8s Secrets doesn't solve...
Mitigate using Just-in-time secrets...

12. Akeyless.io
● The same secrets are also exposed in your entire DevOps platforms...
What K8s Secrets doesn't solve...

13. Poll #2

14. Akeyless.io
Implement Just-in-Time Access / Ephemeral Secrets:
1. Least privileges: Secrets are only created on-demand
2. Short-lived: Secrets expire after usage
Just like One Time Password in your bank account website….
Eliminating the risk of compromised Secrets

15. Akeyless.io
● Unified Secrets Store
○ Plugins to your entire cloud/devops platforms
● Just-in-time Secrets
○ Both for humans as well as machines
● Think about the future:
○ Choose the one that can Seamlessly Scale to your hybrid-cloud and multi-regions
Choose a Secrets Management that provides: