Can Kubernetes Keep
a Secret?
@omerlh
@omerlh
@omerlh
I’m a builder
@omerlh
@omerlh
DevSecOps @
@omerlh
Super-Devs: Full Responsibility
● Writing Code
● Deploying to Production
● Monitoring
https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
@omerlh
Super-Devs Need Help
● Good tools to support them
● Make it harder to do mistakes
● Secure by design
@omerlh
@omerlh
Manifests
Files
Code
A GitOps Solution
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
How do we manage secrets?
@omerlh
Manifests
Files
Code
Secret
A GitOps Solution
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Requirements
 GitOps
 Kubernetes native
 Secure
 “One-way encryption”
@omerlh
Pod is out of scope
● Who can “SSH” into it?
● What is running on the pod?
● Does the code leaked the secrets?
@omerlh
Let’s Go!
@omerlh
First iteration – Kubernetes Secrets
@omerlh
https://kubernetes.io/docs/concepts/configuration/secret/
@omerlh
Requirements
 GitOps
 Kubernetes native
 Secure
@omerlh
@omerlh
GitOps?
@omerlh
File Manifest
@omerlh
Well, that complicates things…
http://i.imgur.com/5ebYy62.jpg
@omerlh
@omerlh
Naive Approach
@omerlh
Encrypted Secrets?
● Secrets that can be committed
● Transparent for the application
● Multiple solutions
○ Helm Secrets
○ Sealed Secrets
@omerlh
A Sealed Secret
@omerlh@omerlh
@omerlh
Issues
● Key Management
○ Sealed Secret – single key-pair in the cluster
○ Helm Secret – based on Mozilla mops (AWS/GCP KMS support)
○ Coupled to a specific cluster/deployment method
@omerlh
Requirements
 GitOps – under some serious limitations
 Kubernetes native
 Secure – depend on usage
@omerlh
Second iteration – Hashicorp Vault
@omerlh
What?
● Secure secrets storage
● Native Kubernetes integration
● Seamless consuming
○ Side-car to generate config files
https://www.vaultproject.io/
@omerlh
@omerlh
@omerlh
DevOps
@omerlh
Naive Approach
@omerlh
Travis Encrypted Secrets
https://docs.travis-ci.com/user/encryption-keys/
@omerlh
Eureka!
http://theunprofessionalblog.blogspot.com/2016/04/whatsapp-this-is-killing-me.html
@omerlh
Third iteration – Kamus
Travis secret encryption – for Kubernetes
https://kamus.soluto.io
@omerlh
Manifests
Files
Code
Secret
A GitOps Solution
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
A Pod has a Name
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Let’s Encrypt Some Secrets!
@omerlh
Manifests
Files
Code
Secret
Encrypted
A GitOps Solution
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
@omerlh
A perfect solution?
 GitOps
 Kubernetes native
? Secure
@omerlh
Let’s talk about security
@omerlh
Permission Model
Encrypt Decrypt
User Yes (Can be
limited)
No
Pod Yes Only it’s own
secrets
@omerlh
Kamus – Threat Model
Encryptor Decryptor
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Mitigations: User
● Secure by default permission model
● Secured CLI
○ Enforce HTTPS
○ Support for certificate pinning
@omerlh
Mitigations: Git
● Strong encryption (using Cloud Provider KMS)
○ HSM protection
○ IP Filtering
● One-way encryption
@omerlh
Mitigations: Pod
● Secure by default permission
model
● In-Memory volume for
decrypted files
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Mitigations: Kamus API
● Separate pods
● Authentication support for encryptor
● Security tests
○ SAST (Checkmarx)
○ DAST (OWASP Zaproxy)
○ Packages scan (Snyk)
@omerlh
Kamus - A perfect solution
 GitOps
 Kubernetes native
 Secure
https://kamus.soluto.io/
@omerlh
How can I use it?
● Simply using helm:
helm install kamus soluto/kamus
● Checkout the install guide for a
secure installation
● Blog post - https://bit.ly/2T2Nhgs
@omerlh
Kamus Roadmap
● Rolling encryption keys
● Quality – improve test coverage
● Non-Kubernetes deployments
● SPIFFE support
@omerlh
Wrapping Up
@omerlh
Solutions
GitOps Kubernetes
Native
Secure
Kubernetes
Secrets
It depends Yes It depends
Vault No Yes Yes
Kamus Yes Yes Yes
@omerlhhttp://www.applestory.biz/hermione-hand-raise-gif.html
Questions?
@omerlh
Feedback appreciated
@omerlh
Can Kubernetes Keep a
Secret?
@omerlh
@omerlh
Kamus Enable Super-Devs to Fly Higher
https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
Thank You
@omerlh
https://solutotlv.com

FTRD - Can Kubernetes Keep a Secret?

Editor's Notes

  • #2 Hey, good morning everyone My name is Omer I want to start this talk by showing gratitude First, to all the people who worked hard on organizing this conf and all the people who are working today so we all could enjoy it - thank you Second, I want to thank the organizers who choose me to speak here, so thank you. It is a big honor <pause> Can kubernetes keep a secret? <pause> Why? Raise you’re hand if you ever worked on a project and you had to deal with credentials: API Key, client secret, certificates etc
  • #3 What you need to do is pass these credentials securely to the platform running your code, so the app in production can use it. Different platforms have different solutions to the problem (sometime good, sometime bad). This talk will focus on Kubernetes – how we can tell a secret to Kubernetes, and make sure only Kubernetes will know it? But first – let me introduce myself quickly, so you could understand what are my credentials and where I’m coming from.
  • #4 I’m a builder, this is what I love doing and doing it from a really early age Doing it professionally for the last 8 years I’m from Israel, married etc Who else is a builder? This talk is for you!
  • #5 Today I’m working at Soluto, our missing is to help people with their technology My job is DevSecOps, or as I see it - helping the entire team to build a more secure software I’m achieving it via many approaches, including education, reviewing and threat modeling – but what I love the most is threat modeling
  • #6 About me slides
  • #7 What you need to do is pass these credentials securely to the platform running your code, so the app in production can use it. Different platforms have different solutions to the problem (sometime good, sometime bad). This talk will focus on Kubernetes – how we can tell a secret to Kubernetes, and make sure only Kubernetes will know it?
  • #8 Explain why it is a challenge – you cant expect one person to manage all secrets Why not solved it manually
  • #9 Explain why it is a challenge – you cant expect one person to manage all secrets Why not solved it manually
  • #10 And that’s why we love GitOps: Git is a tool that all devs are familiar with.
  • #12 And we started to look for solutions. It want not an easy path, and today I want to share with you the process we want through. So, let’s start we talking on what we want.
  • #18 Choose one sentence
  • #19 Security is what we all here love
  • #23 Security features like encryption at rest
  • #24 Encoding is not encrypting Adding native approach
  • #27 Add meme
  • #30 Security is what we all here love
  • #45 Security is what we all here love
  • #46 Battle tested
  • #47 Add attributation
  • #48 Add headlines – encryptor & decryptor
  • #53 We really love Kamus, we’re been using it in production for the past 6 months
  • #56 End of journey meme/image
  • #57 Today I discussed 3 different solutions for secret management on Kubenretes. All are good solutions, depend on your requirments.
  • #60 I started the talk by asking “Can Kubernetes keep a secret?” Now you that yes – Kubernetes can. You just need to use the right tool for you’re use case.
  • #61 For us, it was Kamus What Kamus can do for you?