Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Can Kubernetes Keep a Secret?

895 views

Published on

We’ve all experienced it: you’re working on a task, adding some code, and then you need to store some sensitive configuration value. It could be an API key, client secret or an encryption key ― something that’s highly sensitive and must be kept secret. And this is where things get messy. Usually, secret storage is highly coupled with how the code is deployed, and different platforms have different solutions.

Kubernetes has a promise to simplify this process by using the native secret object, which, as the name implies, can be used to store secrets or sensitive configurations. Unfortunately, Kubernetes secrets are fundamentally broken, and a developer who tries to use them will definitely have some issues.

But no need to worry ― there are solid alternatives for storing secrets securely on Kubernetes platform. One solution is to use Kamus, an open-source, git-ops solution, that created by Soluto, for managing secrets on Kubernetes. Kamus can encrypt a secret so it can be decrypted only by your app on runtime - and not by anyone else.

The first part of this session will cover the challenges faced when using Kubernetes secrets (from a usability and security point of view). The second part will discuss some of the existing solutions (Sealed Secrets, Helm Secrets and others), their pros, and cons, and then feature Kamus: how it works, what problems it solves, how it differs from other solutions, and what threats it can help mitigate (and what threats it can’t).

The talk will cover all that is required to know so you can run Kamus on your own cluster and use it for secret management. Join me for this session to learn how you can build a Kubernetes cluster than can keep a secret ― for real.

Published in: Technology
  • Be the first to comment

Can Kubernetes Keep a Secret?

  1. 1. @omerlh Can Kubernetes Keep a Secret? Omer Levi Hevroni AppSec Cali 2019
  2. 2. @omerlh@omerlh
  3. 3. @omerlh I’m a builder @omerlh
  4. 4. @omerlh DevSecOps @
  5. 5. @omerlh I OWASP • Zap contributor • Proud member • Glue project leader
  6. 6. @omerlh
  7. 7. @omerlh Super-Devs: Full Responsibility ● Writing Code ● Deploying to Production ● Monitoring https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
  8. 8. @omerlh Super-Devs Need Help ● Good tools to support them ● Make it harder to do mistakes ● Secure by design
  9. 9. @omerlh
  10. 10. @omerlh Manifests Files Code A GitOps Solution Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  11. 11. @omerlh How do we manage secrets?
  12. 12. @omerlh Manifests Files Code Secret A GitOps Solution Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  13. 13. @omerlh Requirements  GitOps  Kubernetes native  Secure  “One-way encryption”
  14. 14. @omerlh Pod is out of scope ● Who can “SSH” into it? ● What is running on the pod? ● Does the code leaked the secrets?
  15. 15. @omerlh Let’s Go!
  16. 16. @omerlh First iteration – Kubernetes Secrets
  17. 17. @omerlh https://kubernetes.io/docs/concepts/configuration/secret/
  18. 18. @omerlh Requirements  GitOps  Kubernetes native  Secure
  19. 19. @omerlh@omerlh
  20. 20. @omerlh Let’s take a deeper look…  Producing (dev)  Consuming (application) Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  21. 21. @omerlh Producing - File Manifest?
  22. 22. @omerlh Well, that complicates things… http://i.imgur.com/5ebYy62.jpg @omerlh
  23. 23. @omerlh Producing – Naive Approach
  24. 24. @omerlh Producing - Encrypted Secrets? ● Secrets that can be committed ● Transparent for the application ● Multiple solutions ○ Helm Secrets ○ Sealed Secrets
  25. 25. @omerlh A Sealed Secret
  26. 26. @omerlh@omerlh
  27. 27. @omerlh Issues ● Key Management ○ Sealed Secret – single key-pair in the cluster ○ Helm Secret – based on Mozilla mops (AWS/GCP KMS support) ● Coupling to a specific cluster/deployment method ● Any change to the secret requires decryption
  28. 28. @omerlh Let’s take a deeper look… ⓧ Producing (dev)  Consuming (application)
  29. 29. @omerlh Consuming – Environment Variables
  30. 30. @omerlh The Environment Variable Dispute https://i0.wp.com/www.rogerogreen.com/wp-content/uploads/2015/06/Disputation.jpg
  31. 31. @omerlh • Some log libraries collects env vars • Accessible via /proc • Visible when inspecting docker image • RCE – run env to leaked all env vars. Simpler than finding all sensitive files and exporting them (even with LFI) • Harder to commit accidently • Simpler than files • If you can access /proc or inspect docker images, you can inspect mounted volumes • Better permissions model on windows (thanks @swisshttp!) • Leaked files (thanks @sporkmonger!) The Environment Variable Dispute Cons Pros https://tvtropes.org/pmwiki/pmwiki.php/Main/GoodAngelBadAngel
  32. 32. @omerlhhttps://twitter.com/omerlh/status/1079088158929797121
  33. 33. @omerlh Consuming – Volume Mount
  34. 34. @omerlh Consuming/Producing: Configuration Files Configuration File Base64 Encoder Secret Manifest
  35. 35. @omerlh Let’s take a deeper look… ⓧ Producing (dev) ⓧ Consuming (application) Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  36. 36. @omerlh Requirements  GitOps – under some serious limitations  Kubernetes native  Secure – depend on usage
  37. 37. @omerlh Second iteration – Hashicorp Vault
  38. 38. @omerlh What? ● Secure secrets storage ● Native Kubernetes integration ● Seamless consuming ○ Side-car to generate config files https://www.vaultproject.io/
  39. 39. @omerlh Workflow /my-app/super-sensitive Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  40. 40. @omerlh @omerlh
  41. 41. @omerlh Workflow Access Control /my-app/super-sensitive Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  42. 42. @omerlh Manifests Files Code Secret Vault - Threat Modeling Kubernetes Icons Source: Kubernetes Community, Apache 2 license Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  43. 43. @omerlh Imperfect solution ●Separate storage of secrets and deployment files ○ No single source of truth ●External Permission Model ●Deployment ○ Cloud vendor alternatives (Azure KeyVault, AWS secret manager) ○ Vault users authn/authz
  44. 44. @omerlh Requirements ⓧ GitOps  Kubernetes native  Secure – depend on usage
  45. 45. @omerlh
  46. 46. @omerlh Travis Encrypted Secrets https://docs.travis-ci.com/user/encryption-keys/
  47. 47. @omerlh Eureka! http://theunprofessionalblog.blogspot.com/2016/04/whatsapp-this-is-killing-me.html
  48. 48. @omerlh Third iteration – Kamus Travis secret encryption – for Kubernetes
  49. 49. @omerlh Kamus?
  50. 50. @omerlh https://github.com/Soluto/kamus/tree/master/example
  51. 51. @omerlh A perfect solution?  GitOps  Kubernetes native  Secure
  52. 52. @omerlh Let’s talk about security
  53. 53. @omerlh Permission Model Encrypt Decrypt User Yes (Can be limited) No Pod Yes Only it’s own secrets
  54. 54. @omerlh Kamus – Threat Model Encryptor Decryptor Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  55. 55. @omerlh Mitigations: User ● Secure by default permission model ● Secured CLI ○ Enforce HTTPS ○ Support for certificate pinning
  56. 56. @omerlh Mitigations: Git ● Strong encryption (using Azure KeyVault/GCP KMS) ○ HSM protection ○ IP Filtering ● One-way encryption
  57. 57. @omerlh Mitigations: Pod ● Secure by default permission model ● In-Memory volume for decrypted files Kubernetes Icons Source: Kubernetes Community, Apache 2 license
  58. 58. @omerlh Mitigations: Kamus API ● Separate pods ● Authentication support for encryptor ● Security tests ○ SAST (Checkmarx) ○ DAST (Zaproxy) ○ Packages scan (Snyk)
  59. 59. @omerlh Accepted Risks ●Clear text traffic inside the cluster ●Any pod in the same namespace can mount any service account ○ Pod impersonation ●Service account token never expires
  60. 60. @omerlh Public Threat Model https://github.com/Soluto/kamus/blob/master/docs/features
  61. 61. @omerlh Security.md https://github.com/Soluto/kamus/blob/master/security.md
  62. 62. @omerlh Kamus - A perfect solution  GitOps  Kubernetes native  Secure
  63. 63. @omerlh How can I use it? ● Simply using helm: helm install kamus soluto/kamus ● Checkout the install guide for a secure installation ● Blog post - https://bit.ly/2T2Nhgs
  64. 64. @omerlh Kamus Roadmap ● AWS support ● Custom Resource Descriptor ● Rolling encryption keys ● Quality – improve test coverage ● FaaS
  65. 65. @omerlh Wrapping Up
  66. 66. @omerlh Solutions GitOps Kubernetes Native Secure Kubernetes Secrets It depends Yes It depends Vault No Yes Yes Kamus Yes Yes Yes
  67. 67. @omerlhhttp://www.applestory.biz/hermione-hand-raise-gif.html Questions?
  68. 68. @omerlh Feedback appreciated
  69. 69. @omerlh Can Kubernetes Keep a Secret? @omerlh
  70. 70. @omerlh Kamus Enable Super-Devs to Fly Higher https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
  71. 71. @omerlh Thank You! Omer Levi Hevroni AppSec Cali 2019

×