The document outlines an open house presentation by Jirayut Nimsaeng, CEO of Opsta (Thailand), on Kubernetes secrets management, focusing on best practices in a DevSecOps environment. Key topics include the utilization of HashiCorp Vault for managing sensitive data, integrating with various secret management systems, and auto-injecting values into Kubernetes secrets. It also highlights security measures, multi-tenancy, and the role of policy engines like Kyverno in managing Kubernetes environments.

1. Jirayut Nimsaeng (Dear)
CEO & Founder, Opsta (Thailand) Co.,Ltd.
TD Tech - Open House: The Technology Playground @ Sathorn Square
October 29, 2022 https://bit.ly/opsta-tdtech-vault
Secrets Management on
Production

2. Kubernetes Secrets Management
#whoami
Jirayut Nimsaeng (Dear)
Jirayut has been involved in DevSecOps, Container, Cloud Technology
and Open Source for over 10 years. He has experienced and
succeeded in transforming several companies to deliver greater
values and be more agile.
● He is Founder and CEO of Opsta (Thailand) Co.,Ltd.
● He is Cloud/DevSecOps Transformation Consultant and
Solution Architecture
● He is the first Certified Kubernetes Security Specialist (CKS)
and Certified Kubernetes Administrator (CKA) in Thailand
● He is first Thai Google Cloud Developer Expert (GDE) in
Thailand

3. Kubernetes Secrets Management
Agenda
● Secrets Management in DevSecOps
● HashiCorp Vault
● Kubernetes Secrets
● External Secrets Operator
● Live Demo

4. Kubernetes Secrets Management
Secrets Management in
DevSecOps Flow

5. Kubernetes Secrets Management
What is Secrets?
Secrets authenticate software components like cloud infrastructure,
databases, microservices, third-party APIs, and others – against each other.
● API tokens
● username/password pairs and generic passwords
● database connection URLs
● browser session tokens
services:
mongodb:
image: bitnami/mongodb:5.0.6-debian-10-r46
volumes:
- "./databases:/docker-entrypoint-initdb.d"
environment:
MONGODB_ROOT_PASSWORD: VERYSECUREPASS
MONGODB_EXTRA_USERNAMES
: app
MONGODB_EXTRA_PASSWORDS: SECUREPASS
MONGODB_EXTRA_DATABASES
: app

6. Kubernetes Secrets Management
How leaky was 2021?
https://www.gitguardian.com/files/the-state-of-secrets-sprawl-report-2022

7. Kubernetes Secrets Management
Where to handle Secrets in DevSecOps?

8. Kubernetes Secrets Management
Developer Environments
Developer

9. Kubernetes Secrets Management
Version Control System
VCS

10. Kubernetes Secrets Management
CI/CD & Artifacts
Artifacts
CI CD

11. Kubernetes Secrets Management
Infrastructure
Infrastructure
DEV
UAT
PRD

12. Kubernetes Secrets Management
Secrets Manager

13. Kubernetes Secrets Management
Secrets Management Tools

14. Kubernetes Secrets Management
What we will focus in this session
How we can handle Secrets with
HashiCorp Vault in Kubernetes

15. Kubernetes Secrets Management
HashiCorp Vault

16. Kubernetes Secrets Management
HashiCorp Vault
HashiCorp Vault is an identity-based secrets and encryption
management system. Vault provides encryption services that are
gated by authentication and authorization methods. Using Vault’s
UI, CLI, or HTTP API, access to secrets and other sensitive data
can be securely stored and managed, tightly controlled
(restricted), and auditable.

17. Kubernetes Secrets Management
HashiCorp Vault Architecture

18. Kubernetes Secrets Management
How Vault works?

19. Kubernetes Secrets Management
Kubernetes Secrets

20. Kubernetes Secrets Management
Kubernetes Secret

21. Kubernetes Secrets Management
How to use Kubernetes Secrets
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq
namespace: default
type: Opaque
data:
RabbitPass: cmFiYml0bXE=
stringData:
RabbitPlain: P@ssw0rd
apiVersion: apps/v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
containers:
- name: nginx
image: nginx:latest
env:
- name: RABBITMQ_PASSWORD
valueFrom:
secretKeyRef:
name: rabbitmq
key: RabbitPass
Create Secret Inject Secret into Container
Environment Variable
Read Secret from Code
[NodeJS]
process.env.RABBITMQ_PASSWORD;
[Python]
import os
os.getenv('RABBITMQ_PASSWORD')
[Golang]
package main
import (
"os"
)
os.Getenv("RABBITMQ_PASSWORD")

22. Kubernetes Secrets Management
What we need
● Developer can self-manage secrets in non-production environment
● Admin/Security Team can manage secrets on production environment
● Kubernetes should sync secrets from HashiCorp Vault automatically
● Developer just config their application to use secret with agreed
variable name
● Developer should not directly access to Kubernetes Secrets

23. Kubernetes Secrets Management
There is Vault Agent Sidecar Injector
https://developer.hashicorp.com/vault/docs/platform/k8s/injector

24. Kubernetes Secrets Management
External Secrets
Operator

25. Kubernetes Secrets Management
External Secrets Operator
External Secrets Operator is a Kubernetes operator that integrates
external secret management systems like AWS Secrets Manager, HashiCorp
Vault, Google Secrets Manager, Azure Key Vault and many more. The
operator reads information from external APIs and automatically injects
the values into a Kubernetes Secret.

26. Kubernetes Secrets Management
Why External Secrets Operator?
● Support many Secrets Manager
● Easy to understand for Developer
● Easy to Maintain
● Secure

27. Kubernetes Secrets Management
External Secrets Operator with Vault

28. Kubernetes Secrets Management
Multi Tenancy with Shared ClusterSecretStore
https://external-secrets.io/v0.6.0/guides/multi-tenancy/

29. Kubernetes Secrets Management
Show me your code!

30. Kubernetes Secrets Management
Vault Production on Kubernetes Checklists
❏ Use Official Vault Helm Chart
❏ Don’t run as root
❏ Run with HA mode
❏ Configure End-to-End TLS
❏ Dedicated worker node if possible
❏ Ensure mlock is Enabled
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-security-concerns

31. Kubernetes Secrets Management
Kyverno

32. Kubernetes Secrets Management
Kyverno
Kyverno (Greek for “govern”) is a policy engine designed specifically
for Kubernetes

33. Kubernetes Secrets Management
Sample Kyverno Policy
validationFailureAction: enforce
background: true
rules:
- name: privileged-containers
match:
any:
- resources:
kinds:
- Pod
validate:
message: >-
Privileged mode is disallowed. The fields
spec.containers[*].securityContext.privileged
and spec.initContainers[*].securityContext.privileged
must be unset or set to `false`.
pattern:
spec:
containers:
- =(securityContext):
=(privileged): "false"
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: unmatched-key
spec:
validationFailureAction: enforce
rules:
- name: check-unmatch-namespace-and-key
match:
resources:
kinds:
- ExternalSecret
validate:
message: "key must prefix with namespace name"
pattern:
spec:
dataFrom:
- extract:
key: "{{request.namespace}}/?*"

34. Kubernetes Secrets Management
Further More
● Reloaded Pod when Secrets changed
https://github.com/stakater/Reloader
● Integrate HashiCorp Vault with Databases
● Dynamic or Rotate Secrets
● How to manage Kubernetes secrets with GitOps?

35. Kubernetes Secrets Management
Contact Us
Jirayut Nimsaeng (Dear)
Facebook:
Email:
Website:
fb.me/DearJirayut
jirayut@opsta.co.th
www.opsta.co.th
Founder & CEO