For Meetup August 19th

1. Reproducible Security
with Helm Charts

2. ●
●
●
https://bit.lyChartCenterAugust

3. Deep Datta
deepd@jfrog.com
Community Product Manager
Twitter @DeepDattaX

4. Helm 3 Introduction
Helm Chart Security
Show: Helm Chart Dependency Tools
Agenda

5. Helm 3 Introduction

6. Helm is the Package Manager for Kubernetes
Helm is a package manager for Kubernetes. It helps
users create templated packages called Helm Charts
to include all Kubernetes resources that are required
to deploy a particular application.
Helm then assists with installing the Helm Chart on
Kubernetes:
● Install
● linting
● status
● test
● verify
● deploy
● upgrade
● rollback

7. Helm 2 vs Helm 3
Tiller was removed in Helm 3:
Removal of Tiller

8. Helm 2 vs Helm 3
Helm 3 interacts directly with the Kubernetes API
Role Based Access Controls

9. Here are more improvements to Helm 3:
Dependencies: used to live in a requirements.yaml file, but are now part of the Chart.yaml file.
Releases in namespaces
Three-way strategic merge patch
OCI Registries for charts
Chart validation: JSON Schema support is added
Improved CRD support: Kubernetes Custom Resource Definition (CRD) installations
Library charts: a class of charts called “library charts” are introduced in Helm 3

10. New commands help with monitoring
Helm status [RELEASE]
Displays the status of the named
release
Helm list | Helm ls
Lists all the releases
Helm history [RELEASE]
The history of releases is printed
$ helm history demo-rel
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Mon Oct 3 10:15:13 2016 superseded alpine-0.1.0 1.0 Initial install
2 Mon Oct 3 10:15:13 2016 superseded alpine-0.1.0 1.0 Upgraded successfully
3 Mon Oct 3 10:15:13 2016 superseded alpine-0.1.0 1.0 Rolled back to 2
4 Mon Oct 3 10:15:13 2016 deployed alpine-0.1.0 1.0 Upgraded successfully

11. Let’s talk about the structure of a basic Helm Chart
Helm Charts Summary
When you publish a Helm chart, you
can take care of all the security issues
beforehand.
Helm holds the final package with all of your
previously approved configuration options and
pieces in place and creates an immutable way to
manage security with each chart version and
each build being tracked.
Chart.yaml
Charts
Templates
values.yaml

12. Creating your first chart
Things to keep in mind

13. Let’s create our first helm chart
Helm create demochart

14. Demo Chart
charts
templates
Chart.yaml
values.yaml

15. Chart.yaml
This is where metadata about the chart lives. You also declare dependencies here.
apiVersion: v2
name: demochart
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.0
dependencies:
- name: nginx
version: "1.2.3"
repository: "https://example.com/charts"
- name: memcached
version: "3.2.1"
repository: "https://another.example.com/charts"

16. values.yaml
This is where you define your configurations options for each deployment
replicaCount: 1
image:
repository: nginx
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000

17. Templates Folder
This is where Helm finds the YAML definitions
service.yaml
deployment.yaml
hpa.yaml
Ingress.yaml
Serviceaccount.yaml
helpers.tpl
NOTES.txt

18. service.yaml
Here you can define your set of services for the pods in Kubernetes
apiVersion: v2
kind: Service
metadata:
name: {{ include "demochart.fullname" . }}
labels:
{{- include "demochart.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "demochart.selectorLabels" . | nindent 4 }}

19. deployment.yaml
Generates the metadata of your deployment
apiVersion: v2
kind: Deployment
metadata:
name: {{ include "demochart.fullname" . }}
labels:
{{- include "demochart.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "demochart.selectorLabels" . | nindent 6 }}
template:
metadata:

20. demochart/
Chart.yaml
values.yaml
charts/
templates/
...
The charts/ directory contains subcharts
Charts Folder

21. Helm Chart Security

22. NAVIGATING HELM CHART SECURITY
Set Resource Quotas
GPG and Signing Charts
Verification and Provenance
Pod Security Policies
Role-Based Access Controls
Network Policies
Secrets Management
Helm Lint and
Cert-Manager
Dependencies
ChartCenter Mitigation

23. NAVIGATING HELM CHART SECURITY
Set Resource Quotas
Checksums 101

24. Don’t forget to set resource quotas!
apiVersion: v2
kind: ResourceQuota
metadata:
name: resources-demo
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi

26. SHA-256 and SHA-512 Hash as a Checksum
SHA256 for kubernetes.tar.gz:
f1e15dff8e36899728c6f305713bd33c6bc98655db25154e8761174b2ac434ea
SHA512 for kubernetes.tar.gz:
29ab8fab7645c6ee4583ee45feaae734953d127d1413bdd3f321789607f613646ccf8d67a57c6ce1172a
e18ff9a3135a03294cac70077260388c56382ae0301d

27. NAVIGATING HELM CHART SECURITY
GnuPG
Signing Charts
Verification and
Provenance

28. Step 2: Create the public-private key
gpg --gen-key
Passphrase: *********
Signing with GnuPGP and the Helm-GPG Plugin
Step 1:
brew install gpg
helm plugin install https://github.com/technosophos/helm-gpg

29. Example
This is my public key
--------------------------------------
pub rsa2048 2020-08-10 [SC] [expires: 2022-08-10]
1AD5246C294CD0E06936F7EFA3DB8715C26DE93F
uid [ultimate] Deep Datta <deepd@jfrog.com>
sub rsa2048 2020-08-10 [E] [expires: 2022-08-10]

30. Helm package --sign --key ‘demokey’ --keyring ~/.gnupg/secring.gpg demochart
GNUPG 2.1
Use the following command to transfer your keys into the old file format:
gpg --export-secret-keys >~/.gnupg/secring.gpg
Package and Sign the chart

31. You’ve signed and created a provenance file to track lineage
demochart-0.1.0.tgz
demochart-0.1.0.tgz.prov

32. helm verify demochart-0.1.0.tgz
Signed by: Deep Datta <deepd@jfrog.com>
Using Key With Fingerprint: 1AD5246C294CD0E06936F7EFA3DB8715C26DE93F
Chart Hash Verified:
sha256:c5aa81aae8c139ea2005e842086a05a299881a71687f2933b7275663e56cded1
Now, we’ll verify the signature:

33. NAVIGATING HELM CHART SECURITY
Pod Security
Policies
RBAC
Service Accounts

34. Pod Security Policy (PSP)
When you enable Pod Security Policies, you can control things like:
● The running of privileged containers
● Use of host namespaces
● Use of host networking and ports
● Use of volume types
● Use of the host filesystem
● Requirements for use of a read only root file system
● The user and group IDs of the container
● Escalations of root privileges
kubectl create -f your-new-policy.yaml

35. Disable privileged containers
apiVersion: policy/v1demobeta1
kind: PodSecurityPolicy
metadata:
name: prevent-privileged-containers
spec:
privileged: false
https://resources.whitesourcesoftware.com/blog-whitesource/kubernetes-pod-security-policy

36. Read-only file system
apiVersion: policy/v1demobeta1
kind: PodSecurityPolicy
metadata:
name: read-only-fs
spec:
readOnlyRootFilesystem: true
https://resources.whitesourcesoftware.com/blog-whitesource/kubernetes-pod-security-policy

37. Prevent privilege escalation
apiVersion: policy/v1demobeta1
kind: PodSecurityPolicy
metadata:
name: no-privilege-escalation
spec:
allowPrivilegeEscalation: false
https://resources.whitesourcesoftware.com/blog-whitesource/kubernetes-pod-security-policy

38. Prevent containers from running as root
apiVersion: policy/v1demobeta1
kind: PodSecurityPolicy
metadata:
name: no-privilege-escalation
spec:
MustRunAsNonRoot: true
https://resources.whitesourcesoftware.com/blog-whitesource/kubernetes-pod-security-policy

39. Group your policies together
apiVersion: policy/v1demobeta1
kind: PodSecurityPolicy
Metadata: my-policies
name:
spec:
privileged: false
spec:
readOnlyRootFilesystem: true
spec:
allowPrivilegeEscalation: false
spec:
MustRunAsNonRoot: true
https://resources.whitesourcesoftware.com/blog-whitesource/kubernetes-pod-security-policy

40. Let’s Talk About Pod Access
The desired state of each cluster and access privileges
within each node is highly configurable.
Even pods have security features that can be activated with the admission controller and by
assigning unique privileges to users and groups using:
Role-Based Access Control (RBAC)

41. Role-Based Access Control
Role
ClusterRole
RoleBinding
ClusterRoleBinding
ServiceAccount, User or Group

42. Namespaces:
Default
Kube-System
Content
Role
Ruby
Role
Binding
Get, list, watch,
update, delete

43. Service Accounts
Who is the user working within the pod? You can create service accounts to limit the permissions.
apiVersion: v1
kind: ServiceAccount
metadata:
name: peddling-lighteningbug-mychart
labels:
app.kubernetes.io/name: mychart
helm.sh/chart: mychart-0.1.0
app.kubernetes.io/instance: peddling-lightningbug
app.kubernetes.io/version: "1.1"
app.kubernetes.io/managed-by: Tiller
imagePullSecrets:
- name: acr-auth

44. NAVIGATING HELM CHART SECURITY
Network
Policies

46. Network Policies
To limit the access to the nginx service so that only Pods with the label access: true can query it, create a
NetworkPolicy object as follows:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default.postgres
namespace: default
spec:
podSelector:
matchLabels:
app: postgres
ingress:
- from:
- podSelector:
matchLabels:
app: balance
policyTypes:
- Ingress

47. Network Policies
They control the traffic in and out of pods:
app: postgres
Pod
Ingress
Egress
app: fooapp: balance
Pod
Pod

48. NAVIGATING HELM CHART SECURITY
Secrets
Management
Helm-Secrets
Plugin

49. Don’t store sensitive information (passwords, authentication
credentials, API keys...) in ConfigMaps!
Secrets Management
Secrets
Sensitive data
ConfigMaps
Key:value pairs that not
intended to be hidden

50. env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mariadb-root-password
key: password
These are secret!

51. Secrets Management Best Practices
1 2 3 4
Rotate
credentials
Encode and
Encrypt
secrets
Isolate where
they are used
and where
they are
stored
Log and
monitor the
user of
secrets
Source: https://www.youtube.com/watch?v=DNKcRUyz4Hw&t=215s

52. Helm Secrets Plugin
Usernames, Passwords, Database Credentials, API Tokens, TLS Certificates
We end up putting this in plain text in
many different places
...don’t store this in source control
helm plugin install
https://github.com/futuresimple/helm-secrets

53. Sops step
secrets.yaml diff=sopsdiffer
secrets.*.yaml diff=sopsdiffer
helm_varsCreate a file .sops.yaml inside helm_vars folder.
brew install sops

54. Supply with our key pair value in plain text.
mysecret:password
Lets encrypt our secrets.yaml using Helm-secret plugin.
$ helm secrets enc ~/helm_vars/secrets.yaml
Encrypting secrets.yaml
Encrypted secrets.yaml

55. You can also use Hashicorp Vault for advanced Secrets
Management
https://www.vaultproject.io/

56. NAVIGATING HELM CHART SECURITY
Helm Commands
(Helm Lint)

57. helm lint demochart
==> Linting demochart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
Let’s lint our chart:
helm lint is your go-to tool for verifying that your chart follows best practices
helm install --dry-run --debug
helm get manifest

58. NAVIGATING HELM CHART SECURITY
Certificates

59. Let’s use cert-manager by Jetstack for TLS

60. Install Cert-Manager using Helm Charts

61. TLS with Cert-Manager
Then you’ll need to get a TLS certificate by installing cert-manager:
# Install the CustomResourceDefinition resources separately:
$ kubectl apply --validate=false -f
https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml
# Create the namespace for cert-manager:
$ kubectl create namespace cert-manager
# Install the cert-manager Helm chart from ChartCenter:
$ helm install cert-manager helm install center/jetstack/cert-manager
You can do a final rollout status check with:
$ kubectl -n cert-manager rollout status deploy cert-manager

62. NAVIGATING HELM CHART SECURITY
Dependencies and
Mitigation Tools

63. Creating a subchart
$ cd demochart/charts
$ helm create mysubchart
Creating mysubchart
$ rm -rf mysubchart/templates/*.*

64. Using a trusted registry

65. Checking Chart Dependencies:

66. What is a CVE?

67. ChartCenter Security Tab

68. For Chart Maintainers, Introducing:
security-mitigation.yaml

69. Here is the spec:
## Schema version of this YAML file
schemaVersion: v1
## Overall mitigation summary
summary: text
## External URL if you'd like to link to an external page
securityAdvisoryUrl: URL
## If you want to point us to a file instead of filling out the CVE's here
useMitigationExternalFile: boolean
mitigationExternalFileUrl: URL
## Mitigation notes for individual CVEs
mitigations:
cves:
## Indicates package Uri for which the security mitigation is provided. helm://… || docker://…
affectedPackageUri:
## Which chart versions this cve note belongs to
affectedVersions: mastermind SemVer constraint
## Description / note
description: text
https://github.com/jfrog/chartcenter/blob/master/docs/security-mitigation.yaml

70. Here is an example of what these notes look like on ChartCenter

71. NAVIGATING HELM CHART SECURITY
Deploy your chart!

72. Get a Helm
Chart from
ChartCenter
Package and Sign the
Chart
JFrog
Container
Registry
Deploy and manage our Kubernetes
application on a trusted provider

73. Deploy and manage our Kubernetes
application on Rancher’s custom
catalogs

74. How Charts Create Reproducible Security
Organizations do not have to
replicate each security step.
If teams are distributed throughout the
world and have multiple environments,
such as test, QA, staging and production.
Immutable Configurations can be shared
Feat
Test QA
Stage
Prod
Chart version: 1.5.1

75. HELM CHART SECURITY

76. HELM CHART SECURITY BLUEPRINT
Set Resource Quotas
GPG and Signing Charts
Verification and Provenance
Pod Security Policies
Role-Based Access Controls
Network Policies
Secrets Management
Helm Lint and
Cert-Manager
Dependencies
ChartCenter Mitigation
Deploy Your Chart

77. Q&A
deepd@jfrog.com
Twitter @DeepDattaX
Chartcenter.io

78. ●
●
●
https://bit.lyChartCenterAugust