OWASP Bay Area Meetup - DevSecOps the Kubernetes Way

JIM MANICO Secure Coding Instructor www.manicode.com
DevSecOps the Kubernetes Way

COPYRIGHT ©2018 MANICODE SECURITY
A little background dirt…
@jimmesta
 CTO @ Manicode
 10 years of penetration testing,
teaching, and building security
programs
 OWASP AppSec California organizer
and Santa Barbara chapter founder
 Conference speaker
 Been on both sides of the InfoSec
fence
 Loves Clouds
2

WARNING: Please do not attempt to hack any
computer system without legal permission to do so.
Unauthorized computer hacking is illegal and can
be punishable by a range of penalties including
loss of job, monetary fines and possible imprisonment.
ALSO: The Free and Open Source Software presented in these materials are
examples of good secure development techniques. You may have unknown
legal, licensing or technical issues when making use of Free and Open Source
Software. You should consult your company's policy on the use of Free and Open
Source Software before making use of any software referenced in this material.
3

COPYRIGHT ©2018 MANICODE SECURITY 4
Introduction to DevOps and Common Patterns
Introducing Security to DevOps Environments
Infrastructure as Code
Deploying Containers Securely
Kubernetes Introduction
Attacking and Defending Kubernetes Infrastructure
Managing Secrets in Kubernetes
Logging and Monitoring in Kubernetes
DevSecOps Pipelines

Introductions
6
Who are you?
Why are you here?
What would you like to get out of today?

 All labs are in a Git Repository
 Help Thy Neighbor
 If you discover bugs, typos, etc. let me
know or submit a PR!
 Please keep chaos to a tolerable level
 If you complete every bonus you get a
prize
 If you run into technical difficulties talk to
me over the break
 We have time built in for discussion and
questions
7

github.com/ManicodeSecurity/Defending-DevOps
Lab Material

https://tinyurl.com/owaspsf
GKE Cluster Access

We Have a “Situation”

What the f&%$
is DevOps?
11

“Our research has
uncovered 24 key
capabilities that drive
improvements in software
delivery performance in a
statistically significant way.”
The (Actual) Current State of Affairs

Continuous Delivery Capabilities
Version Control
Deployment Automation
Continuous Integration
Trunk-Based Development
Test Automation
Test Data Management
Shift Left on Security
Continuous Delivery
14

Architecture Capabilities
Loosely Coupled Architecture
Empowered Teams
Customer Feedback
Value Stream
Working in Small Batches
Team Experimentation
15

Lean Management and Monitoring Capabilities
Change Approval Process
Monitoring
Proactive Notification
WIP Limits
Visualizing Work
16

Cultural Capabilities
Supporting Learning
Collaboration Among Teams
Job Satisfaction
Transformational Leadership
17

High Performers vs. Low Performers
46x more frequent code deployments
440x faster lead time from commit to deploy
170x faster mean time to recover from
downtime
5x lower change failure rate
18

High Performing Security Teams
“High-performing teams were more likely to
incorporate information security into the
delivery process. Their infosec personnel
provided feedback at every step of the
software delivery lifecycle, from design
through demos to helping out with test
automation. However, they did so in a way
that did not slow down the development
process…”
19

Hopes and Dreams for Tonight
20
 Give you the tools to move the needle to “High Performer”
 Common consensus (or not?) on DevOps and DevSecOps
 Understand Kubernetes from a technical perspective
 Using Kubernetes to enhance DevSecOps initiatives
 Attack and defend Kubernetes clusters
 Deploy software more confidently
 Exposure to only the best memes
Most importantly, have some fun!

Introduction to DevOps
21

DevOps Anti-Patterns
22
Have the Ops team deal with it.
Time to fire the Ops team!
Let’s hire a DevOps unit!

Anti-Pattern: “Throw it Over the Wall”
23
Development Operations

Anti-Pattern: “DevOps Team Silo”
24
Development OperationsDevOps

Anti-Pattern: “NoOps” Approach
25
Dev
Ops

Anti-Pattern: “Ops Will Handle it”
26
Dev
Ops

Anti-Pattern: “Ops Will Handle it”
27

Development and Operations Collaboration
28

Dev and Ops Fully Shared Responsibilities
29

DevOps-as-a-Service
30
Development Operations DevOps

So…What is DevOps?
31
Process
ToolsPeople
…and some buzzwords

Collaboration Across Teams
Cross-Training
Support Business Agility
Breaking Down Silos
Automation of Repetitive Tasks
“Everything” as Code
Test, Measure, and Monitor
People
Tools
Process

DevOps Processes
33
Automate building the dev and production environment
Automate software testing (including security)
Automate deploying software and services
Automate monitoring and alerting
Tune your tools to become more automated and hands-
off
Build the pipeline slowly and don’t fear failure!
Be careful with sensitive areas which are difficult to
automate (access control, biz logic, complex actions)

Adding the “Sec” to DevOps
34

Windows for automated scanning and
manual testing are shrinking
Continuous delivery scares security teams
Framework, language, infrastructure
fatigue
Security teams are vastly outnumbered
Automated detection of complex issues is
hard work
Third-party code / libraries / APIs / tooling
scattered everywhere

We want to turn this…

Into this!

Competing Forces
38
”The Business”
Development
Operations
Security
”No downtime, ever!”

The Case for DevSecOps
39
Software and product development is rapidly moving
towards Agile, Scrum, DevOps
The “perimeter” as we know it is going away
Traditional security mechanisms are failing to keep up
The demand for security aka “not getting hacked” is
skyrocketing
Security is becoming a marketing tool and selling point

“DevSecOps is the process of incorporating
and enforcing meaningful security controls
without slowing down deployment velocity.”

DevSecOps is a Journey…

Infrastructure as Code
42

Building Infrastructure
43
Is your infrastructure…
Self documenting?
Version controlled?
Capable of continuous delivery?
Integration tested?
Immutable?
Remember: ”It’s all software"

Immutable Infrastructure
44
“Immutable infrastructure is built using components which are replaced
during deployment rather than being updated in place”

Security and Immutable Infrastructure
45
An immutable infrastructure starts with a
“Golden Image” in a version catalog
Security teams have a central location to
validate images as compliant and enforce OS
hardening policies
No more guesswork what is installed
Automation can flag security anomalies vs.
human intervention
“Push Security to the Left”

Simple Immutable Infrastructure
46
Base OS
Version Catalog
Packages
Base Container
Latest Code
Base
Image
0.2
Base
Image
0.2
Base
Image
0.2
Base
Image
0.2
Instance 1
Instance 2
Instance n

Proving Immutability
47
Base OS
Version Catalog
Packages
Base Container
Latest Code
Base
Image
0.2
Base
Image
0.2
Base
Image
0.2
Base
Image
0.2
Instance 1
Instance 2
Instance n
SHA1(Base_Image)
96c5…07e4bb
96c5…07e4bb
96c5…07e4bb

Shellshock?
48
Base OS
Version Catalog
Packages
Base Container
Latest Code
Base
Image
0.2
Base
Image
0.2
Base
Image
0.2
Base
Image
0.2
Instance 1
Instance 2
Instance n

Shellshock?
49
Base OS
Version Catalog
Packages
Base Container
Latest Code
Base
Image
0.3
Base
Image
0.3
Base
Image
0.3
Base
Image
0.3
Instance 1
Instance 2
Instance n
Emergency
Patch!

Grouping & Tagging
Node
Node
Node
Node
DB
Node
Node
Node
Node
Cache
Node
Node
Node
Node
Web
Apache
Monitoring
MySql
Memcache
 Tagging your servers
and containers applies
the required set of
automation
 A base set of for all
servers
 Each server can have
multiple tags
 Map tags to security
requirements

Cattle, not pets.
51

Security Wins
52
Security team now has insight into the entire
system
Infrastructure is auditable and version
controlled, just like source code
Patching can be applied programmatically with
a high level of certainty
Alerting can be built for changes to specific
areas of the infrastructure
– A new firewall rule is created or deleted
– Administrative user is created
– New VPC rolled out
Testing can occur much earlier in the pipeline

Lab 1
001-Lab-Setup
Lab Goals:
• Set up infrastructure
• Get acquainted with
Google Cloud Shell
• Explore the git repo

Containers
54

Containers, Containers, Containers, Containers…
55

VM

Container

OS Virtualization Security Building Blocks
58
Process Security
Process Isolation

Kernel Namespaces
59
Process Security
Process Isolation
Limits what a process can see
–The pid namespace partitions kernel resources such that
one set of processes may be provided with an independent
set of process IDs (PIDs). Each container gets its own
network stack
–Network namespaces create virtual networking interfaces
to allow programs to run on any port without conflict
–Mount namespaces enable the mounting and un-
mounting of filesystems without affecting the host
filesystem
No privileged access to the sockets or interfaces of
another container

PID Namespace
60
Process Security
Process Isolation

Control Groups
61
Ensures each container is provided with its fair
share of memory, CPU, disk I/O and more
DoS anyone?
Released in 2006 in kernel 2.6.24

Docker Engine
62
Client-Server application that includes a
few key components
 Docker Daemon (dockerd)
– Responsible for container
orchestration
REST API
– Used to talk to the Docker daemon
Docker Client (CLI)
– Interface to interact with the Docker
daemon

Docker Engine
63
Process Security
Process Isolation

Docker Security Gotchas
64
Process Security
Process Isolation
Untrusted Users
Sensitive Volume Mounts
API Exposed over HTTP
App Vulnerabilities
Untrusted Images

Container Security Benefits
65
Patching Simplicity
Typically Short Lifespans
One Process Per Container (Ideally)
Isolation from Others

https://blog.heroku.com/exploration-of-security-when-building-docker-containers

It is possible to break out of a Docker container
67

Yes. Docker Images Have Vulnerabilities
68

Docker vulnerability scanning
69

Lab 2
002-Containerizing-An-Application
Lab Goals:
• Run Golang API locally
• Build Docker Image
• Run the Docker Container
• Use Environment
Variables

K8S - A Gentle Introduction
71

72
Kubernetes is an open-source
platform built to automate
deployment, scaling and
orchestration of containers.

73
K8S is portable. Clusters can
be deployed on a public/private
cloud, on prem, and even on
your laptop.

74
K8S is customizable. It is
modular and extensible to fit a
variety of use-cases.

75
K8S is scalable. It provides
self-healing, auto scaling, and
replication out of the box.

cluster

virtual
machines that
Kubernetes
manages
cluster

clusternodemaster node node

clusternodemaster node node
node node node
node node node
node node node
node node
node node node
node node nodemaster
master
node

pod

group of
containers
sharing
storage and
network
pod

podcontainer container container
volume A volume B
network interface

apiVersion: v1
kind: Pod
metadata:
name: redis-rails
spec:
containers:
- name: key-value
image: redis
ports:
- containerPort: 6379
- name: rails-frontend
image: rails
ports:
pod.yaml

pod.yamlnodemaster node node

deployment

ensure N pods
are up and
running
deployment

kind: Deployment
apiVersion: apps/v1
metadata:
name: rails-deployment
labels:
app: rails
spec:
replicas: 4
selector:
matchLabels:
app: rails
template:
metadata:
labels:
app: rails
spec:
containers:
- name: key-value
image: redis
ports:
image: rails
ports:
deploy.yaml

deploy.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: rails-deployment
labels:
app: rails
spec:
replicas: 4
selector:
matchLabels:
app: rails
template:
metadata:
labels:
app: rails
spec:
containers:
- name: key-value
image: redis
ports:
image: rails
ports:

deploy.yamlnodemaster node node

deploy.yamlnodemaster node node
10.0.0.1 10.0.0.2
10.0.0.3
10.0.0.4

abstraction
layer that
enables pod
communication
service

service

servicenodemaster node node
10.0.0.1 10.0.0.2
10.0.0.3
10.0.0.4

servicemaster
service

servicemaster
service
service

servicemaster
service
service
public load balancer
your.site.com

kind: Service
apiVersion: v1
metadata:
name: web-frontend
spec:
ports:
- name: http
port: 80
targetPort: 3000
protocol: TCP
selector:
app: rails
type: LoadBalancer
svc.yaml

kind: Service
apiVersion: v1
metadata:
name: web-frontend
spec:
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
selector:
app: rails
type: LoadBalancer
svc.yaml

Labels
and
Selectors

Metadata (key-
value) which
can be
attached to a
resource
Labels

Used for
identification
such as app
name, tier,
environment
Labels

Provides loose
coupling
between
objects
Selectors

Ingress

configure
external
access to your
cluster
ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: web-ingress
spec:
backend:
serviceName: web-frontend
servicePort: 80
ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: web-ingress-vhosts
rules:
- host: sub.domain.com
http:
paths:
- backend:
serviceName: web-frontend-1
servicePort: 80
- host: other.domain.com
http:
paths:
- backend:
serviceName: web-frontend-2
servicePort: 80
ingress.yaml

manage
different
environments
in the same
cluster
namespace

ns.yaml
kind: Namespace
apiVersion: v1
metadata:
name: development

k8s internals

master
master
API
Server
Scheduler
Controller
Manager
etcd

master
master
API
Server

master
master
Scheduler

master
master
Controller
Manager

master
master
kube-controller-manager
NodeController
ReplicationController
ServiceAccount Controller
(…)

master
kube-controller-manager
NodeController
Responsible for noticing when
nodes go up and down
Responsible for maintaining the
correct number of pods for every
replication controller object
ServiceAccount Controller
Creates default accounts and API
access tokens

master
master
cloud-controller-manager
Node Controller
Route Controller
Service Controller
Volume Controller

master
master
etcd

node
node
kube-proxy kubelet
container runtime

node
node
kube-proxy

node
node
kubelet

node
node
kubelet
No auth on your kubelet API?
You’re gunna have a bad time.https://medium.com/handy-tech/analysis-
of-a-kubernetes-hack-backdooring-
through-kubelet-823be5c3d67c

node
node
container runtime

Kubernetes Isolation Model
https://cloudplatform.googleblog.com/2018/05/Exploring-container-security-Isolation-at-different-layers-of-the-Kubernetes-stack.html?m=1

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
Lifecycle of a Pod in
11(ish) steps
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
1. Developer uses
.yaml to create a
Rails pod using
kubectl
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
2. API Server
validates the request
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
3. The desired state
is persisted to etcd
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
4. etcd responds to
the API Server to
begin scheduling the
Rails pod
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
5. The Scheduler
decides where the
pod should run
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
6. API Server invokes
the kubelet on the
corresponding node
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
7. Kubelet tells the
container runtime
(Docker) to create
the container
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
8. The image is
pulled from an
external repo
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
9. Kubelet informs
the API Server of the
pod status
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
10. etcd is now
updated
etcd

kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB
11.
ensures desired
state is reality
etcd

Lab 3
003-K8S-Cluster-Setup
Lab Goals:
• Launch API in cluster
• Exec to container
• Expose via LoadBalancer
• Using YAML manifests for
deployment

Kubernetes Security Model
151

• The REST API is the
fundamental fabric of
Kubernetes
• All operations and
communications between
components, and external
user commands are REST
API calls that the API
Server handles
• Everything in the
Kubernetes platform is
treated as an API object
and has a corresponding
entry in the API
master
API
Server
Scheduler
Controller
Manager
etcd

apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they
access?)
Admission
Control
(Which
policies are
applied for
this user?
Access
Granted
https://
K8S Security Model

Transport Security
- K8S API typically serves traffic over
TLS
- Self-Signed Cert provisioned on
operators laptop in $USER/.kube/config
apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they
access?)
Admission
Control
(Which
policies are
applied for
this user?
Access
Granted
https://

Authentication
- Supports many authentication modules:
HTTP Basic, OpenID, Tokens, Client Cert,
Keystone
- Multiple modules can be specified
apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they
access?)
Admission
Control
(Which
policies are
applied for
this user?
Access
Granted
https://

Authorization
- Every HTTP request is authorized
get, list, create, update, etc.
- Request attributes are checked against
policy
apiserver
Authentication
(Who can
access the
cluster?
Authorization
(What can
they
access?)
Admission
Control
(Which
policies are
applied for
this user?
Access
Granted
https://

Authorization
--authorization-mode=AlwaysAllow allows all
requests; use if you don’t need authorization
--authorization-mode=ABAC allows for a simple local-
file-based user-configured authorization policy
--authorization-mode=RBAC allows for authorization
to be driven by the Kubernetes API

Admission Controllers
- Intercept requests prior to object creation
- May mutate incoming request to apply
system defaults
apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they
access?)
Admission
Control
(Which
policies are
applied for
this user?
Access
Granted
https://

Admission Controllers
AlwaysPullImages
DenyEscalatingExec
ResourceQuota
NamespaceExists
http://kubernetes.io/docs/admin/admission-controllers/

Kubernetes Authentication and
Authorization
160

In the beginning…
 ABAC was available as a form of enforcing
authentication and authorization
 Difficult to understand
 Most users went full YOLO as cluster admin
 RBAC was promoted to Beta in 1.6
 Broke stuff
 Now we’re here

User Management
 Kubernetes has no “User” object
– AKA no “kubectl create user jboss”
 It is up to the administrator to create users
–Certificate-based Auth (K8s has a CA)
–Token-based Auth
–Basic Auth
–OAuth
Can combine authentication methods

RBAC – The Key Players
Users
you@email.com
Service account
API Resources
Namespaces
Pod
Service
Secrets
…
Operations
Get
List
Delete
Patch

RBAC – Roles
API Resources
Namespaces
Pod
Service
Secrets
Operations
Get
Delete
List
Create
Goal: Establish a set of rules for a set of resources in a particular namespace
namespace = “development”

RBAC – Roles
API Resources
Namespaces
Pod
Service
Secrets
Operations
Get
Delete
List
Create
A role called “pod-reader” may only allow Get/List operations on pods in this namespace

role.yaml
kind: Role
apiVersion:
rbac.authorization.k8s.io/v1
metadata:
namespace: development
name: pod-reader
rules:
- apiGroups: [""]
resources: [”pods"]
verbs: ["get", "list"]

RBAC – Roles
API Resources
Namespaces
Pod
Service
Secrets
Operations
Get
Delete
List
Create
A role called “dev-admin” may allow all operations on all resources in this namespace

role.yaml
kind: Role
apiVersion:
metadata:
name: dev-admin
rules:
- apiGroups: [”*"]
resources: [”*"]
verbs: [”*"]

RBAC – ClusterRoles
API Resources
Namespaces
Pod
Service
Secrets
Operations
Get
Delete
List
Create
Cluster = myawesomek8scluster
Goal: Establish a set of rules for a set of resources in an entire cluster

cr.yaml
kind: ClusterRole
apiVersion:
metadata:
name: pod-reader
rules:
- apiGroups: [""]
resources: [”pods"]
verbs: ["get", "list"]

RBAC – RoleBinding
Roles
pod-reader
list-secrets
dev-admin
Goal: Tie a role to a subject (user) or set of subjects
User: jboss@manicode.com
Group: intern
User: jim@manicode.com
Group: dev-admin
Namespace = development

rb.yaml
kind: RoleBinding
apiVersion:
metadata:
name: read-pods
subjects:
- kind: User
name: jboss@manicode.com
apiGroup:rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader

RBAC – ClusterRoleBinding
Roles
pod-reader
list-secrets
dev-admin
Goal: Tie a role to a subject (user) or set of subjects
User: jboss@manicode.com
Group: intern
User: jim@manicode.com
Group: dev-admin
cluster = myawesomek8scluster

crb.yaml
kind: ClusterRoleBinding
apiVersion:
metadata:
name: read-pods
subjects:
- kind: User
name: jboss@manicode.com
roleRef:
kind: ClusterRole
name: pod-reader

Default RBAC Policies – Discovery Roles
Kubernetes comes with a ClusterRoles and
ClusterRoleBindings out of the box…

Default RBAC Policies – User Facing Roles

Authentication - Service Accounts
#KubernetesSecurityTip: If your pod doesn't need API access
set `AutomountServiceAccountToken: false` for an extra layer of
defense. Especially if you are running an old cluster!
 Every Pod has the directory
/var/run/secrets/kubernetes.io/serviceaccount auto-
mounted unless told otherwise
 With RBAC, the default service account has no
permissions that you don’t grant it
 Be very careful granting permissions to service accounts!

Lab 4
004-K8S-Cluster-Authentication
Lab Goals:
• Build RBAC Policy
• Restrict user Permissions
• Verify RBAC Enforcement

Attacking and Defending Kubernetes
179

Let’s Play a Game - Kubernetes Threat Model
180

Kubernetes Threat Model
181
User Compromise and Insider Threats
• Cluster admin account compromise
• Rogue Employee
• Tenant account compromise leads to the application compromise
Application Vulnerabilities
• Lack of authentication and authorization, both k8s internal and
external
• Weak or incorrect usage of cryptography
• Application UI and API vulnerabilities - remote code execution
(RCE), web vulnerabilities (XSS, CSRF, SSRF, SQL Injection etc.)
• Unsecured third party components accessible through APIs

182
Network and Infrastructure
• Network snooping, ARP spoof attacks
• Compromising infrastructure services (etc. NTP, DNS, SSH)
• Kernel and other operating system vulnerabilities
Application Containers
• Container breakout and unauthorized access control plane and other
containers
• Denial of Service - resource hogging, eating up CPU/Mem/Disk/IO to
impact or even crash other containers
• Compromised or malicious image or pipeline

183
Misconfiguration
• Insecure default configurations - unused open ports, services, not
enforcing system/application limits, failing to implement security
features
• Misuse of passwords, passphrases, TLS private keys (*cough*
checking them into git *cough*. Bad handling include key reuse,
insecure handling of keys, no key rotation, weak passwords, not
using MFA etc.
• Lack of network segmentation - exposing critical systems to various
network attacks

O boy.
186

Attack: Unauthorized Dashboard Access

Defense: Unauthorized Dashboard Access
 Always run RBAC on your cluster
 By default, the Dashboard ServiceAccount has very
limited privileges. Do not grant the Kubernetes dashboard
service account elevated privileges such as root!
 If access is needed, create SAs per user with limited
permissions
Don’t expose to the internet
Don’t be Tesla

Lab 5
5-K8S-Dashboard
Lab Goals:
• Explore K8S Dashboard

Attack: Elevated Pod Privileges
 Pods may be deployed with containers that require
elevated privileges:
– “privileged mode” grants containers the ability to
manipulate the network stack or access devices
– Containers may run as root (User ID = 0)
– Containers may request to mount sensitive volumes or
request write access to volumes
– Containers may request to bind to host ports
– Containers may request elevated Linux capabilities
Compromised containers can take full advantage of these
privileges to attack the cluster and cloud infrastructure

Pod Security Context
 Pod security context is defined in the pod or deployment
manifest
Defines the the privilege and access control for a pod
The security context defined in a pod applies to all containers
within the pod
Examples include:
Enabling seccomp, SELinux, or AppArmor
Defining users and groups containers use to run
Disallowing privilege escalation on containers
Whitelisting certain Linux privileges to the container

priv-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: priv-pod
spec:
securityContext:
privileged: true
containers:
- name: pause
image: k8s.gcr.io/pause
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME"]

Defense 1: gVisor
#KubernetesSecurityTip: When running untrusted workloads in a
cluster, consider implementing strong sandboxing techniques such
as gVisor or Kata containers to prevent escalation attacks.
 gVisor provides a strong
isolation boundary by
intercepting application system
calls and acting as the guest
kernel, all while running in user-
space

Defense 2: Pod Security Policies
 Pod security policies are represented by the
PodSecurityPolicy resource
Defines conditions a pod must meet to be scheduled
Examples include:
Disallow privileged containers from running
Disallow containers that require root privileges
Disallow containers that access certain volume types
Disallow containers that access certain host ports
#KubernetesSecurityTip: Use the PodSecurityPolicy admission
controller to restrict the use of privileged pods in your cluster

psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: my-psp
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: 'MustRunAsNonRoot'
volumes:
- 'configMap'
- 'emptyDir’
- 'secret’
- 'persistentVolumeClaim'

Lab 6
6-Pod-Security-Policy
Lab Goals:
• Block a privileged pod
from launching using
PodSecurityPolicy

Attack: Unauthorized Network Access
#KubernetesSecurityTip: Third-party technologies such as Istio
and Linkerd offer proxy services or ”sidecar” containers which can
help deploy mTLS / proxying throughout your cluster
If you run an API endpoint in your cluster such as Redis
without authentication, other pods may have unrestricted
access to the pod
A compromised pod may be able to read, alter, or delete
data from another pod in the cluster
It is important to isolate these workloads using granular
Network Policies as well as mTLS where appropriate

Attack: Unauthorized Network Access

Defense: Network Policies
The Kubernetes object NetworkPolicy allows you to block
traffic to pods
Acts as a ”pod firewall” where rules are administered by
cluster admins
Best practice is to start with a default “deny all” and only
add what you need
Default Deny – You must build the whitelist

Defense: Network Policies
 Networking Policies are implemented using network
plugins
 Typically need to install an “overlay network” in order to
enforce Network Policies
 Several exist, GKE has beta support for Networking
Policies out of the box (uses Calico)

np-deny-all.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: web-deny-all
spec:
podSelector:
matchLabels:
app: web
ingress: []
https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/01-deny-all-traffic-to-an-application.md

np-limit-traffic.yaml
kind: NetworkPolicy
metadata:
name: api-allow
spec:
podSelector:
matchLabels:
app: bookstore
role: api
ingress:
- from:
- podSelector:
matchLabels:
app: bookstore

limit-egress.yaml
kind: NetworkPolicy
metadata:
name: limit-egress
spec:
podSelector:
matchLabels:
app: foo
policyTypes:
- Egress
egress:
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
- to:
- namespaceSelector: {}

Defense: Istio Service Mesh
 Istio is a service mesh for
microservices (not just Kubernetes)
Offers:
– Monitoring
– Metrics
– Traffic Management and Routing
– Security
– Tracing

Defense: Istio Service Mesh

Istio: Envoy Proxy
 High performance load balancer
 Config management via API
 L7 Visibility
 Rate-limiting, health checks,
retries, etc.
 In Kubernetes…
– Envoy container is injected as
a “sidecar” container
– Controls pod ingress / egress
routing
– Config is via Pilot

Istio: Pilot
 Control plane for distributed
Envoy instances
 Configures Istio configurations
and pushes to other system
components
 System of record for the
service mesh
 Exposes API for service
discovery, load balancing, etc.

Istio: Mixer
 Responsible for providing
policy controls
 Handles telemetry
collection (Grafana,
Prometheus)
 Envoy sidecar calls Mixer
before each request to
perform precondition
checks and report
telemetry

Lab 7
007-Istio
Demo Goals:
• Deploy Istio in our Cluster
• Apply Egress policy
Maybe got time for that?

Access Cloud Metadata
#KubernetesSecurityTip: Use a tool like KIAM or Kube2IAM to
limit access to the AWS Metadata API. Better yet, apply a
NetworkPolicy to stop traffic outbound.
 Simple SSRF can lead to Cloud Metadata leak
 Using curl we can hit the AWS Metadata API endpoint from
a pod and depending on the configuration, sensitive data
may be returned
 http://169.254.169.254/latest/meta-data/iam/security-
credentials/IAM_USER_ROLE_HERE

deny-
all.yaml
kind: NetworkPolicy
metadata:
name: deny-all
namespace: default
spec:
podSelector: {}
egress:
- to:
- podSelector:
matchLabels:
k8s-app: kube-dns
- ports:
- protocol: UDP
port: 53
policyTypes:
- Ingress
- Egress

Attack: Unprotected Kubelet API
#KubernetesSecurityTip: This is a big deal and is not trivial to
address. Some bootstrap tools enable certificate authentication
between the master and nodes by default. Some don’t. YMMV.
The Kubelet handles Master <-> Node communication
By default, the Kubelet API allows for unauthenticated
access to ports 10255 (read-only) and 10250 (read /
write)
If a user has network access to your nodes the
Kubelet API may be exposed

DEMO-Attacking-Kubelet
Demo Goals:
• Explore Kubelet
• SSH to K8s master
• Access sensitive pod data
• Patch cluster

Kubernetes Secrets
214

Kubernetes Secrets
215
Kubernetes Secret objects are designed to store small
amounts of sensitive data such as API keys, tokens, or
passwords
Secrets are only sent to a node if a pod on that node
requires it
Secrets may be exposed to a Pod as a mounted volume
or as an Environment Variable

Kubernetes Secrets
216
Secret data on nodes is stored in tmpfs volumes and not
stored at rest on disk (technically)
Communication between api-server to Kubelet is
encrypted with TLS
Secrets are tied to a particular namespace and must be
encoded using base64

Kubernetes Secrets Risks
217
Secrets are stored in plain text by default in etcd
Very little separation of duties
During etcd replication, secrets are sent in plaintext
People still love pushing secrets to version control
Modifying secrets requires rolling out new objects

Lab 8
008-K8S-Cluster-Secrets
Lab Goals:
• Launch a MySQL
Deployment
• Use Secrets for the
MySQL password

Logging and Monitoring in Kubernetes
219

Application Logging
 Kubernetes offers a very flexible approach to
application logging
 Important to ship logs outside of the cluster as well as
the cloud infrastructure
 kubectl logs works for the command line
 stdout and stderr are supported out of the box

Sidecar Logging Container
 Sidecar listens to stdout and stderr and makes
decisions on where to ship logs

Kubernetes Audit
 A new feature to Kubernetes, auditing provides a
detailed sequence of events that have affected the
Kubernetes cluster
 Performed by the kube-apiserver
 Logs can be (and should be) shipped to a central
logging mechanism for storage
 Configurable by using an audit policy file

audit.yaml
apiVersion: audit.k8s.io/v1beta1 #
kind: Policy
rules:
# Log configmap and secret changes in all
other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps”]

Lab 9
009-Kube-Logs
Lab Goals:
• Explore Kubernetes Audit
• Trigger a log entry in GKE
• View Audit log details

DevSecOps Pipelines
225

Key Goals of AppSec Pipelines
 Optimize the critical resource – AppSec personnel
 Automate all the things that don’t require a human
brain
 Drive up consistency
 Increase tracking of work status
 Increase flow through the system
 Increase visibility and metrics
 Reduce any dev team friction with application security

Pipeline Security
228
Code Committed
Continuous
Integration
Repository
Peer review
Automated test suite
Build
Automation
Continuous
Deployment
Infrastructure QA Testing
(performance, load, etc.)
Configuration management,
artifact creation, db
migrations, etc.
Production
IaaS, PaaS,
On-Prem, etc.
Post-deploy
checks
Monitoring
and alerting

Development (Pre-Commit)
229
Developer laptops are the first line of
defense in a DevSecOps pipeline
Moving security to the left prevents
costly mistakes and vulnerabilities later
Required Git pre-commit hooks can
offer a simple, effective feedback loop
–Static analysis scans in the IDE
–Peer review from security engineers
–Lightweight, threat modeling in
sensitive areas
Code Committed

Git-Secrets
230
https://github.com/awslabs/git-secrets

Brakeman Static Scan Demo
231

Continuous Integration (Commit Stage)
233
Repository
Peer review
Automated test suite
Basic automated testing is performed
after a commit is made
Must be quick and offer instant
feedback
Key place to include security checks
that run in parallel with integration
tests, unit tests, etc.
– Identify risk in third-party components
– Incremental static security scanning
– Alerting on changes to high-risk areas
– Digital signatures for binaries

Continuous Integration (Commit Stage)
234
CI server may include a dedicated
security worker
Third-party dependency checking
performed in CI
–OWASP Dependency Check
–Node Security Project
–Bundler-Audit
–SRC:CLR
Custom alerts set on repositories and
sent to “on-call” security teams
– Is someone changing pw hashing algorithm?
– Is a new password policy enabled?

Continuous Deployment (Acceptance)
235
Build
Automation
Infrastructure QA Testing
(performance, load, etc.)
Configuration management,
artifact creation, db
migrations, etc.
Triggered by successful commit and
passing build
Utilize parallel, out-of-band processes
for heavyweight security tasks
IaaS and Config Management should
provision latest, known-good
environment state (as close to
production as possible)
Security checks during acceptance:
– Comprehensive fuzzing
– Dynamic Scanning (DAST)
– Deep static analysis
– Manual security testing

Zap Baseline scan incorporated into CI
stage of the deployment pipeline
Runs a basic scan scan from a simple
Docker run command
By default will output all results of
passive scan rules
Highly configurable but still struggles in
certain areas
https://github.com/zaproxy/community-scripts/tree/master/api/mass-baseline
Continuous Deployment (Acceptance)

Production (Post-Deployment)
237
After all security checks have passed
and deployment is complete
Security teams job does not stop here:
–Monitoring and Alerting
–Runtime Defense (RASP)
–Red Teaming
–Bug Bounties
–External Assessments
–Web Application Firewalls
–Vulnerability Management
IaaS, PaaS,
On-Prem, etc.
Post-deploy
checks

Lab 10
010-Security-Pipeline
Lab Goals:
• Launch Jenkins in your
cluster
• Launch a private Docker
registry
• Scan images for vulns

JIM MANICO Secure Coding Instructor www.manicode.com
It's been a pleasure.
jmesta@manicode.com

OWASP Bay Area Meetup - DevSecOps the Kubernetes Way

More Related Content

What's hot

Similar to OWASP Bay Area Meetup - DevSecOps the Kubernetes Way

Recently uploaded

OWASP Bay Area Meetup - DevSecOps the Kubernetes Way