AppSec USA 2016 talk on using containers and Kubernetes to manage a variety of security tools. Includes best practices for securing Kubernetes implementations.

1. Containerizing your SOC

2. @jimmesta
OWASP Santa Barbara Founder
AppSec California Organizer
Works at Invoca
Was consulting, now “defensing”
I really like containers

3. Greetings from
Sunny AppSec
California!
Location: Santa Monica, Ca.
Date: January 23-25, 2017
Why: Because Winter
2017.appseccalifornia.org

5. Security Confessions:
A Time for Healing

6. What are employees saying
about your security program?

7. Security as a [ ]

8. Security as a Service?

9. Security as a Magic Unicorn?

10. Security as a Bottleneck

11. Security as a Black Hole

12. Security as a “No” Machine

13. Security as a Hot Potato

14. Security as a PDF Generator

15. We are all under-staffed

16. We are all over budget

17. We are all too busy

18. Can we DevSecOps our way
out of this?

21. Step 1: Install XCode
Command Line Tools
xcode-select --install
Nice! It looks like I get to compile
some stuff.

22. Step 2: Make sure Java
is updated
Dang. My Java is out of whack.
What did I do? I’ll just update…
java --version

23. 20 minutes later...

24. Step 3: Install Homebrew
But I use Macports and ZSH.. where’s
my .bash_profile?
source ~/.bash_profile
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/.../)"
brew tap homebrew/versions

25. Step 4: Update $PATH
and Install Dependencies
Wait a minute. I need a local
Postgres DB to run this thing?
echo PATH=/usr/local/bin:/usr/local/sbin:$PATH >> ~/.bash_profile
brew install nmap && brew install postgresql

26. Step 5: Initialize the DB
What?! Postgres didn’t initialize?
Forget this. Hacking is hard.
cp /user/local/Cellar/postgresql/9.4.0/.../...
initdb /usr/local/var/postgres
launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

27. You just lost a golden
opportunity to foster a
co-worker's interest in
security.

28. How can we make our
security tooling more about
using the tool and less
about maintenance?

29. whitesourcesoftware.com

31. Docker is an Open Source
engine to pack, ship, and run
any application as a lightweight
container.

32. Why U No Just
Virtualbox!?

33. Traditional Virtual Machines
Source: https://www.docker.com/what-docker

34. Docker Infrastructure
Source: https://www.docker.com/what-docker

36. Docker provides a user friendly
API to create containers.

37. Images use layers for efficiency
and speed.

38. Build the Docker image once and
use it all over the place.

39. Minimize concerns around
compatibility and dependencies.

40. So what about our “Securious”
Dev who just wanted to run
Metasploit?

41. Step 1: Install Docker
Nice! That was super
easy to point and click.

42. Step 2: Run Command
That just..worked?
docker run -t -i linuxkonsult/kali-metasploit

43. Step 3: Profit

44. FROM linuxkonsult/kali
MAINTAINER Tom Eklöf "tom@linux-konsult.com"
ENV DEBIAN_FRONTEND noninteractive
ADD ./init.sh /init.sh
RUN apt-get -y update ; apt-get -y --force-yes
install ruby metasploit-framework
CMD /init.sh
Dockerfile

45. #!/bin/bash
/usr/share/metasploit-framework/msfupdate
/usr/share/metasploit-framework/msfconsole
init.sh

46. What could go wrong with
blindly pulling Docker images?

48. Always inspect and vet the
Dockerfile before blindly using it.

49. Tools like Clair, Quay, and Anchore
can help identify vulnerabilities in
images.

51. https://github.com/zaproxy/zaproxy/wiki/Docker
https://hub.docker.com/r/jmbmxer/threadfix/
https://github.com/blacktop/docker-cuckoo
https://github.com/wazuh/docker-ossec-elk
Security is Catching on

52. docker run is useful and all but
how do we get these containers out to
the world for others to use?

54. K8S - A Gentle Introduction

55. Kubernetes is an open-source
platform built to automate
deployment, scaling and
orchestration of containers.

56. K8S is portable. Clusters can be
deployed on a public/private
cloud, on prem, and even on
your laptop.

57. K8S is customizable. It is
modular and extensible to fit
nearly any use-case.

58. K8S is scalable. It provides
self-healing, auto scaling, and
replication.

59. There are others!

60. - Don’t orchestrate for the
sake of orchestration (or
because the cool kids are
doing it)
- Containers first, then
orchestration
- docker-compose does a
fine job for many things

61. Core Concepts

62. https://flic.kr/p/bNpyRp

63. cluster
virtual machines
that Kubernetes
manages

64. clusternodemaster node node

65. clusternodemaster node node
node node node
node node node
node node node
node node
node node node
node node nodemaster
master
node

66. pod
group of
containers
sharing storage
and network

67. podcontainer container container
volume A volume B
network interface

68. pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: owasp-app
spec:
containers:
- name: owasp-app
image: owasp-app
- name: nginx-ssl
image: nginx
ports:
- containerPort: 80
- containerPort: 443

69. pod.yamlnodemaster node node

70. pod.yamlnodemaster node node

71. pod.yamlnodemaster node node

72. deployment
ensure N pods
are up and
running

73. deploy.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: frontend
spec:
replicas: 4
selector:
role: web
template:
metadata:
name: web
labels:
role: web
spec:
containers:
- name: owasp-app
image: owasp-app
- name: nginx-ssl
image: nginx
ports:
- containerPort: 80
- containerPort: 443

74. deploy.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: frontend
spec:
replicas: 4
selector:
role: web
template:
metadata:
name: web
labels:
role: web
spec:
containers:
- name: owasp-app
image: owasp-app
- name: nginx-ssl
image: nginx
ports:
- containerPort: 80
- containerPort: 443

75. deploy.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: frontend
spec:
replicas: 4
selector:
role: web
template:
metadata:
name: web
labels:
role: web
spec:
containers:
- name: owasp-app
image: owasp-app
- name: nginx-ssl
image: nginx
ports:
- containerPort: 80
- containerPort: 443

76. deploy.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: frontend
spec:
replicas: 4
selector:
role: web
template:
metadata:
name: web
labels:
role: web
spec:
containers:
- name: owasp-app
image: owasp-app
- name: nginx-ssl
image: nginx
ports:
- containerPort: 80
- containerPort: 443

77. deploy.yamlnodemaster node node

78. deploy.yamlnodemaster node node

79. deploy.yamlnodemaster node node
10.0.0.1 10.0.0.2
10.0.0.3
10.0.0.4

80. service
abstraction layer
that enables pod
communication

81. servicenodemaster node node
10.0.0.1 10.0.0.2
10.0.0.3
10.0.0.4

82. servicemaster
service

83. servicemaster
service
service

84. servicemaster
service
service
public load balancer
tools.beardsec.com

85. svc.yaml
kind: Service
apiVersion: v1
metadata:
name: web-frontend
spec:
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
selector:
role: web
type: LoadBalancer

86. svc.yaml
kind: Service
apiVersion: v1
metadata:
name: web-frontend
spec:
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
selector:
role: web
type: LoadBalancer

87. svc.yaml
kind: Service
apiVersion: v1
metadata:
name: web-frontend
spec:
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
selector:
role: web
type: LoadBalancer

88. svc.yaml
kind: Service
apiVersion: v1
metadata:
name: web-frontend
spec:
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
selector:
role: web
type: LoadBalancer

89. namespace
manage different
environments in
the same cluster

90. ns.yaml
kind: Namespace
apiVersion: v1
metadata:
name: sec-tools

91. kubectl
master apiserver
HTTPS
scheduler
Replication
controller
node
node kubelet
pod pod pod
proxy External LB

92. Sounds great! What about
security?

93. apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they access?)
Admission
Control
(Which policies
are applied for
this user?
Access
Granted
https://
K8S Security Model

94. - K8S API typically serves traffic over TLS
- Self-Signed Cert provisioned on
operators laptop in $USER/.kube/config
Transport Security
apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they access?)
Admission
Control
(Which policies
are applied for
this user?
Access
Granted
https://

95. - Supports many authentication modules:
HTTP Basic, OpenID, Tokens, Client Cert, Keystone
- Multiple modules can be specified
Authentication
apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they access?)
Admission
Control
(Which policies
are applied for
this user?
Access
Granted
https://

96. - Every HTTP request is authorized
get, list, create, update, etc.
- Request attributes are checked against
policy
Authorization
apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they access?)
Admission
Control
(Which policies
are applied for
this user?
Access
Granted
https://

97. Authorization
--authorization-mode=AlwaysAllow allows all requests;
use if you don’t need authorization.
--authorization-mode=ABAC allows for a simple
local-file-based user-configured authorization policy.
--authorization-mode=RBAC is an experimental
implementation which allows for authorization to be driven by the
Kubernetes API.

98. Role Resource Types
Role
ClusterRole
RoleBinding
ClusterRoleBinding
http://kubernetes.io/docs/admin/authorization/

99. rb.yaml
kind: RoleBinding
apiVersion:
rbac.authorization.k8s.io/v1alpha1
metadata:
name: read-pods
namespace: sec-tools
subjects:
- kind: User
name: jimmy
roleRef:
kind: Role
namespace: sec-tools
name: pod-reader
apiVersion:
rbac.authorization.k8s.io/v1alpha1

100. - Intercept requests prior to object
creation
- May mutate incoming request to apply
system defaults
Admission Controllers
apiserver
Authentication
(Who can
access the
cluster?
kubectl
Authorization
(What can
they access?)
Admission
Control
(Which policies
are applied for
this user?
Access
Granted
https://

101. Admission Controllers
AlwaysPullImages
DenyEscalatingExec
ResourceQuota
http://kubernetes.io/docs/admin/admission-controllers/

102. Secrets Everywhere!

103. K8S Secret Object
- Secrets can only be accessed by pods in
the same namespace
- Secrets are only sent to nodes with pods
that require it
- Not written to disk - stored on tmpfs
- Deleted once dependent pod is removed

104. Buyer Beware
- Secrets are stored in plaintext on the
apiserver (etcd)
- Protect etcd with your life
- Don’t forget what OWASP taught you!
- Secrets in logs, app security, etc.
- Anyone with root on any node can read
secrets by impersonating kubelet

105. Vault
- It works! But no official K8S support
(yet)
- API driven, do what you will
- Customize your deployment

106. #!/bin/bash
PASSWORD="$(vault read -field=value secret/password | base64)"
# Create YAML object from stdin
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
password: "${PASSWORD}"
EOF
```

107. secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: owasp-secrets
type: Opaque
data:
username:d293IHlvdSBkZWNvZGVkIGl0
password: Z29vZCBmb3IgeW91
host:bm90aGluZyBqdWljeSB0aG91Z2g=

108. deploy.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: frontend
spec:
replicas: 4
selector:
role: web
template:
metadata:
name: web
labels:
role: web
spec:
containers:
- name: owasp-app
image: owasp-app
env:
- name: OWASP_PASS
valueFrom:
secretKeyRef:
name: owasp-secrets
key: password
ports:
- containerPort: 443

109. Security Hygiene
- Restrict SSH access to nodes
- Only use trusted images
- Regularly apply updates to your K8S
environment (including kubectl)
- Log all of the things
- Apply SecurityContext to deployments
runAsNonRoot, readOnlyRootFilesystem

110. Demo (sort of)
- 2 node cluster running on GCE
- Kubernetes 1.4

125. - Maintain one K8S cluster
- Deploy and scale security tooling
- DevSecOps all the things
- We are part of this container journey
together
Security can be an enabler

126. Resources
Kubernetes Bootcamp
CloudSOC
Minikube

127. Questions?