Everyone has been told not to run with scissors. Doing so makes one highly exposed to serious damage. Both containers and Kubernetes define a bajillion different toggles how to configure the applications. Rather than using all the proper toggles, developers often run things in containers and in Kubernetes just using the plain defaults. That leaves many capabilities lurking in the applications that just wait to be exploited. This session is highly inspired by Liz Rice’s talk at KubeCon EU 2018, “Running with scissors”. My session will focus on a different angle: how to take the scissors away from the developers so that they do not harm themselves. In this talk, we’ll look at some of the concepts of forcing security of the application workloads both from conceptual and practical points of view. We’ll look at things like security policies, resource quotas, and pod security contexts. We’ll also discuss what they mean for the applications developers are pushing to the Kubernetes cluster.

1. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Taking the scissors away
Make your K8S cluster safe for DevOps

2. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Well, safer at least

3. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Latest CPU level attacs
#ZombieLoad
MDSAttacs
RIDL
Fallout

4. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps Photo by Randy Fath on Unsplash

5. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Me, myself and I
Jussi NummelinName
Current Work
@JNummelin / jnummelinTwitter / Github
Developer / Advocate @ Kontena, Inc.
Previous Tecnotree, Digia, Tieto, Nokia, …
Bio All-around handyman on technical topics
Working with containers & microservices
for ~5 years
Avid fly-fisher
Hockey dad

6. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Importance of releasing often
Go faster than competition
Learn fast
Adapt

7. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Topics
Scissors, scissors everywhere
Securing the cluster kernel
RBAC, the pains and gains
Enforcing workload configuration
No one likes hoarders

8. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
https://www.youtube.com/watch?v=ltrV-Qmh3oY

9. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps

10. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps https://commons.wikimedia.org/wiki/File:Kookie_Studio_Mixer.JPG

11. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
User errors
Bazillion knobs in K8S
Easy to leave open doors
Pods not safe by default

12. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Root vs. non-root
86% of images in Docker Hub use root
Root is root
Userns remap not available in K8S

13. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Capabilities
Fine grained permission checks
Lot of default caps
Privileged == all caps

14. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps

15. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps https://www.flickr.com/photos/jdickert/1270880225

16. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps By Source (WP:NFCC#4), Fair use, https://en.wikipedia.org/w/index.php?curid=19565850

17. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Securing cluster kernel
Use benchmark tools
Lock all doors by default
Auth complexity

18. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Things to watch out
Auth configured properly
Everything has proper TLS
No exposed APIs
...

19. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Audit
API server does auditing
Make sure you store them
Can be pushed to webhook too

20. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
RBAC

21. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
RBAC
Who can do what?
Super fine grained
Scoped

22. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps

23. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
RBAC - Pains
Fine grained controls
Sea of YAML
What can I do?

24. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
RBAC - Helpers
RBAC Manager
kubectl-who-can

25. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Ensuring workload safety

26. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
No root allowed

27. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
No root in images
Requires effort
Easy for greenfield

28. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Pod Security Policy
Control for Pod security aspects
An Admission Controller
Enforcer
Can also set some defaults

29. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps

30. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
PSP - Pains
Easy to cap your cluster
Enforced policy selected via RBAC
When did you last create a pod?

31. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps

32. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
No one likes hoarders

33. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps https://www.flickr.com/photos/71622328@N08/36359861566

34. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Resource Quota
Limit aggregate resource consumption
Scoped per namespace
YAAC - YetAnotherAdmissionController

35. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Limits on
CPU
MEM
Storage
Object counts

36. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps

37. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps

38. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Putting all this together

39. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Secure cluster kernel
Auth
RBAC
ALL components secured

40. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Namespaces FTW, ensure:
Resource Quotas
PSP setup properly
Network Policies
LimitRanges

41. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Customize K8S
Custom operators
Operator SDK
Custom admission webhooks

42. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Operator Framework
Operator running in the cluster
Reacts to changes of specific objects
Can setup ”adjacent” resources
Namespace created à configure it

43. www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps www.kontena.ioJAX DevOps 2019 // Removing the scissors - Making K8S safe for DevOps
Thank you!