My Talk at OWASP Appsec USA 2016

1. Breaking && Securing your ‘Docker’ized environments
Manideep K

2. Shameless Bragging
• Masters Student + Security Researcher at “Carnegie Mellon”
• Authored a book on Info Sec & Ethical Hacking at the age of 20
• Featured in INDIA’s largest news papers and news channels
• Trained 15,000+ people in Infosec including corporates, students & cyber cops
• 10 certifications : ISO 27001:2013 ISMS LA, CCNA, CEH, JNCIP-SEC etc.
• Ex Team Lead – Core Security & Data Analytics at TCS
• Interest areas : Container Security, Application Security etc.
More details on “www.manideepk.com”
2

3. Experience with Docker containers
• Research at Carnegie Mellon University under renowned professors
• Cloud Security Research Intern @Adobe
• Co-Authored CIS Docker 1.12 Benchmark (You will see points from it today)
• Research Intern at Carnegie Mellon
3

4. Agenda
 Introduction
- One minute quick dive on containers
- Container environment
- Risk Areas
 Breaking && Securing
- Images
- Container runtime
- Daemon
- Host
- Communication and Registry security (if time permits)
 Holistic view of entire container pipeline
 Few tips for enterprises
 Future
 Conclusion
 References
 Q&A
4

5. Agenda
 Introduction
- One minute quick dive on containers
- Container environment
- Risk Areas
 Breaking && Securing
- Images
- Container runtime
- Daemon
- Host
 Holistic view of entire container pipeline
 Few tips for enterprises
 Future
 Conclusion
 References
 Q&A
5

6. Containers?
 Lightweight
 Application centric
 No more - “it works on my machine”
 Micro-services
Namespaces
Isolates containers (PID, User, Network, IPC, Mount ,
UTS)
Cgroups
Limits, accounts and isolates resource usage (CPU,
memory etc.)
BUZZ……….!
Are containers brand
new?
Img Ref: www.docker.com
6

7. Container environment (Docker)
Img Ref: Modified version of
image on www.docker.com
7

8. 8
Risk Areas
Images
Container Runtime
Daemon
Host System
Kernel/ Containers Isolation
(Shared kernel)
Communication

9. Container pipeline (Docker)
Client <=> daemon
communication
Communication with public/private registry
Registry’s security
Host security
Daemon security
Containers Images
9

10. Agenda
 Introduction
- One minute quick dive on containers
- Container environment
- Risk Areas
 Breaking && Securing
- Images
- Container runtime
- Daemon
- Host
 Holistic view of entire container pipeline
 Few tips for enterprises
 Future
 Conclusion
 References
 Q&A
10

11. 11
Containers do not contain
53% of decision makers are worried about security of containers*
Containers are not production-ready
Container Security (Docker) developed “a lot” in the past two years, is still
developing and has lot of scope
Docker containers are now “production-ready**” . Google spins up more than
2 billion containers per week
Containers are the “FUTURE”
* Forrester/Red Hat Report , January 2015 ** You have to make them secure

12. Docker says
At Docker we believe in “Secure by Default” *
NCC Group Report on containers says
Docker default security settings are quite good **
I say
(and even you will, by end of this presentation)
“Docker is not secure by default, you have to make it secure…!”
* https://blog.docker.com/2016/04/docker-security/
** https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/april/understanding-and-hardening-linux-
containers/
12

13. My approach for explanation
“Point” (on later slides)
What breaks it & How to secure it
Why this approach?
13

14. Breaking and Securing
“Images”
14

15. Image Layers – Copy on Write (CoW) model
Manifest file of an image inside registry
Image’s lifecycle
(Img Ref: www.docker.com)
Introduction to Images
15

16. • Where can I get Images?
- Docker hub , Docker Private Store
• Can I use them (directly)?
- No, Not Docker hub (at least general images) for enterprises (personal?)
- How about Docker Private store?
• Official images are scanned with Nautilus (general images are not)
• My analysis with images (sampling)
• Banyanops analysis* proved that Docker hub images are prone to severe vulnerabilities
such as Heartbleed, Shellshock etc. (2015 report)
Quick Facts from the report :
30% official images are vulnerable
70% general images are vulnerable
* : Below report was released in 2015. Though Docker initiated Nautilus after that and started mitigating the vulnerabilities in official images, it still has many
unpatched. Also, unofficial Docker hub images are still untouched and are left with lot of vulnerabilities. So, though the percentage of vulnerabilities might
not be exactly the same today, the essence of the report still remains the same**
** : You can open the official images on Docker hub and see the tags section of each image which shows the scan report of Nautilus for that particular image
Report Link: https://banyanops.com/blog/analyzing-docker-hub/
16

17. Lesson Learnt….!
Do Not directly use Docker hub images
(not atleast general images and not at-least for enterprise use)
17

18. Manideep, Then what to do?
Prepare your own Docker Images (Oh Gosh, Lot of security controls to follow….!)
• How to write Dockerfiles (any other similar mechanism such as Packr) securely?
• How to maintain/consume images securely?
18

19. Writing Dockerfiles securely (our “Points” are here….)
1) Follow version pinning mechanisms for base images, packages etc. (Caching issue)
CIS Docker 1.12– 5.27
2) Download packages securely - Ex: GPG (MITM issue) : CIS Docker 1.12 – 4.11
3) Create a user for containers (Priv issue / root issue) : CIS Docker 1.12 – 4.1
4) Do not use any kind of update instructions alone in the Dockerfile (Caching issue)
CIS Docker 1.12 – 4.7
5) Remove setuid, setgid permissions within the images (Priv Escalation) : CIS Docker 1.12 – 4.8
6) Do not install unnecessary packages within the images (Increased attack surface)
CIS Docker 1.12 – 4.3
7) Do not write secrets in Dockerfiles (Information Disclosure). Use Keywhiz, Vault etc.
CIS Docker 1.12 – 4.10
8) Use COPY instead of ADD (Increased attack surface) : CIS Docker 1.12 – 4.9
9) Use HEALTHCHECK command (Best practice) : CIS Docker 1.12 – 4.6
10) Use gosu instead of sudo wherever possible ( TTY and signal forwarding issue)
11) Do not prefer to run multiple applications within one container (Increased attack surface)
19

20. How to maintain / consume images securely?
1) Docker Content Trust (Enable it!)
• Based on Thandy framework and is little complex to understand
• Ensures integrity, authenticity and freshness guarantees to images (Without DCT?)
• Extensive analysis done (Seems secure…)
• Create your own Notary setup
2) Images should be “vulnerability-free” and should be “frequently” scanned with tools like
Twistlock, Nautilus etc. Tools should be capable of doing binary level analysis /hash based
match and just not version string matching.
3) All images / packages within images must be up-to-date (Except compatibility issues)
4) Follow all CIS Docker benchmark standards
20

21. Breaking and Securing
“Container runtime”
21

22. Container runtime security
1) Ensure Docker commands get latest images (Caching issue) : CIS Docker 1.12 – 5.27
2) Enable PIDs cgroup per container (fork bomb) : CIS Docker 1.12 - 5.28
3) Do not use default docker0 bridge (ARP spoofing, MITM etc.) : CIS Docker 1.12 - 5.29
4) Set memory limits per container (DOS) : CIS Docker 1.12 - 5.10
5) Do not use privileged containers (Violation of least privilege) : CIS Docker 1.12 - 5.4
6) Bind incoming traffic to any specific interface (Info disclosure etc.) : CIS Docker 1.12 - 5.13
7) Do not share host’s process, user, UTS, IPC, network namespaces to containers (Info
disclosure etc.)
8) Do not disable capabilities (significance of SYS_TIME, NET_RAW)
9) Prefer to set no new privileges flag (no runtime priv. gain atacks) : CIS Docker 1.12 – 5.25
10) Do not disable seccomp profiles (Container breakout) : CIS Docker 1.12 – 5.21
11) Set CPU shares appropriately (DOS) : CIS Docker 1.12 – 5.11
12) Use Apparmor / SELinux profiles (No access restriction) : CIS Docker 1.12 – 5.1, 5.2
13) Mounts (Violation of least privilege)
14) Beware of non namespaced components : kernel keyring etc. (Container breakout)
15) Adhere CIS Docker 1.12 benchmark
22

23. Breaking and Securing
“Daemon”
23

24. 1) Enable user namespaces
- When user namespaces are enabled (which are disabled by default) root inside the container
will be “no more” root on the host. (Without user namespaces?)
- Enable user namespaces when launching daemon with “- - userns-remap” flag
- Know about the restrictions (no read only filesystem for containers, no - -privileged flag etc.)
2) Disable communication between containers
- By default, all containers are on the same bridge and can communicate (intercept traffic)
- Add “- - icc = false” flag when launching daemon which would prevent container communication
- Use user-defined networks for allowing group of containers to communicate (DO NOT use legacy
container linking method as it leaks secrets stored in environment variables between containers)
3) Enable TLS & authentication for communication between daemon and client
- By default, communication between docker daemon and client are not encrypted and there is
no authentication between them.
- Enable TLS & CA authentication at daemon level which would prevent attacks such as MITM,
server or client impersonation etc.
Daemon Security
24

25. Daemon Security
4) Enable ulimits for your docker daemon (You can do it at runtime per container)
- ulimits are great way to restrict lot of resources such as CPU time limit, max file descriptors
that can be opened, max no of processes etc.
- By default, no ulimits are set . You can enable them by using flags.
Ex: “- -default-ulimit nproc=10”
5) Use authorization plugins
- By default, every user having access to daemon can run any Docker command. Auth plugins
provides granular control on “who runs what” at daemon level.
- You can enable auth plugins while launching daemon with the flag
“- -authorization-plugin=plugin1 –authorization-plugin=plugin2 …………….”
6) Ensure only trusted users will have access to docker daemon as all the members
within the docker group will have effective root privileges
25

26. Breaking and Securing
“The host where containers are hosted”
26

27. Security controls on the host
1) Maintain up-to-date Linux kernel
- They have all vulnerabilities fixed and latest security controls included
- Ex: PIDs cgroup (prevents fork bomb) is only available in kernels > 4.3
2) Apply kernel hardening policies such as GRSecurity, PaX etc.
- These are kernel patch sets which help in preventing multiple kind of memory, access control
exploits etc. These would prevent exploitation even though attacker was able to break out of
container. Also, ensure that these kernel patches do not interfere with your Docker setup (test
with sysctl and then lock down)
3) Harden the host as per your enterprise / CIS / SANS guidelines
4) Keep Docker always up-to-date
27

28. Container pipeline (Holistic View) – Communication & Registry Security
Client <=> daemon
communication
Communication with public/private registry
Registry’s security
Host security
Daemon security
Containers
Images
Img Ref: Modified version of image on www.docker.com
*
*
28

29. Agenda
 Introduction
- One minute quick dive on containers
- Container environment
- Risk Areas
 Breaking && Securing
- Images
- Container runtime
- Daemon
- Host
 Holistic view of entire container pipeline
 Few tips for enterprises
 Future
 Conclusion
 References
 Q&A
29

30. Few tips for enterprises
• Setup your own in-house registry having “secure and ready-to-consume” images
• Prepare customized auth plugins, seccomp, apparmor profiles as per your requirements
• Do not share secrets in environment variables
• Use tools like Twistlock, Scalock etc. to secure your container environments. These tools
provide capabilities like Image lockdown, RBAC, automatic profiling etc.
• Use TPM, TCB etc. in Docker environments (Add docker daemon to TCB)
• Use image optimization techniques (Do not compromise security!)
• Container specific rules to be created in SIEM
• Group containers on VM’s (basing on trust, operational categories)
• Employ least privilege, defense in depth, reduced attack surface principles
• Employ file integrity monitoring tools for critical files
• Prepare separate patch management, vulnerability management etc. procedures for
container environments
• Adhere to latest CIS Docker benchmark or prepare your own internal benchmarks
customizing standard CIS Docker benchmark as per your organizational requirements
30

31. Agenda
 Introduction
- One minute quick dive on containers
- Container environment
- Risk Areas
 Breaking && Securing
- Images
- Container runtime
- Daemon
- Host
 Holistic view of entire container pipeline
 Few tips for enterprises
 Future
 Conclusion
 References
 Q&A
31

32. Future (My 2¢)
• Hardware level Isolation ( Container’s performance + VM security )
- Clear Containers (QEMU+KVM : Very light weight VM ‘s. Though not up to Docker
performance, they are optimizing more).
- Has lot of scope. My area of work for next few days.
• No more shared kernel as in clear containers
• Trusted place for getting Images. Ex: Docker store
• Advancement in container security tools : Twistlock, Scalock are only tools which cover
entire container ecosystems today. They are adding more features
• In-built “proper” hardening of host when Docker installed (GRSEC, PaX etc.)
• Docker Images encryption – similar to VM (Most security controls of VM will be soon
shipped to container ecosystem)
• Portability between container technologies (lot of current work)
• Name spacing the missing components: Kernel keyring, SYS_TIME etc.
• Enhanced security controls on the daemon, images and container runtime
Ex: better default seccomp by in-built profiling etc.
32

33. My Next Plans
• Research on hardware level isolation to containers
• In-depth Research on other container technologies
• Whitepaper (or possibly a book? ) on Docker Security
• Continue contribution to CIS Docker Benchmark (Please join!)
• Will be presenting at RSA conference (poster under RSA Scholar Program), Grehack etc.
• Feel free to reach me if you have similar interests or want to collaborate
33

34. Conclusion
34

35. References
1. https://www.nccgroup.trust/globalassets/our-
research/us/whitepapers/2016/april/ncc_group_understanding_hardening_linux_containers-1-1pdf
2. http://www.oreilly.com/webops-perf/free/files/docker-security.pdf
3. http://container-solutions.com/content/uploads/2015/06/15.06.15_DockerCheatSheet_A2.pdf
4. CIS Docker Benchmark 1.12
5. http://csrc.nist.gov/news_events/cif_2015/research/day1_research_430-530.pdf
6. http://www.slideshare.net/SreenivasMakam/docker-security-overview?qid=49963984-be1a-4322-a3a8-
e5a2963127c0&v=&b=&from_search=1
7. http://www.slideshare.net/Docker/docker-security-workshop-slides
8. http://www.slideshare.net/Docker/securing-the-container-pipeline-at-salesforce-by-cem-gurkok-63493231
9. https://clearlinux.org/features/clear-containers
10. https://arxiv.org/pdf/1501.02967.pdf
11. http://www.slideshare.net/Docker/the-golden-ticket-docker-and-high-security-microservices-by-aaron-
grattafiori?qid=dd633f11-1c78-42ce-86f0-88068690a6ee&v=&b=&from_search=2
12. https://docs.docker.com/engine/security/
13. http://www.slideshare.net/Docker/docker-security-deep-dive-by-ying-li-and-david-lawrence
35

36. Special Thanks to
Adobe
36

37. Q & A time
37

38. Thank you…!
Reach me on www.manideepk.com for any questions
38