Recommended
More Related Content
What's hot
What's hot (20)
Similar to Container security Familiar problems in new technology
Similar to Container security Familiar problems in new technology (20)
More from Frank Victory
More from Frank Victory (12)
Recently uploaded
Recently uploaded (20)
Container security Familiar problems in new technology
- 1. Container Security: Familiar Problems in New Technology Sean Storer Product Marketing Manager
- 2. • Visibility • Vulnerabilities • Testing • Access Control • Organizational Challenges AGENDA 2
- 3. VISIBILITY 3
- 7. • CVE-2019-5736 – Doomsday runC • Container escape that affects the open source command line this vulnerability could allow an attacker-controlled container to gain root-level code execution to the Docker host by overwriting the runC binary • CVE-2019-1003065 – Docker Community Edition Trojan Horse • Docker Desktop Community Edition before 2.1.0.1 allows local users to gain privileges by placing a Trojan horse docker-credential-wincred.exe file in %PROGRAMDATA%DockerDesktopversion-bin as a low-privilege user, and then waiting for an admin or service user to authenticate with Docker, restart Docker, or run 'docker login' to force the command. • CVE-2019-11253 – Billion Laughs Attack • XML parser DoS vulnerability in the API server due to kube-api not performing input validation or putting size restriction on YAML files 7 CONTAINER & KUBERNETES SPECIFIC VULNERABILITIES
- 8. INSECURE DOCKER HUB IMAGES 15.9 40.5 Official Images Community Images Average number of vulnerabilities in Docker Hub1 Source: Tenable, “Sourcing Container Images from Docker Hosts,” 2017 8
- 9. • 0 days get press and hype • Cyber Hygiene is equally important 9 100+ DAY VULNERABILITIES
- 10. 10 TESTING
- 11. SHIFT LEFT WITH SECURITY 11 PREVENT PROTECTSecurity
- 12. IMAGE SCANNING 12 Layer 1 Layer 2 Layer 3 Layer 4 Container Image PRO TIP: BEWARE OF FALSE POSITIVES Higher container image layers often remediate vulnerabilities found in lower layers
- 13. SET RISK THRESHOLDS 13 Write container security policies that align to security goals Notify developers immediately when container images exceed organization risk thresholds RegistryTestBuild Source Control Build Container Unit Tests API Tests Security Tests Push to Registry ⛔️ ⛔️
- 15. • Subject • A subject represents a user, team, organization, or a service account. A subject can be granted a role that defines permitted operations against one or more resource sets. • Role • Roles define what operations can be done by whom. A role is a set of permitted operations against a type of resource, like a container or volume, which is assigned to a user or a team with a grant. • Grant • A grant is made up of a subject, a role, and a resource/resource set. • Grants are effectively Access Control Lists (ACLs) which provide comprehensive access policies for an entire organization when grouped together. 15 ACCESS CONTROL MODEL
- 16. • Namespace • A namespace is a logical area for a Kubernetes cluster. Kubernetes comes with a default namespace for your cluster objects, plus two more namespaces for system and public resources. • You can create custom namespaces • Resource types that users can access in a Kubernetes namespace include pods, deployments, network policies, nodes, services, secrets, and many more 16 ACCESS CONTROL MODEL - ORCHESTRATION
- 17. • Cgroups – limits the resources a process has access to (CPU, Memory, etc.) • Namespaces – limits what a container can see • Default: root on container is also root on the host due to shared kernel • Update your dockerfile to create a uid for each application process • Ensure containers are not running in privileged mode unless absolutely necessary • Only enable docker socket if absolutely necessary • Unix socket for the daemon listens on • Enables direct communication with the daemon from within container 17 CONTAINER ACCESS CONTROL
- 18. • SecComp • Acts as a whitelist for system calls • Disables 44 system calls by default • Only available if Docker kernel has seccomp available – to check: • AppArmor • Linux kernel security module that you can use to restrict the capabilities of processes running on the host operating system • The security profile allows or disallows specific capabilities, such as network access or file read/write/execute 18 CAPABILITY LIMITATION
- 19. • Ensure orchestrator authentication directory isn’t creating orphaned accounts • Default orchestrator accounts tend to be admin • Leverage security zones and namespaces • Groups nodes and prevents orchestrators from scheduling mixed sensitivity workloads on a given node 19 ORCHESTRATOR ACCESS CONTROL
- 20. • CIS Docker Host benchmark • Audits all layers of the stack (host, daemon, runtime etc.) security settings to bring them in line with best practices • Built into some vulnerability scanning tools • NIST SP 800-190 Application Container Security Guide • Contains risks, countermeasures, threat scenarios, security lifecycle considerations • CNCF Kubernetes Security Audit • Commissioned by the community, performed by Trail of Bits • Very open and detailed with findings • Offers a whitepaper and threat modeling scenarios 20 WHERE TO START: STANDARDS AND BEST PRACTICES
- 23. • Scales security influence without scaling headcount • Reinforces the mantra that security is everyone’s responsibility • Gamification and wall of fame can increase adoption • Avoid name and shame • Avoids complete decentralization but minimizes bottlenecks 23 SECURITY CHAMPIONS
- 24. • Most security incidents are not the result of malicious insiders • Development, security & business need to be on the same page • Creates shared ownership • Determines security tradeoffs to cost, scope, and timeline up-front • Helps build security considerations into the earliest phase of development 24 COLLABORATIVE THREAT MODELING
- 25. • Security fundamentals need to be adapted for faster pace of development • Communicate standards and risk thresholds up front if enforcement will be automated • Take advantage of portability and create golden, reusable images • Collaborate to avoid bottlenecks 25 SUMMARY
- 26. Q&A 26
Editor's Notes
- It’s impossible to protect what you don’t know is out there which makes visibility into your environment and specifically visibility that can differentiate containers an essential starting point The easiest way to do this through an infrastructure scan that can detect docker hosts The average host is running 8 containers Once you detect container hosts, it’s important to harden them based on best practices like the CIS benchmark for docker but more importantly, this is where the real work begins It’s easy to think of containers security and focus on the the container itself but containerization is more than just a little container, it’s an entirely new application development and deployment paradigm that comes with an entire new ecosystem to protect
- Docker Daemon checks the client request and communicates with the Docker components in order to perform a service whereas, Docker Engine or Docker is the base engine installed on your host machine to build and run containers using Docker components and services Image - a read-only template with instructions for creating a Docker container. Generally comprised of a base image (minimal version of Ubuntu, linux, etc.) additional layers vary by container purpose and resemble modern web applications the most commonly used images are Nginx for running HTTP servers, redis for caching, and postgres Image also contains the declarative manifest for how the container runs e.g. the dockerfile Container - a runnable instance of an image Registry/Repository - stores Docker images. Docker Hub is a public registry that anyone can use, and Docker is configured to look for images on Docker Hub by default. Can also use private registries or registries provided by cloud platforms AWS ECR, Google container reg, azure container registry Client - the primary way that many Docker users interact with Docker Daemon - listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. A daemon can also communicate with other daemons to manage Docker services
- Per datadog 50% of all deployments run in orchestrated environments Pod - A Pod is the basic execution unit of a Kubernetes application–the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod represents processes running on your Cluster. Node - A node is a worker machine in Kubernetes, Each node contains the services necessary to run pods and is managed by the master components Cluster - A cluster is a set of machines, called nodes, that run containerized applications managed by Kubernetes. A cluster has at least one worker node and at least one master node. Kubectl - a command line interface for running commands against Kubernetes clusters Kubernetes Master API Server -REST API that validates and configures data for API objects such as pods, services, replication controllers Scheduler - Scheduler that manages availability, performance, and capacity. Controller Manager - Daemon that embeds the core control loops shipped with Kubernetes. etcd – distributed storage system that manages cluster state Kubelet - The primary node agent that runs on each node. The kubelet takes a set of PodSpecs and ensures that the described containers are running and healthy. Kube-proxy - Can do simple TCP/UDP stream forwarding or round-robin TCP/UDP forwarding across a set of back-ends
- Vulnerabilities have been present in every software component since time immemorial and now with increased complexity and software defined everything the number of components that can and will have vulnerabilities is only increasing Containerized environments are susceptible to all manner of component and code vulnerabilities as well as some unique ones that we’ll take a look at next
- Runc is a container escape vulnerability runc through 1.0-rc6, as used in Docker before 18.09.2 and other products, allows attackers to overwrite the host runc binary (and consequently obtain host root access) by leveraging the ability to execute a command as root within one of these types of containers: (1) a new container with an attacker-controlled image, or (2) an existing container, to which the attacker previously had write access, that can be attached with docker exec. I highlighted vulnerabilities within docker, docker community, and Kubernetes to show that these vulnerabilities can exist within any stage of container adoption from docker community where it might be one developer or a small team to full fledged orchestrated Cryptojacking Identify front facing systems and websites vulnerable to remote code execution and inject code via API or through webform. Code traverses to container environment Code executes when container is spun up, code is executed and commands are sent directly to the shell Cryptomining malware is downloaded through a wget command
- If you ever needed evidence that “trust but verify” isn’t going away any time soon this should be it. Even official versions of images from reputable sources in the official docker hub repository are riddled with vulnerabilities.
- Cyber hygiene – doing the basics right is fundamental to security, more important the protecting yourself immediately from the next 0 day attack Cyber hygiene becomes more difficult as more and more complexity is introduced to the environment. And containerization is very complex with several moving parts, microservices, and network overlays creating an obfuscation layer
- Increased velocity is generally a byproduct of making more, smaller changes and it’s tempting to say that if there are only small changes being made then testing becomes less important This thinking is a recipe for lots of small vulnerabilities getting through and causing havoc especially in microservices and containers that are meant to be deployed in bunches and built for autoscaling, one vulnerability in a container can quickly become 5 vulnerabilities in your production environment Testing remains very important and done right it can remove security bottlenecks throughout the development process
- there are a number of arguments to be made in favor of finding and fixing vulnerabilities as early as possible and that’s even more relevant with containers There shouldn’t be differences between environments which removes the reason of dev/test/qa being behind production and different From the developer’s laptop there’s generally an automated toolchain used for continuous integration and continuous deployment, as soon as an image is built it should go through automated testing An important one and one that’s generally more convincing across stakeholder units is the economic and time argument for finding and fixing vulnerabilities as soon as possible. This leaves minimal time for the person who wrote the initial code to move on to new projects or move on all together so a new person needs to familiarize themselves with the codebase before implementing any fix. This time and money argument can be very persuasive but it needs to be implemented correctly.
- Ensure the tool you’re using to scan images gives layer level detailed information. A lot of patching happens in development and falls to the developers who don’t want to waste cycles guessing what to patch where and falling victim to false positives Suppose that you have a base image that includes mypkg 1.2.7 (without the fix), so we know it is vulnerable. If the scanner simply reports vulnerabilities layer by layer, it will appear that the issue is present in any child image, even if the package gets replaced with version 1.3.0 in a different layer. Another false positive. Let’s say that there’s a fictional package mypkg version 1.2.7 with a vulnerability (that I’m making up) called CVE-999. The maintainers of mypkg fixed the vulnerability, and the fix made it into the release of mypkg 1.3.0. So the vulnerability database might say that CVE-999 is known to exist in any version of mypkg before 1.3.0. Now, several other fixes and features went into 1.3.0 upstream, and let’s imagine that the owner of the image being scanned didn’t want all of them. Instead, she decided to cherry-pick just the fix for CVE-999. She rebuilt her own version of the mypkg and called it 1.2.7.12345. According to the database, it would seem that the vulnerability is still there, because 1.2.7.12345 is lower than 1.3.0 where the fix is known to have been applied. But it’s a false positive. Do not get complacent with images in repositories that have been scanned and patched before. As with any software, vulnerabilities will constantly be found in the software that makes up your image layers. Consistent scanning, patching, and re-deploying new secure versions of any containers running the image is required. This is where the immutability and portability of containers actually plays well with security. There’s no need to deploy complex patches to running applications, you can simply tear down the container and replace. The orchestrator will automatically pull the most recent (secure) version of the image to rebuild the container.
- Access control here comes down to what different users and components can see and what they can do Access control has two purposes, keep malicious actors out and if they get in, limit what they can do and where they can go to limit the blast radius of any breach Now let’s get into several ways you can implement a containment strategy for your containers
- https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b Avoid running containers as uid 0, if possible. Containers leverage two concepts to govern access control and resource utilization – namespaces and cgroups Namespace – what you can see Cgroups what you can do/what resources you have access to – useful for preventing buffer overflow type attacks by limiting the amount of memory a given process/container is allocated
- In addition to these tools, you can drop individual capabilities from your container as part of the build CAP_SYS_ADMIN is a specially nasty one in terms of security, it grants a wide range of root level permissions SecComp, AppArmor and SELinux are linux kernel security modules that can be applied to containers to limit capabilities but one thing to remember is they’re meant to be broadly applicable. So in a sense these are made for “less privilege” but they’re not aware of how your containers will be used in your environment so even after applying these profiles or instead of applying these profiles if they’re not available it’s recommended that you list and remove individual capabilities that aren’t necessary
- Kubernetes namespaces were developed as a method to help provide workload isolation. Running multiple, potentially multi-tenant, workloads in the same namespace sidesteps the protections of namespaces, resulting in a single large and flat namespace. In Docker EE with Swarm mode, administrators have the ability of influencing these scheduling decisions by using labels that are securely attached to the individual node identities. These labels allow administrators to group nodes together into different security zones limiting the exposure of particularly sensitive workloads and any secrets related to them.
- If you’re just starting out with containers it’s good to have a guide and some best practices, the good news is these exist for every level of the container ecosystem Start with the hosts because if you build bullet proof applications on top of swiss cheese infrastructure you’re still extremely vulnerable Next move on to the containers themselves with a detailed overview of common issues and countermeasures as well as things to keep in mind from the design of your first container to decommissioning