This document discusses vulnerabilities in containerized production environments. It begins by outlining how security mindsets must change when adopting container technologies. It then discusses how data centers are targets for attacks, with the average cost of a data breach being over $3 million. The document outlines how certifications and regulations currently guide security processes, but these may not be fully aware of new technologies like containers. It stresses that container technologies are largely open source, so managing risk requires understanding components beyond just proprietary code. The document provides several best practices for securing containers throughout the development and deployment process. It emphasizes the importance of image health and outlines challenges around managing open source vulnerabilities in containers. Finally, it discusses implications of serverless architectures and provides key takeaw

1. CONFIDENTIAL© 2019 Synopsys, Inc.1
Vulnerabilities in Containerised
Production Environments
Tim Mackey, Senior Technology Evangelist, Synopsys
Changes in Security Mindset When Adopting Container Technologies

2. CONFIDENTIAL© 2019 Synopsys, Inc.2
Data centers are prime targets for attack
Average cost of data breach:
$3.62 million
Lost business:
$4.03 million
Average time to identify
and contain a breach:
257 days
Source: 2017 Cost of Data Breach Study (US
Data), Ponemon Institute
Average cost of data breach:
$3.86 million
Lost business:
$4.20 million
Average time to identify
and contain a breach:
266 days
Source: 2018 Cost of Data Breach Study (US
Data), Ponemon Institute
2017 2018

3. CONFIDENTIAL© 2019 Synopsys, Inc.3
Certifications and regulations guide current processes
On-prem infrastructure policies
• DISA STIG, OVAL Definitions, and XCCDF
• Managed via Chef, Puppet, Ansible,
raw SSH tooling
• Private cloud adopts similar policies
Public cloud infrastructure policies
• Provider responsible for infrastructure security
• Tenant remains responsible for VM security
Some operational risk transferred to provider
Focus for certifications and regulations
• PCI, PII, and PHI
• Process-centric and often not technology aware
• Developers assume compliance and no feedback loop
• NIST 800-137 focused on process
• Continuous monitoring isn’t prescriptive
NIST 800-137

4. CONFIDENTIAL© 2019 Synopsys, Inc.4
Equifax breach focused attention on open source

5. CONFIDENTIAL© 2019 Synopsys, Inc.5
Container technologies are largely open source
Managing risk when you don’t own the stack

6. CONFIDENTIAL© 2019 Synopsys, Inc.6
Modern application
=
Proprietary code
+
Open source components
+
API usage
+
Application
behavior and configuration

7. CONFIDENTIAL© 2019 Synopsys, Inc.8
Risk is a function of the full stack—not just the app
“CNCF Interactive Landscape” application
Cloud Native Computing Foundation provides a web-based
application to browse for partner technology providers.
Application details
• 40K source lines of code
• Node.js application framework
• Containerized and deployed on Kubernetes
• 0.05% of code in use is custom
• 99.5% of code is in the stack
• Risks present at all layers of the stack
Source: CNCF Presentation: How good is our code?

8. CONFIDENTIAL© 2019 Synopsys, Inc.10
Cloud-native design rules
1. Containers are immutable
Create once, run many instances
2. Containers are ephemeral
A container’s life should be only as long
as absolutely required
3. Containers can be sacrificed
- Orchestration systems may terminate a container if needed
- No guarantee of life span
- Don’t store data or logs in containers
4. Containers limit resource access
- Define groups for CPU/RAM access
- Avoid use of ROOT credentials
- Layered file system helps manage storage

9. CONFIDENTIAL© 2019 Synopsys, Inc.11
Designing a container is like developing a new car
DevelopmentOperations
• Engineers design using internal and external components
• Production assembles components into a vehicle
• Vehicle safety and assembly tests ensure compliance
• Vehicle delivery occurs using trusted carriers to dealerships
• Vehicle deployment occurs at time of purchase
• Repair occurs using validated components
• Regulators define governance and compliance criteria

10. CONFIDENTIAL© 2019 Synopsys, Inc.12
At each stage of development, question everything
1. Where does your base image actually come from?
2. What is the health of that base image?
3. You’re updating it at build time, but from what cache?
4. You trust your build servers, but who controls them?
5. Is there any way a foreign container can start in your environment?
6. Who has rights to modify container images?
7. What happens if the base image registry goes away?
8. What happens if the base image tag goes away?
9. When a security disclosure happens, what’s the process to determine impact?
10. How are images being updated and deployed in the face of new security disclosures?
Container usage can trigger IP governance issues—
i.e., `docker push` is potentially a distribution.

11. CONFIDENTIAL© 2019 Synopsys, Inc.13
Image health is critical to application security
Docker Hub Container Scanning Red Hat Container Catalog Health Index

12. CONFIDENTIAL© 2019 Synopsys, Inc.14
Example: How image layer cache affects security
[root@localhost project]# docker history scanner_base:4.2.0
IMAGE CREATED CREATED BY SIZE
COMMENT
395dfd09d6d7 16 hours ago /bin/sh -c #(nop) EXPOSE 9036/tcp 0 B
6ccb88892a15 16 hours ago /bin/sh -c #(nop) ENTRYPOINT ["/scanner" 0 B
604fcdb13b57 16 hours ago /bin/sh -c #(nop) LABEL name=Base scanner 0 B
5bf60b767227 2 days ago /bin/sh -c #(nop) COPY file:0319ebe1148b5cefa 682 B
fe23aeab7fcc 2 days ago /bin/sh -c #(nop) COPY file:e822182ba43798ba0 11.36 kB
9cdc179735ad 2 days ago /bin/sh -c #(nop) COPY file:e822182ba43798ba0 11.36 kB
57bd5e62be14 2 days ago /bin/sh -c #(nop) COPY dir:d9dc3b531575096c83 241.6 MB
a1cb8fd37a68 2 days ago /bin/sh -c #(nop) COPY file:98c69c969ee05b51b 6.14 MB
13855a218a3e 7 days ago /bin/sh -c #(nop) ENV PATH=/scan.cli-4.2.0/b 0 B
885efab8f9b5 7 days ago /bin/sh -c #(nop) ENV APP_HOME=/scan.cli-4.2 0 B
1ed791e999b5 7 days ago /bin/sh -c #(nop) ARG BUILD 0 B
9dcb95a5ceb4 7 days ago /bin/sh -c #(nop) ARG BUILDTIME 0 B
8ada27a4da06 7 days ago /bin/sh -c #(nop) ARG LASTCOMMIT 0 B
7461b836791f 7 days ago /bin/sh -c #(nop) ARG bds_ver 0 B
4020be54fb0f 7 days ago /bin/sh -c yum -y update-minimal --security - 2.632 MB
208a012b6fe4 7 days ago /bin/sh -c #(nop) MAINTAINER Black Duck 0 B
196e0ce0c9fb 6 weeks ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B
<missing> 6 weeks ago /bin/sh -c #(nop) LABEL name=CentOS Base Ima 0 B
<missing> 6 weeks ago /bin/sh -c #(nop) ADD file:1ed4d1a29d09a636dd 196.6 MB

13. CONFIDENTIAL© 2019 Synopsys, Inc.15
Example: System behavior with image consistency
1. Push and tag image version “1.0.1”
2. Pull and run “latest”
3. Scale to 2 replicas
4. Pull and run tag “1.0.1”
5. Scale to 2 replicas
6. Pull and run pull spec “sha256:12345”
7. Scale to 2 replicas
8. Delete tag “1.0.1” in registry
9. Scale “1.0.1” to 3 replicas
10. Push and tag image version “1.0.2”
11. Add node 2
12. Down node 1
Cluster Node 1
Cluster Node 2
“latest”
“latest”
1.0.1
1.0.1
Sha256:12345
Sha256:12345
1.0.1
“latest”
“latest”
Sha256:12345
Sha256:12345

14. CONFIDENTIAL© 2019 Synopsys, Inc.16
Open source vulnerability management is a challenge
64 vulnerabilities per codebase—a 134% increase over last year
Source: Synopsys Open Source Security and Risk Report 2018

15. CONFIDENTIAL© 2019 Synopsys, Inc.17
• Primary objective: Facilitate application deployment
–Abstracts replication controllers and pods from users
• DeploymentConfig
–Defines the conditions for a new deployment
– Configuration and image changes supported
–Defines deployment strategy
– Rolling, recreate, blue/green, A/B, custom
– Requires “readiness” probes for pods
–Rollback via DeploymentConfigRollback
Planning your container patch strategy
kind: "DeploymentConfig"
apiVersion: "v1"
metadata:
name: "frontend"
spec:
template:
metadata:
labels:
name: "frontend"
spec:
containers:
- name: "helloworld"
image: "openshift/helloworld"
triggers:
- type: "ConfigChange"
- type: "ImageChange"
imageChangeParams:
automatic: true
containerNames:
- "helloworld"
from:
kind: "ImageStreamTag"
name: “helloworld:latest"
strategy:
type: "Rolling"
Patch
version
worse
Patch
version
worse
Patch
version
worse

16. CONFIDENTIAL© 2019 Synopsys, Inc.18
Container security paradigm shift
Container security antipatterns
• Allowing interactive login or SSH access
• Requiring elevated privileges
• Installing software or patching existing
software
• Attaching to a running container instance
• Executing multiple processes
• Writing any log data to local disk
Container security patterns
• Limit execution life span
• Use exception-based network access
with default blocked
• Limit resource consumption
• Limit image dependencies
to required components
• Restrict privileges on running containers
• Allow for multiple replicas and concurrent
container operations
• Define a base image approval
and update process
• Use a secrets vault for all sensitive
configuration information

17. CONFIDENTIAL© 2019 Synopsys, Inc.19
Example: Security information flow aids compliance
Suggestion: usage of
red and green for the
outline confuses the
message, i.e. start/stop

18. CONFIDENTIAL© 2019 Synopsys, Inc.20
Augmenting perimeter scan solutions
Threat agents
• Scan networks for weaknesses using toolkits
• Success is a numbers game—zero knowledge of target
• Perimeter defenses can be false positives
• Use multifactors for attack reconnaissance
Infiltration
• Occurs through at least one vector
• Creates beachheads supporting infection, C&C, and lateral movement
• Exploits latent vulnerabilities and misconfigurations
Mitigation powered by information flow
• Can’t exploit what doesn’t exist
• Focus attention on unpatched services
• Open source originates from multiple channels; patches must match
• Recognizes that attack landscape evolves
Global IP
space
Managed
systems
Accessible
systems
Vulnerability
present

19. CONFIDENTIAL© 2019 Synopsys, Inc.21
Example: The tale of CVE-2017-5638 and Equifax
1,649 days
Code bug
introduced
August
2012
Struts 2.3
released
November
2012
Struts 2.5
released
May
2016
Patches
available
March 6
2017
7 days
March 7
2017
Disclosure
published
NVD details
March 14
2017
78 days
Hacks
successful
May 13
2017
Hacks
discovered
July 29
2017

20. CONFIDENTIAL© 2019 Synopsys, Inc.22
Does the “serverless” model change things?
Definition: A model under which an application is
decomposed into stateless activities or functions
run on cloud infrastructure
Or: RPC.Next
Process
• Cloud provider defines a runtime
• Runtime is implemented in a container
• Developer creates an app-function targeting runtime
• Developer defines an API endpoint for
the app-function
• Cloud provider supplies API gateway
Security implications
• Authorization and authentication required
for protected data accesses
• All data passes on “the wire,” which affects certifications
• API contracts are critical, as are data reviews
• Security response process must include security awareness
from cloud provider
Web App
Authorizer
Data StoreUser Service
External APIContent Service
10100
001100
API Gateway

21. CONFIDENTIAL© 2019 Synopsys, Inc.23
Key takeaways
Reassess processes when technology stacks change
• Identify opportunities to reduce business risk with new technologies
• Cloud-native paradigms are perfect to derisk in the face of adversity
• Containers aren’t VMs and offer opportunities to improve risk posture
Reduce risks of noncompliance
• Implement continuous monitoring of all deployed apps, complete with dependency inventory
• Leverage immutability properties of containers to simplify security
• Validate running containers against orchestration to eliminate rogue activities
Define clear technology risk acceptance criteria
• Ensure criteria are understood in Ops, Development, and Procurement
• Train all development and operations teams to identify changes in risk
• Document decisions affecting risk acceptance at all points in the SDLC

22. Thank You