The document discusses the complexities and best practices of managing containerized applications in cloud environments, emphasizing the importance of secure configurations, continuous monitoring, and clear documentation. It stresses the need for version-controlled configurations, proper access controls, and the segregation of images in CI/CD pipelines to enhance security. Additionally, the document highlights the significance of detecting and addressing misconfigurations and advocates for tools that assist in maintaining container security.

1. Containing the cloud
Wes Widner
@kai5263499

3. Containers
represent
complexity
● Every piece is
important
● Notes help us
manage complexity
● Expect every
component involved
to show their work
Docker
Registry
K8S
Master
etcd
K8S Worker
K8S Worker
K8S Worker
Pod
Container
Pod
Container
Data plane

4. Designed for
developers
Everyone loves having
the playground to
themselves
● Dockerfiles help
eliminate
snowflakes
● “Containers don’t
contain” - Daniel
Walsh RedHat

5. My friend said “Suggest to President
Obama that the team should deploy to
production every day.”
-Bill Higgins
healthcare.gov tech surge

6. Developers and
ops need
different images
● Templated Dockerfiles to
share structure
● Multi-stage builds to share
data
● Debug images for developers
● Separate repos for everyone

7. Where do images
come from?
● Production images should
come from CI/CD pipelines
● Repos in registries
should be segregated
● Everything should be
logged

8. What’s in your
prod layers?
● Layer construction is open
for inspection
● Add forensics information to
LABELs
● Seperate services by
function
● Understand that the kernel
is your attack surface
● Lint like crazy

9. Continuous image
scanning
● It’s easy to bake unwanted
things into images
● What’s clean today may not be
clean tomorrow
● We need a way to
retroactively scan images
● It’s also useful to just know
what an image contains at
build time

10. What happens
when the cloud
breaks?
● Let forensics be your
guide to a secure cloud
● Pretend the cloud has
melted down and put
everything in place
that you would want to
have

12. Misconfiguration
is the root of
all evil
● By definition, you
don’t mean to do it
● This is an even bigger
problem with
declarative systems
● How long does it take
for you to detect a
misconfiguration?

13. Configuration as
code
● Even without formal code
reviews, version
controlled configurations
are useful
● One-off configuration
updates are not only bad
form, they’re an incident
waiting to happen

14. Try not to let
people break
assumptions
● Allowing containers to
change means they will
● Mutable containers make
forensics needlessly hard

15. Least privileged
workers
● Privilege separation is hard
● There is not an official standard
for container security policies
● Workers can be tainted to run
sensitive pods
● Another option is multiple clusters
● Workers don’t need operational
secrets

16. Monitoring your
orchestrator
● Kubernetes security
solutions fall into three
broad categories
● Admin API control
● eBPF for tagging packets
and syscalls
● Hypervisor shims
● Runner-up category for
auditd

17. A little chaos
is good for you
● Chaos engines are not only
good for development.
They’re also good for
security
● They help smoke out false
assumptions
● Cloud fuzzing is a thing

18. How mature is your cloud?
❏ Properly configured access
controls to registries and
API servers
❏ Tagged packets from pods,
continuous image scanning
❏ Execution logs
❏ Images developed and pushed
from a build system
❏ Version controlled
Dockerfiles, configs, and
config values
What’s your Okta?

19. https://github.com/kai5263499/
container-security-awesome
More resources

20. ● Write everything down
● Trust but verify
● Kick the tires
● Pyramid of truth
Roadmap