This document discusses potential security issues with container-based datacenter platforms like DC/OS and Mesos. It describes how an initial compromise of a vulnerable web application inside a container could allow an attacker to pivot and gain further access by exploiting features of Mesos DNS, the Mesos master and agents, Marathon and Chronos frameworks, and the Etcd configuration store. It recommends steps like disabling risky Mesos DNS features, enforcing authentication, and logically segmenting networks and configuration stores.

1. SESSION ID:SESSION ID:
#RSAC
Bryce Kunz
Orchestration Ownage:
Exploiting Container-Centric
Datacenter Platforms
CSV-R03
Senior Threat Specialist
Adobe
Mike Mellor
Director, Information Security
Adobe

2. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Intro
2
Mike Mellor
Director, Information Security @ Adobe
Bryce Kunz
Senior Threat Specialist @ Adobe

3. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Containers - The Future is Now!
3
2016 Surveys:
15-16% of all organizations are already using containers in production
35% organizations have done a proof-of-concept
The Future is Now!
Containers are in production now
Containers are continuing to grow in popularity

4. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Containers appear more secure
4
The biggest drivers:
39% to increase developer efficiency and
36% to support microservices
Organizations want to avoid cloud platform lock-in
2016 Surveys:
Many (42%) organizations gain value in the “secure/isolated”
capabilities that containers provide

5. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
But managing Containers feels complex
5
2016 Survey:
The more exposure an organization has to containers,
The more complexities are exposed.
Respondents said they found containers…
“too complex to integrate into existing environments,”
and require “too many skilled resources to manage.”

6. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
And are very challenging to manage at scale
6
2016 Survey:
The #1 challenge of containers, according to the 53% of respondents
who are either using or evaluating containers, is …
“Container Management.”

7. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Probable Security Nightmare
7
“Too Complex” + Challenging to Manage
=
Probable Security Nightmare
“Complexity the Worst Enemy of Security” - Bruce Schneier

8. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Container and Cluster Management Options
8
Technology Design Pros Cons
Public Cloud
Container Services
Container Centric Easy, Scalable Vendor Lock-in;
Proprietary
Docker Swarm Docker Centric Native Clustering Limited by API
Kubernetes Clusters of
Containerized Apps
Works w/ Docker;
Mounts persistent
volumes
Custom overlay
requires more
specialization
Mesos & DC/OS Cluster
Management
Works w/ Docker,
Kubernetes, &
Native Apps;
Very Flexible
Additional layers
adds more
complexities

9. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Cluster Management
9
CoreOS
Linux OS
Many servers in…
DataCenter
AWS
Azure
etc…
How do we
effectively use all of
these resources?
Datacenter, Azure, AWS,
GCE, etc…

10. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Mesos Master & Agents
10
Mesos Master
5050/TCP by default
Distributes Tasks
Mesos Agent
5051/TCP by default
Executes Tasks
CoreOS
Linux OS
Datacenter, Azure, AWS,
GCE, etc…
Master
AgentAgent

11. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Mesos is the Kernel of DC/OS
11
Mesos is the kernel
of the distributed
operating system
known as DC/OS
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Master
AgentAgent

12. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Frameworks
12
Frameworks
provide the logic
Init Jobs
— Marathon
Cron Jobs
Chronos
MetronomeDatacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Master
AgentAgent

13. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Supporting: Configuration Stores
13
Configuration Stores
keep everyone on the
same page
Zoo Keeper
Etcd
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Supporting:
Master
AgentAgent

14. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Supporting: Discovery
14
Discovery
Enables the finding of
other services within
the cluster
Mesos DNS
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Supporting:
Master
AgentAgent

15. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
DC/OS Design
15
Containers w/ Apps
Docker Containers
Web Apps
etc…
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App

16. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Internet Accessible Containers
16
Containers w/ Apps
Public
Internet Accessible
Private
Internal
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet

17. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Scenario
17
Initial Access (RCE)
Via a vulnerable web
application
Into a container
As limited user (e.g.
www-data)
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

18. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Scenario: RCE via web app within a container
18
e.g. JBoss, Tomcat, OSGi Console, Axis2, etc…

19. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Recon via Mesos DNS
19
Query via pivot:
Mesos DNS
53/UDP & TCP
— DNS service
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

20. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
.mesos TLD
20
The easy way to find services within the cluster

21. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Recon via Mesos DNS
21
Query via pivot:
Mesos DNS
8123/TCP by default
— DNS via REST API
Service Discover
— within the Cluster
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

22. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Undocumented?
22
/v1/enumerate -> all mesos dns information

23. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Enumerate Mesos DNS using REST API
23
/v1/enumerate -> all mesos dns information

24. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Find IP & RHP TCP ports of all services
24
/v1/enumerate -> all mesos dns information

25. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure: Disable Risky Mesos DNS Features
25
Disable the…
“AXFR”
“Enumerate”
API Calls
• Harder for attacker to
discover all services
• Applications shouldn’t
commonly be using
these API callsDatacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

26. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Recon via Mesos Master
26
Query via pivot:
Mesos Master
5050/TCP by default
Distributes Tasks
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

27. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Enumerate Mesos Master
27
Request via the REST API

28. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Enumerate Mesos Master
28
Response: json w/ all Mesos Agent’s IP addresses within the cluster

29. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Recon via Mesos DNS
29
Query via pivot:
Mesos Agent
5051/TCP by default
Executes Tasks
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

30. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Enumerate Mesos Agent
30
Request via the REST API

31. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Enumerate Mesos Agent
31
Response: json w/ what containers are currently running on the server
(i.e. basic0012)

32. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure: Logical Internal Network Segmentation
32
Separates out the
network into zones:
Apps w/ Data
Management
Commonly with
Calico,
Canal, or
FlannelDatacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App

33. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secrets via Configuration Store
33
Etcd
RHP/TCP by default
— 2379/TCP
client/server
— 2380/TCP peers
Configuration Store
— Core OS Fleets Units
— Applications
ZooKeeper
2181/TCP by default
— Binary Protocol
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

34. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Enumerate Etcd
34
Request via the REST API recursively

35. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Enumerate Etcd
35
Response: json frequently containing secrets including credentials

36. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure: Separate Configuration Stores
36
Separate out the
configuration stores into
zones:
Apps w/ Data
Management
Enforce separation via…
Authentication
Credentials and
Logical Network
Segmentation
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

37. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Frameworks
37
Marathon
Long Running Services
— e.g. Containers
Ensures always
running
Chronos
Cron for the Cluster
Batch Jobs
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

38. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
RCE via Marathon Jobs
38
Request via the REST API

39. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
RCE via Marathon Jobs
39
Marathon
Long Running Services
— e.g. Containers
Ensures always
running
Chronos
Cron for the Cluster
Batch Jobs
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE
RCE

40. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
RCE via Marathon Jobs
40
Response: json with the malicious job status

41. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
RCE via Chronos Jobs
41
Marathon
Long Running Services
— e.g. Containers
Ensures always
running
Chronos
Cron for the Cluster
Batch Jobs
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE
RCE

42. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure: Enforce Authentication
42
Applications must…
support and be
configured to use
authentication as well
securely store and use
credentials
be deployed securely
and/or retrieve
credentials securely
Alert on brute force
attempts
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

43. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Creds via MitM with ARP Spoofing
43
Another Container
has the Creds for
Marathon
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE

44. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Creds via MitM with ARP Spoofing
44
Attacker uses ARP
spoofing to redirect
that containers
traffic to the
compromised
container
Attacker collect the
credentials
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE ARP

45. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Creds via MitM with ARP Spoofing
45
Attacker can now
create malicious
Marathon jobs
Negating
authentication
security controls
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE ARP
RCE

46. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure: TLS for Internal Communications
46
Enable TLS w/ valid
certificates for strong
HTTPS communications
Anything using
credentials needs TLS!
Validate Certificates
Fail closed on bad
certificates
Alert on certificates
errors
Datacenter, Azure, AWS,
GCE, etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
Cntr
App
Cntr
App
Cntr
App
Cntr
App
Internet
RCE ARP

47. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Strategic Actions
47
Next week:
Assess which services you can enable Authentication & TLS on
— w/o breaking your existing applications within the cluster
Three months from now:
Implement Authentication & TLS on safe services and frameworks
— Focusing on services responsible for orchestration within the cluster
Deploy separate services where possible for Apps that do not support TLS & Auth
Six months from now:
Retrofit all Applications within the cluster to use TLS & Authentication
Enforce the use of TLS & Authentication internal everywhere (disable clear-text)

48. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Big Picture
48
Container Adoption Is Maturing, especially in Enterprises
Enterprises are using containers in production.

49. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Big Picture
49
Pivoting from a compromised service within the cluster
No container breakout / 0day / exploit needed J
May enable an attacker to completely compromise the cluster

50. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Big Picture
50
Looking Beyond the Border
with a Defense in Depth strategy
Secures the Future & the cluster

51. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Thank you!
51
Thank you!

52. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
Future Research
52
Testing MitM from compromised container
NCC Group’s report states this is possible for co-hosted containers
Test downgrade HTTPS communications
Can we downgrade from HTTPS to HTTP and capture creds from another container?
Test Certs (e.g. can cert pinning be enabled?) to REST APIs
Can we MitM and impersonate the API service?
Test Authentication Brute force attacks
Fairly certain there are no lockouts, can we enable better authentication security?
— Write module to brute-force and guess creds
Test Logical Network Segmentation Tools
Calico, Canal, Flannel
— Note: these should work as advertised but probably we should independently verify

53. Presenter’s Company
Logo – replace or
delete on master slide
#RSAC
References
53
https://www.cloudfoundry.org/wp-content/uploads/2016/06/Cloud-
Foundry-2016-Container-Report.pdf
https://clusterhq.com/assets/pdfs/state-of-container-usage-june-
2016.pdf
http://www.rightscale.com/blog/cloud-industry-insights/new-devops-
trends-2016-state-cloud-survey