In May, a security researcher reported a vulnerability in a Shopify microservice and demonstrated how it could be used to access keys from the Google Cloud metadata API. This could have led to a cluster takeover, for which Shopify awarded $25k through its bug bounty program. Shane will share his experience responding to the report, analyzing Kubernetes audit logs, and hardening the cluster to block the escalation path. Together, Greg and Shane will describe some example Kubernetes audit log queries that can help discover unusual activity in the form of Kubernetes API access, and assess the impact of credential exposure, such as in this report. The collection of example queries will be released for use by the Kubernetes community and we will also share some hardening best practices. https://kccna18.sched.com/event/GrZf/shopifys-25k-bug-report-and-the-cluster-takeover-that-didnt-happen-greg-castle-google-shane-lawrence-shopify

1. Shopify’s $25K
Bug Report
and the cluster takeover
that didn’t happen
Shane Lawrence
Security Infrastructure Engineer
Twitter: @shaneplawrence
Github: @shane-lawrence
Shopify
Greg Castle
GKE Security Tech Lead
Twitter: @mrgcastle
Github: @destijl
Google
North America 2018

2. A production security story
Introduction Bug report Detection TakeawaysExchange Attack & defense

3. Introduction

4. Who is shopify.com?
2017 data
Shopify’s background
600K+
merchants
26B+
$processed
80K+
requests per second
peak traffic

5. Shopify cloud platform
• Scalable
• Application developers don’t need to learn k8
• Self-serve with guardrails & paved roads
• Security*** by default

6. Shopify’s bug bounty programs
• 330+ hackers over 3+ years
• Merchants and buyers protected
• $1,000,000+ paid
• hackerone.com/shopify

7. Bug report

8. Security report and responses
7:39pm
Report (goo.gl/dqynDa)
from André Baptista
(0xacb): vuln in
Exchange app
7:50pm
Incident declared
8:00pm
Cloudsec and app dev
teams contacted
8:43pm
Merged commit to
disable vulnerable
feature
9:27pm
Investigation and
cleanup started
(rotate credentials,
contact Google,
investigate logs)
1 hour

9. Exchange

10. Image of
page
Request
screenshot
Request storefront
What is
Exchange?
Marketplace for buying
& selling stores
Webpage
3. Test store
Test store frontpage
2. Screenshot service
Headless browser
1. Exchange app
Create listing
Screenshots

11. Attack & defense

12. cluster control
X
kubeletkube-env
Google service
account token
SSRF
The attack
Security
researcher

13. Server Side Request Forgery (SSRF)

14. Attack:
Weaponize SSRF
Existing workflow
Image of
page
Request
screenshot
Request storefront
2. Screenshot service
Headless browser
1. Exchange app
Create listing
Screenshots
Webpage
4. Metadata service
3. Test store
Test store frontpage

15. Webpage
4. Metadata service
1. Exchange app
Create listing
Screenshots
Request
screenshot
Request storefront
3. Test store
Exploit page
2. Screenshot service
Headless browser
Attack:
Weaponize SSRF
Got token for the VM’s Google
service account

16. Request token
4. Metadata server
Default SA token
v1
Attack:
Weaponize SSRF
Got token for the VM’s Google
service account
1. Exchange app
Create listing
Screenshots
Webpage
Request storefront
3. Test store
Exploit page
2. Screenshot service
Headless browser
Request
screenshot

17. Sidebar: What is this Google SA?
Node (VM)
Metadata server
Service
account
Pod
Token
Token
Google
APIs

18. Demo
Token attack

19. 403: header
required
4. Metadata server
Default SA token
v1
1. Exchange app
Create listing
Screenshots
Defense:
Require header
Metadata server requires header:
Metadata-Flavor: Google
Webpage
Request storefront
3. Test store
Exploit page
Request
screenshot
2. Screenshot service
Headless browser

20. Token
4. Metadata server
Default SA token
v1
v1beta1
1. Exchange app
Create listing
Screenshots
Attack: Use old
API version
Beta API: no request
header required :(
Webpage
Request storefront
3. Test store
Exploit page
Request
screenshot
2. Screenshot service
Headless browser
Image of
token

21. Defense: Disable old API versions
• Beta API known issue: APIs still in use
• Disabled by default in new 1.12+ clusters
• Opt-in now: “disable-legacy-endpoints=true”
• goo.gl/JsdJbL for details

22. Defense: Least priv on token
• Default SA least privilege from 1.10+
• May vary if users have granted extra privs
• Shopify had minimal privs for log/mon/debug
• Token not useful to researcher

23. 5. Metadata server
Default SA token
v1
v1beta1
kube-env
Request
screenshot
Request storefront
Webpage
2. Screenshot service
Headless browser
kube-env
1. Exchange app
Create listing
Screenshots
Attack: What
other metadata?
Metadata server = trust
bootstrap for nodes
Export static key from
“kube-env”
Image of
kube-env
3. Test store
Exploit page

24. Demo
kube-env attack

25. Attack: Kubelet bootstrap key
Kube-env
Security researcher machine
CA.crt client.crt client.pem GKE K8s
API Server
kubectl

26. Request
screenshot
Request storefront
Defense: Metadata
concealment
Now: metadata concealment
(Beta) goo.gl/u6rrMT
Future: K8s TPM trust bootstrap
Webpage
3. Test store
Exploit page
2. Screenshot service
Headless browser
403 Forbidden
4. Metadata proxy
Whitelist/blacklist
1. Exchange app
Create listing
Screenshots
5. Metadata server
Default SA token
v1
v1beta1
kube-env

27. Defense: Minimize kubelet privs
• RBAC on (ABAC off): GKE default
• Node Authorization on: GKE default
• Audit role bindings:
GKE “kubelet-cluster-admin” (not actually
cluster admin) binding if upgraded cluster
https://kubernetes.io/docs/tasks/administer-clus
ter/securing-a-cluster/

28. Demo
Defenses

29. Detection

30. What’s in the logs?
• K8s API audit logs: goo.gl/d8YebH
• Content depends on audit policy
• GKE: g.co/gke/auditlogging

31. Filter logs for kubelet user

32. Filter logs for kubelet user

33. Create deployment failed

34. Create deployment

35. Exec into exchange pod

36. Exec into exchange pod

37. Node CSR creation

38. Takeaways

39. Shopify’s response
1 day
• Disable vulnerable service.
• Start rotating credentials.
• Pay $ to researcher.
1 week
• Analyze audit logs.
• Clean up RBAC.
1 month
• Prevent unwanted redirects.
• Re-enable screenshot service.
• Deploy metadata proxy.
• Pay $$$ to researcher.
• Disclose vulnerability.

40. Lessons learned: K8s advice
• Follow cloud provider hardening advice (GKE: g.co/gke/hardening)
• Block off/filter access to any privileged network endpoints
• Run RBAC and Node Authorization (GKE default)
• Apply least privilege for K8s service accounts
• Audit role bindings, especially upgraded clusters
• Collect API logs and have them available to query (GKE default)

41. Links and references
Shopify bug bounty: hackerone.com/shopify
Bug report details: goo.gl/dqynDa
GKE disable old APIs: goo.gl/JsdJbL
GKE metadata conceal: goo.gl/u6rrMT
K8s API audit logs: goo.gl/d8YebH
GKE logging: g.co/gke/auditlogging
Shane Lawrence
Security Infrastructure Engineer
Twitter: @shaneplawrence
Github: @shane-lawrence
Shopify
Greg Castle
GKE Security Tech Lead
Twitter: @mrgcastle
Github: @destijl
Google

42. Thank you

43. Reference
Log queries

44. Example log queries
• Broad strokes to get you started
• No standard language for queries like this
• SQL seems most standard
• But includes some BigQuery-isms for unpacking repeated fields
• Validation/tweaking on production clusters needed
• Mostly intended to point out interesting values and fields

45. RBAC Changes (excl system)
SELECT
timestamp,
protopayload_auditlog.methodName AS method,
protopayload_auditlog.resourceName AS resource,
protopayload_auditlog.authenticationInfo.principalEmail AS suid,
authzinfo.granted AS granted,
protopayload_auditlog.requestMetadata.callerIp AS saddr
FROM
`gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`,
UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo
WHERE
protopayload_auditlog.methodName LIKE " io.k8s.authorization.rbac.v1%"
AND NOT protopayload_auditlog.authenticationInfo.principalEmail LIKE " system:%"
LIMIT 100
Similarly, use these methodName strings for specific changes to roles or bindings:
“io.k8s.authorization.rbac.v1.roles.%”
“io.k8s.authorization.rbac.v1.rolebindings.%”
“io.k8s.authorization.rbac.v1.clusterroles.%”
“io.k8s.authorization.rbac.v1.clusterrolebindings.%”

46. Creating CSRs via K8s API
SELECT
timestamp,
protopayload_auditlog.methodName AS method,
protopayload_auditlog.resourceName AS resource,
protopayload_auditlog.authenticationInfo.principalEmail AS suid,
authzinfo.granted AS granted,
protopayload_auditlog.requestMetadata.callerIp AS saddr
FROM
`gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`,
UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo
WHERE
protoPayload_auditlog.resourceName LIKE
"certificates.k8s.io/v1beta1/certificatesigningrequests%"
LIMIT 100

47. Unauth’d web requests
SELECT
timestamp,
protopayload_auditlog.methodName AS method,
protopayload_auditlog.resourceName AS resource,
protopayload_auditlog.authenticationInfo.principalEmail AS suid,
authzinfo.granted AS granted,
protopayload_auditlog.requestMetadata.callerIp AS saddr
FROM
`gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`,
UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo
WHERE
protopayload_auditlog.authenticationInfo.principalEmail = " system:anonymous"
LIMIT 100

48. Kubelet bootstrap identity calls
(GKE specific)
SELECT
timestamp,
protopayload_auditlog.methodName AS method,
protopayload_auditlog.resourceName AS resource,
protopayload_auditlog.authenticationInfo.principalEmail AS suid,
authzinfo.granted AS granted,
protopayload_auditlog.requestMetadata.callerIp AS saddr
FROM
`gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`,
UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo
WHERE protopayload_auditlog.authenticationInfo.principalEmail LIKE " kubelet"
LIMIT 100

49. Node authenticated requests
SELECT
timestamp,
protopayload_auditlog.methodName AS method,
protopayload_auditlog.resourceName AS resource,
protopayload_auditlog.authenticationInfo.principalEmail AS suid,
authzinfo.granted AS granted,
protopayload_auditlog.requestMetadata.callerIp AS saddr
FROM
`gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`,
UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo
WHERE protopayload_auditlog.authenticationInfo.principalEmail LIKE " system:node%"
LIMIT 100

50. Calls outside IP range
SELECT
timestamp,
protopayload_auditlog.methodName AS method,
protopayload_auditlog.resourceName AS resource,
protopayload_auditlog.authenticationInfo.principalEmail AS suid,
authzinfo.granted AS granted,
protopayload_auditlog.requestMetadata.callerIp AS saddr
FROM
`gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`,
UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo
WHERE
NOT protopayload_auditlog.requestMetadata. callerIp="127.0.0.1"
AND NOT protopayload_auditlog.requestMetadata. callerIp="::1"
AND protopayload_auditlog.requestMetadata. callerIp NOT LIKE "8.8%"
LIMIT 100