Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Shopify’s $25k Bug Report, and the Cluster Takeover That Didn’t Happen

270 views

Published on

https://kccna18.sched.com/event/GrZf/shopifys-25k-bug-report-and-the-cluster-takeover-that-didnt-happen-greg-castle-google-shane-lawrence-shopify
https://youtu.be/2XCm7vveU5A

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Shopify’s $25k Bug Report, and the Cluster Takeover That Didn’t Happen

  1. 1. Shopify’s $25K Bug Report and the cluster takeover that didn’t happen Shane Lawrence Security Infrastructure Engineer Twitter: @shaneplawrence Github: @shane-lawrence Shopify Greg Castle GKE Security Tech Lead Twitter: @mrgcastle Github: @destijl Google North America 2018
  2. 2. A production security story Introduction Bug report Detection TakeawaysExchange Attack & defense
  3. 3. Introduction
  4. 4. Who is shopify.com? 2017 data Shopify’s background 600K+ merchants 26B+ $processed 80K+ requests per second peak traffic
  5. 5. Shopify cloud platform • Scalable • Application developers don’t need to learn k8 • Self-serve with guardrails & paved roads • Security*** by default
  6. 6. Shopify’s bug bounty programs •   330+ hackers over 3+ years •   Merchants and buyers protected •   $1,000,000+ paid •   hackerone.com/shopify
  7. 7. Bug report
  8. 8. Security report and responses 7:39pm Report (goo.gl/dqynDa) from André Baptista (0xacb): vuln in Exchange app 7:50pm Incident declared 8:00pm Cloudsec and app dev teams contacted 8:43pm Merged commit to disable vulnerable feature 9:27pm Investigation and cleanup started (rotate credentials, contact Google, investigate logs) 1 hour
  9. 9. Exchange
  10. 10. Image of page Request screenshot Request storefront What is Exchange? Marketplace for buying & selling stores Webpage 3. Test store Test store frontpage 2. Screenshot service Headless browser 1. Exchange app Create listing Screenshots
  11. 11. Attack & defense
  12. 12. cluster control X kubeletkube-env Google service account token SSRF The attack Security researcher
  13. 13. Server Side Request Forgery (SSRF)
  14. 14. Attack: Weaponize SSRF Existing workflow Image of page Request screenshot Request storefront 2. Screenshot service Headless browser 1. Exchange app Create listing Screenshots Webpage 4. Metadata service 3. Test store Test store frontpage
  15. 15. Webpage 4. Metadata service 1. Exchange app Create listing Screenshots Request screenshot Request storefront 3. Test store Exploit page 2. Screenshot service Headless browser Attack: Weaponize SSRF Got token for the VM’s Google service account
  16. 16. Request token 4. Metadata server Default SA token v1 Attack: Weaponize SSRF Got token for the VM’s Google service account 1. Exchange app Create listing Screenshots Webpage Request storefront 3. Test store Exploit page 2. Screenshot service Headless browser Request screenshot
  17. 17. Sidebar: What is this Google SA? Node (VM) Metadata server Service account Pod Token Token Google APIs
  18. 18. Demo Token attack
  19. 19. 403: header required 4. Metadata server Default SA token v1 1. Exchange app Create listing Screenshots Defense: Require header Metadata server requires header: Metadata-Flavor: Google Webpage Request storefront 3. Test store Exploit page Request screenshot 2. Screenshot service Headless browser
  20. 20. Token 4. Metadata server Default SA token v1 v1beta1 1. Exchange app Create listing Screenshots Attack: Use old API version Beta API: no request header required :( Webpage Request storefront 3. Test store Exploit page Request screenshot 2. Screenshot service Headless browser Image of token
  21. 21. Defense: Disable old API versions • Beta API known issue: APIs still in use • Disabled by default in new 1.12+ clusters • Opt-in now: “disable-legacy-endpoints=true” • goo.gl/JsdJbL for details
  22. 22. Defense: Least priv on token • Default SA least privilege from 1.10+ • May vary if users have granted extra privs • Shopify had minimal privs for log/mon/debug • Token not useful to researcher
  23. 23. 5. Metadata server Default SA token v1 v1beta1 kube-env Request screenshot Request storefront Webpage 2. Screenshot service Headless browser kube-env 1. Exchange app Create listing Screenshots Attack: What other metadata? Metadata server = trust bootstrap for nodes Export static key from “kube-env” Image of kube-env 3. Test store Exploit page
  24. 24. Demo kube-env attack
  25. 25. Attack: Kubelet bootstrap key Kube-env Security researcher machine CA.crt client.crt client.pem GKE K8s API Server kubectl
  26. 26. Request screenshot Request storefront Defense: Metadata concealment Now: metadata concealment (Beta) goo.gl/u6rrMT Future: K8s TPM trust bootstrap Webpage 3. Test store Exploit page 2. Screenshot service Headless browser 403 Forbidden 4. Metadata proxy Whitelist/blacklist 1. Exchange app Create listing Screenshots 5. Metadata server Default SA token v1 v1beta1 kube-env
  27. 27. Defense: Minimize kubelet privs • RBAC on (ABAC off): GKE default • Node Authorization on: GKE default • Audit role bindings: GKE “kubelet-cluster-admin” (not actually cluster admin) binding if upgraded cluster https://kubernetes.io/docs/tasks/administer-clus ter/securing-a-cluster/
  28. 28. Demo Defenses
  29. 29. Detection
  30. 30. What’s in the logs? • K8s API audit logs: goo.gl/d8YebH • Content depends on audit policy • GKE: g.co/gke/auditlogging
  31. 31. Filter logs for kubelet user
  32. 32. Filter logs for kubelet user
  33. 33. Create deployment failed
  34. 34. Create deployment
  35. 35. Exec into exchange pod
  36. 36. Exec into exchange pod
  37. 37. Node CSR creation
  38. 38. Takeaways
  39. 39. Shopify’s response 1 day • Disable vulnerable service. • Start rotating credentials. • Pay $ to researcher. 1 week • Analyze audit logs. • Clean up RBAC. 1 month • Prevent unwanted redirects. • Re-enable screenshot service. • Deploy metadata proxy. • Pay $$$ to researcher. • Disclose vulnerability.
  40. 40. Lessons learned: K8s advice • Follow cloud provider hardening advice (GKE: g.co/gke/hardening) • Block off/filter access to any privileged network endpoints • Run RBAC and Node Authorization (GKE default) • Apply least privilege for K8s service accounts • Audit role bindings, especially upgraded clusters • Collect API logs and have them available to query (GKE default)
  41. 41. Links and references Shopify bug bounty: hackerone.com/shopify Bug report details: goo.gl/dqynDa GKE disable old APIs: goo.gl/JsdJbL GKE metadata conceal: goo.gl/u6rrMT K8s API audit logs: goo.gl/d8YebH GKE logging: g.co/gke/auditlogging Shane Lawrence Security Infrastructure Engineer Twitter: @shaneplawrence Github: @shane-lawrence Shopify Greg Castle GKE Security Tech Lead Twitter: @mrgcastle Github: @destijl Google
  42. 42. Thank you
  43. 43. Reference Log queries
  44. 44. Example log queries • Broad strokes to get you started • No standard language for queries like this • SQL seems most standard • But includes some BigQuery-isms for unpacking repeated fields • Validation/tweaking on production clusters needed • Mostly intended to point out interesting values and fields
  45. 45. RBAC Changes (excl system) SELECT timestamp, protopayload_auditlog.methodName AS method, protopayload_auditlog.resourceName AS resource, protopayload_auditlog.authenticationInfo.principalEmail AS suid, authzinfo.granted AS granted, protopayload_auditlog.requestMetadata.callerIp AS saddr FROM `gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`, UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo WHERE protopayload_auditlog.methodName LIKE " io.k8s.authorization.rbac.v1%" AND NOT protopayload_auditlog.authenticationInfo.principalEmail LIKE " system:%" LIMIT 100 Similarly, use these methodName strings for specific changes to roles or bindings: “io.k8s.authorization.rbac.v1.roles.%” “io.k8s.authorization.rbac.v1.rolebindings.%” “io.k8s.authorization.rbac.v1.clusterroles.%” “io.k8s.authorization.rbac.v1.clusterrolebindings.%”
  46. 46. Creating CSRs via K8s API SELECT timestamp, protopayload_auditlog.methodName AS method, protopayload_auditlog.resourceName AS resource, protopayload_auditlog.authenticationInfo.principalEmail AS suid, authzinfo.granted AS granted, protopayload_auditlog.requestMetadata.callerIp AS saddr FROM `gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`, UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo WHERE protoPayload_auditlog.resourceName LIKE "certificates.k8s.io/v1beta1/certificatesigningrequests%" LIMIT 100
  47. 47. Unauth’d web requests SELECT timestamp, protopayload_auditlog.methodName AS method, protopayload_auditlog.resourceName AS resource, protopayload_auditlog.authenticationInfo.principalEmail AS suid, authzinfo.granted AS granted, protopayload_auditlog.requestMetadata.callerIp AS saddr FROM `gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`, UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo WHERE protopayload_auditlog.authenticationInfo.principalEmail = " system:anonymous" LIMIT 100
  48. 48. Kubelet bootstrap identity calls (GKE specific) SELECT timestamp, protopayload_auditlog.methodName AS method, protopayload_auditlog.resourceName AS resource, protopayload_auditlog.authenticationInfo.principalEmail AS suid, authzinfo.granted AS granted, protopayload_auditlog.requestMetadata.callerIp AS saddr FROM `gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`, UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo WHERE protopayload_auditlog.authenticationInfo.principalEmail LIKE " kubelet" LIMIT 100
  49. 49. Node authenticated requests SELECT timestamp, protopayload_auditlog.methodName AS method, protopayload_auditlog.resourceName AS resource, protopayload_auditlog.authenticationInfo.principalEmail AS suid, authzinfo.granted AS granted, protopayload_auditlog.requestMetadata.callerIp AS saddr FROM `gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`, UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo WHERE protopayload_auditlog.authenticationInfo.principalEmail LIKE " system:node%" LIMIT 100
  50. 50. Calls outside IP range SELECT timestamp, protopayload_auditlog.methodName AS method, protopayload_auditlog.resourceName AS resource, protopayload_auditlog.authenticationInfo.principalEmail AS suid, authzinfo.granted AS granted, protopayload_auditlog.requestMetadata.callerIp AS saddr FROM `gcastle-gke-dev.kubecon2018.cloudaudit_googleapis_com_activity_*`, UNNEST(protopayload_auditlog.authorizationInfo) AS authzinfo WHERE NOT protopayload_auditlog.requestMetadata. callerIp="127.0.0.1" AND NOT protopayload_auditlog.requestMetadata. callerIp="::1" AND protopayload_auditlog.requestMetadata. callerIp NOT LIKE "8.8%" LIMIT 100

×