Presentation by Marco Slaviero at BlackHat USA in 2010. This presentation is about mining information from memchached. The presentation begins with a brief introduction to memcached. go-derper.rb, a tool developed by the presenter for hacking memchaced servers is introduced and a few memchached mining examples are given. The presentation ends with a brief discussion on serialized objects exposed in the chache.

1. Cache on delivery
mining memcached
marco@sensepost.com

3. The need for caching
• Large percentage of data remains relatively
constant
• Wikipedia page contents
• Youtube video links
• FB Profile data
• Poorly designed solutions regenerate data
on each request
• Don’t regenerate, rather regurgitate

4. The need for caching
• Large percentage of data remains relatively
constant
• Wikipedia page contents
• Youtube video links
• FB Profile data
• Poorly designed solutions regenerate data
on each request
• Don’t regenerate, rather regurgitate

5. Memcached
• memcached.org
• Written for LJ (2003) by Brad
Fitzpatrick
• Non-persistent network-based KV
store
• Why do we care? Mom&pop don’t
need the cache.

6. Memcached
• memcached.org
• Written for LJ (2003) by Brad
Fitzpatrick
• Non-persistent network-based KV
store
• Why do we care? Mom&pop don’t
need the cache.

7. Basic KV
• Slabs are fixed size • Users don’t care about slabs
• Dstvalue size
by
slab determined • Miners care about slabs

8. Trivial protocol
• ASCII-based
• Long-lived
• Tiny command set
• set
• get
• stats
• ...
• ????
Binary and UDP protocols also
exist, these were not touched.

9. Trivial protocol
• ASCII-based
• Long-lived
• Tiny command set
• set
• get
• stats
• ...
• ????
Binary and UDP protocols also
exist, these were not touched.

10. Trivial protocol
• ASCII-based
• Long-lived
• Tiny command set
• set
• get
• stats
• ...
• ????
Binary and UDP protocols also
exist, these were not touched.

11. Trivial protocol
• ASCII-based
• Long-lived
• Tiny command set
• set
• get
• stats
• ...
• ????
Binary and UDP protocols also
exist, these were not touched.

12. Trivial protocol
• ASCII-based
• Long-lived
• Tiny command set
• set
• get
• stats
• ...
• ????
Binary and UDP protocols also
exist, these were not touched.

13. 1998
• Blank ‘sa’
• Anonymous ftp
• system/manager

14. Goals
• Connect to memcached
• Find all slabs
• Retrieve keynames from each slab
• Retrieve each key

15. Lies, damn lies, and
stats
stats slabs
STAT 1:chunk_size 80
•
<...>
stats cmd has subcmds STAT 2:chunk_size 104
<...>
STAT 3:chunk_size 136
• items <...>
STAT 4:chunk_size
<...>
176
• slabs
STAT 6:chunk_size
<...>
STAT 8:chunk_size
280
440
•
<...>
... STAT 9:chunk_size 552
<...>
STAT 9:cas_badval 0
STAT active_slabs 7
This gets us the slabs_ids

16. Retrieving key names
Rely on two
{poorly|un}
documented
features

17. Retrieving key names
Feature #1:
Remote enabling of
debug mode

18. Retrieving key names
Feature #2:
“stats cachedump”

19. Retrieving key names
Feature #2:
“stats cachedump”

20. Retrieving key names
Feature #2:
“stats cachedump”
Slabs ID

21. Retrieving key names
Feature #2:
“stats cachedump”
Key limit

22. Retrieving key names
Feature #2:
“stats cachedump”
Key list

23. Retrieving key names
Feature #2:
“stats cachedump”
This gets us key names

24. And this gets us?
• No need for complex hacks. Memcached serves up
all its data for us.
• What to do in an exposed cache?
• Mine
• SQLi is too hard for me
• Overwrite
• Client-side
• Server-side

25. Mining the cache
• go-derper.rb – memcached miner
• Retrieves up to k keys from each slab and
their contents, store on disk
• Applies regexes and filters matches in a
hits file
• Supports easy overwriting of cache
entries
• [demo]

26. Two issues

27. Two issues
• Finding caches
• Again with the
simple approach
• Pick a cloud
network, scan for
memcacheds on
port 11211 with
a mod’ed .nse

28. Two issues
• Linking apps to
caches
• Who’s %$!#ing
cache is this?
• Cached high scores
suck. Where’s the
good stuff?
• Is it live?
http://www.rhythm.com/~keith/autoStereoGrams/vortexas.gif

29. Results #1
IPs scanned 2^16
# of caches found 229
Retrieved Items 7.3GB
Average uptime ~50days
Total bandwidth used 9PB
Total entry count 288 million
Total Bytes stored 136TB
Highest bandwidth 247TB
Highest entry count 133 million
Highest Bytes Stored 19.3GB

30. Results #2
• HTML • Objects found
• JavaScript • Serialized Java
• Data • Pickled Python
• Email • Ruby ActiveRecord
• Passwords • .Net Object
(clear-text,
crypt’ed, MD5) • JSON

31. Globworld

32. Globworld

33. Globworld

34. Globworld

35. Gowalla

36. Gowalla

37. Gowalla

38. Gowalla

39. Gowalla

40. Gowalla

41. Bit.ly Pro

42. Bit.ly Pro

43. Bit.ly Pro

44. PBS

45. PBS

46. PBS

47. PBS

48. Sidebar: serialized objs
• Python’s pickle intentionally insecure
• But they’re exposed!
• Pickle shellcode
cos
system
(S'echo hostname'
tR.
• [demo]

51. Sidebar: serialized objs
• Python’s pickle intentionally insecure
• But they’re exposed!
• Pickle shellcode
cos
system
(S'echo hostname'
tR.
• [demo]

52. Fixes?
• FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW.
FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW.
FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW.
(VPC)
• Hack code to disable stats facility (but doesn’t prevent
key brute-force)
• Hack code to disable remote enabling of debug features
• Switch to SASL
• Requires binary protocol
• Not supported by a number of memcached libs
• Salt your passwords with a proper scheme (PHK’s MD5
or Bcrypt)
• Also, FW.

53. Random thoughts
• This can’t be new
• Inject tracker images / strings
• Trace Refers / hit Google
• Key guessing or prediction
• Your data ends up in places you never
expected.

54. Places to keep looking
• Improve data detection/sifting/filtering
• Spread the search past a single provider
• Caching providers (?!?!)
• Other cache software
• Other infrastructure software

55. Questions?
www.sensepost.com/labs/tools/poc/go-derper