Heartache and Heartbleed: An inside look a the aftermath of Heartbleed. Presented at 31c3 in Hamburg on December 29, 2014.

1. Heartache and Heartbleed
An inside look at the aftermath of Heartbleed
31c3
Nick Sullivan
@grittygrease

2. CloudFlare Reverse Proxy
2

3. CloudFlare’s Global Network
3

4. Application Layer
• DNS (TCP & UDP port 53)
• HTTP (TCP port 80)
• HTTPS (TCP port 443) - powered by OpenSSL
• Every machine can serve every site
4

5. Customers
5

6. Customers
6

7. It started with a tweet?
7
- Russell Brandom (The Verge)

8. More like
• Do you use TLS heartbeats?
• What is a TLS heartbeat again?
• They’re stupid, you probably don’t need them. Consider turning them off.
• Ok, I’ll compile OpenSSL with -DOPENSSL_NO_HEARTBEATS
8

9. Then it happened
• April 7, 10:27 PDT — OpenSSL publishes advisory
• OpenSSL notification hit #1 on Hacker News
• CloudFlare releases standard “Customer sites are patched” blog post
9

10. Then it REALLY happened
10

11. 11
HEARTBLEED

12. Mass-media
• Codenomicon launches heartbleed.com with logo
• Heartbleed hits the mainstream press
• #heartbleedvirus trending on Twitter
• My mom calls me
12

13. Things to do
13

14. Heartbleed Scanner
14
• Filippo Valsorda’s tool in Go
• Sends a benign heartbeat (~100 bytes)
• Hosted on AWS

15. 15
April 8th (requests/minute)

16. 16
April 8th to April 21

17. 17
203,190,914 tests
Total:
in the first 14 days

18. % of hosts vulnerable

19. Meanwhile, at CloudFlare…
• Log every heartbeat with a
mismatched length
• Don’t look at data until 31c3
19

20. Logs from April 9th
20
Malformed Heartbeats Message Size
69% 16384
20% 121
2% 0
8% All other
ssltest.py?
filippo.io

21. Logs from April 14-16
21
Malformed Heartbeats Message Size
66% 16384
22% 69-131
5% 0
7% All other
ssltest.py?
filippo.io
IP range
1% of all scans

22. Logs from April 14-16
22
ssltest.py
filippo.io
IP range

23. Why is Heartbleed so dangerous?
23
• One request gets attacker server data
• Typically not logged — doesn’t leave a trace
• 1.5 million CloudFlare sites share memory
• Login session cookies
• SSL/TLS private keys(???)

24. 24

25. What does the code say?
25
• Key allocated when process starts
• Copies of keys made at computation time
• OpenSSL bignum library clears allocated memory
• So on a single-threaded server, keys should be safe, right?

26. The CloudFlare Heartbleed challenge
• Let’s crowdsource an answer!
• Standard nginx on digital ocean with vulnerable OpenSSL
• Proof of private key by signing individualized message
26

27. 27
Trolling

28. Challenge Solved
28

29. 29

30. Challenge Solved
30
1. Fedor Indutny (@indutny) Developer
2. Ilkka Mattila, Information Security Adviser
3. Rubin Xu (@xurubin), Security PhD Student
4. Ben Murphy (@benmmurphy), Security Researcher
5. Steve Hunter (@nonaxiomatic)
6. Xavier Martin (@xav), Security Researcher
7. no name given
8. Jeremi Gosney (@jmgosney), CEO, Stricture Group
9. Michele Guerini Rocco (@Rnhmjoj), Student
10.David Gervais (@davidgervais), Software Engineer
11.Christian Bürgi (@buergich)
12.Daniel Burkard (@hiptomcat)

31. • Results: solved in under 10 hours
• Private keys are vulnerable
31

32. How it was solved
• Part of the the private key was on the heap. But why?
• There was a second bug in OpenSSL
32

33. Second OpenSSL bug
33
• Computation uses temporary variables
• Private key can be derived from them
• Some temporary variables were not wiped

34. Cleaning up the mess
34

35. How it was solved - RSA basics
• Two prime numbers P & Q
• Public key, including P x Q
• Finding P or Q can get you the private key
35

36. How it was solved
• Take every 128byte block
• Attempt to divide into public RSA key
• Coppersmith’s attack (only requires partial prime factor)
36

37. 37“Revocation”

38. Revoking 100,000 SSL certificates in 24 hours
38

39. Revoking 100,000 SSL certificates in 24 hours
39

40. How revocation works
CRL
OCSP
CRLSets
40

41. How revocation works
CRL
OCSP
CRLSets
41

42. Revoking 100,000 SSL certificates in 24 hours
• GlobalSign CRL grew from 22KB to 4.7MB
• 30Gbps + 100Gbps waves every three hours
42

43. How revocation works
CRL
OCSP
CRLSets
43

44. OCSP is broken
• OCSP hard fail breaks captive portals
• Soft fail can be circumvented via network manipulation
• Chrome does not check OCSP
44

45. How revocation works
CRL
OCSP
CRLSets
45

46. CRLSets are broken
• Single vendor control
• Only EV certs
• Updates when browser is updated
• None of 100,000+ certs were in CRLSets
• cloudflarechallenge.com was added manually
46

47. Most efficient revocation code ever
Chromium Issue 267913003
47

48. How revocation works
CRL
OCSP
CRLSets
48

49. Revocation solutions
• Shorter certificate expiration periods?
• OCSP Must-staple?
• Certificate Transparency?
49

50. Things we did
50

51. Conclusions
• Disclosure in open source are hard
• Many “attacks” were scans
• Crowdsourcing was effective
• Revocation needs a solution
51

52. Heartache and Heartbleed
An inside look at the aftermath of Heartbleed