Presentation made at DrupalGov Canberra 2014 where I talked about how the Heartbleed OpenSSL vulnerability affected our systems and the path we took to patching the vulnerability.

1. Tim Hilliard
DrupalGov Canberra
August 22nd 2014
at

2. This session
1. Hearbleed case study
2. Q & A with:
• Tim Hilliard (Cloud Eng)
• Adam Malone (Support)
• Chris O’Neill (Support)
• Phil Ingrim (Ops)

3. About Me
Cloud Platform Engineer
@big_bear84
or timhilliard on
github, d.o., etc.

4. Act 1: Technology

5. How it all started
8 April 9:30 AM AEST
ZZZzzz….
!
Me == Sleeping

6. How it all started
8 April 9:26 AM AEST

8. How it all started
8 April 9:33 AM AEST

9. How it all started

10. Risk assessment
Lucid:
[00:35:27] root@bal-2.dev:~# openssl version
OpenSSL 0.9.8k 25 Mar 2009
!
Precise:
[00:34:37] root@master.dev:~# openssl version
OpenSSL 1.0.1 14 Mar 2012

11. Where’s Wally OpenSSL
8000 EC2 Machines:
- 99.9% of them puppetized
- Candidates:
- Balancers
- SVN Servers
- Appliances
- ELBs
- 3rd party AMIs
- Unique little snowflakes
(Jira, Crucible,…)

12. Stack
Web tier
Other services
DB, shared filesystem, memcache
Balancers
Varnish
Port 80 Port 443
Nginx
ELBs
Internet
Here!

13. Stack
Web tier
Other services
DB, shared filesystem, memcache
Balancers
Varnish
Port 80 Port 443
Nginx
ELBs
Internet
Here!
and here!

14. Support
11:52:32 Adam Malone: hi QQ opes, I here ther is a
heartbloom security issue with ssh. Is this being treated
with high urgencies (p1) we need to escalade this if
possible

15. Let the patching begin

16. Rollout
Australia:
!
Con:
- Spiders
- Snakes
!
Pro:
- Ops is awake

17. Rollout

18. Rollout
We did not fail over EIPs to passive balancers when
upgrading Nginx.
!
Failing over an EIP leaves the IP disassociated for up to about
3 minutes. Upgrading Nginx in place takes as long as it takes
to restart Nginx. So a matter of seconds.
!
Linux package management ++

19. Rollout
• Rollout across the entire platform took ~4 hours
• ~800 balancers to upgrade.

20. Scan
www
https://filippo.io/Heartbleed/

21. Waiting on ELBs…

22. Internal Certificates

23. Suddenly:
“reverse” Heartbleed

24. Act 2: Communication

25. Internal
• Pre-determined chat rooms
• Dial-in conference bridges
• A communication plan
Thanks SSAE-16, PCI and FedRAMP… I guess :)

26. Statuspage + Twitter
* Powered by StatusPage.io

27. Documentation
https://docs.acquia.com/articles/heartbleed-acquia-cloud

28. Proactive communication
Phone calls by Acquia support, TAMs, …

29. Since then:
Post mortem

30. Since then:
Incident Commander
(shamelessly stolen from Heroku)
http://en.wikipedia.org/wiki/Incident_command_system

31. Since then:
Dedicated resource to vet security threats

32. Since then:
Clean up intranet docs

33. Since then:
Additional tooling

34. We’re hiring
(shameless self promotion)
bit.ly/acquiajobs

35. Q & A
• Tim Hilliard (@big_bear84) (d.o./u/timhilliard)
• Phil Ingrim (he’s in ops so doesn’t have ANY social media)
• Chris O’Neill (@cjoneill)
• Adam Malone (@adammalone) (d.o/u/typhonius)