DevOps Boston - Heartbleed at Acquia

751 views

Published on

A presentation I gave at DevOps Boston on how we handled the Heartbleed bug at Acquia

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
751
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

DevOps Boston - Heartbleed at Acquia

  1. 1. Marc Seeger (@rb2k)
 Boston Devops Meetup
 May 20th 2014 at
  2. 2. Act 1: Technology
  3. 3. How it all started 7:24 PM
  4. 4. How it all started 7:30 PM
  5. 5. How it all started 7:26 PM
  6. 6. How it all started 7:33 PM
  7. 7. How it all started
  8. 8. Quick risk assessment Lucid: [00:35:27] root@bal-2.dev:~# openssl version OpenSSL 0.9.8k 25 Mar 2009 ! Precise: [00:34:37] root@master.dev:~# openssl version OpenSSL 1.0.1 14 Mar 2012
  9. 9. Where’s Waldo OpenSSL 8000 EC2 Machines: - 99.9% of them puppetized - Candidates: - Balancers - SVN Servers - Appliances - ELBs - 3rd party AMIs - Unique little snowflakes
 (Jira, Crucible,…)
  10. 10. Let the patching begin
  11. 11. Rollout Australia: ! Con: - Spiders - Snakes ! Pro: - Ops is awake
  12. 12. Rollout
  13. 13. Scan www
  14. 14. Waiting on ELBs…
  15. 15. Internal Certificates
  16. 16. Suddenly: “reverse” Heartbleed
  17. 17. Act 2: Communication
  18. 18. Internal • Pre-determined chat rooms • Dial-in conference bridges • A communication plan Thanks SSAE-16, PCI and FedRAMP… I guess :)
  19. 19. Statuspage + Twitter * Powered by StatusPage.io *
  20. 20. Documentation https://docs.acquia.com/articles/heartbleed-acquia-cloud
  21. 21. Proactive communication Phone calls by Acquia support, TAMs, …
  22. 22. Since then: Post mortem
  23. 23. Since then: Incident Commander (shamelessly stolen from Heroku) http://en.wikipedia.org/wiki/Incident_command_system
  24. 24. Since then: Dedicated resource to vet security threats
  25. 25. Since then: Clean up intranet docs
  26. 26. Since then: Additional tooling
  27. 27. We’re hiring (shameless self promotion) bit.ly/acquiajobs

×