Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Edge 2014: Bypass Surgery - Akamai's Heartbleed Response Case Study

6,588 views

Published on

Bypass Surgery - Akamai's Heartbleed Response Case Study by Brian Sniffen, Chief Security Architect, Akamai Technologies

In April of this year, the inevitable happened and Akamai's network was determined to be vulnerable to the Heartbleed bug. The practice of information security is both about preventing vulnerabilities and mitigating vulnerabilities when they're found. In this case study, Akamai Chief Security Architect Brian Sniffen will walk through Akamai's response to the Heartbleed vulnerability and provide insights into the lessons to be learned for improved security processes and incident response.

Akamai Edge is the premier event for Internet innovators, tech professionals and online business pioneers who together are forging a Faster Forward World. At Edge, the architects, experts and implementers of the most innovative global online businesses gather face-to-face for an invaluable three days of sharing, learning and together pushing the limits of the Faster Forward World. Learn more at: http://www.akamai.com/edge

Published in: Technology
  • Be the first to comment

Edge 2014: Bypass Surgery - Akamai's Heartbleed Response Case Study

  1. 1. Bypass Surgery and Other Tales Brian Sniffen
  2. 2. ©2014 AKAMAI | FASTER FORWARDTM Akamai Security Research & Architecture • Crypto engineering expertise • Technical backstop • Product review • Akamai Architecture Group seat • Safety engineering • Incident management
  3. 3. ©2014 AKAMAI | FASTER FORWARDTM 2014: The Year of Vulnerabilities “You people in InfoSec have become the Product Managers!” Yes, and we can’t wait to get out of that role. • Heartbleed • INRIA-Prosecco Cookies • Shellshock 2009 2010 2011 2012 2013 2014 1 0 0 1 1 3+ 2008 1
  4. 4. ©2014 AKAMAI | FASTER FORWARDTM Akamai Incident Management Principles • Technical Incident Manager (TIM) coordinates all work • Incident Executive communicates upwards, shields TIM • GSS Business Incident Leads manage customer comms • No single-point harm can cause a Severity 1 Incident • A hot meal and 6 hours sleep fix more problems than an all-nighter • If the TIM becomes an SME, get a new TIM
  5. 5. ©2014 AKAMAI | FASTER FORWARDTM We tell ourselves who we wish to be: • Akamai says thank you. • Akamai doesn't respond to name calling, but does respond to the useful technical content. • Akamai presents itself as a responsible and respected member of the Internet community. • Akamai will use this incident to improve both its own security and the general security of the Internet. • Akamai can laugh at itself.
  6. 6. ©2014 AKAMAI | FASTER FORWARDTM Heartbleed mail From: Brian Sniffen Date: 7 Apr 2014 21:34:08 Subject: Sev 1: Heartbleed Will, I'm contacting you because you're the Ghost SME on call. I'm looking for evidence to refute the statement: "The Heartbleed bug can’t extract long-term customer private keys from a Ghost; we put them only in a wired, mmaped page way lower on the stack."
  7. 7. ©2014 AKAMAI | FASTER FORWARDTM Heartbleed Timeline April 1: Notice; QA begins April 4: last Akamai Deployed Systems patched April 7, 1pm: Public Notice April 7, 6pm: What did we leak? April 8, 1am: Working exploit in Akamai lab April 9–12: Hastily publish Akamai Secure Allocator April 13, 11pm: Begin cert rotations & revocations
  8. 8. ©2014 AKAMAI | FASTER FORWARDTM “Don’t worry, we restored the old functionality” April 14, 6am: “Why is this message in the old log format?” A “Manual Change” had restored an old version.
  9. 9. ©2014 AKAMAI | FASTER FORWARDTM The Akamai Secure Memory Allocator • 1999 code • One author, three redactors • State machine inspired by CLOS “advice” system Turns out it works fine Code Secure Heap mmap’d file Long-term Allocations Heap
  10. 10. 70% 90% 95% # of certs ©2014 AKAMAI | FASTER FORWARDTM Cert Revocation Progress 21 Apr 28 Apr 5 May 12 May 19 May 26 May 2 Jun 9 Jun 16 Jun 23 Jun 30 Jun
  11. 11. ©2014 AKAMAI | FASTER FORWARDTM Learning from Heartbleed Nobody’s paying for OpenSSL! Practice in mass, fast, patching Practice in releasing helpful patches Simplicity promotes safety.
  12. 12. ©2014 AKAMAI | FASTER FORWARDTM Shellshock Timeline Sep 23, 12pm: Notice from Florian Weimer, Debian Security Sep 23, 9pm: Manual change: replace bash wish dash; Patches started Sep 24, 5am: WAF rule in place SSH command= systems made safe Sep 24, 12pm: Public Notice Sep 25: “Kobrin Patch” to remove dangerous feature Sep 28: bash mostly replaced with dash on deployed network
  13. 13. ©2014 AKAMAI | FASTER FORWARDTM Bash patches Pre-release: • Embargoed patch: 195 lines, 7 files (1/6 CVEs) • Kobrin patch: 2 lines, 1 file (6/6 CVEs) Post-release: • NetBSD patch: 3 lines, 2 files (6/6 CVEs) • Fixed patch: 164 lines, 11 files (6/6 CVEs) • Apple patch: unpublished (exposure unclear)
  14. 14. ©2014 AKAMAI | FASTER FORWARDTM SSH command= limits ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFnHfYyS5onAN72oFpaopm+/yKbRy/ TCwpt7Tmw3lk0P bts@tereva.evenmere.org-2014-03-19 command="/a/bin/akamai_run suspend" ssh-ed25519 AAAAB3NzaC1yc2EAAAADAQABAAABAQDKVmNk8leXjKkWZUHQjJIT zrX+n1aa1xfBwK9Yp42q bsniffen@akamai.com-example-2014-10-01 V=“() { :;}; /bin/bash” ssh example.com :
  15. 15. ©2014 AKAMAI | FASTER FORWARDTM Akamai Shellshock exposures Client ssh Authgate Server 1 Server 2 Perforce Server 3 https ssh Web Kerberos CGI
  16. 16. ©2014 AKAMAI | FASTER FORWARDTM Solaris 10 “We don’t have any Solaris 10 admins”
  17. 17. ©2014 AKAMAI | FASTER FORWARDTM Who’s looking? 13000 IPs probing per day
  18. 18. ©2014 AKAMAI | FASTER FORWARDTM Learnings from Shellshock Nobody’s paying for Bash. And it was written in the 1980s! Simplicity promotes safety.
  19. 19. ©2014 AKAMAI | FASTER FORWARDTM The New Normal • Two or three internet-wide patching incidents per year • Enterprise-wide compliance takes months • Trust less code. • Trust code less. • Treat upstream code like you wrote it? • Homework: set up 24/7 contacts and Security contacts

×