Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Life after the hack

2,823 views

Published on

Your Drupal site has just been hacked. Stakeholders are piled around you. What do you do now ?

Published in: Software
  • Be the first to comment

  • Be the first to like this

Life after the hack

  1. 1. Life after the hack OSInetFrédéric G. MARAND (fgm)
  2. 2. 2/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr Topics • 1 Intro : setting the stage • 2 Snapshotting • 3 Maintaining presence • 4 Crisis communication • 5 Rebuild, don’t repair • 6 Using forensics tools • 7 Back online
  3. 3. 4/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.1 Some fact checking first • In this room … • Who has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a contingency plan ? • Who read node 2365547 ?
  4. 4. 5/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
  5. 5. 6/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.2 Can you say that again ? I.A.N.A.L. So be sure to get one !
  6. 6. 7/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.3 Whence do I speak ? • Drupal org member since 2005 (fgm) • Drupal consultant, not a site building agency • Worked on fixing broken (in) sites since 2008 • Auditing • Fixing technical flaws • Addressing intrusions / exploits • Mostly Media and Government sites (.fr) • Provisional member of the Security Team
  7. 7. 8/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.4 Setting the stage • 10:00 The daily scrum has just begun. • 10:01 Phones rings : someone noticed your site has been defaced and is warning you • 10:02 Twitter and Reddit start buzzing • 10:05 Phones ring all over the place, with journalists and the various C-level execs on the other end, your mailbox is filling with warnings • What is your next step ?
  8. 8. 9/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.5 Get ready • Pad 1 : discovery log • all your work steps • all your findings / observations • with timestamps and numbers • Pad 2 : remedies ideas • cross-refer pad 1 numbers • all your ideas for fixing the breach • all your ideas for further hardening
  9. 9. 11/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 2.1 Forensic copy : why ? • First temptation : restore and resume • But you’re still vulnerable • So you need to diagnose • Analyzing means modifying • So preserve the « crime scene » • Snapshot everything
  10. 10. 12/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 2.2 Snapshots : pull the plug • Prevents interference • Shutdown handlers, SIGPWR • Self-destructing code on network loss • Easy on VMs But… • Bare remote servers • Further data loss • Journaled FS • Databases • Service interruption
  11. 11. 13/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 2.3 Snapshots : what ? Not just the main DB • Reverse Proxy logs • Web fronts • DB servers • File servers And also… • External logs (SaaS) • External transactions • IDS/firewall logs The site may just be an attack vector
  12. 12. 15/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.1 Maintaining presence 1 • Yes • Don’t tip off hackers • Keep generating short-term value • No • Increasing damage • Responsibility • Legal • Financial • Moral As though intrusion had not been detected
  13. 13. 16/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.2 Attacker workflow Evolved • Break in • Dig for gold • Implant zombie • Wait for implant migration to archives • Activate • Profit • Alt : Need for Speed • Use exploit ASAP • While it lasts • Usually least loss • Alt : hidden steal • Valuable content • Identity data • Close the door
  14. 14. 17/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.3 Maintaining presence 2 • Limited static site • Best with prior work • Minimal subset • Possibly taken from RP cache • Very little load : can run on RP heads • Working limited site • Alternate infra • Alternate tech • Updates ? • Content created during this step Safe fallback mode
  15. 15. 18/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.4 Maintaining presence 3 When all else fails • Social networks • Always there • Also authoritative for audience • Still needs some preparation : • Accounts access • Include them in long-term communication
  16. 16. 20/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 4.1 Communicating : from tech • Stakeholders • Chain up to CxO level in most cases • Prepare next steps, do not overreach • Fear of reprisal ? Gag orders, SLAPP… • Protection • France : whistleblower protection (Sapin 2) • Italy : Dec. 385 01/09/93 sect 52bis (banks) • US : Anti-SLAPP • Many other countries have similar rules
  17. 17. 21/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 4.2 : Communication : C-level • Legal counsel (first) • Crisis Management specialists • Law enforcement • EU countries typically have specialized units for « cybercrime » • Other sites • On same server • On same network • Online business partners
  18. 18. 22/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 4.3 Communication : privacy • In many cases personal data leaks • will happen, or... • unprovable they did not happen • Operational constraints • Commerce : PCI/DSS (12 steps etc) • Health : (US) HIPAA Subtitle D E2.80.93 • Public image damage control • A french example
  19. 19. 24/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.1 Rebuild : keep, rollback or ? • Restore and restart same ? • Still just as vulnerable • Keep and fix ? • lots of time and effort reviewing • never completely trusted : not just Drupal • Throw away ? • Event sites, past lines of biz, post-M&A... • Can a static version suffice ? • From RP snapshots : recent content
  20. 20. 25/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.2 Rebuild : restore • Needs backups from before the hack • Do you know when it happened ? • Remember attacker workflow « wait » • GFS, continuous incremental, 15 min ? • How much can you lose ? • FLOSS solutions : Amanda, Bacula, custom • Unprepared emergency ? • Preproduction, CI builds...
  21. 21. 26/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.3 Rebuild : sources + export • Easy and reliable, but assumes : • Code-driven development process • Reliable data export system in place • Flat content exports • Content + assets repositories • Still need to add the fixes • Delay can be a problem on high-volume sites • Bulk handling, Incremental loading
  22. 22. 27/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.4 Rebuild : other cases • Ad hoc « traditional » build process • Longer, less reliable • Too long to be a chance to fix the process • From scratch • Too long in most cases • Do it as a complement after the fix • Not NOW
  23. 23. 29/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6 Forensics : switching hats
  24. 24. 30/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.1 Forensics : first, think ! • How did you become aware of hack ? • What did it take to succeed ? • Cast your net wide, think big • « Unlikely » vs « impossible » • Priority : • Easiest attacks first • OWASP 10 • GIYF : search your Pad 1 patterns
  25. 25. 31/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.2 Forensics : keep in mind • /anything/ may be erased after success • But most of the time, not /everything/ will • Anything you do leaves its own traces • Work on copies of the snapshots • You can restart from fresh copies anytime • There maybe more than one exploit
  26. 26. 32/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.3 Forensics : classics • Code files : • lax permissions • filesystem traversal issues • Remote payload execution by upload • Nginx without extra hardening • .htaccess won’t do much good • In-DB PHP • PHP module • Eval-uated code
  27. 27. 33/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.4 Forensics : non-Drupal • Filesystem : • <user>/www-data outside /sites • www-data/www-data suspicious • x bit on files below docroot • timestamps • outside sites/*/files = install • exploits > install • meld with fresh build from sources • Also check outside docroot
  28. 28. 34/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.5 Forensics : Drupal modules • Code signing/diffing : • Hacked! • D7 : md5check, file_integrity • Finding DB PHP • QA (github) • Misc • security_review
  29. 29. 35/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.6 Forensics : DB • Quick wins : • users.email!= users.init • review roles, accounts with admin roles • On corp. sites, users.email domains • match users accounts with SSO data • Diff DB snapshot with live • Especially menu_router : file_put_contents, assert • Altova DatabaseSpy content compare
  30. 30. 36/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.7 Forensics : sessions • Sessions should be in persistent storage • Remember when you pulled the plug • Were your sessions in Memcache ? • sessions.timestamp vs users_field_data : created/changed/access/login • for intranets : sessions.hostname
  31. 31. 37/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.8 Forensics : logs • You use off-site logs, right ? • SaaS : Loggly, Logmatic, Logsene, Logz.io, Papertrail, Scalyr…. • Remote ELK • On site ? • dblog {watchdog} • syslog → follow the redirects • mongodb_watchdog • Application/WS logs
  32. 32. 38/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.9 Forensics : sleuth tools • Software • Guidance Software : Encase • AccessData : Ultimate Forensics Toolkit (FTK) • Consider certified consultants
  33. 33. 40/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.1 Live again : restoring prod • Recheck Pad 1 findings vs new build • Usually, reset passwords. On D7 : • update users set pass = concat('ZZZ', sha(concat(pass, md5(rand()))) ); • Prepare marketing/social copy
  34. 34. 41/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.2 L8R : future-readiness
  35. 35. 42/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.3 L8R : disaster prevention • Developer education on security • Security Team mailing list • https://twitter.com/drupalsecurity • https://www.drupal.org/security/rss.xml • http://crackingdrupal.com/
  36. 36. 43/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.4 L8R : disaster prevention • Security process • Analyse sec. releases to understand fixes • Look for similar flaw in custom code • Take part in contrib for more expertise • Quality process • Systematic peer code reviews • Code-driver maintenance + dev process • Automatic quality tools in CI • Contrib updates scheduling
  37. 37. 44/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.5 Continuous improvement • You can’t improve what you don’t measure • Get time metrics from Pad 1 • Build contigency plan from Pad 2 • Plan for periodic intrusion simulations
  38. 38. 45/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr

×