Life after the hack

1,895 views

Published on

Your Drupal site has just been hacked. Stakeholders are piled around you. What do you do now ?

Published in: Software
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,895
On SlideShare
0
From Embeds
0
Number of Embeds
1,237
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Life after the hack

  1. 1. Life after the hack OSInetFrédéric G. MARAND (fgm)
  2. 2. 2/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr Topics • 1 Intro : setting the stage • 2 Snapshotting • 3 Maintaining presence • 4 Crisis communication • 5 Rebuild, don’t repair • 6 Using forensics tools • 7 Back online
  3. 3. 4/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.1 Some fact checking first • In this room … • Who has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a contingency plan ? • Who read node 2365547 ?
  4. 4. 5/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
  5. 5. 6/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.2 Can you say that again ? I.A.N.A.L. So be sure to get one !
  6. 6. 7/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.3 Whence do I speak ? • Drupal org member since 2005 (fgm) • Drupal consultant, not a site building agency • Worked on fixing broken (in) sites since 2008 • Auditing • Fixing technical flaws • Addressing intrusions / exploits • Mostly Media and Government sites (.fr) • Provisional member of the Security Team
  7. 7. 8/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.4 Setting the stage • 10:00 The daily scrum has just begun. • 10:01 Phones rings : someone noticed your site has been defaced and is warning you • 10:02 Twitter and Reddit start buzzing • 10:05 Phones ring all over the place, with journalists and the various C-level execs on the other end, your mailbox is filling with warnings • What is your next step ?
  8. 8. 9/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 1.5 Get ready • Pad 1 : discovery log • all your work steps • all your findings / observations • with timestamps and numbers • Pad 2 : remedies ideas • cross-refer pad 1 numbers • all your ideas for fixing the breach • all your ideas for further hardening
  9. 9. 11/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 2.1 Forensic copy : why ? • First temptation : restore and resume • But you’re still vulnerable • So you need to diagnose • Analyzing means modifying • So preserve the « crime scene » • Snapshot everything
  10. 10. 12/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 2.2 Snapshots : pull the plug • Prevents interference • Shutdown handlers, SIGPWR • Self-destructing code on network loss • Easy on VMs But… • Bare remote servers • Further data loss • Journaled FS • Databases • Service interruption
  11. 11. 13/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 2.3 Snapshots : what ? Not just the main DB • Reverse Proxy logs • Web fronts • DB servers • File servers And also… • External logs (SaaS) • External transactions • IDS/firewall logs The site may just be an attack vector
  12. 12. 15/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.1 Maintaining presence 1 • Yes • Don’t tip off hackers • Keep generating short-term value • No • Increasing damage • Responsibility • Legal • Financial • Moral As though intrusion had not been detected
  13. 13. 16/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.2 Attacker workflow Evolved • Break in • Dig for gold • Implant zombie • Wait for implant migration to archives • Activate • Profit • Alt : Need for Speed • Use exploit ASAP • While it lasts • Usually least loss • Alt : hidden steal • Valuable content • Identity data • Close the door
  14. 14. 17/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.3 Maintaining presence 2 • Limited static site • Best with prior work • Minimal subset • Possibly taken from RP cache • Very little load : can run on RP heads • Working limited site • Alternate infra • Alternate tech • Updates ? • Content created during this step Safe fallback mode
  15. 15. 18/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 3.4 Maintaining presence 3 When all else fails • Social networks • Always there • Also authoritative for audience • Still needs some preparation : • Accounts access • Include them in long-term communication
  16. 16. 20/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 4.1 Communicating : from tech • Stakeholders • Chain up to CxO level in most cases • Prepare next steps, do not overreach • Fear of reprisal ? Gag orders, SLAPP… • Protection • France : whistleblower protection (Sapin 2) • Italy : Dec. 385 01/09/93 sect 52bis (banks) • US : Anti-SLAPP • Many other countries have similar rules
  17. 17. 21/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 4.2 : Communication : C-level • Legal counsel (first) • Crisis Management specialists • Law enforcement • EU countries typically have specialized units for « cybercrime » • Other sites • On same server • On same network • Online business partners
  18. 18. 22/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 4.3 Communication : privacy • In many cases personal data leaks • will happen, or... • unprovable they did not happen • Operational constraints • Commerce : PCI/DSS (12 steps etc) • Health : (US) HIPAA Subtitle D E2.80.93 • Public image damage control • A french example
  19. 19. 24/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.1 Rebuild : keep, rollback or ? • Restore and restart same ? • Still just as vulnerable • Keep and fix ? • lots of time and effort reviewing • never completely trusted : not just Drupal • Throw away ? • Event sites, past lines of biz, post-M&A... • Can a static version suffice ? • From RP snapshots : recent content
  20. 20. 25/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.2 Rebuild : restore • Needs backups from before the hack • Do you know when it happened ? • Remember attacker workflow « wait » • GFS, continuous incremental, 15 min ? • How much can you lose ? • FLOSS solutions : Amanda, Bacula, custom • Unprepared emergency ? • Preproduction, CI builds...
  21. 21. 26/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.3 Rebuild : sources + export • Easy and reliable, but assumes : • Code-driven development process • Reliable data export system in place • Flat content exports • Content + assets repositories • Still need to add the fixes • Delay can be a problem on high-volume sites • Bulk handling, Incremental loading
  22. 22. 27/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 5.4 Rebuild : other cases • Ad hoc « traditional » build process • Longer, less reliable • Too long to be a chance to fix the process • From scratch • Too long in most cases • Do it as a complement after the fix • Not NOW
  23. 23. 29/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6 Forensics : switching hats
  24. 24. 30/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.1 Forensics : first, think ! • How did you become aware of hack ? • What did it take to succeed ? • Cast your net wide, think big • « Unlikely » vs « impossible » • Priority : • Easiest attacks first • OWASP 10 • GIYF : search your Pad 1 patterns
  25. 25. 31/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.2 Forensics : keep in mind • /anything/ may be erased after success • But most of the time, not /everything/ will • Anything you do leaves its own traces • Work on copies of the snapshots • You can restart from fresh copies anytime • There maybe more than one exploit
  26. 26. 32/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.3 Forensics : classics • Code files : • lax permissions • filesystem traversal issues • Remote payload execution by upload • Nginx without extra hardening • .htaccess won’t do much good • In-DB PHP • PHP module • Eval-uated code
  27. 27. 33/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.4 Forensics : non-Drupal • Filesystem : • <user>/www-data outside /sites • www-data/www-data suspicious • x bit on files below docroot • timestamps • outside sites/*/files = install • exploits > install • meld with fresh build from sources • Also check outside docroot
  28. 28. 34/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.5 Forensics : Drupal modules • Code signing/diffing : • Hacked! • D7 : md5check, file_integrity • Finding DB PHP • QA (github) • Misc • security_review
  29. 29. 35/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.6 Forensics : DB • Quick wins : • users.email!= users.init • review roles, accounts with admin roles • On corp. sites, users.email domains • match users accounts with SSO data • Diff DB snapshot with live • Especially menu_router : file_put_contents, assert • Altova DatabaseSpy content compare
  30. 30. 36/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.7 Forensics : sessions • Sessions should be in persistent storage • Remember when you pulled the plug • Were your sessions in Memcache ? • sessions.timestamp vs users_field_data : created/changed/access/login • for intranets : sessions.hostname
  31. 31. 37/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.8 Forensics : logs • You use off-site logs, right ? • SaaS : Loggly, Logmatic, Logsene, Logz.io, Papertrail, Scalyr…. • Remote ELK • On site ? • dblog {watchdog} • syslog → follow the redirects • mongodb_watchdog • Application/WS logs
  32. 32. 38/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 6.9 Forensics : sleuth tools • Software • Guidance Software : Encase • AccessData : Ultimate Forensics Toolkit (FTK) • Consider certified consultants
  33. 33. 40/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.1 Live again : restoring prod • Recheck Pad 1 findings vs new build • Usually, reset passwords. On D7 : • update users set pass = concat('ZZZ', sha(concat(pass, md5(rand()))) ); • Prepare marketing/social copy
  34. 34. 41/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.2 L8R : future-readiness
  35. 35. 42/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.3 L8R : disaster prevention • Developer education on security • Security Team mailing list • https://twitter.com/drupalsecurity • https://www.drupal.org/security/rss.xml • http://crackingdrupal.com/
  36. 36. 43/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.4 L8R : disaster prevention • Security process • Analyse sec. releases to understand fixes • Look for similar flaw in custom code • Take part in contrib for more expertise • Quality process • Systematic peer code reviews • Code-driver maintenance + dev process • Automatic quality tools in CI • Contrib updates scheduling
  37. 37. 44/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr 7.5 Continuous improvement • You can’t improve what you don’t measure • Get time metrics from Pad 1 • Build contigency plan from Pad 2 • Plan for periodic intrusion simulations
  38. 38. 45/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr

×