Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to investigate and recover from a security breach in WordPress


Published on

Talk given at the first ever WordCamp Nordic on March 8th, 2019.

How to investigate and recover from a security breach – real-life experiences with WordPress

Sometimes the bad guys get in, despite all the protections and precautions. If that happens, there are many techniques that can be used to stop further damage, track down what the intruder did and how they got in. Finally the site needs to be cleaned up and re-opened for visitors. In this talk the most important techniques are presented along with real-life examples when they were used.

Published in: Software
  • Login to see the comments

  • Be the first to like this

How to investigate and recover from a security breach in WordPress

  1. 1. How to Investigate and Recover from a Security Breach Real-life Experiences with WordPress Otto Kekäläinen @ottokekalainen WordCamp Nordic March 8, 2019
  2. 2. ● A CEO who codes at ● Written WP themes and plugins, contributed to WordPress Core, MySQL, MariaDB, Debian, Ubuntu, Linux kernel, AppArmor… ● Linux and open source advocate Otto Kekäläinen
  3. 3. I’ve spoken many times about what WordPress site owners should focus on to keep their site secure...äläinen
  4. 4. ...but not today. This talk is different.
  5. 5. This talk is about Friday, November 9th 2018.
  6. 6. Premium hosting and upkeep for WordPress HTTP/2 TESTED UPDATES 24/7 UPKEEP
  7. 7. Upkeep: If a site goes down, we bring it up again. Covers security incidents.
  8. 8. 2018-11-09 11:37:48 <redacted> ALERT ! ! ! 2018-11-09 11:40:26 <redacted> ALERT ! ! ! 2018-11-09 11:40:42 <redacted> ALERT ! ! ! 2018-11-09 11:42:37 <redacted> ALERT ! ! ! Just one ordinary Friday (not even 13th!)
  9. 9. Weird siteurl – on all 4 sites! Mistake by site admin? – No way Targeted attack on one and same company? – Plausible, but weird modus of operandi Security breach? – Definitely! $ wp option get siteurl
  10. 10. High alert – 4 sites down for investigation 1. First responder notifies security officer on-call 2. Process list saved and further PHP execution frozen 3. Customer notified about on-going security incident 4. Response escalation: 3 investigators working in parallel
  11. 11. 11:55
  12. 12. Security breach investigation questions ● What is happening? Is it stopped? ● What happened before? When did this start? ● Is there malicious code somewhere? Backdoors planted? ● What files or database contents has changed? Which changes are malicious? ● Who did what? What IP addresses and other identifiers are linked to what actions?
  13. 13. Security breach investigation questions ● How did they get in? ● What level of access did they gain? ● What data could have leaked? ● What was their motive? ● What damage was caused?
  14. 14. Investigation and recovery steps 1. Make a new backup 2. Compare backups wp-backup-list-changes diff -ur wordpress backup/wordpress 3. Check last WP and SSH logins Store current state Reveal file and database changes Detect unauthorized use based on anomalies in timestamps or IP geolocation
  15. 15. Investigation and recovery steps 4. wp core verify-checksums wp plugin verify-checksums --all wp package install seravo/wp-checksum wp checksum all --details Compare WordPress core, plugin and theme files to their original versions as downloaded from
  16. 16. Modified plugin code found ..but was a false alert, modification most likely a mistake by real plugin author who released two plugin variants published with same version number. $ wp checksum diff plugin entry-views inc/widget-entry-views.php Executing diff /tmp/1541763665-4CBDYu.tmp wordpress/htdocs/wp-content/plugins/entry-views/inc/widget-entry-views.php 49c49 < $this->WP_Widget( --- > parent::__construct(
  17. 17. 13:31
  18. 18. Investigation and recovery steps 5. wp user list 6. wp db query 'SELECT post_modified, id, post_title, post_name, post_type FROM wp_posts ORDER BY id DESC LIMIT 50;' View recent new users View recent new contents
  19. 19. Two suspected attacker user accounts Variants of trollherten and different .ru email addresses found on multiple of the investigated sites. $ wp user list +----+---------------+--------------+----------------------+---------------------+---------------+ | ID | user_login | display_name | user_email | user_registered | roles | +----+---------------+--------------+----------------------+---------------------+---------------+ | 12 | t2trollherten | | | 2018-11-08 13:36:03 | administrator | | 13 | t3trollherten | | | 2018-11-08 14:42:09 | administrator | +----+---------------+--------------+----------------------+---------------------+---------------+
  20. 20. Bingo! usernames, timestamps, IP addresses, email +----+---------------+--------------+----------------------+---------------------+---------------+ | ID | user_login | display_name | user_email | user_registered | roles | +----+---------------+--------------+----------------------+---------------------+---------------+ | 12 | t2trollherten | | | 2018-11-08 13:36:03 | administrator | | 13 | t3trollherten | | | 2018-11-08 14:42:09 | administrator | +----+---------------+--------------+----------------------+---------------------+---------------+ These can be given to grep /data/log for log data mining
  21. 21. The entry - - [08/Nov/2018:15:36:02 +0200] "GET / HTTP/1.1" 200 19027 "-" 0.301 - - [08/Nov/2018:15:36:02 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.220 - - [08/Nov/2018:15:36:03 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.258 - - [08/Nov/2018:15:36:04 +0200] "POST /wp-login.php?action=register HTTP/1.1" 302 5 "-" 0.648 - - [08/Nov/2018:15:36:04 +0200] "GET /wp-login.php?checkemail=registered HTTP/1.1" 200 1463 "https://<redacted>/wp-login.php?action=register" 0.129 - - [08/Nov/2018:15:36:04 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.163 - - [08/Nov/2018:15:36:04 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.167 $ host domain name pointer User agent was: "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
  22. 22. POST /wp-admin/admin-ajax.php ● Seravo does not log POST requests for good reasons ● So what was the payload that granted magic powers to the attacker? ● Luckily we have other PHP and database logs...
  23. 23. Anomalies in database use ● Weird empty WordPress options value updates ● Unusual requests to database table wpgdprc_access_requests ● What plugin does that belong to? $ grep -rF wpgdprc_access_requests wp-gdpr-compliance/ wp-gdpr-compliance/Includes/AccessRequest.php: return $wpdb->base_prefix . 'wpgdprc_access_requests'; wp-gdpr-compliance/uninstall.php: $wpdb->query("DROP TABLE IF EXISTS `{$wpdb->base_prefix}wpgdprc_access_requests`");
  24. 24. 14:03
  25. 25. Hmm.. Recent wp-gdpr-compliance plugin code changes smell like SQL injection fixes
  26. 26. Point of entry known ● The plugin WP GDPR Compliance Plugin most likely route ● Fix: remove it from all 4 sites $ wp plugin deactivate --uninstall wp-gdpr-compliance
  27. 27. 14:35
  28. 28. More information started coming in ● When the US woke up (in European afternoon) and published blogs the Sucuri RSS feed we subscribe showed interesting stuff: h-wp-gdpr-compliance-plugin-vulnerability.html ● Then more and more other reports were found: a. wp-gdpr-compliance-plugin-exploited-in-the-wild/ b. c.
  29. 29. Vulnerability details ● A SQL injection flaw in WP GDPR Compliance allowed a remote attacker to set arbitrary WP option values a. First allow anybody to register with users_can_register=1 b. Then set default_role=”administrator” for all new users c. Register an account, log in and do whatever an admin can do ● Reported to by Adrian Mörchen / ● Fixed in WP GDPR Compliance version 1.4.3
  30. 30. Fix issue globally for all our customers commit 2ffb891415628ead16263e1fa09d78dac9e5dcdd Author: Ville Korhonen Date: Fri Nov 9 14:51:18 2018 +0200 Add WP GDPR Compliance plugin to urgent updates WP GDPR Compliance < 1.4.3 has critical SQL injection flaw which allows simple privilege escalation. <> Added to Seravo’s update systems as an urgent update
  31. 31. 14:51
  32. 32. Investigation and recovery steps 7. Based on findings, clean up the site a. Recover clean version from backups b. Remove malicious code and content manually 8. As a precaution, reset all WordPress user sessions and passwords wp-reset-all-passwords In this case option A was not possible, but luckily option B was quite easy as backups showed only one potential malware file was injected.
  33. 33. Investigation and recovery steps 9. As extra precaution, scan the site for malware one more time when it is otherwise deemed to be clean Using Seravo’s custom made WordPress/PHP malware scanner
  34. 34. 2018-11-09 15:26:22 <redacted> RESOLVED 2018-11-09 15:17:49 <redacted> RESOLVED 2018-11-09 15:29:03 <redacted> RESOLVED 2018-11-09 15:20:24 <redacted> RESOLVED All sites clean and finally back online
  35. 35. Investigation and recovery steps 10. Elevated monitoring and follow-up for site once it has been re-opened, just in case there was more attack avenues not discovered during the investigation. During the investigation Seravo sent 8 status update e-mails to the site owner and the customer mobilized their own team to support the effort and they also sent us valuable additional information. A few additional emails from Seravo to the customer followed over the weekend and next week to confirm all necessary measures had been completed.
  36. 36. Notification e-mail from new registration of ‘trollherten’ users ● Later we found out the site owner did get an email notification from WordPress about the new user named “trollherten” but since the e-mail was vague and did not contain any alarming information, the person who read the e-mail ignored it.
  37. 37. Luckily this was not a targeted attack ● Most likely the attacker just wanted to own the site and use it to redirect traffic, spam, mount more attacks against other sites etc. ● The site itself or the data it had was not the target and most likely not used.
  38. 38. Be prepared: no security is perfect ● No plugin author makes perfect code. ● All plugins on the site were updated a week earlier, the vulnerability was used close to zero-day. ● Unreasonable for site admin to read deeply all notification e-mails. ● Fact: sometimes even good security isn’t enough. One also needs to have a security incident response plan. ● We do. Do you?
  39. 39. Thank you! @ottokekalainen