1.
How to Investigate and
Recover from a Security Breach
Real-life Experiences with WordPress
Otto Kekäläinen
@ottokekalainen
WordCamp Nordic
March 8, 2019

2.
● A CEO who codes at Seravo.com
● Written WP themes and plugins,
contributed to WordPress Core,
MySQL, MariaDB, Debian, Ubuntu,
Linux kernel, AppArmor…
● Linux and open source advocate
Otto Kekäläinen

3.
I’ve spoken many
times about what
WordPress site
owners should
focus on to keep
their site secure...
wordpress.tv/?s=otto+kekäläinen

4.
...but not today.
This talk is
different.

5.
This talk is about Friday,
November 9th
2018.

6.
Premium hosting
and upkeep for
WordPress
HTTP/2
TESTED
UPDATES
24/7 UPKEEP

7.
Upkeep:
If a site goes down,
we bring it up again.
Covers security
incidents.

8.
2018-11-09 11:37:48 <redacted>.seravo.com ALERT ! ! !
2018-11-09 11:40:26 <redacted>.seravo.com ALERT ! ! !
2018-11-09 11:40:42 <redacted>.seravo.com ALERT ! ! !
2018-11-09 11:42:37 <redacted>.seravo.com ALERT ! ! !
Just one ordinary Friday (not even 13th!)

9.
Weird siteurl – on all 4 sites!
Mistake by site admin? – No way
Targeted attack on one and same company? –
Plausible, but weird modus of operandi
Security breach? – Definitely!
$ wp option get siteurl
http://erealitatea.net

10.
High alert – 4 sites down for investigation
1. First responder notifies security officer on-call
2. Process list saved and further PHP execution frozen
3. Customer notified about on-going security incident
4. Response escalation: 3 investigators working in parallel

11.
11:55

12.
Security breach investigation questions
● What is happening? Is it stopped?
● What happened before? When did this start?
● Is there malicious code somewhere? Backdoors
planted?
● What files or database contents has changed? Which
changes are malicious?
● Who did what? What IP addresses and other
identifiers are linked to what actions?

13.
Security breach investigation questions
● How did they get in?
● What level of access did they gain?
● What data could have leaked?
● What was their motive?
● What damage was caused?

14.
Investigation and recovery steps
1. Make a new backup
2. Compare backups
wp-backup-list-changes
diff -ur wordpress backup/wordpress
3. Check last WP and SSH logins
Store current state
Reveal file and database
changes
Detect unauthorized use based
on anomalies in timestamps or IP
geolocation

15.
Investigation and recovery steps
4. wp core verify-checksums
wp plugin verify-checksums --all
wp package install seravo/wp-checksum
wp checksum all --details
Compare WordPress core,
plugin and theme files to
their original versions as
downloaded from wp.org

16.
Modified plugin code found
..but was a false alert, modification most likely a mistake
by real plugin author who released two plugin variants
published with same version number.
$ wp checksum diff plugin entry-views inc/widget-entry-views.php
Executing diff /tmp/1541763665-4CBDYu.tmp
wordpress/htdocs/wp-content/plugins/entry-views/inc/widget-entry-views.php
49c49
< $this->WP_Widget(
---
> parent::__construct(

17.
13:31

18.
Investigation and recovery steps
5. wp user list
6. wp db query
'SELECT post_modified, id, post_title,
post_name, post_type FROM wp_posts
ORDER BY id DESC LIMIT 50;'
View recent new users
View recent new contents

19.
Two suspected attacker user accounts
Variants of trollherten and different .ru email addresses
found on multiple of the investigated sites.
$ wp user list
+----+---------------+--------------+----------------------+---------------------+---------------+
| ID | user_login | display_name | user_email | user_registered | roles |
+----+---------------+--------------+----------------------+---------------------+---------------+
| 12 | t2trollherten | | trollherten@mail.com | 2018-11-08 13:36:03 | administrator |
| 13 | t3trollherten | | t3trollherten@bk.ru | 2018-11-08 14:42:09 | administrator |
+----+---------------+--------------+----------------------+---------------------+---------------+

20.
Bingo!
usernames, timestamps,
IP addresses, email
+----+---------------+--------------+----------------------+---------------------+---------------+
| ID | user_login | display_name | user_email | user_registered | roles |
+----+---------------+--------------+----------------------+---------------------+---------------+
| 12 | t2trollherten | | trollherten@mail.com | 2018-11-08 13:36:03 | administrator |
| 13 | t3trollherten | | t3trollherten@bk.ru | 2018-11-08 14:42:09 | administrator |
+----+---------------+--------------+----------------------+---------------------+---------------+
These can be given to grep /data/log for log data mining

21.
The entry
109.234.37.214 - - [08/Nov/2018:15:36:02 +0200] "GET / HTTP/1.1" 200 19027 "-" 0.301
109.234.37.214 - - [08/Nov/2018:15:36:02 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.220
109.234.37.214 - - [08/Nov/2018:15:36:03 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.258
109.234.37.214 - - [08/Nov/2018:15:36:04 +0200] "POST /wp-login.php?action=register HTTP/1.1" 302 5 "-" 0.648
109.234.37.214 - - [08/Nov/2018:15:36:04 +0200] "GET /wp-login.php?checkemail=registered HTTP/1.1" 200 1463
"https://<redacted>/wp-login.php?action=register" 0.129
109.234.37.214 - - [08/Nov/2018:15:36:04 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.163
109.234.37.214 - - [08/Nov/2018:15:36:04 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 36 "-" 0.167
$ host 109.234.37.214
214.37.234.109.in-addr.arpa domain name pointer host-109-234-37-214.hosted-by-vdsina.ru.
User agent was:
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"

22.
POST /wp-admin/admin-ajax.php
● Seravo does not log POST requests for good reasons
● So what was the payload that granted magic powers
to the attacker?
● Luckily we have other PHP and database logs...

23.
Anomalies in database use
● Weird empty WordPress options value updates
● Unusual requests to database table wpgdprc_access_requests
● What plugin does that belong to?
$ grep -rF wpgdprc_access_requests wp-gdpr-compliance/
wp-gdpr-compliance/Includes/AccessRequest.php:
return $wpdb->base_prefix . 'wpgdprc_access_requests';
wp-gdpr-compliance/uninstall.php:
$wpdb->query("DROP TABLE IF EXISTS
`{$wpdb->base_prefix}wpgdprc_access_requests`");

24.
14:03

25.
Hmm..
Recent
wp-gdpr-compliance
plugin code
changes smell
like SQL injection
fixes

26.
Point of entry known
● The plugin WP GDPR Compliance Plugin most likely route
● Fix: remove it from all 4 sites
$ wp plugin deactivate --uninstall wp-gdpr-compliance

27.
14:35

28.
More information started coming in
● When the US woke up (in European afternoon) and published blogs the
Sucuri RSS feed we subscribe showed interesting stuff:
blog.sucuri.net/2018/11/erealitatea-net-hack-corrupts-websites-wit
h-wp-gdpr-compliance-plugin-vulnerability.html
● Then more and more other reports were found:
a. www.wordfence.com/blog/2018/11/privilege-escalation-flaw-in-
wp-gdpr-compliance-plugin-exploited-in-the-wild/
b. vitalisec.blogspot.com/2018/11/wp-gdpr-plugin-attack.html
c. wpvulndb.com/vulnerabilities/9144

29.
Vulnerability details
● A SQL injection flaw in WP GDPR Compliance allowed a remote
attacker to set arbitrary WP option values
a. First allow anybody to register with users_can_register=1
b. Then set default_role=”administrator” for all new users
c. Register an account, log in and do whatever an admin can do
● Reported to wpvulndb.com by Adrian Mörchen / moewe.io
● Fixed in WP GDPR Compliance version 1.4.3

30.
Fix issue globally for all our customers
commit 2ffb891415628ead16263e1fa09d78dac9e5dcdd
Author: Ville Korhonen
Date: Fri Nov 9 14:51:18 2018 +0200
Add WP GDPR Compliance plugin to urgent updates
WP GDPR Compliance < 1.4.3 has critical SQL injection
flaw which allows simple privilege escalation.
<https://plugins.trac.wordpress.org/changeset/1970313>
Added to Seravo’s update systems as an urgent update

31.
14:51

32.
Investigation and recovery steps
7. Based on findings, clean up the site
a. Recover clean version from backups
b. Remove malicious code and content
manually
8. As a precaution, reset all WordPress user
sessions and passwords
wp-reset-all-passwords
In this case option A was not
possible, but luckily option B
was quite easy as backups
showed only one potential
malware file was injected.

33.
Investigation and recovery steps
9. As extra precaution, scan the site for
malware one more time when it is
otherwise deemed to be clean
Using Seravo’s custom
made WordPress/PHP
malware scanner

34.
2018-11-09 15:26:22 <redacted>.seravo.com RESOLVED
2018-11-09 15:17:49 <redacted>.seravo.com RESOLVED
2018-11-09 15:29:03 <redacted>.seravo.com RESOLVED
2018-11-09 15:20:24 <redacted>.seravo.com RESOLVED
All sites clean and finally back online

35.
Investigation and recovery steps
10. Elevated monitoring and follow-up for
site once it has been re-opened, just in
case there was more attack avenues not
discovered during the investigation.
During the investigation Seravo sent 8 status update e-mails to the site
owner and the customer mobilized their own team to support the effort and
they also sent us valuable additional information. A few additional emails
from Seravo to the customer followed over the weekend and next week to
confirm all necessary measures had been completed.

36.
Notification e-mail from new registration of
‘trollherten’ users
● Later we found out the site owner did get an email
notification from WordPress about the new user
named “trollherten” but since the e-mail was vague
and did not contain any alarming information, the
person who read the e-mail ignored it.

37.
Luckily this was not a targeted attack
● Most likely the attacker just wanted to own the site
and use it to redirect traffic, spam, mount more
attacks against other sites etc.
● The site itself or the data it had was not the target and
most likely not used.

38.
Be prepared: no security is perfect
● No plugin author makes perfect code.
● All plugins on the site were updated a
week earlier, the vulnerability was used
close to zero-day.
● Unreasonable for site admin to read
deeply all notification e-mails.
● Fact: sometimes even good security isn’t
enough. One also needs to have a
security incident response plan.
● We do. Do you?

39.
Thank you!
@ottokekalainen