The document presents a talk by Wim Godden, founder of cu.be solutions, and discusses various aspects of PHP and database management, particularly focusing on the importance of efficient database queries and indexing to enhance performance. It details common mistakes, slow query detection, and the impact of inefficient coding practices on application performance. The presentation emphasizes the critical nature of database design and query optimization in the development of scalable applications.

1. Wim Godden
Cu.be Solutions
@wimgtr
Beyond PHP :
It's not (just) about the code

2. Who am I ?
Wim Godden (@wimgtr)

3. Where I'm from

4. Who am I ?
Wim Godden (@wimgtr)
Founder of Cu.be Solutions (http://cu.be)
Open Source developer since 1997
Creator of PHPCompatibility
Speaker at PHP and Open Source conferences

5. Cu.be Solutions ?
Open source consultancy
PHP-centered (Symfony, Oro, ...)
Training courses
High-speed redundant network (BGP, OSPF, VRRP)
High scalability development
Nginx + extensions
MySQL Cluster
Projects :
mostly IT & Telecom companies
lots of public-facing apps/sites

6. Who are you ?
Developers ?
Anyone setup a MySQL master-slave (source-replica) ?
Anyone setup a site/app on separate web and database server ?
→ How much traffic between them ?

7. The topic
Things we take for granted
Famous last words : "It should work just fine"
Works fine today
→ might fail tomorrow
Most common mistakes
PHP code ↔ PHP ecosystem

8. It starts with...
… code !
First up : database

9. Database queries – complexity
SELECT DISTINCT n.nid, n.uid, n.title, n.type, e.event_start, e.event_start AS event_start_orig,
e.event_end, e.event_end AS event_end_orig, e.timezone, e.has_time, e.has_end_date, tz.offset
AS offset, tz.offset_dst AS offset_dst, tz.dst_region, tz.is_dst, e.event_start - INTERVAL IF(tz.is_dst,
tz.offset_dst, tz.offset) HOUR_SECOND AS event_start_utc, e.event_end - INTERVAL IF(tz.is_dst,
tz.offset_dst, tz.offset) HOUR_SECOND AS event_end_utc, e.event_start - INTERVAL IF(tz.is_dst,
tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0 SECOND AS event_start_user,
e.event_end - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0
SECOND AS event_end_user, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset)
HOUR_SECOND + INTERVAL 0 SECOND AS event_start_site, e.event_end - INTERVAL
IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0 SECOND AS event_end_site,
tz.name as timezone_name FROM node n INNER JOIN event e ON n.nid = e.nid INNER JOIN
event_timezones tz ON tz.timezone = e.timezone INNER JOIN node_access na ON na.nid = n.nid
LEFT JOIN domain_access da ON n.nid = da.nid LEFT JOIN node i18n ON n.tnid > 0 AND n.tnid =
i18n.tnid AND i18n.language = 'en' WHERE (na.grant_view >= 1 AND ((na.gid = 0 AND na.realm =
'all'))) AND ((da.realm = "domain_id" AND da.gid = 4) OR (da.realm = "domain_site" AND da.gid =
0)) AND (n.language ='en' OR n.language ='' OR n.language IS NULL OR n.language = 'is' AND
i18n.nid IS NULL) AND ( n.status = 1 AND ((e.event_start >= '2010-01-31 00:00:00' AND
e.event_start <= '2010-03-01 23:59:59') OR (e.event_end >= '2010-01-31 00:00:00' AND
e.event_end <= '2010-03-01 23:59:59') OR (e.event_start <= '2010-01-31 00:00:00' AND
e.event_end >= '2010-03-01 23:59:59')) ) GROUP BY n.nid HAVING (event_start >= '2010-02-01
00:00:00' AND event_start <= '2010-02-28 23:59:59') OR (event_end >= '2010-02-01 00:00:00' AND
event_end <= '2010-02-28 23:59:59') OR (event_start <= '2010-02-01 00:00:00' AND event_end >=
'2010-02-28 23:59:59') ORDER BY event_start ASC;

10. Database - indexing
'select id from stock where status = 2 order by qty'
→ aggregate index on (status, qty)
'select id from stock where status > 2 order by qty'
→ aggregate index on (status, qty) ?
→ Depends :
- Btree : yes
- Hash : range selection stops use of aggregate index
→ separate index on status and qty (since recent versions)

11. Database - indexing
Indexes make database faster
→ Let's index everything !
→ DON'T :
Insert/update/delete → Index modification
Each select → evaluation of all indexes
"Relational schema design is based on data
but index design is based on queries"
- Bill Karwin,
author of “SQL Antipatterns”

12. Databases – detecting problematic queries
Slow query log
→ SET GLOBAL slow_query_log = ON;
Queries not using indexes
→ In my.cnf/my.ini : 'log_queries_not_using_indexes'
General query log
→ SET GLOBAL general_log = ON;
→ Turn it off quickly !
Percona Toolkit
pt-query-digest

13. Databases - pt-query-digest
# Profile
# Rank Response time Calls R/Call Item
# ==== ================ ===== ======= ======
# 1 16526.2542 98.2% 1208 13.6806 SELECT output_option
# 2 0.8312 0.0% 6412 0.0001 SELECT poller_output poller_item
# 3 0.6811 0.0% 6416 0.0001 SELECT poller_time
# 4 0.2805 0.0% 149 0.0019 SELECT wp_terms wp_term_taxonomy wp_term_relationships
# 5 0.1999 0.0% 51 0.0039 SELECT UNION wp_pp_daily_summary wp_pp_hourly_summary
# 6 0.1956 0.0% 89 0.0022 UPDATE wp_options
# MISC 302.8137 1.8% 3853 0.0002 <147 ITEMS>

14. # Query 2: 0.26 QPS, 0.00x concurrency, ID 0x92F3B1B361FB0E5B at byte 14081299
# This item is included in the report because it matches --limit.
# Scores: Apdex = 1.00 [1.0], V/M = 0.00
# Query_time sparkline: | _^ |
# Time range: 2011-12-28 18:42:47 to 19:03:10
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 1 312
# Exec time 50 4s 5ms 25ms 13ms 20ms 4ms 12ms
# Lock time 3 32ms 43us 163us 103us 131us 19us 98us
# Rows sent 59 62.41k 203 231 204.82 202.40 3.99 202.40
# Rows examine 13 73.63k 238 296 241.67 246.02 10.15 234.30
# Rows affecte 0 0 0 0 0 0 0 0
# Rows read 59 62.41k 203 231 204.82 202.40 3.99 202.40
# Bytes sent 53 24.85M 46.52k 84.36k 81.56k 83.83k 7.31k 79.83k
# Merge passes 0 0 0 0 0 0 0 0
# Tmp tables 0 0 0 0 0 0 0 0
# Tmp disk tbl 0 0 0 0 0 0 0 0
# Tmp tbl size 0 0 0 0 0 0 0 0
# Query size 0 21.63k 71 71 71 71 0 71
# InnoDB:
# IO r bytes 0 0 0 0 0 0 0 0
# IO r ops 0 0 0 0 0 0 0 0
# IO r wait 0 0 0 0 0 0 0 0
# pages distin 40 11.77k 34 44 38.62 38.53 1.87 38.53
# queue wait 0 0 0 0 0 0 0 0
# rec lock wai 0 0 0 0 0 0 0 0
# Boolean:
# Full scan 100% yes, 0% no
# String:
# Databases wp_blog_one (264/84%), wp_blog_tw… (36/11%)... 1 more
# Hosts
# InnoDB trxID 86B40B (1/0%), 86B430 (1/0%), 86B44A (1/0%)... 309 more
# Last errno 0
# Users wp_blog_one (264/84%), wp_blog_two (36/11%)... 1 more
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms
# 10ms ################################################################
# 100ms
# 1s
# 10s+
# Tables
# SHOW TABLE STATUS FROM `wp_blog_one ` LIKE 'wp_options'G
# SHOW CREATE TABLE `wp_blog_one `.`wp_options`G

15. Databases – next step : explain
explain <query>
"How will MySQL execute the query"

16. Databases – next step : explain
+-----------+------+---------------+------+---------+------+--------+-------------+
| TABLE | TYPE | possible_keys | KEY | key_len | REF | ROWS | Extra |
+-----------+------+---------------+------+---------+------+--------+-------------+
| employees | ALL | NULL | NULL | NULL | NULL | 299809 | USING WHERE |
+-----------+------+---------------+------+---------+------+--------+-------------+
+------------+-------+-------------------------------+---------+---------+-------+------+-------+
| table | type | possible_keys | key | key_len | ref | rows | Extra |
+------------+-------+-------------------------------+---------+---------+-------+------+-------+
| itdevice | const | PRIMARY,fk_device_devicetype1 | PRIMARY | 4 | const | 1 | |
| devicetype | const | PRIMARY | PRIMARY | 4 | const | 1 | |
+------------+-------+-------------------------------+---------+---------+-------+------+-------+

17. Databases – next step : explain
Type of lookup
'system', 'const' and 'ref' = good
'ALL' = bad
Extra info
Using index = good
Using filesort = usually bad

18. Databases – covering indexes
mysql> explain select * from product where category=5 and stock=1;
+----+-------+---------------+---------------+---------+------+------------+
| id | TYPE | possible_keys | KEY | key_len | ROWS | Extra |
+----+-------+---------------+---------------+---------+------+------------+
| 1 | ref | categorystock | categorystock | 8 | 1200 | |
●
+----+-------+---------------+---------------+---------+------+------------+
+--------------+---------------+------+-----+------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------+------+-----+------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| category | int(11) | YES | MUL | NULL | |
| stock | int(11) | YES | MUL | NULL | |
| description | varchar(255) | YES | | NULL | |
...
mysql> show index from product;
+----------+------------+---------------------+--------------+---------------|
| Table | Non_unique | Key_name | Seq_in_index | Column_name |
+----------+------------+---------------------+--------------+---------------+
| product | 0 | PRIMARY | 1 | id |
| product | 1 | categorystock | 1 | category |
| product | 1 | categorystock | 2 | stock |
...

19. Databases – covering indexes
mysql>explain select category, stock, id from product where category=5 and stock=1;
+----+-------+---------------+---------------+---------+------+-------------+
| id | TYPE | possible_keys | KEY | key_len | ROWS | Extra |
+----+-------+---------------+---------------+---------+------+-------------+
| 1 | ref | categorystock | categorystock | 8 | 1200 | Using index |
+----+-------+---------------+---------------+---------+------+-------------
+--------------+---------------+------+-----+------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------+------+-----+------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| category | int(11) | YES | MUL | NULL | |
| stock | int(11) | YES | MUL | NULL | |
| description | varchar(255) | YES | | NULL | |
...
mysql> show index from product;
+----------+------------+---------------------+--------------+---------------|
| Table | Non_unique | Key_name | Seq_in_index | Column_name |
+----------+------------+---------------------+--------------+---------------+
| product | 0 | PRIMARY | 1 | id |
| product | 1 | categorystock | 1 | category |
| product | 1 | categorystock | 2 | stock |
...

20. Databases – when to use / not to use
Good at :
Fetching data
Storing data
Searching through data
Bad at :
select `name` from `room` where ceiling(`avgNoOfPeople`) = 8
→ full table scan
→ creates temporary table
select `name` from `room` where avgNoOfPeople > 7 and
avgNoOfPeople <= 8
→ Avoid functions that run across every row
→ Avoid functions in where statement

21. For / foreach (N+1 problem)
$customers = CustomerQuery::create()
->filterByState('MN')
->find();
foreach ($customers as $customer) {
$contacts = ContactsQuery::create()
->filterByCustomerid($customer->getId())
->find();
foreach ($contacts as $contact) {
doSomestuffWith($contact);
}
}

22. Joins
$contacts = mysql_query("
select
contacts.*
from
customer
join contact
on contact.customerid = customer.id
where
state = 'MN'
");
while ($contact = mysql_fetch_array($contacts)) {
doSomeStuffWith($contact);
}
(or the ORM equivalent)

23. Better...
10001 → 1 query
Sadly : people still produce code with query loops
Usually :
Growth not anticipated
Internal app → Public app

24. The origins of this talk
Customers :
Projects we built
Projects we didn't build, but got pulled into
Fixes
Changes
Infrastructure migration
15 years of 'how to cause mayhem with a few lines of code'

25. Client X
Jobs search site
Monitor job views :
Daily hits
Weekly hits
Monthly hits
Which user saw which job

26. Client X
Originally : when user viewed job details
Now : when job is in search result
Search for 'php' → 50 jobs = 50 jobs to be updated
→ 50 updates for shown_today
→ 50 updates for shown_week
→ 50 updates for shown_month
→ 50 inserts for shown_user
= 200 queries for 1 search !

27. Client X : the code
foreach ($jobs as $job) {
$db->query("
insert into shown_today(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_week(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_month(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number +
1
");
$db->query("
insert into shown_user(
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}

28. Client X : the graph

29. Client X : the numbers
600-1000 inserts/sec (peaks up to 1600)
400-1000 updates/sec (peaks up to 2600)
16 core machine

30. Client X : panic !
Mail : "MySQL replicate is more than 5 minutes behind source"
We set it up → who did they blame ?
Wait a second !

31. Client X : what's causing those peaks ?

32. Client X : possible cause ?
Code changes ?
→ According to developers : none
Action : turn on general log, analyze with pt-query-digest
→ 50+-fold increase in 4 queries
→ Developers : 'Oops we did make a change'
After 3 days : 2,5 days behind
Every hour : 50 min extra lag

33. Client X : But why is the replica lagging ?
Source Replica
File :
master-bin-xxxx.log
File :
master-bin-xxxx.log
Replica I/O thread
Binlog dump
thread
Replica
SQL
thread

34. Client X : Source

35. Client X : Replica

36. Client X : fix ?
foreach ($jobs as $job) {
$db->query("
insert into shown_today(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_week(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_month(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number +
1
");
$db->query("
insert into shown_user(
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}

37. Client X : the code change
insert into shown_today values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_week values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_month values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_user values (5, 23, "2015-10-12 12:01:00"), (8, 23, "2015-10-12
12:01:00"), … ;

38. Client X : the code change
$todayQuery = "
insert into shown_today(
jobId,
number
) values ";
foreach ($jobs as $job) {
$todayQuery .= "(" . $job['id'] . ", 1),";
}
$todayQuery = substr($todayQuery, 0, strlen($todayQuery) - 1);
$todayQuery .= "
on duplicate key
update
number = number + 1
";
$db->query($todayQuery);

39. Client X : the chosen solution
$db->autocommit(false);
foreach ($jobs as $job) {
$db->query("
insert into shown_today(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_week(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_month(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_user(
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}
$db->commit();

40. Client X : conclusion
For loops are bad (we already knew that)
Add master/slave (source/replica) and it gets much worse
Use transactions : it will provide huge performance increase
Better yet : use MariaDB 10 or higher → slave_parallel_threads
Result : replica caught up 5 days later

41. Network trouble : more than just traffic
Customer Y
150.000 visits/day
News ticker :
XML feed from other site (owned by same customer)
Cached for 15 min

42. Customer Y – fetching the feed
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
unlink(APP_DIR . '/tmp/cacheFile.xml');
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
file_get_contents('http://www.scrambledsitename.be/xml/feed.xml')
);
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');
What's wrong with this code ?

43. Customer Y – no feed without the source
Feed source

44. Customer Y – no feed without the source
Feed source

45. Customer Y : timeout
default_socket_timeout : 60 sec by default
Each visitor : 60 sec wait time
People keep hitting refresh → more load
More active connections → more load
Apache hits maximum connections → entire site down

46. Customer Y – fetching the feed
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
unlink(APP_DIR . '/tmp/cacheFile.xml');
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
file_get_contents('http://www.scrambledsitename.be/xml/feed.xml')
);
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');

47. Customer Y : timeout fix
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
unlink(APP_DIR . '/tmp/cacheFile.xml');
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
file_get_contents(
'http://www.scrambledsitename.be/xml/feed.xml',
false,
$context
)
);
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');

48. Customer Y : don't delete from cache
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
unlink(APP_DIR . '/tmp/cacheFile.xml');
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
file_get_contents(
'http://www.scrambledsitename.be/xml/feed.xml',
false,
$context
)
);
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');

49. Customer Y : don't delete from cache
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
file_get_contents(
'http://www.scrambledsitename.be/xml/feed.xml',
false,
$context
)
);
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');

50. Customer Y : don't delete from cache
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
$feed = file_get_contents(
'http://www.scrambledsitename.be/xml/feed.xml',
false,
$context
);
if ($feed !== false) {
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
$feed
);
}
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');

51. Customer Y : process early
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
$feed = file_get_contents(
'http://www.scrambledsitename.be/xml/feed.xml',
false,
$context
);
if ($feed !== false) {
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
ParseXmlFeed($feed)
);
}
}

52. Customer Y : file_[get|put]_contents atomicity
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
$feed = file_get_contents(
'http://www.scrambledsitename.be/xml/feed.xml',
false,
$context
);
if ($feed !== false) {
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
ParseXmlFeed($feed)
);
}
}
Relying on user → concurrent requests → possible data corruption
Better : run every 15min through cronjob

53. Network resources
Use timeouts for all :
fopen
curl
SOAP
…
Data source trusted ?
→ setup a webservice
→ let them push updates when their feed changes
→ less load on data source
→ no timeout issues
Add logging → early detection

54. Logging
Logging = good
Logging in PHP using fopen
→ bad idea : locking issues
→ Use monolog : file, syslog, mail, Pushover, HipChat, Graylog,
Rollbar, ElasticSearch (and 50 more)
For Firefox : FirePHP (add-on for Firebug)
Debug logging = bad on production
Watch your logs !
Don't log on slow disks → I/O bottlenecks

55. File system : I/O bottlenecks
Causes :
Excessive writes (database updates, logfiles, swapping, …)
Excessive reads (non-indexed database queries, swapping, small file
system cache, …)
How to detect ?
top
iostat
See iowait ? Stop worrying about php, fix the I/O problem !
Cpu(s): 0.2%us, 3.0%sy, 0.0%ni, 61.4%id, 35.5%wa, 0.0%hi
avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 0.96 53.70 0.00 45.24
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 120.40 0.00 123289.60 0 616448
sdb 2.10 0.00 4378.10 0 18215
dm-0 4.20 0.00 36.80 0 184
dm-1 0.00 0.00 0.00 0 0

56. Web DB
Client Z - a recent horror story
Prod
Test
Prod-backup.sql
Prod
Test
mysqldump prod >
mysql test <

57. Web DB
A recent horror story
Prod
Test
Prod-backup.sql
Prod
Test
mysqldump –opt prod >
mysql test <
CREATE DATABASE IF NOT EXISTS `prod` ….;
USE `prod`;
...

58. A recent horror story
Friday 05:00-06:30 – backup runs
Friday 13:30 – prod imported from backup but fails
Friday 14:45 – prod imported from backup but fails
Friday 16:30 – prod imported from backup
Friday 18:00 to Monday 05:00 – business as usual
Monday 05:00-06:30 – backup runs
Monday 12:00 – prod imported from backup
Monday 16:30 – panic !

59. A recent horror story
Fri Sat Sun Mon
05:00 12:00 19:00 05:00 12:00 14:00

60. A recent horror story
Recovery to the exact second
Restore backup
Replay binary logs to that point
Have binary logs
Even if you don’t have a replica server
Make sure you have > 24h rollover !
Don’t use mysqldump → xtrabackup / mariabackup

61. Much more than code
DB
server
Webserver
User
Network
XML feed

62. Look beyond PHP (or Perl, Ruby, Python, ...) !

63. Questions ?

64. Questions ?

65. Contact
Twitter @wimgtr
Slides http://www.slideshare.net/wimg
E-mail wim@cu.be
Thanks !