Beyond php - it's not (just) about the code

Wim Godden
Cu.be Solutions
@wimgtr
Beyond PHP :
It's not (just) about the code

Who am I ?
Wim Godden (@wimgtr)

Who am I ?
Wim Godden (@wimgtr)
Founder of Cu.be Solutions (http://cu.be)
Open Source developer since 1997
Creator of PHPCompatibility
Speaker at PHP and Open Source conferences

Cu.be Solutions ?
Open source consultancy
PHP-centered (Symfony, Oro, ...)
Training courses
High-speed redundant network (BGP, OSPF, VRRP)
High scalability development
Nginx + extensions
MySQL Cluster
Projects :
mostly IT & Telecom companies
lots of public-facing apps/sites

Who are you ?
Developers ?
Anyone setup a MySQL master-slave (source-replica) ?
Anyone setup a site/app on separate web and database server ?
→ How much traffic between them ?

The topic
Things we take for granted
Famous last words : "It should work just fine"
Works fine today
→ might fail tomorrow
Most common mistakes
PHP code ↔ PHP ecosystem

It starts with...
… code !
First up : database

Database queries – complexity
SELECT DISTINCT n.nid, n.uid, n.title, n.type, e.event_start, e.event_start AS event_start_orig,
e.event_end, e.event_end AS event_end_orig, e.timezone, e.has_time, e.has_end_date, tz.offset
AS offset, tz.offset_dst AS offset_dst, tz.dst_region, tz.is_dst, e.event_start - INTERVAL IF(tz.is_dst,
tz.offset_dst, tz.offset) HOUR_SECOND AS event_start_utc, e.event_end - INTERVAL IF(tz.is_dst,
tz.offset_dst, tz.offset) HOUR_SECOND AS event_end_utc, e.event_start - INTERVAL IF(tz.is_dst,
tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0 SECOND AS event_start_user,
e.event_end - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0
SECOND AS event_end_user, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset)
HOUR_SECOND + INTERVAL 0 SECOND AS event_start_site, e.event_end - INTERVAL
IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0 SECOND AS event_end_site,
tz.name as timezone_name FROM node n INNER JOIN event e ON n.nid = e.nid INNER JOIN
event_timezones tz ON tz.timezone = e.timezone INNER JOIN node_access na ON na.nid = n.nid
LEFT JOIN domain_access da ON n.nid = da.nid LEFT JOIN node i18n ON n.tnid > 0 AND n.tnid =
i18n.tnid AND i18n.language = 'en' WHERE (na.grant_view >= 1 AND ((na.gid = 0 AND na.realm =
'all'))) AND ((da.realm = "domain_id" AND da.gid = 4) OR (da.realm = "domain_site" AND da.gid =
0)) AND (n.language ='en' OR n.language ='' OR n.language IS NULL OR n.language = 'is' AND
i18n.nid IS NULL) AND ( n.status = 1 AND ((e.event_start >= '2010-01-31 00:00:00' AND
e.event_start <= '2010-03-01 23:59:59') OR (e.event_end >= '2010-01-31 00:00:00' AND
e.event_end <= '2010-03-01 23:59:59') OR (e.event_start <= '2010-01-31 00:00:00' AND
e.event_end >= '2010-03-01 23:59:59')) ) GROUP BY n.nid HAVING (event_start >= '2010-02-01
00:00:00' AND event_start <= '2010-02-28 23:59:59') OR (event_end >= '2010-02-01 00:00:00' AND
event_end <= '2010-02-28 23:59:59') OR (event_start <= '2010-02-01 00:00:00' AND event_end >=
'2010-02-28 23:59:59') ORDER BY event_start ASC;

Database - indexing
'select id from stock where status = 2 order by qty'
→ aggregate index on (status, qty)
'select id from stock where status > 2 order by qty'
→ aggregate index on (status, qty) ?
→ Depends :
- Btree : yes
- Hash : range selection stops use of aggregate index
→ separate index on status and qty (since recent versions)

Database - indexing
Indexes make database faster
→ Let's index everything !
→ DON'T :
Insert/update/delete → Index modification
Each select → evaluation of all indexes
"Relational schema design is based on data
but index design is based on queries"
- Bill Karwin,
author of “SQL Antipatterns”

Databases – detecting problematic queries
Slow query log
→ SET GLOBAL slow_query_log = ON;
Queries not using indexes
→ In my.cnf/my.ini : 'log_queries_not_using_indexes'
General query log
→ SET GLOBAL general_log = ON;
→ Turn it off quickly !
Percona Toolkit
pt-query-digest

Databases - pt-query-digest
# Profile
# Rank Response time Calls R/Call Item
# ==== ================ ===== ======= ======
# 1 16526.2542 98.2% 1208 13.6806 SELECT output_option
# 2 0.8312 0.0% 6412 0.0001 SELECT poller_output poller_item
# 3 0.6811 0.0% 6416 0.0001 SELECT poller_time
# 4 0.2805 0.0% 149 0.0019 SELECT wp_terms wp_term_taxonomy wp_term_relationships
# 5 0.1999 0.0% 51 0.0039 SELECT UNION wp_pp_daily_summary wp_pp_hourly_summary
# 6 0.1956 0.0% 89 0.0022 UPDATE wp_options
# MISC 302.8137 1.8% 3853 0.0002 <147 ITEMS>

# Query 2: 0.26 QPS, 0.00x concurrency, ID 0x92F3B1B361FB0E5B at byte 14081299
# This item is included in the report because it matches --limit.
# Scores: Apdex = 1.00 [1.0], V/M = 0.00
# Query_time sparkline: | _^ |
# Time range: 2011-12-28 18:42:47 to 19:03:10
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 1 312
# Exec time 50 4s 5ms 25ms 13ms 20ms 4ms 12ms
# Lock time 3 32ms 43us 163us 103us 131us 19us 98us
# Rows sent 59 62.41k 203 231 204.82 202.40 3.99 202.40
# Rows examine 13 73.63k 238 296 241.67 246.02 10.15 234.30
# Rows affecte 0 0 0 0 0 0 0 0
# Rows read 59 62.41k 203 231 204.82 202.40 3.99 202.40
# Bytes sent 53 24.85M 46.52k 84.36k 81.56k 83.83k 7.31k 79.83k
# Merge passes 0 0 0 0 0 0 0 0
# Tmp tables 0 0 0 0 0 0 0 0
# Tmp disk tbl 0 0 0 0 0 0 0 0
# Tmp tbl size 0 0 0 0 0 0 0 0
# Query size 0 21.63k 71 71 71 71 0 71
# InnoDB:
# IO r bytes 0 0 0 0 0 0 0 0
# IO r ops 0 0 0 0 0 0 0 0
# IO r wait 0 0 0 0 0 0 0 0
# pages distin 40 11.77k 34 44 38.62 38.53 1.87 38.53
# queue wait 0 0 0 0 0 0 0 0
# rec lock wai 0 0 0 0 0 0 0 0
# Boolean:
# Full scan 100% yes, 0% no
# String:
# Databases wp_blog_one (264/84%), wp_blog_tw… (36/11%)... 1 more
# Hosts
# InnoDB trxID 86B40B (1/0%), 86B430 (1/0%), 86B44A (1/0%)... 309 more
# Last errno 0
# Users wp_blog_one (264/84%), wp_blog_two (36/11%)... 1 more
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms
# 10ms ################################################################
# 100ms
# 1s
# 10s+
# Tables
# SHOW TABLE STATUS FROM `wp_blog_one ` LIKE 'wp_options'G
# SHOW CREATE TABLE `wp_blog_one `.`wp_options`G

Databases – next step : explain
explain <query>
"How will MySQL execute the query"

Type of lookup
'system', 'const' and 'ref' = good
'ALL' = bad
Extra info
Using index = good
Using filesort = usually bad

Databases – covering indexes
mysql> explain select * from product where category=5 and stock=1;
+----+-------+---------------+---------------+---------+------+------------+
| id | TYPE | possible_keys | KEY | key_len | ROWS | Extra |
+----+-------+---------------+---------------+---------+------+------------+
| 1 | ref | categorystock | categorystock | 8 | 1200 | |
●
+----+-------+---------------+---------------+---------+------+------------+
+--------------+---------------+------+-----+------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------+------+-----+------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| category | int(11) | YES | MUL | NULL | |
| stock | int(11) | YES | MUL | NULL | |
| description | varchar(255) | YES | | NULL | |
...
mysql> show index from product;
+----------+------------+---------------------+--------------+---------------|
| Table | Non_unique | Key_name | Seq_in_index | Column_name |
+----------+------------+---------------------+--------------+---------------+
| product | 0 | PRIMARY | 1 | id |
| product | 1 | categorystock | 1 | category |
| product | 1 | categorystock | 2 | stock |
...

Databases – covering indexes
mysql>explain select category, stock, id from product where category=5 and stock=1;
+----+-------+---------------+---------------+---------+------+-------------+
| id | TYPE | possible_keys | KEY | key_len | ROWS | Extra |
+----+-------+---------------+---------------+---------+------+-------------+
| 1 | ref | categorystock | categorystock | 8 | 1200 | Using index |
+----+-------+---------------+---------------+---------+------+-------------
+--------------+---------------+------+-----+------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------+------+-----+------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| category | int(11) | YES | MUL | NULL | |
| stock | int(11) | YES | MUL | NULL | |
| description | varchar(255) | YES | | NULL | |
...
mysql> show index from product;
+----------+------------+---------------------+--------------+---------------|
| Table | Non_unique | Key_name | Seq_in_index | Column_name |
+----------+------------+---------------------+--------------+---------------+
| product | 0 | PRIMARY | 1 | id |
| product | 1 | categorystock | 1 | category |
| product | 1 | categorystock | 2 | stock |
...

Databases – when to use / not to use
Good at :
Fetching data
Storing data
Searching through data
Bad at :
select `name` from `room` where ceiling(`avgNoOfPeople`) = 8
→ full table scan
→ creates temporary table
select `name` from `room` where avgNoOfPeople > 7 and
avgNoOfPeople <= 8
→ Avoid functions that run across every row
→ Avoid functions in where statement

For / foreach (N+1 problem)
$customers = CustomerQuery::create()
->filterByState('MN')
->find();
foreach ($customers as $customer) {
$contacts = ContactsQuery::create()
->filterByCustomerid($customer->getId())
->find();
foreach ($contacts as $contact) {
doSomestuffWith($contact);
}
}

Joins
$contacts = mysql_query("
select
contacts.*
from
customer
join contact
on contact.customerid = customer.id
where
state = 'MN'
");
while ($contact = mysql_fetch_array($contacts)) {
doSomeStuffWith($contact);
}
(or the ORM equivalent)

Better...
10001 → 1 query
Sadly : people still produce code with query loops
Usually :
Growth not anticipated
Internal app → Public app

The origins of this talk
Customers :
Projects we built
Projects we didn't build, but got pulled into
Fixes
Changes
Infrastructure migration
15 years of 'how to cause mayhem with a few lines of code'

Client X
Jobs search site
Monitor job views :
Daily hits
Weekly hits
Monthly hits
Which user saw which job

Client X
Originally : when user viewed job details
Now : when job is in search result
Search for 'php' → 50 jobs = 50 jobs to be updated
→ 50 updates for shown_today
→ 50 updates for shown_week
→ 50 updates for shown_month
→ 50 inserts for shown_user
= 200 queries for 1 search !

Client X : the code
foreach ($jobs as $job) {
$db->query("
insert into shown_today(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_week(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_month(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number +
1
");
$db->query("
insert into shown_user(
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}

Client X : the numbers
600-1000 inserts/sec (peaks up to 1600)
400-1000 updates/sec (peaks up to 2600)
16 core machine

Client X : panic !
Mail : "MySQL replicate is more than 5 minutes behind source"
We set it up → who did they blame ?
Wait a second !

Client X : what's causing those peaks ?

Client X : possible cause ?
Code changes ?
→ According to developers : none
Action : turn on general log, analyze with pt-query-digest
→ 50+-fold increase in 4 queries
→ Developers : 'Oops we did make a change'
After 3 days : 2,5 days behind
Every hour : 50 min extra lag

Client X : But why is the replica lagging ?
Source Replica
File :
master-bin-xxxx.log
File :
master-bin-xxxx.log
Replica I/O thread
Binlog dump
thread
Replica
SQL
thread

Client X : fix ?
$db->query("
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number +
1
");
$db->query("
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}

Client X : the code change
insert into shown_today values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_week values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_month values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_user values (5, 23, "2015-10-12 12:01:00"), (8, 23, "2015-10-12
12:01:00"), … ;

Client X : the code change
$todayQuery = "
jobId,
number
) values ";
$todayQuery .= "(" . $job['id'] . ", 1),";
}
$todayQuery = substr($todayQuery, 0, strlen($todayQuery) - 1);
$todayQuery .= "
on duplicate key
update
number = number + 1
";
$db->query($todayQuery);

Client X : the chosen solution
$db->autocommit(false);
$db->query("
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}
$db->commit();

Client X : conclusion
For loops are bad (we already knew that)
Add master/slave (source/replica) and it gets much worse
Use transactions : it will provide huge performance increase
Better yet : use MariaDB 10 or higher → slave_parallel_threads
Result : replica caught up 5 days later

Network trouble : more than just traffic
Customer Y
150.000 visits/day
News ticker :
XML feed from other site (owned by same customer)
Cached for 15 min

Customer Y – fetching the feed
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
unlink(APP_DIR . '/tmp/cacheFile.xml');
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
file_get_contents('http://www.scrambledsitename.be/xml/feed.xml')
);
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');
What's wrong with this code ?

Customer Y – no feed without the source
Feed source

Customer Y : timeout
default_socket_timeout : 60 sec by default
Each visitor : 60 sec wait time
People keep hitting refresh → more load
More active connections → more load
Apache hits maximum connections → entire site down

Customer Y – fetching the feed
file_put_contents(
file_get_contents('http://www.scrambledsitename.be/xml/feed.xml')
);
}

Customer Y : timeout fix
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
file_put_contents(
file_get_contents(
'http://www.scrambledsitename.be/xml/feed.xml',
false,
$context
)
);
}

Customer Y : don't delete from cache
array(
'http' => array(
'timeout' => 5
)
)
);
file_put_contents(
file_get_contents(
false,
$context
)
);
}

array(
'http' => array(
'timeout' => 5
)
)
);
file_put_contents(
file_get_contents(
false,
$context
)
);
}

array(
'http' => array(
'timeout' => 5
)
)
);
$feed = file_get_contents(
false,
$context
);
if ($feed !== false) {
file_put_contents(
$feed
);
}
}

Customer Y : process early
array(
'http' => array(
'timeout' => 5
)
)
);
false,
$context
);
file_put_contents(
ParseXmlFeed($feed)
);
}
}

Customer Y : file_[get|put]_contents atomicity
false,
$context
);
file_put_contents(
ParseXmlFeed($feed)
);
}
}
Relying on user → concurrent requests → possible data corruption
Better : run every 15min through cronjob

Network resources
Use timeouts for all :
fopen
curl
SOAP
…
Data source trusted ?
→ setup a webservice
→ let them push updates when their feed changes
→ less load on data source
→ no timeout issues
Add logging → early detection

Logging
Logging = good
Logging in PHP using fopen
→ bad idea : locking issues
→ Use monolog : file, syslog, mail, Pushover, HipChat, Graylog,
Rollbar, ElasticSearch (and 50 more)
For Firefox : FirePHP (add-on for Firebug)
Debug logging = bad on production
Watch your logs !
Don't log on slow disks → I/O bottlenecks

File system : I/O bottlenecks
Causes :
Excessive writes (database updates, logfiles, swapping, …)
Excessive reads (non-indexed database queries, swapping, small file
system cache, …)
How to detect ?
top
iostat
See iowait ? Stop worrying about php, fix the I/O problem !
Cpu(s): 0.2%us, 3.0%sy, 0.0%ni, 61.4%id, 35.5%wa, 0.0%hi
avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 0.96 53.70 0.00 45.24
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 120.40 0.00 123289.60 0 616448
sdb 2.10 0.00 4378.10 0 18215
dm-0 4.20 0.00 36.80 0 184
dm-1 0.00 0.00 0.00 0 0

Web DB
Client Z - a recent horror story
Prod
Test
Prod-backup.sql
Prod
Test
mysqldump prod >
mysql test <

Web DB
A recent horror story
Prod
Test
Prod-backup.sql
Prod
Test
mysqldump –opt prod >
mysql test <
CREATE DATABASE IF NOT EXISTS `prod` ….;
USE `prod`;
...

Friday 05:00-06:30 – backup runs
Friday 13:30 – prod imported from backup but fails
Friday 14:45 – prod imported from backup but fails
Friday 16:30 – prod imported from backup
Friday 18:00 to Monday 05:00 – business as usual
Monday 05:00-06:30 – backup runs
Monday 12:00 – prod imported from backup
Monday 16:30 – panic !

Fri Sat Sun Mon
05:00 12:00 19:00 05:00 12:00 14:00

Recovery to the exact second
Restore backup
Replay binary logs to that point
Have binary logs
Even if you don’t have a replica server
Make sure you have > 24h rollover !
Don’t use mysqldump → xtrabackup / mariabackup

Much more than code
DB
server
Webserver
User
Network
XML feed

Look beyond PHP (or Perl, Ruby, Python, ...) !

Contact
Twitter @wimgtr
Slides http://www.slideshare.net/wimg
E-mail wim@cu.be
Thanks !

Beyond php - it's not (just) about the code

Recommended

Recommended

More Related Content

Similar to Beyond php - it's not (just) about the code

Similar to Beyond php - it's not (just) about the code (20)

More from Wim Godden

More from Wim Godden (20)

Recently uploaded

Recently uploaded (20)

Beyond php - it's not (just) about the code