7. DB Optimization – Logging Queries
mysql> SET GLOBAL slow_query_log = ON;
mysql> SET GLOBAL long_query_time = 0;
mysql> SET GLOBAL log_queries_not_using_indexes = ON;
Eli Aschkenasy - @EliAschkenasy - #JDNL16
8. DB Optimization – Logging Queries
Change to ON
Eli Aschkenasy - @EliAschkenasy - #JDNL16
9. DB Optimization – Analyzing Queries
Percona Toolkit for MySQL
Eli Aschkenasy - @EliAschkenasy - #JDNL16
10. DB Optimization – Analyzing Queries
http://nk.gl/slow_queries/analyze
For those of us who don’t want to commit.
Eli Aschkenasy - @EliAschkenasy - #JDNL16
11. WITH THIS DATA
WE CAN CHECK THE
IMPLEMENTATION
Eli Aschkenasy - @EliAschkenasy - #JDNL16
13. DB Optimization
Here is my schema, what indexes do I need?
Eli Aschkenasy - @EliAschkenasy - #JDNL16
14. DB Optimization
Index choice depends on the queries
you run, not the data you have!
relational schema
design is based on
DATA
index
design is based on
QUERIES
Eli Aschkenasy - @EliAschkenasy - #JDNL16
15. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
16. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
works
Simple/Compound Searches
Eli Aschkenasy - @EliAschkenasy - #JDNL16
17. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
works
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
AND f_name = ‘Hans’
works
Simple/Compound Searches
Eli Aschkenasy - @EliAschkenasy - #JDNL16
18. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
works
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
AND f_name = ‘Hans’
works
SELECT * FROM PhoneBook
WHERE f_name = ‘Hans’
doesn‘t work
Simple/Compound Searches
Eli Aschkenasy - @EliAschkenasy - #JDNL16
19. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name LIKE ‘Ku%’
works
Range Lookup
Eli Aschkenasy - @EliAschkenasy - #JDNL16
20. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name LIKE ‘Ku%’
works
SELECT * FROM PhoneBook
WHERE l_name = ‘Ku%’
AND f_name = ‘Hans’
doesn‘t work
Range Lookup
Eli Aschkenasy - @EliAschkenasy - #JDNL16
21. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name LIKE ‘Ku%’
works
SELECT * FROM PhoneBook
WHERE l_name = ‘Ku%’
AND f_name = ‘Hans’
doesn‘t work
Range Lookup
Tip #1:
Remember: Any Range Comparison breaks index
benefit for subsequent columns
Eli Aschkenasy - @EliAschkenasy - #JDNL16
22. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
ORDER BY f_name
works
Sorting
Eli Aschkenasy - @EliAschkenasy - #JDNL16
23. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
ORDER BY f_name
works
Sorting
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
ORDER BY phone
doesn‘t work
Eli Aschkenasy - @EliAschkenasy - #JDNL16
24. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
ORDER BY f_name
works
Sorting
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
ORDER BY phone
doesn‘t work
Tip #2:
Remember: Index can only benefit the sorting of the
column immediately following last column used for
search Eli Aschkenasy - @EliAschkenasy - #JDNL16
25. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT phone FROM PhoneBook
WHERE l_name = ‘Kuijpers’
AND f_name = ‘Hans’
works
Index – Only Search
Because phone is included in index
we can select it without overhead
even though we didn’t include it in
the search
Eli Aschkenasy - @EliAschkenasy - #JDNL16
26. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT phone FROM PhoneBook
WHERE l_name = ‘Kuijpers’
AND f_name = ‘Hans’
works
Index – Only Search
Tip #3:
Putting columns in index even if they are not part of
the query might speed things up – COVERING INDEX
Because phone is included in index
we can select it without overhead
even though we didn’t include it in
the search
Eli Aschkenasy - @EliAschkenasy - #JDNL16
27. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
idx (l_name, f_name, phone)
SELECT * FROM PhoneBook
WHERE l_name = ‘Kuijpers’
OR f_name = ‘Hans’
doesn‘t work
Disjoint Selector
additional index (f_name, phone)
would be required
Eli Aschkenasy - @EliAschkenasy - #JDNL16
29. Index Primer
Insurance Company:
PolicyNumber
Agent
Type
idx (policy, name, type)
SELECT * FROM InsuranceCompany
WHERE type =
‘Vervoerdersaansprakelijkheidsverzekering’
Index Size
Type Options:
1. Vervoerdersaansprakelijkheidsverzekering
2. Bestuurdersaansprakelijkheidsverzekering
3. Overeenstemmingsbeoordelingsprocedures
Eli Aschkenasy - @EliAschkenasy - #JDNL16
30. Index Primer
Insurance Company:
PolicyNumber
Agent
Type
idx (type(1))
SELECT * FROM InsuranceCompany
WHERE type =
‘Vervoerdersaansprakelijkheidsverzekering’
Index Size
Type Options:
1. Vervoerdersaansprakelijkheidsverzekering
2. Bestuurdersaansprakelijkheidsverzekering
3. Overeenstemmingsbeoordelingsprocedures
Eli Aschkenasy - @EliAschkenasy - #JDNL16
31. Index Primer
Insurance Company:
PolicyNumber
Agent
Type
idx (type(1))
SELECT * FROM InsuranceCompany
WHERE type =
‘Vervoerdersaansprakelijkheidsverzekering’
Index Size
Type Options:
1. Vervoerdersaansprakelijkheidsverzekering
2. Bestuurdersaansprakelijkheidsverzekering
3. Overeenstemmingsbeoordelingsprocedures
Trick #1:
Use only part of the column as length of the index,
but remember that it will break the ‘covering’
property. Eli Aschkenasy - @EliAschkenasy - #JDNL16
32. Index Primer
Insurance Company:
PolicyNumber
Agent
Type
idx (type(1))
SELECT * FROM InsuranceCompany
WHERE type =
‘Vervoerdersaansprakelijkheidsverzekering’
Index Size
Type Options:
1. Vervoerdersaansprakelijkheidsverzekering
2. Bestuurdersaansprakelijkheidsverzekering
3. Overeenstemmingsbeoordelingsprocedures
Trick #1a:
Specificity Check
SELECT
COUNT(DISTINCT(type)) AS total,
COUNT(DISTINCT(LEFT(type,10))) AS t10,
COUNT(DISTINCT(LEFT(type,20))) AS t20
FROM Insurance Company;
Eli Aschkenasy - @EliAschkenasy - #JDNL16
33. Index Primer
Insurance Company:
PolicyNumber
Agent
Type
idx (type(1))
SELECT * FROM InsuranceCompany
WHERE type =
‘Vervoerdersaansprakelijkheidsverzekering’
Index Estimation
Type Options:
1. Vervoerdersaansprakelijkheidsverzekering
2. Bestuurdersaansprakelijkheidsverzekering
3. Overeenstemmingsbeoordelingsprocedures
Trick #1b:
Cardinality
ANALYZE TABLES;
Eli Aschkenasy - @EliAschkenasy - #JDNL16
34. Index Primer
Phone Book:
Last Name (ASC)
First Name (ASC)
Phone Number (RAND)
MySQL 5.5
idx (l_name) – traditional
idx (l_name, f_name, phone) – covering
Range Lookup
SELECT phone FROM PhoneBook
WHERE l_name = ‘Kuijpers’
AND f_name LIKE ‘%Hans%’
MySQL 5.6
idx (l_name, f_name)
Range access l_name, Filter clause on
f_name (only read if full row match)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
35. Index Strategy
Choose Index order which benefits more queries
• SELECT * FROM t WHERE a=1 AND b=2
• SELECT * FROM t WHERE a>1 AND b=2
Eli Aschkenasy - @EliAschkenasy - #JDNL16
36. Index Strategy
Choose Index order which benefits more queries
• SELECT * FROM t WHERE a=1 AND b=2
• SELECT * FROM t WHERE a>1 AND b=2
KEY (b, a) is better than KEY (a, b)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
37. Index Strategy (Trick #2)
Choose Index order which benefits more queries
• SELECT * FROM t WHERE a=1 AND b=2
• SELECT * FROM t WHERE a>1 AND b=2
KEY (b, a) is better than KEY (a, b)
Index order (without restrictions) should be most
selective to least selective
Eli Aschkenasy - @EliAschkenasy - #JDNL16
38. Index Strategy (Trick #1)
• SELECT * FROM t WHERE a=1 AND b=2
• SELECT * FROM t WHERE a>1 AND b=2
KEY (a, b)
• SELECT * FROM t WHERE a BETWEEN 2 AND 4 AND b=2
Eli Aschkenasy - @EliAschkenasy - #JDNL16
39. Index Strategy (Trick #1)
• SELECT * FROM t WHERE a=1 AND b=2
• SELECT * FROM t WHERE a>1 AND b=2
KEY (a, b)
• SELECT * FROM t WHERE a BETWEEN 2 AND 4 AND b=2
KEY (a, b)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
40. Query Optimization (Trick #1)
• SELECT * FROM t WHERE a=1 AND b=2
• SELECT * FROM t WHERE a>1 AND b=2
KEY (a, b)
• SELECT * FROM t WHERE a BETWEEN 2 AND 4 AND b=2
KEY (a, b)
• SELECT * FROM t WHERE a IN(2,3,4) AND b=2
KEY (a, b)
Enumerating Ranges will ensure the usage of both key parts
Eli Aschkenasy - @EliAschkenasy - #JDNL16
41. Query Optimization (Trick #2)
CREATE TABLE profile(
id INT,
gender CHAR(1),
age TINYINT(3),
city VARCHAR(100),
PRIMARY KEY(id)
);
SELECT id FROM profile
WHERE city = “Amsterdam”
AND age BETWEEN 35 AND 40
KEY (city, age, id)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
42. Query Optimization (Trick #2)
CREATE TABLE profile(
id INT,
gender CHAR(1),
age TINYINT(3),
city VARCHAR(100),
PRIMARY KEY(id)
);
SELECT id FROM profile
WHERE city = “Amsterdam”
AND age BETWEEN 35 AND 40
KEY (city, age, id)
SELECT id FROM profile
WHERE city = “Amsterdam”
AND gender = “F”
AND age BETWEEN 35 AND 40
KEY (city, gender, age, id)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
43. Index Strategy (Trick #3)
CREATE TABLE profile(
id INT,
gender CHAR(1),
age TINYINT(3),
city VARCHAR(100),
PRIMARY KEY(id)
);
SELECT id FROM profile
WHERE city = “Amsterdam”
AND age BETWEEN 35 AND 40
KEY (city, gender, age, id)
SELECT id FROM profile
WHERE city = “Amsterdam”
AND gender = “F”
AND age BETWEEN 35 AND 40
KEY (city, gender, age, id)
Trick #3:
Use as little indexes as possible. A little query rewrite
could save massive overhead on the index-side.
Eli Aschkenasy - @EliAschkenasy - #JDNL16
44. Query Optimization (Trick #2)
SELECT id FROM profile
WHERE city = “Amsterdam”
AND age BETWEEN 35 AND 40
KEY (city, gender, age, id)
SELECT id FROM profile
WHERE city = “Amsterdam”
AND gender = “F”
AND age BETWEEN 35 AND 40
KEY (city, gender, age, id)
SELECT id FROM profile
WHERE city = “Amsterdam”
AND gender IN(‘M’,’F’,’O’)
AND age IN(35,36,37,38,39,40)
KEY (city, gender, age, id)
SELECT id FROM profile
WHERE city = “Amsterdam”
AND gender = “F”
AND age IN(35,36,37,38,39,40)
KEY (city, gender, age, id)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
45. Resource Handling (Trick #1)
SELECT id, name, picture, … , age
FROM profile
WHERE city = “Amsterdam”
AND gender IN(‘M’,’F’,’O’)
AND age IN(35,36,37,38,39,40)
ORDER BY rating
LIMIT 100, 10
(page 11)
Retrieves 110 rows and discards 100
Eli Aschkenasy - @EliAschkenasy - #JDNL16
46. Resource Handling (Trick #1)
SELECT id, name, picture, … , age
FROM profile
WHERE city = “Amsterdam”
AND gender IN(‘M’,’F’,’O’)
AND age IN(35,36,37,38,39,40)
ORDER BY rating
LIMIT 100, 10
(page 11)
Retrieves 110 rows and discards 100
SELECT id, name, picture, … , age
FROM profile
INNER JOIN(
SELECT id
FROM profile
WHERE city = “Amsterdam”
AND gender IN(‘M’,’F’,’O’)
AND age IN(35,36,37,38,39,40)
ORDER BY rating
LIMIT 100, 10)
AS x
USING (id);
(page 11)
Retrieves 110 ids and discards 100
Retrieves only 10 rows of data
Eli Aschkenasy - @EliAschkenasy - #JDNL16
47. Query Optimization (Trick #3)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT id FROM url
WHERE url="http://www.joomladagen.nl";
CREATE TABLE pseudohash (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
url VARCHAR(255) NOT NULL,
url_crc INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY(id)
);
CREATE TRIGGER pseudohash_crc_ins
BEFORE INSERT ON pseudohash FOR EACH ROW BEGIN
SET NEW.url_crc=crc32(NEW.url);
CREATE TRIGGER pseudohash_crc_upd
BEFORE UPDATE ON pseudohash FOR EACH ROW BEGIN
SET NEW.url_crc=crc32(NEW.url);
SELECT id FROM url
WHERE url_crc=CRC32("http://www.joomladagen.nl")
AND url="http://www.joomladagen.nl";
48. Query Optimization (Trick #3)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT id FROM url
WHERE url="http://www.joomladagen.nl";
CREATE TABLE pseudohash (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
url VARCHAR(255) NOT NULL,
url_crc INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY(id)
);
CREATE TRIGGER pseudohash_crc_ins
BEFORE INSERT ON pseudohash FOR EACH ROW BEGIN
SET NEW.url_crc=crc32(NEW.url);
CREATE TRIGGER pseudohash_crc_upd
BEFORE UPDATE ON pseudohash FOR EACH ROW BEGIN
SET NEW.url_crc=crc32(NEW.url);
SELECT id FROM url
WHERE url_crc=CRC32("http://www.joomladagen.nl")
AND url="http://www.joomladagen.nl";
Fixed length index
Fixed length data
49. Resource Handling (Trick #2)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT *
FROM comment AS c
LEFT JOIN article AS a ON a.id_comment = c.id
LEFT JOIN users AS u ON a.created_by = u.id
WHERE a.title=‘joomladagen‘
AND u.name = ‘Hans Kuijpers’;
50. Resource Handling (Trick #2)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT *
FROM comment AS c
LEFT JOIN article AS a ON a.id_comment = c.id
LEFT JOIN users AS u ON a.created_by = u.id
WHERE a.title=‘joomladagen‘
AND u.name = ‘Jisse Reitsma’;
51. Resource Handling (Trick #2)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT *
FROM comment AS c
LEFT JOIN article AS a ON a.id_comment = c.id
LEFT JOIN users AS u ON a.created_by = u.id
WHERE a.title=‘joomladagen‘
AND u.name = ‘Jisse Reitsma’;
CREATE TABLE comment (
id INT(11) NOT NULL…,
tag VARCHAR(255)…,
importance TINYINT(3)…,
image BLOB…,
comment TEXT…,
created DATETIME…,
created_by INT(11)…,
modified…,
….,
….,
publish_up DATETIME,
PRIMARY KEY(`id`)
) ENGINE=InnoDB;
52. Resource Handling (Trick #2)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT *
FROM comment AS c
LEFT JOIN article AS a ON a.id_comment = c.id
LEFT JOIN users AS u ON a.created_by = u.id
WHERE a.title=‘joomladagen‘
AND u.name = ‘Jisse Reitsma’;
CREATE TABLE comment (
id INT(11) NOT NULL…,
tag VARCHAR(255)…,
importance TINYINT(3)…,
image BLOB…,
comment TEXT…,
created DATETIME…,
created_by INT(11)…,
modified…,
….,
….,
publish_up DATETIME,
PRIMARY KEY(`id`)
) ENGINE=InnoDB;
53. Resource Handling (Trick #2)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT c.comment
FROM comment AS c
LEFT JOIN article AS a ON a.id_comment = c.id
LEFT JOIN users AS u ON a.created_by = u.id
WHERE a.title=‘joomladagen‘
AND u.name = ‘Jisse Reitsma’;
CREATE TABLE comment (
id INT(11) NOT NULL…,
tag VARCHAR(255)…,
importance TINYINT(3)…,
image BLOB…,
comment TEXT…,
created DATETIME…,
created_by INT(11)…,
modified…,
….,
….,
publish_up DATETIME,
PRIMARY KEY(`id`)
) ENGINE=InnoDB;
54. Resource Handling (Trick #2a)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT SUBSTRING(c.comment, 0, 25) AS short_comment
FROM comment AS c
LEFT JOIN article AS a ON a.id_comment = c.id
LEFT JOIN users AS u ON a.created_by = u.id
WHERE a.title=‘joomladagen‘
AND u.name = ‘Jisse Reitsma’;
CREATE TABLE comment (
id INT(11) NOT NULL…,
tag VARCHAR(255)…,
importance TINYINT(3)…,
image BLOB…,
comment TEXT…,
created DATETIME…,
created_by INT(11)…,
modified…,
….,
….,
publish_up DATETIME,
PRIMARY KEY(`id`)
) ENGINE=InnoDB;
55. Query Optimization (Trick #4)
Eli Aschkenasy - @EliAschkenasy - #JDNL16
SELECT SUBSTRING(c.comment, 0, 25) AS short_comment
FROM comment AS c
LEFT JOIN article AS a ON a.id_comment = c.id
LEFT JOIN users AS u ON a.created_by = u.id
WHERE a.title=‘joomladagen‘
AND u.name = ‘Jisse Reitsma’;
SELECT id FROM users
WHERE name = ‘Jisse Reitsma’; 57125
SELECT id FROM article
WHERE title = ‘Joomladagen’
AND created_by = 57125; (123,223,934,1145)
SELECT SUBSTRING(comment, 0, 25) AS short_comment FROM comment
WHERE id IN(123,223,934,1145);
Trick #4:
Query cache validation of 3 tables.
Complex queries reduce likelihood of repetition.
57. MySQL 5.7 – JSON
Eli Aschkenasy - @EliAschkenasy - #JDNL16
CREATE TABLE profile(
id INT(11)…,
profile TEXT…,
PRIMARY KEY (`id`)
) ENGINE = InnoDB;
INSERT INTO profile (id, profile)
VALUES (1, ‘{“handle”: “@EliAschkenasy”}’);
58. MySQL 5.7 – JSON
Eli Aschkenasy - @EliAschkenasy - #JDNL16
CREATE TABLE profile(
id INT(11)…,
profile TEXT…,
PRIMARY KEY (`id`)
) ENGINE = InnoDB;
INSERT INTO profile (id, profile)
VALUES (1, ‘{“handle”: “@EliAschkenasy”}’);
SELECT id FROM profile
WHERE field_name REGEXP '"handle":"([^"]*)@EliAschkenasy([^"]*)"';
59. MySQL 5.7 – JSON
Eli Aschkenasy - @EliAschkenasy - #JDNL16
CREATE TABLE profile(
id INT(11)…,
profile TEXT…,
PRIMARY KEY (`id`)
) ENGINE = InnoDB;
INSERT INTO profile (id, profile)
VALUES (1, ‘{“handle”: “@EliAschkenasy”}’);
SELECT id FROM profile
WHERE field_name REGEXP '"handle":"([^"]*)@EliAschkenasy([^"]*)"';
Full Table Scan (not cacheable just like full text search)
REGEX
60. MySQL 5.7 – JSON
Eli Aschkenasy - @EliAschkenasy - #JDNL16
CREATE TABLE profile(
id INT(11)…,
profile JSON,
PRIMARY KEY (`id`)
) ENGINE = InnoDB;
INSERT INTO profile (id, profile)
VALUES (1, ‘{“id”: 1, “handle”: “@EliAschkenasy”}’);
SELECT JSON_EXTRACT(profile, ‘$.handle’) FROM profile
WHERE id = 1; - @EliAschkenasy
61. MySQL 5.7 – JSON
Eli Aschkenasy - @EliAschkenasy - #JDNL16
CREATE TABLE profile(
id INT(11)…,
profile JSON,
PRIMARY KEY (`id`)
) ENGINE = InnoDB;
INSERT INTO profile (id, profile)
VALUES (1, ‘{“id”: 1, “handle”: “@EliAschkenasy”}’);
SELECT JSON_EXTRACT(profile, ‘$.handle’) FROM profile
WHERE id = 1; - @EliAschkenasy
JSON data type consumes about 5% more memory
JSON columns can’t have a default value
JSON columns can’t be indexed
BETWEEN, IN(), GREATEST(), LEAST() Are NOT supported yet!