Exploration of MySQL's inner workings. Talk - https://joind.in/event/scotlandphp-2021/mysql-with

1. MySQL with🔔&🎵
@thomas_shone

2. 🪠 How it started

3. 󰱖People
Xin Wu
Simon Mudd
Daniël van Eeden
Mattias Jonsson
Kristian Köhntopp (@isotopp)

4. 📦 Cargo Culting

5. ⚙ Inner Workings

6. 🛠 Cool Tools

7. 👶 My First Table
CREATE TABLE users (
email VARCHAR(255) NOT NULL UNIQUE,
name VARCHAR(300),
password VARCHAR(72),
address TEXT,
country_iso VARCHAR(2),
language_iso VARCHAR(2),
province VARCHAR(64),
joined DATETIME
);

8. PRIMARY KEY TIMESTAMP DATETIME VARCHAR vs CHAR
INDEX Singular vs Plural Language vs Locale TEXT
Splitting DEFAULTs NULLiness Email length
ENUMs Bcrypt Length Assumptions Normalization
🎲 Bingo Card
bob.smith+tag1+tag2@gmail.com === bobsmith@gmail.com
bob.smith@gmail.com === bobsmith@gmail.com

9. 👶 My First Table
CREATE TABLE users (
email VARCHAR(255) NOT NULL UNIQUE,
name VARCHAR(255),
password VARCHAR(72),
address TEXT,
country_iso VARCHAR(2),
language_iso VARCHAR(2),
province VARCHAR(64),
joined DATETIME
);
What engine is used?
What is the primary key?

10. ⚙MyISAM vs InnoDB
Transactions
Row level locking
ACID compliant
Faster reads

11. SELECT TABLE_SCHEMA, TABLE_NAME
FROM information_schema.TABLES
WHERE ENGINE = 'myISAM'

12. 👶 My First Table
CREATE TABLE users (
email VARCHAR(255) NOT NULL UNIQUE,
name VARCHAR(255),
password VARCHAR(72),
address TEXT,
country_iso VARCHAR(2),
language_iso VARCHAR(2),
province VARCHAR(64),
joined DATETIME
) ENGINE=InnoDB;
What engine is used?
What is a clustered index?
What happens without a
NOT NULL UNIQUE?
Resource: https://blog.jcole.us/2013/05/02/how-does-innodb-behave-without-a-primary-key/
What is the primary key?

13. SELECT database_name, table_name
FROM mysql.innodb_index_stats
WHERE index_name = 'GEN_CLUST_INDEX'

14. So is a 🔑 PRIMARY KEY needed?

15. 󰰁Relationships
CREATE TABLE users (
email VARCHAR(255) NOT NULL UNIQUE,
…
);
CREATE TABLE posts (
email VARCHAR(255),
title VARCHAR(255) NOT NULL UNIQUE,
slug VARCHAR(255) NOT NULL UNIQUE,
content TEXT,
status ENUM('draft', 'published')
);

16. SET_VAR(sql_require_primary_key=ON)
Resource: https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_sql_require_primary_key

17. What to use as a PRIMARY KEY?
● INT UNSIGNED NOT NULL AUTO_INCREMENT
● UUID as BINARY(16)
● TEXT

18. 📅 Dates, Times & Timezones

19. HERE BE DRAGONS!

20. Formats
● DATE
○ Min: 1000-01-01
○ Max: 9999-12-31
● DATETIME
○ Min: 1000-01-01 00:00:00
○ Max: 9999-12-31 23:59:59
● TIMESTAMP
○ Min: 1970-01-01 00:00:01
○ Max: 2038-01-19 03:14:07
● TIME
○ Min: -838:59:59
○ Max: 838:59:59

21. 🕑 UTC vs Timezone?
Resource: https://dev.mysql.com/doc/refman/8.0/en/datetime.html

22. SELECT @@GLOBAL.time_zone,
@@SESSION.time_zone;
SET GLOBAL time_zone = timezone;
SET time_zone = timezone;
Resource: https://dev.mysql.com/doc/refman/8.0/en/time-zone-support.html

23. 🔎 Performant Search

24. Physical Storage of InnoDB

25. Physical Storage of InnoDB Index

26. 🧮 1,048,576 rows only needs a tree depth of 10

27. Physical Storage of InnoDB Index

28. 📇 Identifying Index Needs

29. Is there a need?
● User complaints?
● Personal experience and frustrations?
● Resource hogging?
● Slow running processes
● Treat optimizing queries like you’d optimize anything

30. > EXPLAIN SELECT * FROM users WHERE name = "Bob Smith";
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-----------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-----------------------------+
| 1 | SIMPLE | users | NULL | ALL | NULL | NULL | NULL | NULL | 1582 | 1.0 | Using where; Using filesort |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-----------------------------+
> EXPLAIN SELECT id, email FROM users WHERE email = "bob@smith.com";
+----+-------------+-------+------------+-------+---------------+-----------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+-----------+---------+------+------+----------+-------------+
| 1 | SIMPLE | users | NULL | INDEX | email | email | 256 | NULL | 1 | 100.0 | Using index |
+----+-------------+-------+------------+-------+---------------+-----------+---------+------+------+----------+-------------+
> EXPLAIN SELECT * FROM users WHERE id = 123456;
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | PRIMARY | users | NULL | CONST | PRIMARY | PRIMARY | 4 | NULL | 1 | 100.0 | NULL |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
EXPLAIN your queries
Resource: https://dev.mysql.com/doc/refman/5.7/en/explain-output.html

31. # Hints to use one of the following indexes
SELECT * FROM table USE INDEX (index_name,...) WHERE ...;
# Same as above but assumes a full table scan is very expensive
SELECT * FROM table FORCE INDEX (index_name,...) WHERE ...;
# Ensures that certain indexes aren’t being used
SELECT * FROM table IGNORE INDEX (index_name,...) WHERE ...;
Resource: https://dev.mysql.com/doc/refman/8.0/en/index-hints.html
Use the FORCE INDEX

32. SHOW [GLOBAL | SESSION] VARIABLES
[LIKE 'pattern' | WHERE expr]
SET variable = expr [, variable = expr] ...
> SET slow_query_log = 1;
> SET long_query_time = 10; # Default
> SET min_examined_row_limit = 100;
> SET log_queries_not_using_indexes = 'ON';
> SET slow_query_log_file = 'host_name-slow.log'; # Default
Enable Logging
Resource: https://dev.mysql.com/doc/refman/5.6/en/slow-query-log.html

33. $ vim /etc/mysql/conf.d/mysql.cnf # depends on distro
[mysqld]
performance_schema=ON
$ service mysqld restart # depends on distro too!
$ mysql
> SHOW VARIABLES WHERE Variable_name = 'performance_schema';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| performance_schema | ON |
+--------------------+-------+
Performance Schema
Resource: https://dev.mysql.com/doc/refman/5.7/en/performance-schema.html

34. > SELECT DIGEST_TEXT, SUM_SELECT_SCAN
FROM performance_schema.events_statements_summary_by_digest
WHERE SUM_SELECT_SCAN > 0 AND SCHEMA_NAME = [schema] ORDER BY SUM_SELECT_SCAN DESC;
+----------------------------------------------+-----------------+
| DIGEST_TEXT | SUM_SELECT_SCAN |
+----------------------------------------------+-----------------+
| SELECT `*` FROM `users` WHERE ( `name` = ? ) | 700 |
+----------------------------------------------+-----------------+
> SELECT DIGEST_TEXT, SUM_ROWS_EXAMINED / SUM_ROWS_SENT AS RATIO
FROM performance_schema.events_statements_summary_by_digest
WHERE SCHEMA_NAME = [schema] ORDER BY RATIO DESC;
+---------------------------------------------------------------+-------+
| DIGEST_TEXT | RATIO |
+---------------------------------------------------------------+-------+
| SELECT `*` FROM `users` WHERE ( `name` = ? AND `joined` > ? ) | 10730 |
+---------------------------------------------------------------+-------+
events_statements_summary_by_digest
Resource: https://dev.mysql.com/doc/refman/5.7/en/performance-schema.html

35. 🏗 Index Creation Options

36. Creation
● Combining columns
○ INDEX idx_columnA_columnB (columnA, columnB)
● Truncating columns
○ INDEX idx_columnC (columnC(72))
● Primary key considerations
○ Primary key is post-pended on to all indexes.
○ The size of your primary key impacts your index size.

37. CREATE TABLE users (
id UNSIGNED NOT NULL AUTO_INCREMENT,
email VARCHAR(255),
name VARCHAR(255),
INDEX idx_email_name (email, name),
INDEX idx_name_email (name, email),
PRIMARY KEY (id)
);
SELECT * TABLE users WHERE email='bob@smith.com';
Which index will be used?

38. Creation
CREATE TABLE users (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
name VARCHAR(255),
email VARCHAR(255) NOT NULL UNIQUE,
password VARCHAR(72),
address TEXT,
country_iso VARCHAR(2),
province VARCHAR(64)
INDEX idx_email (email),
INDEX idx_name (name),
INDEX idx_country_iso (country_iso),
INDEX idx_province (province),
PRIMARY KEY (id)
);

39. Original
CREATE TABLE users (
email VARCHAR(255) NOT NULL UNIQUE,
name VARCHAR(255),
password VARCHAR(72),
address TEXT,
country_iso VARCHAR(2),
language_iso VARCHAR(2),
province VARCHAR(64),
joined DATETIME
);

40. Second Attempt
CREATE TABLE users (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
name VARCHAR(255),
email VARCHAR(255) NOT NULL UNIQUE,
password VARCHAR(72),
address TEXT,
country_iso VARCHAR(2),
province VARCHAR(64)
UNIQUE idx_email (email),
INDEX idx_name (name(16)),
INDEX idx_country_iso_province (country_iso, province(16)),
PRIMARY KEY (id)
);

41. 🧮 How are Indexes are Selected?

42. Query to Index Selection
1. MySQL will use only ONE index for the WHERE statement
a. Have you hinted/forced an index?
b. Do you have an index that matches the WHERE statement?
c. Do you have one (or more) index(es) that matches part of the WHERE statement
i. We can use left most part of multiple column indexes.
ii. Index with smallest row result is picked.
2. ORDER/GROUP BY uses available indexes of the LEFT MOST
column
GROUP BY columnA, columnB ORDER BY columnC, columnD
Resource: https://dev.mysql.com/doc/refman/8.0/en/mysql-indexes.html

43. 📈Evaluating an Index

44. Evaluating Indexes
● Same tools as the Identification process
● Is the index redundant?
SELECT table_name, redundant_index_name, dominant_index_name
FROM sys.schema_redundant_indexes
WHERE table_schema = [schema]
● Is the index even used?
SELECT object_schema, object_name, index_name
FROM performance_schema.table_io_waits_summary_by_index_usage
WHERE index_name IS NOT NULL
AND index_name != 'PRIMARY'
AND count_star = 0
AND object_schema = [schema]

45. 󰰁Relationships

46. Relationships
CREATE TABLE users (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
…
);
CREATE TABLE posts (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
author_user_id INT UNSIGNED NOT NULL,
…
FOREIGN KEY (author_user_id)
REFERENCES users(id)
ON DELETE CASCADE
);

47. 🔑FOREIGN KEY or not 💀FOREIGN KEY?

48. Using foreign keys
Cons
● Locks
● Performance
● Distributed systems (multiple databases)
● 15 year old bug (foreign keys and triggers)
● Logic in database not code
Pros
● Data consistency on INSERT and DELETE
● Enforced type constraints between fields

49. ⚖Scaling

50. Cargo Culting Text
column VARCHAR(255) NOT NULL DEFAULT ''

51. Data Types
Size of Field = header + body
VARCHAR(64) = 1 byte header (to store 64)
+ up to 64 characters
1 character = 1-4 bytes (depending on encoding)
-------------------------------------------------
Minimum = 1 byte
Maximum = 257 bytes (using utf8mb4)
Resource: https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html

52. 🛠 Cool Tools

53. 🪑 MySQL Workbench
Resource: https://www.mysql.com/products/workbench/

54. 🗃 mysqldump
Resource: https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html
$ vim backup_database.sh
# Quick and dirty MySQL backup solution
DATETIME=$(date +%Y%m%d-%H%M%S)
# Which database to backup automatically
DATABASE=db_name
# Use file based password handling
mysqldump [options] $(DATABASE) | tar cvzf > db_name.$(DATETIME).sql.tar.gz
$ crontab -e
# Every day at 4am
0 4 * * * backup_database.sh

55. 🕵 Cadfael
Resource: https://github.com/xsist10/cadfael
$ composer global require cadfael/cadfael # or get the phar
$ ./vendor/bin/cadfael run
--host [host] # Where is the database host?
--username [username] # MySQL username
--performance_schema # Run performance schema checks
[schema1] [...schema2] # Which schemas to check?
+----------------------+-------------------------------+---------+--------------------------------------------------------------------------------------+
| Check | Entity | Status | Message |
+----------------------+-------------------------------+---------+--------------------------------------------------------------------------------------+
| SaneInnoDbPrimaryKey | table_with_insane_primary_key | Warning | In InnoDB tables, the PRIMARY KEY is appended to other indexes. |
| | | | If the PRIMARY KEY is big, other indexes will use more space. |
| | | | Maybe turn your PRIMARY KEY into UNIQUE and add an auto_increment PRIMARY KEY. |
| | | | Reference: https://dev.mysql.com/doc/refman/5.7/en/innodb-index-types.html |
| EmptyTable | empty_table | Warning | Table contains no records. |
| RedundantIndexes | table_with_insane_primary_key | Concern | Redundant index `full_name` (superseded by `full_name_height_in_cm`). |
| | | | A redundant index can probably drop it (unless it's a UNIQUE, in which case the |
| | | | dominant index might be a better candidate for reworking). |
| | | | Reference: https://dev.mysql.com/doc/refman/8.0/en/sys-schema-redundant-indexes.html |
+----------------------+-------------------------------+---------+--------------------------------------------------------------------------------------+

56. 🔄Online Schema Change
Perform schema changes on really large tables without locking using OSC.
Resource: https://github.com/facebookincubator/OnlineSchemaChange/wiki/How-OSC-works
MyTable MyTable_new
copy
MyTable MyTable_old
rename
MyTable
MyTable_new rename
Atomic

57. ⏭ So what did I cowardly skip?

58. 🙋Questions?
@thomas_shone