MySQL Scaling Presentation

Strategic MySQL Planning for Complexity & Growth Or MySQL Scaling for Dummies Thomas Falgout Sr. Developer Yahoo!

The Basics
Ways to squeeze extra performance:
Query Optimization
Schema Definition
Hardware
Software
MySQL Config
Cache
Monitoring

Query Optimizations
KNOW EVERY QUERY!
Developers usually don't write proper SQL
SQL Statements will directly affect your performance
Example:
Repeating SQL Statements for no benefit
1000 very quick unnecessary queries vs 1 slow query

Query Optimizations
Don't Develop in an island
Talk it over.
Your coworkers might surprise you.
In a good way.

Query Optimizations - EXPLAIN
Only works for SELECT
Quickly show you how much data your queries are processing
Could spend an entire session on just this.
But won't.
Read for yourself.
http://dev.mysql.com/doc/refman/5.1/en/explain.html
http://dev.mysql.com/doc/refman/5.1/en/using-explain.html
http://www.percona.com/files//presentations/EXPLAIN_demystified.pdf
LUUUUUCCCYYYYYYY!!!!!

Query Optimizations
SELECTS
SELECT * is bad
Hurts performance (cache, maintainability, bandwidth)
Joining two 100,000 row tables gets big
“ SELECT COUNT(*) FROM users”
Performs full table scan on Innodb
Faster on MyISAM (stored in table info)

Query Optimizations
INSERTS
Can you batch INSERT?
INSERT INTO a VALUES (1,23),(2,34),(4,33);
To speed up INSERT operations that require multiple statements, LOCK your tables
Can you use LOAD DATA? (Import via CSV)

Query Optimizations
UPDATE
Updates are usually pretty good.
Not much to say.

Query Optimizations
DELETE
DELETES can be costly.
Sometimes causes DB to rebuild index
IDEAS
Add a “to_delete” column and then delete batch
If having to delete A LOT of rows
SELECT INTO rows to keep into temp table
DROP table
RENAME tmp table to table

Query Optimizations
Find queries that do COUNT(*) or create tables with results
Create tables with results
Aggregate data hourly/daily/per update/etc.

Query Optimization – Handling Logs
Don't talk directly to DB
Send updates through daemon to load data
Don't update for each request
Aggregate per 1000 req/2 min/app start
UPDATE counts SET count = count + 42 WHERE id=1

Schema Definition - MyISAM
Original Engine for MySQL
Does not handle transactions
Faster to Write, Slightly faster to Read
Uses separate Data (.MYD), Index (.MYI) and Frame (.frm)

Schema Definition - INNODB
2 nd Engine for MySQL – From Oracle/Innobase
Handles transactions
START TRANS; INSERT xxx; COMMIT
Uses one file for both data and index for entire DB (Can change in my.cnf to separate to table)
Slower to write and read since it stores data per transaction.
Can be tuned to be almost as fast as MyISAM

Schema Definition - BLACKHOLE
Fastest Engine available
Writes Binary logs to disk
Doesn't store data
Useful for offloading replication from the master
Blackhole Sun. Won't you come. Wash away my data.

Schema Definitions
If you can, stick to one storage engine
Falcon
New Transactional Engine
http://forge.mysql.com/wiki/Falcon

Schema Definition - Size
Make your data Small
http://dev.mysql.com/doc/refman/5.1/en/data-size.html
http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html

Schema Definition – Size
TINYINT – 1 Byte
SMALLINT – 2 Bytes
MEDIUMINT – 3 Bytes
INT – 4 Bytes
BIGINT – 8 Bytes
http://dev.mysql.com/doc/refman/5.0/en/numeric-types.html

Schema Definition - Size
CHAR(N) = N + 1 byte. 2 bytes if N > 255 bytes
VARCHAR(N) = X + 1 byte. X = Number of actual characters in string.
N & X = 1 Byte. Unless using utf8 or a multi-byte format
Same rules apply for BINARY & VARBINARY.
Extra data is TRUNCATED!!!!!!
That's bad.
http://dev.mysql.com/doc/refman/5.0/en/string-types.html

Schema Definition - VARCHAR
VARCHAR(255) is usually the sign of a careless design
VARCHAR(255) is 4.x limitation
When creating a temp table, it allocates the greatest size of data (255)

Schema Definition - INT
BIGINT is not needed for AUTO_INCREMENT
INT UNSIGNED = 4.3 Billion values.
If you hit this, you should be partitioning
Unless you have a reason, all of your INT columns should be UNSIGNED
Adds data integrity for negative values

Schema Definition - Date
TIMESTAMP = 4 bytes
DATETIME = 8 bytes
DATE = 3 bytes
TIME = 3 bytes
???
INT as UNIX Timestamp = 4 bytes

Schema Definition – Storing IP's
IP = 4 Bytes
192.168.101.201 = 11000000.10101000.01100101.11001001
INT UNSIGNED = 4 Bytes!
INET_NTOA => NumberTOAddress
INET_ATON => AddressTONumber
Don't use VARCHAR(15)
Avg of 12 bytes vs 4 bytes

Schema Definition – IP Addr
mysql> SELECT INET_ATON('192.168.101.201'); +------------------------------+ | INET_ATON('192.168.101.201') | +------------------------------+ | 3232261577 | +------------------------------+ mysql> SELECT ... WHERE ip_col BETWEEN INET_ATON('12.13.14.15') AND INET_ATON('13.14.15.16')

Schema Definition - ENUM
Use Codes for enum
m / f instead of male / female
Must alter table to add an enum

Schema Definition - BLOB
BE WARY!
SELECT <field> will return THE WHOLE THING!
Stores first as 768 Bytes, and then increments into 16k data pages per row per TEXT/BLOB field

Hardware
DB typically require hot on RAM before CPU
Separate your App and DB
Keep them close.
Network latency affects experience

Hardware
Split up READS vs WRITES
Create a Slave to replicate your DB to
Send all READS to Slave
Send all WRITES to Master

Hardware – MySQL Cluster
Realtime Transactional DB Engine
Provides HA w/ no Single Point of Failure
Requires 3 servers (2 Data, 1 Mgmt Nodes)
Inserts on one DB are replicated to others
Selects can be performed on any DB
http://dev.mysql.com/tech-resources/articles/mysql-cluster-for-two-servers.html
http://www.lod.com/whitepapers/mysql-cluster-howto.html

Software – MySQL Proxy
Program between your client and your DB
Can be used for
Load Balancing
Failover
Query Analysis
Query Filtering / Modification
http://forge.mysql.com/wiki/MySQL_Proxy

Software – Maatkit
Series of Tools for monitoring & administration
Table Checksum (Compare Master/Slave)
Table Sync (Resolve Differences)
Visual Explain (Explain! Visually!)
Heartbeat (Shows Slave Replication Delay)
And Much, Much More!
http://www.maatkit.org/

Scaling Reads
Get More Servers
Point Slaves to Master
Put Slaves Behind Load Balancer

Scaling Writes
No Easy button
Batch Insert / Update
Don't Delete
Partition your Data
If you're at this point, this presentation probably won't help you.
Read Cal Henderson's Scaling MySQL

MySQL Config
Do you know what those config variables do?
SHOW VARIABLES
key_buffer_size
table_cache
innodb_buffer_pool_size
query_cache_limit
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

MySQL Config
key_buffer_size
MyISAM Only
Size of cache for MyISAM Indexes
Can check performance with SHOW STATUS
key_read_requests
key_reads
key_write_requests
key_writes
key_reads/key_read_requests < 0.01 == GOOD

MySQL Config
table_cache / table_open_cache (5.1.3)
Number of open tables for all threads
This dictates the number of open file descriptors

MySQL Config
INNODB only
Size in bytes of the memory buffer InnoDB uses to cache data and indexes.
Default 8MB

MySQL Config
Query Cache
The query cache stores the text of a SELECT statement with the corresponding result that was sent to the client
Great for tables that don't change often
Doesn't return stale data
Cache is invalidated if returned entry is modified
Another reason why SELECT(*) is bad

MySQL Config
Memory
INNODB – Let MySQL use all memory
MyISAM – Leave more memory for OS page caches

MySQL Config
MySQL Slow Log
Used to log any queries that take longer than x seconds
Use long_query_time to specify x
~1 sec is good starting point
http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html

Caching - External
Memcache
Used by all the big players (Y!, Facebook, etc.)
Originally created by LiveJournal to handle scale
EASY Button
$memcache_obj = memcache_connect("localhost", 11211);
$memcache_obj->add('var_key', 'test variable');
$var = $memcache_obj->get('var_key');

Caching - Internal
Query Cache
http://dev.mysql.com/doc/refman/5.1/en/query-cache.html
key_buffer_size
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html
http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.htm

Benchmarking - mysqlslap
Mysqlslap
mysqlslap -u root -concurrency=10000 -iterations=5 -number-of-queries=10000 -query=”SELECT name FROM city WHERE city_id=12”

Benchmarking – Apache Bench
Built for testing webservices
Like mysqlslap
GOOD NEWS
Benchmark whole web app
BAD NEWS
Benchmark whole web app
http://httpd.apache.org/docs/2.2/programs/ab.html

Monitoring
Monitoring is as important as benchmarking
Basics:
SHOW FULL PROCESSLIST
SHOW GLOBAL STATUS
SHOW GLOBAL VARIABLES

Monitoring Tools
Cacti – RRDtool based networking graphing tool
MUNIN – Similar to Cacti, but easier to maintain
Nagios – More of a notification tool
MySQL Enterprise Dashboard – Notification & Monitoring

Other
Write Proper SQL – Naming Convention
customer_id, order_id NOT id
Be descriptive
Avoid reserve words (date, time, timestamp)

Other
NEVER test on a production system
Goal of test environment:
NOT to test your software
Break your software.

Other
SET sql_mode = 'STRICT_TRANS_TABLES'
Forces you to use good data

The Most Important Lesson This part is absolutely important that you get. It's time to wake up.

The Most Important Lesson When you need to, Ignore everything I've suggested.

Thanks Thanks for listening.

Links Flickr - Scaling MySQL http://www.scribd.com/doc/21593/Scaling-PHPMySQL Very Basic PHP & MySQL: http://www.cs.gsu.edu/~stefan/teaching/csc-6370/notes/phpmysql.pdf JPIPES! http://jpipes.com/presentations/joinfu/joinfu_part_one.pdf http://jpipes.com/presentations/joinfu/joinfu_part_two.pdf http://jpipes.com/presentations/dqm/legend_drunken_query_master.pdf EXPLAIN: http://www.percona.com/files//presentations/EXPLAIN_demystified.pdf Query Optimization: http://www.percona.com/files//presentations/Zurich2007-QueryOptimization.pdf Server Settings Tuning: http://www.percona.com/files//presentations/UC2007-MySQL-Server-Settings-Tuning.pdf http://www.flupps.org/mysql/MySQL_Microscope.ppt http://dev.mysql.com/doc/refman/5.0/en/working-with-null.html http://www.devshed.com/c/a/MySQL/MySQL-Server-Tuning-Tips-and-Tricks/ http://www.planetmysql.org/

MySQL Scaling Presentation

by Tommy Falgout

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 26

Statistics

MySQL Scaling Presentation Presentation Transcript

http://www.slideshare.net	24
http://static.slidesharecdn.com	2