1. Common MySQL Scalability Mistakes - 2010.10
Common MySQL
Scalability Mistakes
Ronald Bradford
OTN LAD Tour
South America
2010.10
www.RonaldBradford.com
Ronald Bradford
http://ronaldbradford.com

2. Common MySQL Scalability Mistakes - 2010.10
9

3. Common MySQL Scalability Mistakes - 2010.10
My website seems to
freeze or responds
randomly?
PROBLEM

4. Common MySQL Scalability Mistakes - 2010.10
The default MyISAM
storage engine uses
exclusive table locks for
DML.
CAUSE

5. Common MySQL Scalability Mistakes - 2010.10
Optimize blocking query
performance
Use a transactional engine
with MVCC and row based
locking to address the
LOCKING issue
SOLUTION

6. Common MySQL Scalability Mistakes - 2010.10
End user report that selects all
customer, order, order lines and
order history data and performs
poor joins. This takes shared read
locks blocking future write locks
then future reads.
EXAMPLE

7. Common MySQL Scalability Mistakes - 2010.10
MySQL is unique in that it offers
different mechanisms for storing
and retrieving data, each with
strengths and weaknesses.
The DEFAULT is not always the
best.
WHY

8. Common MySQL Scalability Mistakes - 2010.10
MySQL PROCESSLIST
Blocked have State = Locked
Blocker - Same table, larger Time
HOW
mysql> SHOW PROCESSLIST;
+----+------+-----------+-------+---------+------+------------+---------------
| Id | User | Host | db | Command | Time | State | Info
+----+------+-----------+-------+---------+------+------------+---------------
| 13 | app1 | localhost | odtug | Query | 144 | User sleep | UPDATE emp ...
| 14 | app1 | localhost | odtug | Query | 116 | Locked | select c from emp
| 15 | app1 | localhost | odtug | Query | 89 | Locked | select c from emp

9. Common MySQL Scalability Mistakes - 2010.10
Optimize Blocker
Indexes
Limit query
Summary table
Change storage engine
HOW

10. Common MySQL Scalability Mistakes - 2010.10
8

11. Common MySQL Scalability Mistakes - 2010.10
Why is my database so
large?
PROBLEM

12. Common MySQL Scalability Mistakes - 2010.10
Don’t store large static
objects in the database
SOLUTION

13. Common MySQL Scalability Mistakes - 2010.10
80% of data is email content/
attachments
60% of data is PDF documents
30% of data is uncompressed large
XML objects
EXAMPLE

14. Common MySQL Scalability Mistakes - 2010.10
Maximize memory usage
for important data
Reduce database recovery
time
WHY

15. Common MySQL Scalability Mistakes - 2010.10
Compress large text data
90% saving on XML data
Store static data in files
Avoids DB handling overhead
HOW

16. Common MySQL Scalability Mistakes - 2010.10
HOW
Table Size per schema
# Schema Table Usage
SELECT table_schema,table_name,engine,row_format, table_rows, avg_row_length,
(data_length+index_length)/1024/1024 as total_mb,
(data_length)/1024/1024 as data_mb,
(index_length)/1024/1024 as index_mb,
CURDATE() AS today
FROM information_schema.tables
WHERE table_schema = DATABASE()
ORDER BY 7 DESC;
http://ronaldbradford.com/mysql-dba/

17. Common MySQL Scalability Mistakes - 2010.10
7

18. Common MySQL Scalability Mistakes - 2010.10
I can't access my website?
PROBLEM

19. Common MySQL Scalability Mistakes - 2010.10
TRUE STORY

20. Common MySQL Scalability Mistakes - 2010.10
Question:
TRUE STORY

21. Common MySQL Scalability Mistakes - 2010.10
Question:
How do you know when your server
is down or not accessible?
TRUE STORY

22. Common MySQL Scalability Mistakes - 2010.10
Question:
How do you know when your server
is down or not accessible?
Answer:
TRUE STORY

23. Common MySQL Scalability Mistakes - 2010.10
Question:
How do you know when your server
is down or not accessible?
Answer:
The users will let us know.
TRUE STORY

24. Common MySQL Scalability Mistakes - 2010.10
SOLUTION
Integrated monitoring
including graphical
interface, real time
analysis and notification

25. Common MySQL Scalability Mistakes - 2010.10
Monitoring/Alerting
Graphical
Historical
Necessary
Generally missing/incomplete
Useless for real-time analysis
HOW

26. Common MySQL Scalability Mistakes - 2010.10
Dashboard
The state of NOW
Sampling at 1s/3s/5s
e.g. 0.1% of throughput
HOW

27. Common MySQL Scalability Mistakes - 2010.10
Instrumentation
Important to business viability
e.g. orders per minute
page load time
Seamless implementation
i.e. no code changes to view real-time
extensible
HOW

28. Common MySQL Scalability Mistakes - 2010.10
6

29. Common MySQL Scalability Mistakes - 2010.10
My replication slave can't
keep up?
PROBLEM

30. Common MySQL Scalability Mistakes - 2010.10
Know the weakest link(s)
of MySQL replication and
don't exceed that,
or cheat.
SOLUTION

31. Common MySQL Scalability Mistakes - 2010.10
If replication can't catchup,
slaves are useless.
Backup & recovery may
also suffer.
WHY

32. Common MySQL Scalability Mistakes - 2010.10
Master
DML Statement
Write Data/Redo Log
Write Binary Log
Return OK to client
HOW

33. Common MySQL Scalability Mistakes - 2010.10
Slave
Detect master log change
Retrieve binary log entry
Write relay log (IO_THREAD)
Read relay log
Apply DML (SQL_THREAD)
Write Data/redo log
HOW

34. Common MySQL Scalability Mistakes - 2010.10
Replication workarounds
Restrict queries executed
--ignore
Different storage engines
Different index structures
HOW

35. Common MySQL Scalability Mistakes - 2010.10
Advanced workarounds
RAID 0 (large number of slaves)
Pre fetch thread
EXPERT TIP

36. Common MySQL Scalability Mistakes - 2010.10
5

37. Common MySQL Scalability Mistakes - 2010.10
My server has crashed with
a hard drive failure
PROBLEM

38. Common MySQL Scalability Mistakes - 2010.10
Question:
Have you ever performed a database recovery?
Answer:
No, why?
TRUE STORY

39. Common MySQL Scalability Mistakes - 2010.10
Consultant:
Do you know that your daily backups only
recover the data up to that time, e.g. 1am. You
know you have lost all your sales and changes
since then.
Customer:
No, I didn’t know that.
TRUE STORY

40. Common MySQL Scalability Mistakes - 2010.10
Have a DR Plan
Documented
Tested
Timed
Verified - End to End
SOLUTION

41. Common MySQL Scalability Mistakes - 2010.10
Do you pass the MySQL backup/
recovery quiz?
http://rb42.com/mysql-backup-quiz
HOW

42. Common MySQL Scalability Mistakes - 2010.10
1. Do you have MySQL backups in place?
2. Do you backup ALL your MySQL data?
3. Do you have consistent MySQL backups?
4. Do you have backups that include both static snapshot and point in
time transactions?
5. Do you review your backup logs EVERY SINGLE day or have tested
backup monitoring in place?
6. Do you perform a test recovery of your static backup?
7. Do you perform a test recovery to point in time?
8. Do you time your backup and recovery process and review over time?
9. Do you have off-site copies of your backups?
10. Do you backup your primary binary logs?
QUIZ
http://rb42.com/mysql-backup-quiz

43. Common MySQL Scalability Mistakes - 2010.10
4

44. Common MySQL Scalability Mistakes - 2010.10
Why is my database
executing 1,200 qps
for 50 users?
PROBLEM

45. Common MySQL Scalability Mistakes - 2010.10
Determine what queries
are running and why they
are running?
SOLUTION

46. Common MySQL Scalability Mistakes - 2010.10
Excessive SQL statements
Duplicate
Redundant
Cachable
Row at a time (RAT)
CAUSE

47. Common MySQL Scalability Mistakes - 2010.10
Reducing SQL load both
improves performance now
and provides greater
capacity as you scale
WHY

48. Common MySQL Scalability Mistakes - 2010.10
EXAMPLE
http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/
SELECT * FROM activities_theme WHERE theme_parent_id=0
SELECT * FROM activities_theme WHERE theme_parent_id=1
SELECT * FROM activities_theme WHERE theme_parent_id=2
SELECT * FROM activities_theme WHERE theme_parent_id=11
SELECT * FROM activities_theme WHERE theme_parent_id=16
SELECT *
FROM activities_theme
WHERE theme_parent_id in (0,1,2,11,16)
CAT
RAT

49. Common MySQL Scalability Mistakes - 2010.10
EXAMPLE
http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/
5 Query SELECT * FROM `artist`
5 Query SELECT * FROM `artist`
5 Query SELECT * FROM `artist`
5 Query SELECT * FROM `artist`
5 Query SELECT * FROM `artist`
5 Query SELECT * FROM `artist` WHERE (ArtistID = 196 )
5 Query SELECT * FROM `artist` WHERE (ArtistID = 2188 )
5 Query SELECT * FROM `artist`
5 Query SELECT * FROM `artist`
5 Query SELECT * FROM `artist`
Duplicate
Unnecessary
Only 2 queries
necessary or 1 CAT

50. Common MySQL Scalability Mistakes - 2010.10
EXAMPLE
http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/
SELECT pages_id, pages_livestats_code, pages_title,
pages_parent, pages_exhibid, pages_theme,
pages_accession_num
FROM pages WHERE pages_id = 0
5 minutes
6000 executions
0 is out of bounds
Unnecessary

51. Common MySQL Scalability Mistakes - 2010.10
Capture & Analyze
DML is easy
SELECT is harder
HOW
http://www.slideshare.net/ronaldbradford/capturing-analyzing-and-optimizing-mysql

52. Common MySQL Scalability Mistakes - 2010.10
MySQL Binary Log (Archive Redo)
mysqlbinlog
mk-query-digest
One Liner
HOW
http://ronaldbradford.com/blog/mysql-dml-stats-per-table-2009-09-09/

53. Common MySQL Scalability Mistakes - 2010.10
HOW
55463 update sessions
25574 insert into sessions
12820 update items
11636 insert into item_categories
7532 update users
5168 delete from item_categories
4076 update extended_item_infos
3701 insert into sphinx_new_items
3701 insert into mini_items
2190 update sweet_bars
1922 update chat_users
1662 update item_shipping_infos
1265 update search_terms
1260 insert into images
931 delete from item_shipping_infos
825 update booths
713 update booth_stats
574 update topics
540 update offers
81k of 141k
Top 2 queries = 57%

54. Common MySQL Scalability Mistakes - 2010.10
Process List
General Log
tcpdump
Application
HOW

55. Common MySQL Scalability Mistakes - 2010.10
Capturing, Analyzing and Optimizing
your SQL
http://www.slideshare.net/ronaldbradford/capturing-analyzing-and-
optimizing-mysql
HOW

56. Common MySQL Scalability Mistakes - 2010.10
/* Comment your queries */
The more products you have, the
more developers you have, the
more time you spend in code
identification before you can even
determine a resolution
EXPERT TIP

57. Common MySQL Scalability Mistakes - 2010.10
3

58. Common MySQL Scalability Mistakes - 2010.10
The database is slow.
My webpage takes five
seconds to load.
PROBLEM

59. Common MySQL Scalability Mistakes - 2010.10
Evaluate the time taken in
the database and all
stages in between
SOLUTION

60. Common MySQL Scalability Mistakes - 2010.10
Client example showed a webpage
taking 5 seconds to load. The html
component was taking only 400 ms.
Any MySQL performance
improvement will only tune 8% of
the total time.
EXAMPLE

61. Common MySQL Scalability Mistakes - 2010.10
Performance is important
to end user
Performance is perception
WHY

62. Common MySQL Scalability Mistakes - 2010.10
Firebug - http://getfirebug.com/
Httpwatch - http://httpwatch.com/
Page speed - http://code.google.com/speed/page-speed/
YSlow - http://developer.yahoo.com/yslow/
wget/curl
Application code instrumentation
HOW

63. Common MySQL Scalability Mistakes - 2010.10
http://www.stevesouders.com/
http://developer.yahoo.com/
performance/rules.html
EXPERT TIP

64. Common MySQL Scalability Mistakes - 2010.10
2

65. Common MySQL Scalability Mistakes - 2010.10
I want to add new H/W.
How do I change my
application to support this?
PROBLEM

66. Common MySQL Scalability Mistakes - 2010.10
Develop a seamless
integration that requires no
code changes, no downtime
and very little additional
physical resources.
SOLUTION

67. Common MySQL Scalability Mistakes - 2010.10
Integrated monitoring and
instrumentation
Deployed from Day 1
HOW

68. Common MySQL Scalability Mistakes - 2010.10
Seamless automated server
deployment
Version Control
Build & Release
Runtime config management
Automated discovery
HOW

69. Common MySQL Scalability Mistakes - 2010.10
API
One code path for business
functionality
Implied business documentation
Enforced data exchange standard
Testability
HOW

70. Common MySQL Scalability Mistakes - 2010.10
Different levels of data availability
Read & Write
Read Only
No Access
Cached
HOW

71. Common MySQL Scalability Mistakes - 2010.10
Different principles for scalability
Read Scalability
Write Scalability
Caching
HOW

72. Common MySQL Scalability Mistakes - 2010.10
Successful MySQL Scalability
1. Integrated monitoring & instrumentation
2. Seamless automated server deployment
3. Disaster is inevitable
4. Application Programming Interface
5. Support different levels of data availability
6. Support different scalability principles
REFERENCE
http://ronaldbradford.com/blog/successful-mysql-scalability-presentation-2010-09-17/

73. Common MySQL Scalability Mistakes - 2010.10
1

74. Common MySQL Scalability Mistakes - 2010.10
My website is slow?
PROBLEM

75. Common MySQL Scalability Mistakes - 2010.10
Seek professional advice.
Hire for example Ronald Bradford.
20+ years of system architecture, database design and
performance tuning.
Employment as Consultant for Oracle Corporation (96-99)
Employment as Senior Consultant for MySQL Inc (06-08)
SOLUTION

76. Common MySQL Scalability Mistakes - 2010.10
R

77. Common MySQL Scalability Mistakes - 2010.10
Monitoring. Before, during and after NOW.
You may not be able to predict the future
but you can preempt the future.
Choose the best product and features for
you needs.
The best SQL statement is the one
you never have to execute.
RECAP

78. Common MySQL Scalability Mistakes - 2010.10
3 levels of real time data access.
Read/Write, Read and no access
3 aspects of scalability.
Read, Write and Caching
Operate below 90% capacity. That 10% is
your insurance.
RECAP

79. Common MySQL Scalability Mistakes - 2010.10
http://ronaldbradford.com
me@ronaldbradford.com
CONTACT