MySQL Tricks
Tips and Tricks from the Phorum Team
Brian Moon, Lead Developer - Phorum
   2008 MySQL Conference & Expo
    ...
What is Phorum?
• Open Source message board system
• In use on MySQL.com and Innodb.com
• Developers obsessive about perfo...
Quick Tips
SQL_COUNT_FOUND_ROWS
Saves the total row count a query matched
NOT including any LIMIT clause. YMMV

Usage:

SELECT SQL_CO...
Extra Layers
Don’t use PHP based abstraction layers
(PEAR::DB, ADOdb, etc.)

Remember this is a talk about speed. Calling
...
Never Use Subqueries

• MySQL Subqueries are still a work in
  progress
• The inner query is run once for every row
  in t...
Be Careful With Joins

• Only join on indexed columns
• Do not use expression in your joins
  inner join t2 on t1.field1+1...
Buffered vs. Unbuffered
What Are
     Buffered Queries?
• Data returned from the database is held in
  memory for use by other MySQL functions.
• ...
What Are
  Unbuffered Queries?
• Data is only sent from the server when it is
  requested.
• Some MySQL functions will not...
When To Use
    Unbuffered Queries
Anywhere you have code like this:

while($row = mysql_fetch_assoc($res)){
    $my_array...
When NOT To Use
  Unbuffered Queries
• Using mysql_num_rows() and other
  functions that use the buffered results
• Read f...
How to use
  Unbuffered Queries
• Older MySQL Extension offers the
  mysql_unbuffered_query function

• For MySQL Improved...
You Can Pick Your Keys
Forcing your keys

• The MySQL optimizer has rules
• Sometimes, you need to break those rules
• You have to know your data...
select
     message_id, subject
from
     phorum_messages
where
     forum_id in (12,14,16,17,18,
     19,20,21,22,23,24,2...
********* 1. row *********
           id: 1
  select_type: SIMPLE
        table: phorum_messages
         type: ref
possib...
select
     message_id, subject
from
     phorum_messages use key (primary)
where
     forum_id in (12,14,16,17,18,
     1...
********* 1. row *********
           id: 1
  select_type: SIMPLE
        table: phorum_messages
         type: index
poss...
I Know My Data
• The majority of the rows in the table match
  the status in the query.
• Also, the majority of the rows w...
Test Your Queries

• If EXPLAIN looks wrong, try using FORCE
  INDEX with another key.
• Actually run the query. Sometimes...
Full-Text > LIKE
Full-Text Search
• Pros
 • It is internal to MySQL
 • It is better than complicated LIKE queries
• Cons
 • Tricky with oth...
Working With Full-Text
• Understand that MySQL will use only the
  FullText index for a query FullText queries
• Mixing ot...
Creating The
        Temporary Table
CREATE TEMPORARY TABLE
foo ( KEY (message_id) ) ENGINE=HEAP
SELECT message_id
FROM   ...
Joining The
        Temporary Table
SELECT SQL_CALC_FOUND_ROWS *
FROM   phorum_messages
INNER JOIN foo USING (message_id)
...
Adding other searches

• Search by author, subject, etc.
• Create more temporary tables
• Merge them together in yet anoth...
Alternatives
• Use old fashioned LIKE clauses (Phorum
  has this for poor users with bad hosts)
• Use an external system
 ...
PHP/MySQL Secret
PHP/MySQL Secret
• The current PHP mysql extensions use
  libmysqlclient
• This code is not “part of PHP”
• Therefore, it ...
mysqlnd, A New Hope
• mysqlnd “uses many of the proven and
  stable PHP internal functions”
• That means it respects memor...
# ./php-with-lib bigselect.php
All OK!


# ./php-with-nd bigselect.php

Fatal error: Allowed memory size of
2097152 bytes ...
MySQL Tricks
Tips and Tricks from the Phorum Team
Brian Moon, Lead Developer - Phorum
   2008 MySQL Conference & Expo
    ...
Upcoming SlideShare
Loading in...5
×

Phorum MySQL tricks

48,014

Published on

Published in: Technology
1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total Views
48,014
On Slideshare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
76
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

Phorum MySQL tricks

  1. 1. MySQL Tricks Tips and Tricks from the Phorum Team Brian Moon, Lead Developer - Phorum 2008 MySQL Conference & Expo http://www.phorum.org/
  2. 2. What is Phorum? • Open Source message board system • In use on MySQL.com and Innodb.com • Developers obsessive about performance Come by out booth in the DotOrg Pavilion! Just look for the sleepy dog. DOH!
  3. 3. Quick Tips
  4. 4. SQL_COUNT_FOUND_ROWS Saves the total row count a query matched NOT including any LIMIT clause. YMMV Usage: SELECT SQL_COUNT_FOUND_ROWS from really_big_table LIMIT 0, 20 select found_rows()
  5. 5. Extra Layers Don’t use PHP based abstraction layers (PEAR::DB, ADOdb, etc.) Remember this is a talk about speed. Calling an extra function before the real MySQL function is called will always be slower. PDO is a C extension and is not slow.
  6. 6. Never Use Subqueries • MySQL Subqueries are still a work in progress • The inner query is run once for every row in the outer query • Never say never? There is progress in this area. Stay tuned to the MySQL docs.
  7. 7. Be Careful With Joins • Only join on indexed columns • Do not use expression in your joins inner join t2 on t1.field1+10=t2.field2 • Very often, two queries are faster even with the programming overhead to merge them
  8. 8. Buffered vs. Unbuffered
  9. 9. What Are Buffered Queries? • Data returned from the database is held in memory for use by other MySQL functions. • Required for some MySQL functions to work. e.g. mysql_num_rows, mysql_seek • This is the default behavior and most commonly used.
  10. 10. What Are Unbuffered Queries? • Data is only sent from the server when it is requested. • Some MySQL functions will not work. e.g. mysql_num_rows, mysql_seek • The results must be fully read and the result freed before another query can run.
  11. 11. When To Use Unbuffered Queries Anywhere you have code like this: while($row = mysql_fetch_assoc($res)){ $my_array[] = $row; } Which for me, is all over the place. They are especially useful for large datasets.
  12. 12. When NOT To Use Unbuffered Queries • Using mysql_num_rows() and other functions that use the buffered results • Read from multiple result sets at one time • Complex operations are done while reading data into user land
  13. 13. How to use Unbuffered Queries • Older MySQL Extension offers the mysql_unbuffered_query function • For MySQL Improved, pass MYSQLI_USE_RESULT to the third parameter of mysqli_query. • For PDO, use PDOStatement::fetchAll()
  14. 14. You Can Pick Your Keys
  15. 15. Forcing your keys • The MySQL optimizer has rules • Sometimes, you need to break those rules • You have to know your data • You have to test a lot
  16. 16. select message_id, subject from phorum_messages where forum_id in (12,14,16,17,18, 19,20,21,22,23,24,25,26,28, 57,58,59,61,62,63,64,65,66,67) and status=2 and moved=0 ORDER BY message_id DESC limit 50; 50 rows in set (1.20 sec)
  17. 17. ********* 1. row ********* id: 1 select_type: SIMPLE table: phorum_messages type: ref possible_keys: status_forum,foru..... key: status_forum key_len: 1 ref: const rows: 33985 Extra: Using where; Using filesort
  18. 18. select message_id, subject from phorum_messages use key (primary) where forum_id in (12,14,16,17,18, 19,20,21,22,23,24,25,26,28, 57,58,59,61,62,63,64,65,66,67) and status=2 and moved=0 ORDER BY message_id DESC limit 50; 50 rows in set (0.00 sec)
  19. 19. ********* 1. row ********* id: 1 select_type: SIMPLE table: phorum_messages type: index possible_keys: NULL key: PRIMARY key_len: 4 ref: NULL rows: 67970 Extra: Using where
  20. 20. I Know My Data • The majority of the rows in the table match the status in the query. • Also, the majority of the rows will match the large IN clause. • So, we can scan on a fast key and most of the rows will match the other things in the where clause.
  21. 21. Test Your Queries • If EXPLAIN looks wrong, try using FORCE INDEX with another key. • Actually run the query. Sometimes EXPLAIN is confusing. • Test how the query will scale when it is being run 100 times per second.
  22. 22. Full-Text > LIKE
  23. 23. Full-Text Search • Pros • It is internal to MySQL • It is better than complicated LIKE queries • Cons • Tricky with other indexes • Not all hosts enable it • Only works with MyISAM
  24. 24. Working With Full-Text • Understand that MySQL will use only the FullText index for a query FullText queries • Mixing other things into the where clause or ordering by ANY fields slows it down • Returning loads of data to the application makes things slow and uses memory • Phorum uses temporary heap tables to get around these issues
  25. 25. Creating The Temporary Table CREATE TEMPORARY TABLE foo ( KEY (message_id) ) ENGINE=HEAP SELECT message_id FROM phorum_search WHERE MATCH (search_text) AGAINST ('+some +search' IN BOOLEAN MODE);
  26. 26. Joining The Temporary Table SELECT SQL_CALC_FOUND_ROWS * FROM phorum_messages INNER JOIN foo USING (message_id) WHERE status=2 and forum_id in (12,14,16,17,18, 19,20,21,22,23,24,25,26,28, 57,58,59,61,62,63,64,65,66,67) and datestamp > 1200390957 ORDER BY datestamp DESC LIMIT 0, 30;
  27. 27. Adding other searches • Search by author, subject, etc. • Create more temporary tables • Merge them together in yet another table • Yes, this really is fast (unless the MySQL server is badly configured)
  28. 28. Alternatives • Use old fashioned LIKE clauses (Phorum has this for poor users with bad hosts) • Use an external system • Sphinx Search - They are in the DotOrg Pavilion as well. • Lucene - Java based full text engine. There is a Zend class for this as well.
  29. 29. PHP/MySQL Secret
  30. 30. PHP/MySQL Secret • The current PHP mysql extensions use libmysqlclient • This code is not “part of PHP” • Therefore, it does not have to follow the PHP rules • Things have to be copied over and over
  31. 31. mysqlnd, A New Hope • mysqlnd “uses many of the proven and stable PHP internal functions” • That means it respects memory limits • It also means it can move data from MySQL directly into PHP data structures • Available starting with PHP 5.3
  32. 32. # ./php-with-lib bigselect.php All OK! # ./php-with-nd bigselect.php Fatal error: Allowed memory size of 2097152 bytes exhausted (tried to allocate 800000 bytes) in /home/brian/ bigselect.php on line 14 The first script used the memory. You just have no control over it!
  33. 33. MySQL Tricks Tips and Tricks from the Phorum Team Brian Moon, Lead Developer - Phorum 2008 MySQL Conference & Expo http://www.phorum.org/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×