Mysqlnd query cache plugin statistics and tuning

2,940 views

Published on

Query caching boosts the performance of PHP MySQL applications. Caching can be done on the database server or at the web clients. The mysqlnd plugin adds query caching to all PHP MySQL extension! It is fast, transparent and supports Memcache, APC, SQLite. Learn how to use its rich sets of performance statistics and how to identify cache candidates.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,940
On SlideShare
0
From Embeds
0
Number of Embeds
151
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mysqlnd query cache plugin statistics and tuning

  1. 1. MySQL native driver for PHP: Dig deeper with QC statistics
  2. 2. mysqlnd_qc: Dig deeper with statistics Ulf Wendel, Andrey Hristov MySQL Connectors Team Sun Microsystems
  3. 3. Table of Contents <ul><li>Overview </li><ul><li>Sources of statistics </li></ul><li>Core statistics </li><ul><li>Configuration and Access
  4. 4. Listing </li></ul><li>Advanced </li><ul><li>Query statistics
  5. 5. Storage handler statistic </li></ul></ul>
  6. 6. Tuning screws (statistics) all over mysqlnd, mysqlnd_qc core, mysqlnd_qc storage handler <?php /* Any PHP MySQL application */ ?> ext/*mysql* ext/*mysql* mysqlnd mysqlnd Query Cache Plugin (mysqlnd_qc core) Storage handler (mysqlnd_qc)
  7. 7. The 150 statistics of mysqlnd <ul><li>Collected by: mysqlnd
  8. 8. Scope: process, connection </li><ul><li>Process: mysqli_get_client_stats(), phpinfo()
  9. 9. Connection: mysqli_get_connection_stats()
  10. 10. Aggregated values from all MySQL PHP APIs </li></ul><li>Contents: wide range </li><ul><li>Network related
  11. 11. Result set related
  12. 12. Connection related
  13. 13. Client-Server Protocol related </li></ul></ul>
  14. 14. 20+ statistics of the query cache core <ul><li>Collected by: query cache plugin
  15. 15. Scope: process </li><ul><li>mysqlnd_qc_get_core_stats()
  16. 16. Aggregated values from all MySQL PHP APIs </li></ul><li>Contents: wide range </li><ul><li>Cache usage related
  17. 17. Network related
  18. 18. Timings </li></ul></ul>
  19. 19. Query back traces and statistics <ul><li>Collected by: query cache plugin
  20. 20. Scope: process </li><ul><li>mysqlnd_qc_get_query_trace_log()
  21. 21. mysqlnd_qc_get_normalized_query_trace_log() </li></ul><li>Contents: </li><ul><li>Source code back trace showing query origin (mysqlnd_qc_get_query_trace_log())
  22. 22. Cache related e.g. hit, miss, added, occurences...
  23. 23. Timings </li></ul></ul>
  24. 24. Query cache storage handler stats <ul><li>Collected by: query cache storage handler
  25. 25. Scope: cache entry (= request or process) </li><ul><li>Depends on storage handler
  26. 26. mysqlnd_qc_get_cache_info()
  27. 27. Aggregated values from all MySQL PHP APIs </li></ul><li>Contents: none or assorted </li><ul><li>Depends on storage handler
  28. 28. APC: timings, hit ratio, result set size
  29. 29. Default: like APC plus result set meta data </li></ul></ul>
  30. 30. Runtime configuration
  31. 31. Table of Contents <ul><li>Overview </li><ul><li>Sources of statistics </li></ul><li>Core statistics </li><ul><li>Configuration and Access
  32. 32. Listing </li></ul><li>Advanced </li><ul><li>Query statistics
  33. 33. Storage handler statistic </li></ul></ul>
  34. 34. Core statistics: basic monitoring <ul><li>Measure cache efficiency </li><ul><li>Determine if query caching gives a speed-up
  35. 35. Check cache hit/miss ratio
  36. 36. Check how many queries get cached
  37. 37. Check collisions/slam defense efficiency </li></ul><li>Available with all handlers
  38. 38. Aggregated data </li><ul><li>You cannot measure individual queries </li></ul></ul>
  39. 39. Accessing core statistics Change runtime configuration to collect statistics! array mysqlnd_qc_get_core_statistics() Returns a list of statistics collected by the core of the query cache plugin, if the PHP configuration setting mysqlnd_qc_collect_statistics is set to 1. The statistics are provided by the core and therefore available with all build-in storage handlers and when using user-defined storage handlers.
  40. 40. Runtime configuration <ul><li>mysqlnd_qc.collect_statistics </li><ul><li>PHP_INI_ALL, default “0”, boolean
  41. 41. enable/disable all core statistics </li></ul><li>mysqlnd_qc.time_statistics </li><ul><li>PHP_INI_ALL, default “1”, boolean
  42. 42. use of gettimeofday() system call
  43. 43. enable/disable: *run_time*, *store_time* </li></ul><li>mysqlnd_qc.slam_defense </li><ul><li>PHP_INI_SYSTEM, default “0”, boolean
  44. 44. enable/disable: slam_stale_* </li></ul></ul>
  45. 45. Scope of core statistics <ul><li>PHP process </li><ul><li>Data from one or multiple requests
  46. 46. Aggregated for all PHP MySQL extensions: ext/mysql, ext/mysqli, PDO_MySQL
  47. 47. Aggregated for all cache entries
  48. 48. Watch out when interpreting the figures – different processes may show different figures! </li></ul></ul>
  49. 49. Lifting scope/life time limitations <ul><li>Manually collect statistics at script end </li><ul><li>php configuration directive: auto_append_file
  50. 50. register_shutdown_function() </li></ul><li>Choose persistent storage medium </li><ul><li>MySQL – fire and forget with ASYNC
  51. 51. Files
  52. 52. Network
  53. 53. Memcache, APC, ... </li></ul></ul>
  54. 54. Manual aggregation using MySQL Fire and forget - works well for all kinds of QC stats! array mysqlnd_qc_get_core_statistics() Returns a list of statistics collected by the core of the query cache plugin, if the PHP configuration setting mysqlnd_qc_collect_statistics is set to 1. The statistics are provided by the core and therefore available with all build-in storage handlers and when using user-defined storage handlers.
  55. 55. Cache access statistics (I) <ul><li>cache_hit </li><ul><li>Statement is considered cacheable...
  56. 56. … and cached data has been reused
  57. 57. … and we had a miss but the statement got cached by someone else while we process it </li></ul></ul>
  58. 58. Cache access statistics (II) <ul><li>cache_miss </li><ul><li>Statement is considered cacheable...
  59. 59. .. and has been added to cache right now
  60. 60. ...but cache_no_table = 1 prevented caching
  61. 61. ...but unbuffered result set used
  62. 62. ...but buffered result set was empty </li></ul><li>cache_put </li><ul><li>Statement is considered cacheable and has been added to cache </li></ul></ul>
  63. 63. Cache access statistics (III) <ul><li>cache_put </li><ul><li>Statement is considered cacheable and has been added to cache
  64. 64. Take care when calculating derived statistics. Handler with a storage life time beyond process scope may report cache_put = 0 together with cache_hit > 0, if another process has filled the cache. You may want to use “num_entries” from mysqlnd_cache_info(), given the handler supports it (Default, APC). </li></ul></ul>
  65. 65. Query related statistics (I) <ul><li>query_should_cache </li><ul><li>Statement is considered cacheable based on query string analysis
  66. 66. It is yet unknown if it will end up in the cache! </li></ul><li>query_should_not_cache </li><ul><li>Statement is considered not cacheable based on query string analysis </li></ul><li>Derived: # of queries inspected by QC </li><ul><li>query_should_cache + query_should_not_cache </li></ul></ul>
  67. 67. Query related statistics (II) <ul><li>query_not_cached </li><ul><li>Statement is considered not cacheable...
  68. 68. … or storage handler has returned no hash key </li></ul><li>Derived: # of empty hash keys </li><ul><li>query_not_cached – query_should_cache
  69. 69. A value >0 indicates storage issues (errors, cache full, … - details are unknown) </li></ul></ul>
  70. 70. Query related statistics (III) <ul><li>query_could_cache </li><ul><li>Statement is considered cacheable...
  71. 71. … and statement executed without errors
  72. 72. … and meta data shows at least one column
  73. 73. It may or may not be in the cache already
  74. 74. It may or may not be added to cache later on </li></ul></ul>
  75. 75. Query related statistics (IV) <ul><li>query_found_in_cache </li><ul><li>Statement is considered cacheable....
  76. 76. … and we have found it in the cache ...
  77. 77. … but we have not yet replayed cached data
  78. 78. No cache hit yet: client might not fetch data
  79. 79. No cache hit yet: cached data may be faulty </li></ul></ul>
  80. 80. Query related statistics (V) <ul><li>query_uncached_other </li><ul><li>Statement is considered cacheable ...
  81. 81. … it may or may not be in the cache already
  82. 82. … but either replaying cached data failed, no result set is available or some other error happened </li></ul></ul>
  83. 83. Query related statistics (VI) <ul><li>query_uncached_no_table </li><ul><li>Query would have been cached if cache_no_table = 1 </li></ul></ul>
  84. 84. Query related statistics (VII) <ul><li>query_uncached_use_result </li><ul><li>Query would have been cached if a buffered result set had been used.
  85. 85. Incremented together with cache_miss </li></ul><li>Derived: # of cacheable queries if no issues: </li><ul><li>query_uncached_other + query_uncached_no_table + query_uncached_use_result </li></ul></ul>
  86. 86. Run times (I) <ul><li>query_aggr_run_time_cache_hit </li><ul><li>Aggregated run times (ms) of queries which are considered as a cache hit </li></ul><li>Derived: average run time of a cached query </li><ul><li>query_aggr_run_time_cache_hit / cache_hit </li></ul></ul>
  87. 87. Run times (II) <ul><li>query_aggr_run_time_cache_put </li><ul><li>Aggregated run times (ms) of queries which are considered as a cache put </li></ul><li>Derived: average run time improvement of cached queries (Default handler only!) </li><ul><li>(query_aggr_run_time_cache_put / cache_put) / (query_aggr_run_time_cache_hit / cache_hit) </li></ul></ul>
  88. 88. Run times (III) <ul><li>query_aggr_run_time_total </li><ul><li>Aggregated run time (ms) of all queries run by the query cache </li></ul><li>Derived: aggr. run time of uncached queries </li><ul><li>query_aggr_run_time_total – query_aggr_run_time_cache_hit </li></ul><li>Derived: avg. run time of uncached queries </li><ul><li>(query_aggr_run_time_total – query_aggr_run_time_cache_hit) / cache_miss </li></ul></ul>
  89. 89. Store times (I) <ul><li>query_aggr_store_time_cache_hit </li><ul><li>Aggregated store times (ms) of queries which are considered as a cache hit </li></ul><li>Derived: average store time for a cached query </li><ul><li>query_aggr_store_time_cache_hit / cache_hit </li></ul></ul>
  90. 90. Store times (II) <ul><li>query_aggr_store_time_cache_put </li><ul><li>Aggregated store times (ms) of queries which are considered as a cache put </li></ul><li>Derived: average store time improvement of a cached query (Default handler only!) </li><ul><li>(query_aggr_store_time_cache_put / cache_put) / (query_aggr_store_time_cache_hit / cache_hit) </li></ul></ul>
  91. 91. Store times (III) <ul><li>query_aggr_store_time_total </li><ul><li>Aggregated store time (ms) of all queries run by the query cache </li></ul><li>Derived: aggr. store time of uncached queries </li><ul><li>query_aggr_store_time_total – query_aggr_store_time_cache_hit </li></ul><li>Derived: avg. store time of uncached queries </li><ul><li>(query_aggr_store_time_total – query_aggr_store_time_cache_hit) / cache_miss </li></ul></ul>
  92. 92. Network traffic (I) <ul><li>receive_bytes_recorded </li><ul><li>Recorded traffic from MySQL to PHP
  93. 93. The data may or may not be added to the cache at some point (depends on hit or miss) </li></ul><li>receive_bytes_replayed </li><ul><li>Replayed recorded traffic from MySQL to PHP
  94. 94. This is the total amount of incoming traffic saved by using the query cache plugin </li></ul></ul>
  95. 95. Network traffic (II) <ul><li>send_bytes_recorded </li><ul><li>Recorded traffic from PHP to MySQL
  96. 96. The data may or may not be added to the cache at some point (depends on hit or miss) </li></ul><li>send_bytes_replayed </li><ul><li>Replayed recorded traffic from PHP to MySQL
  97. 97. This is the total amount of outgoing traffic saved by using the query cache plugin </li></ul></ul>
  98. 98. Network traffic (III) <ul><li>Derived: total network traffic savings in MB </li><ul><li>(receive_bytes_replayed + send_bytes_replayed) / 1024 / 1024 </li></ul></ul>
  99. 99. Slam defense <ul><li>slam_stale_refresh </li><ul><li>Number of cache misses which triggered serving stale data until the client causing the cache miss has refreshed the cache entry </li></ul><li>slam_stale_hit </li><ul><li>Number of cache hits while a stale cache entry gets refreshed </li></ul></ul>
  100. 100. Table of Contents <ul><li>Overview </li><ul><li>Sources of statistics </li></ul><li>Core statistics </li><ul><li>Configuration and Access
  101. 101. Listing </li></ul><li>Advanced </li><ul><li>Query statistics
  102. 102. Storage handler statistic </li></ul></ul>
  103. 103. Query back trace php.ini setting: mysqlnd_qc.collect_query_trace = 1 array mysqlnd_qc_get_query_trace_log() Returns a list query back traces for every query that has been inspected by the query cache regardless if the query ended up being cached or not. The trace tells you where a query has been issues (see also debug_backtrace()). Together with the back trace you get run and store times and information on if the query has been cached.
  104. 104. Runtime configuration <ul><li>mysqlnd_qc.collect_query_trace </li><ul><li>PHP_INI_SYSTEM, default “0”, boolean
  105. 105. enable/disable query back trace </li></ul><li>mysqlnd_qc.query_trace_bt_depth </li><ul><li>PHP_INI_SYSTEM, default “3”, integer
  106. 106. limits code back trace depth </li></ul></ul>
  107. 107. Scope of the query back trace <ul><li>PHP Process </li><ul><li>Data from one or multiple requests
  108. 108. Data from all PHP MySQL APIs: ext/mysql, ext/mysqli, PDO_MySQL </li></ul></ul>
  109. 109. Query back trace array elements (I) <ul><li>query </li><ul><li>string: SQL statement
  110. 110. Trace contains all inspected queries
  111. 111. Trace contains cached and uncached queries
  112. 112. Same SQL statement can appear multiple times! </li></ul><li>origin </li><ul><li>string: code back trace
  113. 113. Back trace to the origin of the query
  114. 114. Back trace begins with, e.g. mysqli_query()
  115. 115. ini setting: mysqlnd_qc.query_trace_bt_depth </li></ul></ul>
  116. 116. Query back trace array elements (II) <ul><li>run_time </li><ul><li>int: query run time
  117. 117. ini setting: mysqlnd_qc.time_statistics
  118. 118. (enabled by default) </li></ul><li>store_time </li><ul><li>int: time required to fetch and store result set
  119. 119. ini setting: mysqlnd_qc.time_statistics
  120. 120. (enabled by default) </li></ul></ul>
  121. 121. Query back trace array elements (III) <ul><li>eligible_for_caching </li><ul><li>boolean
  122. 122. Set to true if the query qualifies for caching
  123. 123. according to the query analysis
  124. 124. The query might or might not be cached </li></ul><li>no_table </li><ul><li>boolean
  125. 125. Set to true if any column from the result set
  126. 126. has no table name, e.g. SELECT NOW() </li></ul></ul>
  127. 127. Query back trace array elements (IV) <ul><li>was_added </li><ul><li>boolean
  128. 128. Set to true if the query has been added to the
  129. 129. cache. </li></ul><li>was_already_in_cahce </li><ul><li>boolean
  130. 130. Set to true in case of a cache hit. </li></ul></ul>
  131. 131. Normalized query back trace mysqlnd_qc.collect_normalized_query_trace = 1 array mysqlnd_qc_get_normalized_query_trace_log() Similar to mysqlnd_qc_get_query_trace_log() but with SQL statements normalized and aggregated by the normalized query string. Normalization refers to replacing actual parameters, for example in “WHERE a > 1”, with questionmarks like “ WHERE a > ?”. “WHERE a > ?” will match any value for “?”, for example “1”, “2”, “'abc'” but not other identifiers.
  132. 132. Runtime configuration <ul><li>mysqlnd_qc.collect_normalized_query_trace </li><ul><li>PHP_INI_SYSTEM, default “0”, boolean
  133. 133. enable/disable normalized query trace </li></ul></ul>
  134. 134. Scope of the normalized query log <ul><li>PHP Process </li><ul><li>Data from one or multiple requests
  135. 135. Data from all PHP MySQL APIs: ext/mysql, ext/mysqli, PDO_MySQL
  136. 136. Aggregated by normalized query </li></ul></ul>
  137. 137. Normalized query trace elements (I) <ul><li>query </li><ul><li>string: normalized SQL statement
  138. 138. Trace contains all inspected queries
  139. 139. Aggregated by normalized SQL statement
  140. 140. Normalized: parameter replaced by placeholder
  141. 141. SELECT 1, SELECT 'a' -> SELECT ? </li></ul><li>occurences </li><ul><li>int
  142. 142. Number of SQL statements summarized
  143. 143. in this record set </li></ul></ul>
  144. 144. Normalized query trace elements (II) <ul><li>eligible_for_caching </li><ul><li>boolean
  145. 145. Set to true if the query qualifies for caching
  146. 146. according to the query analysis
  147. 147. The query might or might not be cached </li></ul></ul>
  148. 148. Normalized query trace elements (III) <ul><li>min_run_time
  149. 149. avg_run_time
  150. 150. max_run_time </li><ul><li>int
  151. 151. Minimum, average and maximum run time of all queries aggregated in this record set
  152. 152. ini setting: mysqlnd_qc.time_statistics
  153. 153. (enabled by default) </li></ul></ul>
  154. 154. Normalized query trace elements (IV) <ul><li>min_store_time
  155. 155. avg_store_time
  156. 156. max_store_time </li><ul><li>int
  157. 157. Minimum, average and maximum store time of all queries aggregated in this record set
  158. 158. ini setting: mysqlnd_qc.time_statistics
  159. 159. (enabled by default) </li></ul></ul>
  160. 160. Storage handler statistics <ul><li>Optional – handler duty!
  161. 161. Default handler </li><ul><li>Result set size
  162. 162. Access statistics
  163. 163. Meta data </li></ul><li>APC </li><ul><li>Access statistics </li></ul><li>Memcache </li><ul><li>no handler statistics available </li></ul></ul>
  164. 164. Accessing storage handler statistics User storage handler need to provide their own API! array mysqlnd_qc_get_cache_info() Returns information on the active handler and, if available, storage handler statistics. Storage handler statistics are provided under the “data” key of the hash. Different storage handler return different statistics, if any. User defined storage handler cannot return their statistics through this function. They need to provide their own API for accessing storage handler level statistics.
  165. 165. “Hot” tables and guessing speed-up <ul><li>Set myslqnd_qc.cache_by_default = 1 </li><ul><li>Accept and ignore “false” results </li></ul><li>Set mysqlnd_qc.collect_statistics = 1
  166. 166. Set mysqlnd_qc.time_statistics = 1
  167. 167. Analyze meta data of cache entries </li><ul><li>mysqlnd_qc_get_cache_info() -> “data” hash
  168. 168. Default and APC handler provide meta data
  169. 169. Search and account table names from meta data
  170. 170. Analyze recorded run and store times </li></ul></ul>
  171. 171. “Hot” tables and guessing speed-up <ul><li>Set myslqnd_qc.cache_by_default = 0
  172. 172. Set mysqlnd_qc.collect_statistics = 1
  173. 173. Set mysqlnd_qc.time_statistics = 1
  174. 174. Read user handler specific presentation! :-) </li></ul>
  175. 175. Runtime configuration <ul><li>mysqlnd_qc.collect_statistics </li><ul><li>no impact on storage handler statistics
  176. 176. background: storage format with/wo statistics </li></ul><li>mysqlnd_qc.time_statistics </li><ul><li>use of gettimeofday() system call
  177. 177. enable/disable: all timings
  178. 178. background: handler make use of core function </li></ul></ul>
  179. 179. Scope of handler statistics <ul><li>Scope of handler storage </li><ul><li>Default: Process (one or multiple requests)
  180. 180. APC: machine (multiple processes)
  181. 181. Data from all PHP MySQL APIs: ext/mysql, ext/mysqli, PDO_MySQL </li></ul></ul>
  182. 182. Default handler: result set size <ul><li>rows </li><ul><li>Number of rows </li></ul><li>stored_size </li><ul><li>Size in bytes of the raw network traffic result set
  183. 183. added to the cache
  184. 184. See http://blog.ulf-wendel.de/?p=198 for
  185. 185. high-level discussion of the raw network format </li></ul></ul>
  186. 186. Default handler: access and timings <ul><li>cache_hits </li><ul><li>Number of cache hits </li></ul><li>run_time </li><ul><li>Run time of the uncached query
  187. 187. Set once when adding the query to the cache </li></ul><li>store_time </li><ul><li>Store time of the uncached query
  188. 188. Set once when adding the query to the cache </li></ul></ul>
  189. 189. Default handler: timings <ul><li>min_run_time
  190. 190. avg_run_time
  191. 191. max_run_time </li><ul><li>Minimum, average and maximum query run
  192. 192. time measured during a cache hit </li></ul></ul>
  193. 193. Default handler: timings <ul><li>min_store_time
  194. 194. avg_store_time
  195. 195. max_store_time </li><ul><li>Minimum, average and maximum store
  196. 196. time measured during a cache hit </li></ul></ul>
  197. 197. APC handler: access, size, timings <ul><li>cache_hits </li><ul><li>Number of cache hits </li></ul><li>rows </li><ul><li>Number of rows </li></ul><li>run_time </li><ul><li>Run time of the uncached query
  198. 198. Set once when adding the query to the cache </li></ul><li>store_time </li><ul><li>Store time of the uncached query
  199. 199. Set once when adding the query to the cache </li></ul></ul>
  200. 200. APC handler: timings <ul><li>min_run_time
  201. 201. avg_run_time
  202. 202. max_run_time
  203. 203. min_store_time
  204. 204. avg_store_time
  205. 205. max_store_time </li><ul><li>See Default handler statistics
  206. 206. NULL if no cache hit </li></ul></ul>
  207. 207. Tooling: Cache Monitor web script <ul><li>web/mysqlnd_qc_monitor.php
  208. 208. Demonstrates usage of statistics
  209. 209. Not a production level tool
  210. 210. Basics only </li><ul><li>Configuration
  211. 211. Per-process statistics
  212. 212. Cache contents (storage handler statistics)
  213. 213. Per-machine statistics using slow auto_append example script </li></ul></ul>
  214. 214. Tooling: per machine statistics <ul><li>web/auto_append_persist_qc_stats.php
  215. 215. auto_append to persist per-process data
  216. 216. Demonstrates aggregation of core statistics
  217. 217. Not a production level tool </li><ul><li>Slow
  218. 218. MySQL storage only </li></ul></ul>
  219. 219. The End Feedback: ulf.wendel@sun.com The End Feedback: [email_address] , [email_address]

×