Your SlideShare is downloading. ×
Mysqlnd query cache plugin statistics and tuning
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Mysqlnd query cache plugin statistics and tuning

2,330
views

Published on

Query caching boosts the performance of PHP MySQL applications. Caching can be done on the database server or at the web clients. The mysqlnd plugin adds query caching to all PHP MySQL extension! It …

Query caching boosts the performance of PHP MySQL applications. Caching can be done on the database server or at the web clients. The mysqlnd plugin adds query caching to all PHP MySQL extension! It is fast, transparent and supports Memcache, APC, SQLite. Learn how to use its rich sets of performance statistics and how to identify cache candidates.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,330
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. MySQL native driver for PHP: Dig deeper with QC statistics
  • 2. mysqlnd_qc: Dig deeper with statistics Ulf Wendel, Andrey Hristov MySQL Connectors Team Sun Microsystems
  • 3. Table of Contents
    • Overview
      • Sources of statistics
    • Core statistics
      • Configuration and Access
      • 4. Listing
    • Advanced
      • Query statistics
      • 5. Storage handler statistic
  • 6. Tuning screws (statistics) all over mysqlnd, mysqlnd_qc core, mysqlnd_qc storage handler <?php /* Any PHP MySQL application */ ?> ext/*mysql* ext/*mysql* mysqlnd mysqlnd Query Cache Plugin (mysqlnd_qc core) Storage handler (mysqlnd_qc)
  • 7. The 150 statistics of mysqlnd
    • Collected by: mysqlnd
    • 8. Scope: process, connection
      • Process: mysqli_get_client_stats(), phpinfo()
      • 9. Connection: mysqli_get_connection_stats()
      • 10. Aggregated values from all MySQL PHP APIs
    • Contents: wide range
      • Network related
      • 11. Result set related
      • 12. Connection related
      • 13. Client-Server Protocol related
  • 14. 20+ statistics of the query cache core
    • Collected by: query cache plugin
    • 15. Scope: process
      • mysqlnd_qc_get_core_stats()
      • 16. Aggregated values from all MySQL PHP APIs
    • Contents: wide range
      • Cache usage related
      • 17. Network related
      • 18. Timings
  • 19. Query back traces and statistics
    • Collected by: query cache plugin
    • 20. Scope: process
      • mysqlnd_qc_get_query_trace_log()
      • 21. mysqlnd_qc_get_normalized_query_trace_log()
    • Contents:
      • Source code back trace showing query origin (mysqlnd_qc_get_query_trace_log())
      • 22. Cache related e.g. hit, miss, added, occurences...
      • 23. Timings
  • 24. Query cache storage handler stats
    • Collected by: query cache storage handler
    • 25. Scope: cache entry (= request or process)
      • Depends on storage handler
      • 26. mysqlnd_qc_get_cache_info()
      • 27. Aggregated values from all MySQL PHP APIs
    • Contents: none or assorted
      • Depends on storage handler
      • 28. APC: timings, hit ratio, result set size
      • 29. Default: like APC plus result set meta data
  • 30. Runtime configuration
  • 31. Table of Contents
    • Overview
      • Sources of statistics
    • Core statistics
      • Configuration and Access
      • 32. Listing
    • Advanced
      • Query statistics
      • 33. Storage handler statistic
  • 34. Core statistics: basic monitoring
    • Measure cache efficiency
      • Determine if query caching gives a speed-up
      • 35. Check cache hit/miss ratio
      • 36. Check how many queries get cached
      • 37. Check collisions/slam defense efficiency
    • Available with all handlers
    • 38. Aggregated data
      • You cannot measure individual queries
  • 39. Accessing core statistics Change runtime configuration to collect statistics! array mysqlnd_qc_get_core_statistics() Returns a list of statistics collected by the core of the query cache plugin, if the PHP configuration setting mysqlnd_qc_collect_statistics is set to 1. The statistics are provided by the core and therefore available with all build-in storage handlers and when using user-defined storage handlers.
  • 40. Runtime configuration
    • mysqlnd_qc.collect_statistics
      • PHP_INI_ALL, default “0”, boolean
      • 41. enable/disable all core statistics
    • mysqlnd_qc.time_statistics
      • PHP_INI_ALL, default “1”, boolean
      • 42. use of gettimeofday() system call
      • 43. enable/disable: *run_time*, *store_time*
    • mysqlnd_qc.slam_defense
      • PHP_INI_SYSTEM, default “0”, boolean
      • 44. enable/disable: slam_stale_*
  • 45. Scope of core statistics
    • PHP process
      • Data from one or multiple requests
      • 46. Aggregated for all PHP MySQL extensions: ext/mysql, ext/mysqli, PDO_MySQL
      • 47. Aggregated for all cache entries
      • 48. Watch out when interpreting the figures – different processes may show different figures!
  • 49. Lifting scope/life time limitations
    • Manually collect statistics at script end
      • php configuration directive: auto_append_file
      • 50. register_shutdown_function()
    • Choose persistent storage medium
      • MySQL – fire and forget with ASYNC
      • 51. Files
      • 52. Network
      • 53. Memcache, APC, ...
  • 54. Manual aggregation using MySQL Fire and forget - works well for all kinds of QC stats! array mysqlnd_qc_get_core_statistics() Returns a list of statistics collected by the core of the query cache plugin, if the PHP configuration setting mysqlnd_qc_collect_statistics is set to 1. The statistics are provided by the core and therefore available with all build-in storage handlers and when using user-defined storage handlers.
  • 55. Cache access statistics (I)
    • cache_hit
      • Statement is considered cacheable...
      • 56. … and cached data has been reused
      • 57. … and we had a miss but the statement got cached by someone else while we process it
  • 58. Cache access statistics (II)
    • cache_miss
      • Statement is considered cacheable...
      • 59. .. and has been added to cache right now
      • 60. ...but cache_no_table = 1 prevented caching
      • 61. ...but unbuffered result set used
      • 62. ...but buffered result set was empty
    • cache_put
      • Statement is considered cacheable and has been added to cache
  • 63. Cache access statistics (III)
    • cache_put
      • Statement is considered cacheable and has been added to cache
      • 64. Take care when calculating derived statistics. Handler with a storage life time beyond process scope may report cache_put = 0 together with cache_hit > 0, if another process has filled the cache. You may want to use “num_entries” from mysqlnd_cache_info(), given the handler supports it (Default, APC).
  • 65. Query related statistics (I)
    • query_should_cache
      • Statement is considered cacheable based on query string analysis
      • 66. It is yet unknown if it will end up in the cache!
    • query_should_not_cache
      • Statement is considered not cacheable based on query string analysis
    • Derived: # of queries inspected by QC
      • query_should_cache + query_should_not_cache
  • 67. Query related statistics (II)
    • query_not_cached
      • Statement is considered not cacheable...
      • 68. … or storage handler has returned no hash key
    • Derived: # of empty hash keys
      • query_not_cached – query_should_cache
      • 69. A value >0 indicates storage issues (errors, cache full, … - details are unknown)
  • 70. Query related statistics (III)
    • query_could_cache
      • Statement is considered cacheable...
      • 71. … and statement executed without errors
      • 72. … and meta data shows at least one column
      • 73. It may or may not be in the cache already
      • 74. It may or may not be added to cache later on
  • 75. Query related statistics (IV)
    • query_found_in_cache
      • Statement is considered cacheable....
      • 76. … and we have found it in the cache ...
      • 77. … but we have not yet replayed cached data
      • 78. No cache hit yet: client might not fetch data
      • 79. No cache hit yet: cached data may be faulty
  • 80. Query related statistics (V)
    • query_uncached_other
      • Statement is considered cacheable ...
      • 81. … it may or may not be in the cache already
      • 82. … but either replaying cached data failed, no result set is available or some other error happened
  • 83. Query related statistics (VI)
    • query_uncached_no_table
      • Query would have been cached if cache_no_table = 1
  • 84. Query related statistics (VII)
    • query_uncached_use_result
      • Query would have been cached if a buffered result set had been used.
      • 85. Incremented together with cache_miss
    • Derived: # of cacheable queries if no issues:
      • query_uncached_other + query_uncached_no_table + query_uncached_use_result
  • 86. Run times (I)
    • query_aggr_run_time_cache_hit
      • Aggregated run times (ms) of queries which are considered as a cache hit
    • Derived: average run time of a cached query
      • query_aggr_run_time_cache_hit / cache_hit
  • 87. Run times (II)
    • query_aggr_run_time_cache_put
      • Aggregated run times (ms) of queries which are considered as a cache put
    • Derived: average run time improvement of cached queries (Default handler only!)
      • (query_aggr_run_time_cache_put / cache_put) / (query_aggr_run_time_cache_hit / cache_hit)
  • 88. Run times (III)
    • query_aggr_run_time_total
      • Aggregated run time (ms) of all queries run by the query cache
    • Derived: aggr. run time of uncached queries
      • query_aggr_run_time_total – query_aggr_run_time_cache_hit
    • Derived: avg. run time of uncached queries
      • (query_aggr_run_time_total – query_aggr_run_time_cache_hit) / cache_miss
  • 89. Store times (I)
    • query_aggr_store_time_cache_hit
      • Aggregated store times (ms) of queries which are considered as a cache hit
    • Derived: average store time for a cached query
      • query_aggr_store_time_cache_hit / cache_hit
  • 90. Store times (II)
    • query_aggr_store_time_cache_put
      • Aggregated store times (ms) of queries which are considered as a cache put
    • Derived: average store time improvement of a cached query (Default handler only!)
      • (query_aggr_store_time_cache_put / cache_put) / (query_aggr_store_time_cache_hit / cache_hit)
  • 91. Store times (III)
    • query_aggr_store_time_total
      • Aggregated store time (ms) of all queries run by the query cache
    • Derived: aggr. store time of uncached queries
      • query_aggr_store_time_total – query_aggr_store_time_cache_hit
    • Derived: avg. store time of uncached queries
      • (query_aggr_store_time_total – query_aggr_store_time_cache_hit) / cache_miss
  • 92. Network traffic (I)
    • receive_bytes_recorded
      • Recorded traffic from MySQL to PHP
      • 93. The data may or may not be added to the cache at some point (depends on hit or miss)
    • receive_bytes_replayed
      • Replayed recorded traffic from MySQL to PHP
      • 94. This is the total amount of incoming traffic saved by using the query cache plugin
  • 95. Network traffic (II)
    • send_bytes_recorded
      • Recorded traffic from PHP to MySQL
      • 96. The data may or may not be added to the cache at some point (depends on hit or miss)
    • send_bytes_replayed
      • Replayed recorded traffic from PHP to MySQL
      • 97. This is the total amount of outgoing traffic saved by using the query cache plugin
  • 98. Network traffic (III)
    • Derived: total network traffic savings in MB
      • (receive_bytes_replayed + send_bytes_replayed) / 1024 / 1024
  • 99. Slam defense
    • slam_stale_refresh
      • Number of cache misses which triggered serving stale data until the client causing the cache miss has refreshed the cache entry
    • slam_stale_hit
      • Number of cache hits while a stale cache entry gets refreshed
  • 100. Table of Contents
    • Overview
      • Sources of statistics
    • Core statistics
      • Configuration and Access
      • 101. Listing
    • Advanced
      • Query statistics
      • 102. Storage handler statistic
  • 103. Query back trace php.ini setting: mysqlnd_qc.collect_query_trace = 1 array mysqlnd_qc_get_query_trace_log() Returns a list query back traces for every query that has been inspected by the query cache regardless if the query ended up being cached or not. The trace tells you where a query has been issues (see also debug_backtrace()). Together with the back trace you get run and store times and information on if the query has been cached.
  • 104. Runtime configuration
    • mysqlnd_qc.collect_query_trace
      • PHP_INI_SYSTEM, default “0”, boolean
      • 105. enable/disable query back trace
    • mysqlnd_qc.query_trace_bt_depth
      • PHP_INI_SYSTEM, default “3”, integer
      • 106. limits code back trace depth
  • 107. Scope of the query back trace
    • PHP Process
      • Data from one or multiple requests
      • 108. Data from all PHP MySQL APIs: ext/mysql, ext/mysqli, PDO_MySQL
  • 109. Query back trace array elements (I)
    • query
      • string: SQL statement
      • 110. Trace contains all inspected queries
      • 111. Trace contains cached and uncached queries
      • 112. Same SQL statement can appear multiple times!
    • origin
      • string: code back trace
      • 113. Back trace to the origin of the query
      • 114. Back trace begins with, e.g. mysqli_query()
      • 115. ini setting: mysqlnd_qc.query_trace_bt_depth
  • 116. Query back trace array elements (II)
    • run_time
      • int: query run time
      • 117. ini setting: mysqlnd_qc.time_statistics
      • 118. (enabled by default)
    • store_time
      • int: time required to fetch and store result set
      • 119. ini setting: mysqlnd_qc.time_statistics
      • 120. (enabled by default)
  • 121. Query back trace array elements (III)
    • eligible_for_caching
      • boolean
      • 122. Set to true if the query qualifies for caching
      • 123. according to the query analysis
      • 124. The query might or might not be cached
    • no_table
      • boolean
      • 125. Set to true if any column from the result set
      • 126. has no table name, e.g. SELECT NOW()
  • 127. Query back trace array elements (IV)
    • was_added
      • boolean
      • 128. Set to true if the query has been added to the
      • 129. cache.
    • was_already_in_cahce
      • boolean
      • 130. Set to true in case of a cache hit.
  • 131. Normalized query back trace mysqlnd_qc.collect_normalized_query_trace = 1 array mysqlnd_qc_get_normalized_query_trace_log() Similar to mysqlnd_qc_get_query_trace_log() but with SQL statements normalized and aggregated by the normalized query string. Normalization refers to replacing actual parameters, for example in “WHERE a > 1”, with questionmarks like “ WHERE a > ?”. “WHERE a > ?” will match any value for “?”, for example “1”, “2”, “'abc'” but not other identifiers.
  • 132. Runtime configuration
    • mysqlnd_qc.collect_normalized_query_trace
      • PHP_INI_SYSTEM, default “0”, boolean
      • 133. enable/disable normalized query trace
  • 134. Scope of the normalized query log
    • PHP Process
      • Data from one or multiple requests
      • 135. Data from all PHP MySQL APIs: ext/mysql, ext/mysqli, PDO_MySQL
      • 136. Aggregated by normalized query
  • 137. Normalized query trace elements (I)
    • query
      • string: normalized SQL statement
      • 138. Trace contains all inspected queries
      • 139. Aggregated by normalized SQL statement
      • 140. Normalized: parameter replaced by placeholder
      • 141. SELECT 1, SELECT 'a' -> SELECT ?
    • occurences
      • int
      • 142. Number of SQL statements summarized
      • 143. in this record set
  • 144. Normalized query trace elements (II)
    • eligible_for_caching
      • boolean
      • 145. Set to true if the query qualifies for caching
      • 146. according to the query analysis
      • 147. The query might or might not be cached
  • 148. Normalized query trace elements (III)
    • min_run_time
    • 149. avg_run_time
    • 150. max_run_time
      • int
      • 151. Minimum, average and maximum run time of all queries aggregated in this record set
      • 152. ini setting: mysqlnd_qc.time_statistics
      • 153. (enabled by default)
  • 154. Normalized query trace elements (IV)
    • min_store_time
    • 155. avg_store_time
    • 156. max_store_time
      • int
      • 157. Minimum, average and maximum store time of all queries aggregated in this record set
      • 158. ini setting: mysqlnd_qc.time_statistics
      • 159. (enabled by default)
  • 160. Storage handler statistics
    • Optional – handler duty!
    • 161. Default handler
    • APC
      • Access statistics
    • Memcache
      • no handler statistics available
  • 164. Accessing storage handler statistics User storage handler need to provide their own API! array mysqlnd_qc_get_cache_info() Returns information on the active handler and, if available, storage handler statistics. Storage handler statistics are provided under the “data” key of the hash. Different storage handler return different statistics, if any. User defined storage handler cannot return their statistics through this function. They need to provide their own API for accessing storage handler level statistics.
  • 165. “Hot” tables and guessing speed-up
    • Set myslqnd_qc.cache_by_default = 1
      • Accept and ignore “false” results
    • Set mysqlnd_qc.collect_statistics = 1
    • 166. Set mysqlnd_qc.time_statistics = 1
    • 167. Analyze meta data of cache entries
      • mysqlnd_qc_get_cache_info() -> “data” hash
      • 168. Default and APC handler provide meta data
      • 169. Search and account table names from meta data
      • 170. Analyze recorded run and store times
  • 171. “Hot” tables and guessing speed-up
    • Set myslqnd_qc.cache_by_default = 0
    • 172. Set mysqlnd_qc.collect_statistics = 1
    • 173. Set mysqlnd_qc.time_statistics = 1
    • 174. Read user handler specific presentation! :-)
  • 175. Runtime configuration
    • mysqlnd_qc.collect_statistics
      • no impact on storage handler statistics
      • 176. background: storage format with/wo statistics
    • mysqlnd_qc.time_statistics
      • use of gettimeofday() system call
      • 177. enable/disable: all timings
      • 178. background: handler make use of core function
  • 179. Scope of handler statistics
    • Scope of handler storage
      • Default: Process (one or multiple requests)
      • 180. APC: machine (multiple processes)
      • 181. Data from all PHP MySQL APIs: ext/mysql, ext/mysqli, PDO_MySQL
  • 182. Default handler: result set size
    • rows
      • Number of rows
    • stored_size
      • Size in bytes of the raw network traffic result set
      • 183. added to the cache
      • 184. See http://blog.ulf-wendel.de/?p=198 for
      • 185. high-level discussion of the raw network format
  • 186. Default handler: access and timings
    • cache_hits
      • Number of cache hits
    • run_time
      • Run time of the uncached query
      • 187. Set once when adding the query to the cache
    • store_time
      • Store time of the uncached query
      • 188. Set once when adding the query to the cache
  • 189. Default handler: timings
    • min_run_time
    • 190. avg_run_time
    • 191. max_run_time
      • Minimum, average and maximum query run
      • 192. time measured during a cache hit
  • 193. Default handler: timings
    • min_store_time
    • 194. avg_store_time
    • 195. max_store_time
      • Minimum, average and maximum store
      • 196. time measured during a cache hit
  • 197. APC handler: access, size, timings
    • cache_hits
      • Number of cache hits
    • rows
      • Number of rows
    • run_time
      • Run time of the uncached query
      • 198. Set once when adding the query to the cache
    • store_time
      • Store time of the uncached query
      • 199. Set once when adding the query to the cache
  • 200. APC handler: timings
  • 207. Tooling: Cache Monitor web script
    • web/mysqlnd_qc_monitor.php
    • 208. Demonstrates usage of statistics
    • 209. Not a production level tool
    • 210. Basics only
      • Configuration
      • 211. Per-process statistics
      • 212. Cache contents (storage handler statistics)
      • 213. Per-machine statistics using slow auto_append example script
  • 214. Tooling: per machine statistics
    • web/auto_append_persist_qc_stats.php
    • 215. auto_append to persist per-process data
    • 216. Demonstrates aggregation of core statistics
    • 217. Not a production level tool
  • 219. The End Feedback: ulf.wendel@sun.com The End Feedback: [email_address] , [email_address]