Built-in query caching for all PHP MySQL extensions/APIs
Upcoming SlideShare
Loading in...5
×
 

Built-in query caching for all PHP MySQL extensions/APIs

on

  • 9,899 views

Query caching boosts the performance of PHP MySQL applications. Caching can be done on the database server or at the web clients. A new mysqlnd plugin adds query caching to all PHP MySQL extension: ...

Query caching boosts the performance of PHP MySQL applications. Caching can be done on the database server or at the web clients. A new mysqlnd plugin adds query caching to all PHP MySQL extension: written in C, immediately usable with any PHP application because of no API changes, supports Memcache, APC, SQLite and main memory storage, integrates itself smoothless into existing PHP deployment infrastructure, helps you to scale by client, ... Enjoy!

Statistics

Views

Total Views
9,899
Views on SlideShare
8,224
Embed Views
1,675

Actions

Likes
6
Downloads
96
Comments
0

11 Embeds 1,675

http://blog.ulf-wendel.de 1511
http://www.planet-php.net 77
http://blogs.vinuthomas.com 54
http://www.planet-php.org 9
http://www.phpeye.com 8
http://planet-php.org 6
http://translate.googleusercontent.com 4
http://static.slidesharecdn.com 3
http://xss.yandex.net 1
http://planet-php.net 1
http://translate.yandex.net 1
More...

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Built-in query caching for all PHP MySQL extensions/APIs Built-in query caching for all PHP MySQL extensions/APIs Presentation Transcript

  • MySQL native driver for PHP: The query cache plugin
  • mysqlnd_qc: The query cache plugin Ulf Wendel, Andrey Hristov MySQL Connectors Team Sun Microsystems
  • Table of Contents
    • At a glance
      • Key Features
      • Basic Usage
    • Architecture
      • MySQLnd Plugins
      • Cache Architecture
      • Storage handler
    • Usage
      • Advanced Usage
      • Configuration
  • Key Features
    • Transparent: no user API changes
      • supports ext/mysqli
      • supports PDO_MYSQL
      • supports ext/mysql
    • Invalidation: TTL, custom
      • Per query TTL (Time to Live)
      • custom: user callbacks
    • Storage handler: build-in, custom
      • Default (Hash), APC, Memcache
      • custom: user callbacks
  • Possibly asked questions
    • Can stale data be served?
      • Sure, it is Time to live (TTL) by default! But you can implement your own invalidation strategy.
    • Which PHP versions are supported?
      • PHP 5.3.3-dev or newer. At the time of writing the query cache is in prototype stage.
    • Where can I get it?
      • http://forge.mysql.com/wiki/MySQLnd_Query_Cache_Plugin_for_PHP
  • Only buffered queries can be cached
    • ext/mysqli
      • mysqli_query()
      • mysqli_real_query() + mysqli_store_result()
    • PDO_MYSQL
      • PDO::query() if PDO::ATTR_EMULATE_PREPARES = 1
      • (this is the default)
    • ext/mysql
      • mysql_query()
  • Uncachable queries
    • ext/mysqli
      • mysqli_real_query() + mysqli_use_result()
      • mysqli_query() + MYSQLI_ASYNC
      • mysqli_stmt_*()
    • PDO_MYSQL
      • PDO::query() if PDO::ATTR_EMULATE_PREPARES = 0
    • ext/mysql
      • mysql_unbuffered_query()
  • Benchmarks?
    • Please run your own!
      • Too many variables, sorry... for now ;-)
      • Chart compares handler – no more
  • f Basic Usage $mysqli = new mysqli($host, $user, $passwd, $db, $port, $socket);; // Cached: SQL hint used $res = $mysqli->query("/*qc=on*/" . "SELECT id, label FROM test"); var_dump($res->fetch_all(MYSQLI_ASSOC)); $res->free(); SQL hint /*qc=on*/ to control caching // Uncached: no SQL hint $res = $mysqli->query("SELECT id, label FROM test"); var_dump($res->fetch_all(MYSQLI_ASSOC)); $res->free();
  • Installation
    • Download source
      • http://forge.mysql.com/wiki/MySQLnd_Query_Cache_Plugin_for_PHP
    • Copy extension into PHP 5.3.3+ source tree
      • cp -R query_cache my_php/ext/mysqlnd_qc
    • Enable query cache and build
      • ./buildconf --force
      • ./configure --help | grep mysql
      • ./configure --help | grep qc
      • make clean && make
  • Compared to MySQL Server Cache
    • Different location!
      • Less work for the server
      • Shorter distance to the app (round trip times)
      • Scale by client
      • No server overload
      • Less sophisticated invalidation (stale data!)
  • Compared to Memcache
    • Memcache is no cache
      • Memcache is a storage medium
      • Often used for storing cached data
      • We offer a Memcache storage handler
    • Application vs. driver based caching
      • almost no application changes needed
      • caching of raw data instead of processed data
  • Table of Contents
    • At a glance
      • Key Features
      • Basic Usage
    • Architecture
      • MySQLnd Plugins
      • Cache Architecture
      • Storage handler
    • Usage
      • Customization
      • Configuration
  • How PHP connects to MySQL PHP MySQL Server Library: implements MySQL Client-Server Protocol PHP API for PHP applications MySQL native driver for PHP / MySQL Client Library
  • Inside PHP (on the C level!) PHP Extensions Zend Engine PDO ext/mysql ext/mysqli SAPI PDO_MYSQL PDO_XYZ MySQL Client Library (libmysql) or MySQL native driver for PHP (default as of PHP 5.3)
  • PHP 5.3.2: mysqlnd plugin interface <?php /* Any PHP MySQL application */ ?> ext/*mysql* ext/*mysql* Plugin MySQL native driver for PHP C plugins operate transparently: no user API changes
  • “Object-orientation” in mysqlnd <?php mysqli_query($link, … ); ?> Inside mysqlnd: “objects” of data and function pointer mysqlnd_connection connection_methods host user password ... connection_methods query() store_result() use_result() free_result() ...
  • Plugins as “Proxies” or “Decorators” <?php mysqli_query($link, … ); ?> Plugins replace or extend internal mysqlnd objects connection_methods query() store_result() use_result() free_result() ... Plugin query() store_result() use_result() free_result() ... mysqlnd_connection connection_methods host user password ...
  • Plugins can change everything
    • Connection methods
      • sending queries, fetching results
      • metadata access
      • utility calls
    • Prepared Statement methods
      • sending queries, fetching results
    • Network calls
      • send and fetch raw MySQL protocol data
  • C level support for Plugins
    • API calls
      • Plugin registration
      • management of function tables
    • Piggybacking data
      • via connection object
      • via result set object
      • via statistics hash
  • Query cache: proxy-style plugin Cache MySQL <?php /* Any PHP MySQL application */ ?> ext/*mysql* ext/*mysql* mysqlnd mysqlnd Cache miss: record wire data, cache hit: replay Query Cache Plugin
  • Query Cache: Miss mysqlnd orig. mysqlnd_conn object mysqlnd_qc object query() Should cache? Yes! Is cached? No! Activate data recorder query() Send query to MySQL Deactivate recorder Cache wire data Decode data Fetch reply
  • Query Cache: Hit mysqlnd mysqlnd_qc object query() Should cache? Yes! Is cached? Yes! Fetch data from cache Decode data
  • Storage handler responsibilities
    • Storage
      • Scope: request, process, machine, multi-machine
      • Location: distance to cache
      • Replacement strategy
      • Slam defense strategy
    • Decide what to cache
      • Detect SELECT statements
      • Parse SQL-Hints
    • Extended statistics
      • Storage statistics, traces, timings
  • Cache storage handler (I)
    • Default (Hash)
      • Scope: process (single or multiple requests)
      • Location: same process
      • User can flush cache
      • Handler maintains storage statistics
    • APC
      • Scope: machine (multiple requests)
      • Location: local
      • User can flush cache
      • Handler maintains storage statistics
  • Cache storage handler (II)
    • Memcache
      • Scope: can be shared between machines
      • Location: can be local or remote
      • User cannot flush cache
      • No handler statistics
    • NOP
      • No operation
      • Internal and not exposed to PHP land
  • Cache storage handler (III)
    • SQLite
      • Scope, if using “:memory:”: process (single or multiple requests)
      • Scope, if using file: single (multiple) machine
      • Location: can be local or remote
      • User can flush cache
      • Handler maintains storage statistics
    • Berkeley DB
      • theoretically usable via the new SQLite C API compatibility wrapper
  • Cache storage handler (IV)
    • User (procedural and object oriented)
      • It is all up to you!
    • User (class mysqlnd_qc_handler_default)
      • Default handler behaviour
      • Can be specialized by user
      • User can specialize one, many or all functions
    • See also extra presentation!
  • Cache Architecture Summary
    • Transparent from a user perspective
      • No user API changes
      • Works with all user APIs
    • Stores raw wire data
      • Pro: Simplicity – low risk of bugs!
      • Con: Client always needs to decode
    • Storage handler
      • Choice of life-span and scope
      • Choice of distance to client
  • Cache Miss Client 2...100 Client 2...100 Client 2...100 BTW, plan your cache carefully! What if a shared cache entry expires? Client 1 Client 2...n Cache Hit MySQL Client 2...100 Client 2...100 Client 2...100 Client 1 Client 2...n MySQL
  • Cache Miss Optimize storage for reuse? Memory consumption versus peak loads Client 1 Client 2 MySQL Client 2...100 Client 2...100 Client 2...100 Client 1 Client 2...n MySQL Cache Miss Cache Hit Client 3..n
  • Table of Contents
    • At a glance
      • Key Features
      • Basic Usage
    • Architecture
      • MySQLnd Plugins
      • Cache Architecture
      • Storage handler
    • Usage
      • Advanced Usage
      • Configuration
  • Customization: user handler
    • Procedural and OOP interfaces
    • Control which query gets cached
    • Control where to store cached data
    • Implement your own invalidation strategy
    • Maintain your own statistics
  • Procedural and OOP user handler
    • mysqlnd_qc_set_user_handlers()
      • procedural
      • you must implement all storage handler functions
    • extending class mysqlnd_qc_handler_default
      • object oriented
      • customize build-in default handler
      • no need to spezialize all handler functions
    • implementing interface mysqlnd_qc_handler
      • object oriented
      • you must implement all storage handler functions
  • Procedural user storage handler void mysqlnd_qc_set_user_handlers( string get_hash_key, string find_query_in_cache, string return_to_cache, string add_query_to_cache_if_not_exists, string query_is_select, string update_cache_stats, string clear_cache ) Yes, this API may be ugly. BUT: Prototype! BUT: See extra presentation for vodoo! mysqlnd_qc_set_user_handlers()
  • User storage handler interface (I) function get_hash($host_info, $port, $user, $db, $query) { /* returns string */ return $key; } function find($key) { /* returns boolean */ return $found; } function return_to_cache($key) { /* void – leave empty */ } function add($key, $data, $ttl, $runtime, $store_time, $row_count) { /* returns boolean */ return $added_to_cache; }
  • User storage handler interface (II) function query_is_select($query) { /* returns mixed - boolean false if the query shall not be cached boolean true or double 0 to use mysqlnd_qc.ttl default double >= 0 to set TTL different from mysqlnd_qc.ttl */ return $to_cache_or_not; } function update_stats($key, $run_time, $store_time) { /* void – data to update your per query cache statistics *// } function clear_cache() { /* returns boolean */ return $cache_has_been_flushed; }
  • Object oriented storage handler
    • implementing interface mysqlnd_qc_handler
    • extending mysqlnd_qc_handler_default
    • See also extra presentation!
  • API calls
    • mysqlnd_qc_change_handler()
    • mysqlnd_qc_clear_cache()
    • mysqlnd_qc_get_cache_info()
    • mysqlnd_qc_get_core_stats()
    • mysqlnd_qc_get_handler()
    • mysqlnd_qc_get_normalized_query_trace_log()
    • mysqlnd_qc_get_query_trace_log()
    • mysqlnd_qc_set_user_handlers()
  • Changing the storage handler bool mysqlnd_qc_change_handler(string handler) Changes the storage handler. Returns false if the current handler cannot be shutdown or the requested handler cannot be initialized. Failing to change the handler should be considered as a fatal error unless the change fails because the requested handler is unknown.
  • Procedural user storage handler void mysqlnd_qc_set_user_handlers( string get_hash_key, string find_query_in_cache, string return_to_cache, string add_query_to_cache_if_not_exists, string query_is_select, string update_cache_stats, string clear_cache ) Sets the function names of a user defined storage handler and puts them into use. See also extra presentation!
  • Available handlers array mysqlnd_qc_get_handlers() Returns a list of available handler and their versions. The handler “default”, “user”, the class “ myslqnd_qc_handler_default” are always available. “ apc” and “memcache” will be reported if support for those handlers has been enabled at compile time.
  • Flushing the cache The function is not supported by the Memcache handler! bool mysqlnd_qc_clear_cache() Returns true if the handler supports the operation, and has flushed the cache.
  • Cache info and handler statistics // Run some queries and generate cache hits $res = $mysqli->query(&quot;/*qc=1*/&quot; . &quot;SELECT id, label FROM test&quot;); var_dump($res->fetch_all(MYSQLI_ASSOC)); C-based handler cache information and statistics var_dump(mysqlnd_qc_get_cache_info()); array(4) { [&quot;num_entries&quot;]=> int(%d) [&quot;handler&quot;]=> string(7) &quot;default&quot; [&quot;handler_version&quot;]=> string(5) &quot;1.0.0&quot; [&quot;data&quot;]=> array(%d) { [&quot;%s”] => array(2) { [“statistics”] => array(...), [“metadata”] => array(...) } [, ...] }
  • Cache hits, time statistics and more
    • Remember: handler specific statistics
      • Memcache: no statistics at all
      • APC: no metadata
    • If available:
      • num_entries: number of cache entries
      • [“data”][key][“statistics”] : run/store time statistics
      • [“data”][key][“statistics”] : cache hits
      • [“data”][key][“metadata”]: result set meta data
    • See also extra presentation!
  • Core statistics php.ini setting: mysqlnd_qc_collect_statistics = 1 array mysqlnd_qc_get_core_statistics() Returns a list 20+ statistics collected by the core of the query cache plugin, if the PHP configuration setting mysqlnd_qc_collect_statistics is set to 1. The statistics are provided by the core and therefore available with all storage handlers and when using user-defined storage handlers. The statistics cover cache accesses, failures, network traffic as well as aggregated run and store times.
  • Query back trace php.ini setting: mysqlnd_qc.query_trace = 1 array mysqlnd_qc_get_query_trace_log() Returns a query back trace for every query that has been inspected by the query cache regardless if the query ended up being cached or not. The trace tells you where a query has been issues (see also debug_backtrace()). Together with the back trace you get run and store times and information on if the query has been cached.
  • Normalized query back trace mysqlnd_qc.collect_normalized_query_trace = 1 array mysqlnd_qc_get_normalized_query_trace_log() Similar to mysqlnd_qc_get_query_trace_log() but with SQL statements normalized and aggregated by the normalized query string. Normalization refers to replacing actual parameters, for example in “WHERE a > 1”, with questionmarks like “ WHERE a > ?”. “WHERE a > ?” will match any value for “?”, for example “1”, “2”, “'abc'” but not other identifiers.
  • Runtime configuration
  • Runtime configuration (cont.)
  • Basics
    • mysqlnd_qc.ttl
      • Time to live in seconds
      • Handler can overrule setting for individual queries
    • mysqlnd_qc.cache_by_default
      • Cache every query, even those without SQL hint?
      • Evaluated by the core, handler cannot overrule!
    • mysqlnd_qc.cache_no_table
      • Cache queries with no table name im their meta data, e.g SELECT SLEEP(1)?
  • Performance
    • mysqlnd_qc.use_request_time
      • Use PHP global request time to avoid gettimeofday() system calls?
      • Pitfall – see APC docs on apc.use_request_time
    • mysqlnd_qc.time_statistics
      • Collect run time and store time statistics using gettimeofday() system call?
    • mysqlnd_qc.std_data_copy
      • Default handler: copy cached wire data?
      • EXPERIMENTAL – use default of “0”
  • Statistics and Slam defense
    • mysqlnd_qc.collect_statistics
      • Collect statistics for mysqlnd_qc_get_core_stats()?
    • mysqlnd_qc.slam_defense
      • Activates handler based slam defense if available
  • Query traces
    • mysqlnd_qc.collect_query_trace
      • Collect query back traces?
    • mysqlnd_qc.query_trace_bt_depth
      • Maximum back trace depth
    • mysqlnd_qc.collect_normalized_query_trace
      • Collect aggregated normalized query traces?
  • APC storage handler
    • mysqlnd_qc.use_request_time = apc.use_request time
      • Use the same timer for TTL invalidation
    • mysqlnd_qc.apc_prefix
      • Key prefix
      • APC handler stores data in the APC user cache
      • Users can manipulate the cache entries!
  • Memcache storage handler
    • mysqlnd_qc.memc_server
      • Memcache server host
      • Cannot be changed at run time
    • mysqlnd_qc.memc_port
      • Memcache server port
  • SQLite storage handler
    • mysqlnd_qc.sqlite_data_file
      • SQLite data file
      • You should use SQLite as an in-memory storage
  • Exported PHP constants
    • MYSQLND_QC_ENABLE_SWITCH
      • SQL-hint to enable caching, e.g. “qc=on”
      • Only this exact string will recognized!
    • MYSQLND_QC_DISABLE_SWITCH
      • SQL-hint to disable caching, e.g. “qc=off”
      • Only this exact string will be recognized!
    • MYSQLND_QC_TTL_SWITCH
      • SQL-hint for setting per query TTL, e.g. “qc_ttl=”
      • All constants can be changed as compile time!
  • Exported PHP classes class mysqlnd_qc_handler_default { public function init() {} public function is_select(...) {} public function get_hash_key(...) {} public function return_to_cache(...) {} public function add_to_cache(...) {} public function find_in_cache(...) {} public function update_cache_stats(...) {} public function get_stats(...) {} public function clear_cache() {} public function shutdown() {} }
  • Exported PHP interfaces interface mysqlnd_qc_handler { public function is_select(...) {} public function get_hash_key(...) {} public function return_to_cache(...) {} public function add_to_cache(...) {} public function find_in_cache(...) {} public function update_cache_stats(...) {} public function get_stats(...) {} public function clear_cache() {} }
    • Presentations
      • A query cache plugin -
      • this is what you look at ;-)
      • Query cache plugin benchmark impressions
      • Dig deeper with QC statistics
      • Developing user storage handler
    Further reading
  • The End Feedback: ulf.wendel@sun.com The End Feedback: [email_address] , [email_address]