• Save
PostgreSQL Query Cache - "pqc"
Upcoming SlideShare
Loading in...5
×
 

PostgreSQL Query Cache - "pqc"

on

  • 20,003 views

PostgreSQL Query Cache enables to improve query performance “extremely” (x10~x100) faster by caching result responses from backend.

PostgreSQL Query Cache enables to improve query performance “extremely” (x10~x100) faster by caching result responses from backend.

Statistics

Views

Total Views
20,003
Views on SlideShare
16,980
Embed Views
3,023

Actions

Likes
13
Downloads
0
Comments
2

46 Embeds 3,023

http://pgsnaga.blogspot.com 1660
http://wpcomwidgets.com 419
http://pgsnaga.blogspot.in 93
http://pgsnaga.blogspot.com.br 91
http://pgsnaga.blogspot.co.uk 72
http://pgsnaga.blogspot.fr 68
http://pgsnaga.blogspot.de 65
http://pgsnaga.blogspot.ca 56
http://pgsnaga.blogspot.it 51
http://pgsnaga.blogspot.com.es 48
http://pgsnaga.blogspot.ru 44
http://pgsnaga.blogspot.com.au 36
http://pgsnaga.blogspot.kr 28
http://pgsnaga.blogspot.nl 24
http://pgsnaga.blogspot.pt 24
http://pgsnaga.blogspot.jp 22
http://pgsnaga.blogspot.hu 20
http://pgsnaga.blogspot.ro 17
http://pgsnaga.blogspot.ch 15
http://pgsnaga.blogspot.cz 15
http://pgsnaga.blogspot.se 12
http://pgsnaga.blogspot.co.nz 12
http://pgsnaga.blogspot.fi 11
http://pgsnaga.blogspot.hk 11
http://pgsnaga.blogspot.tw 11
http://pgsnaga.blogspot.be 10
http://pgsnaga.blogspot.dk 9
http://pgsnaga.blogspot.mx 8
http://pgsnaga.blogspot.co.il 8
http://paper.li 7
http://pgsnaga.blogspot.co.at 7
http://pgsnaga.blogspot.com.ar 7
http://pgsnaga.blogspot.no 7
http://www.slashdocs.com 6
http://pgsnaga.blogspot.gr 6
http://christopherjcoleman.wordpress.com 5
http://translate.googleusercontent.com 5
http://pgsnaga.blogspot.sg 4
http://pgsnaga.blogspot.sk 2
http://pgsnaga.blogspot.ie 1
http://webcache.googleusercontent.com 1
https://p.yammer.com 1
http://pgsnaga.blogspot.com.tr 1
http://pgsnaga.blogspot.ae 1
http://192.0.0.188 1
http://code.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    PostgreSQL Query Cache - "pqc" PostgreSQL Query Cache - "pqc" Presentation Transcript

    • PostgreSQL Query Cache “pqc” March/1/2011 Satoshi Nagayasu Uptime Technologies, LLC.Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Overview• By caching result responses from backend, a query cache daemon (`pqcd’) makes query response faster “extremely” (x10~x100). – Delegates queries in front of the backend, like a proxy. – Waits connections on the different port (9999 by default). – Intercepts and caches SELECT query results. (Query Cache) – Manages lifecycle of the query cache. Server side Direct execution PostgreSQL PostgreSQL client backend(s) Execution pqcd with the query cache Cache Memory Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Tech Specs• Supports simple query executions, and prepared statements. – Supports V3 protocol only. (V2 is not supported)• Supports “Active cache mode” – All SELECT query results would be stored in the query cache. – User specified query would not be cached.• Supports “Passive cache mode” – All SELECT query results would NOT be cached. – Only user specified SELECT queries would be cached. – And/or queries which took longer execution time than the user-specified duration, would be cached. (Not implemented yet)• Supports cache invalidation (expiration or refreshing) – By using the cache expiration timeout. – By using “hint” with SELECT statements. – By using “hint” to expire all query cache. (Not implemented yet) Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • The Query Cache Flow Overview• After `pqcd’ receives a SELECT statement, if some result is available in the query cache, `pqcd’ returns the result from the query cache. If not available, it relays the query to the backend. Receive a query from frontend Is cache No available? Yes Not expired No Invalidate Forward the query yet? the cache to backend Yes Fetch a result Retreive a result from query cache from the backend Store a result Send to the frontend in the query cache Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Implementation• Using KVS (Memcached) to hold the query cache. – “libmemcached” as a memcached client library for C. – A SQL statement as a key, and response messages as a value.• `pqcd’, a query cache daemon, is implemented as a derivative of pgpool. – Un-used feature (code) removed, and several “hook” routines added. Server side PostgreSQL PostgreSQL client pqcd backend(s) Hook routines libmemcached Memcached Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Deployment• Required packages – libevent 1.4.14b (required by memcached) – memcached 1.4.5 – libmemcached 0.43• How to install – automake – ./configure --prefix=$PREFIX – make – sudo make install – cd $PREFIX/etc – cp pqcd_hba.conf.sample pqcd_hba.conf• Starting pqcd – $PREFIX/bin/pqcd• Stopping pqcd – $PREFIX/bin/pqcd stop Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Deployment (with RPMs)• Required RPM – uqc-libevent-1.4.14b-1.i386.rpm – uqc-memcached-1.4.5-1.i386.rpm – uqc-libmemcached-0.43-1.i386.rpm – uqc-querycache-20110223-1.i386.rpm• Configuration files – /opt/uptime/querycache/etc/pqcd.conf – /opt/uptime/querycache/etc/pqcd_hba.conf• Starting pqcd – /opt/uptime/querycache/bin/pqcd• Stopping pqcd – /opt/uptime/querycache/bin/pqcd stop Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Configurations• `pqcd.conf’ may be configured as you need. (in $PREFIX/etc)• memcached_bin – A path name to Memcached executable. (default is “/opt/uptime/querycache/bin/memcached”)• query_cache_mode – The query cache mode. “active” or “passive”. (default is “active”)• query_cache_expiration – The timeout value for query cache expiration (in seconds, default is 30) Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Cache hints• Add as a comment at the beginning of SELECT statements – /* cache:refresh */SELECT * FROM … – <slash> <asterrisk> <space> <hint> <space> <asterisk> <slash>• cache:on (default in Active cache mode) – Looks at the cache first. If not found, then executes on the backend, and puts the result into the cache.• cache:off (default in Passive cache mode) – Doesn’t look at the cache. Execute on the backend first, and doesn’t put the result into the cache.• cache:refresh – Doesn’t look at the cache first. Executes on the backend, and puts the result into the cache.• cache:expire – Invalidates query cache for specified statement. (Not implemented yet)• cache:expireall – Invalidates all query cache. (Not implemented yet) Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • ExamplesCopyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Query execution and the cache A regular scan takes >400ms. 2nd execution completes in 0.5ms with the cache. Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • A hint to handle the cache. Add a hint to disable the query cache. Longer time. Without a hint, the query cache again. Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Record updates and the cache Delete all records. Still an old value left in the cache. Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Refreshing the query cache Add a hint to refresh the cache. The value in the cache refreshed. With hitting the cache. Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • TODO• To Do – Handle a transaction state correctly on “Z (ready for query)” response. – Make expandable cache values for >8192bytes. – Make enable to handle cache key with >250 charactors. (limitation from Memcached) – Automatic query caching by watching duration time under the Passive cache mode. – Run benchmarks!• Done – PreparedStatement. – Cache hints. – Rename the binary to ‘pqcd’ in Makefile – Start/stop Memcached from pqcd. – Invalidation query cache by timeout. (query_cache_expiration option) – Completes a pgbench run. – Remove V2 protocol code. (prevent to be conneced with V2 at startup) – `memcached_bin’ option added in pqcd.conf. – Put a connected database name into the cache key. – Remove un-used code (replication, master-slave, dual server under connection pooling) – “Keep it simple, stupid!” – Make it public as an opensource software! Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.
    • Contact:Uptime Technologies, LLC.E-Mail: contact@uptime.jpWeb: http://www.uptime.jp/ Copyright 2010-2011 Uptime Technologies, LLC. All rights reserved.