Scaling PostgreSQL with Skytools

5,758 views
5,510 views

Published on

Published in: Technology
1 Comment
17 Likes
Statistics
Notes
No Downloads
Views
Total views
5,758
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
94
Comments
1
Likes
17
Embeds 0
No embeds

No notes for slide

Scaling PostgreSQL with Skytools

  1. 1. Scaling with SkyTools & More Scaling-Out Postgres with Skype’s Open-Source ToolsetGavin M. RoySeptember 14th, 2011
  2. 2. About Me• PostgreSQL ~ 6.5• CTO @myYearbook.com • Scaled initial infrastructure • Not as involved day-to-day database operational and development• Twitter: @Crad
  3. 3. Scaling?
  4. 4. ConcurrencyRequests per Second 6am 8am 10am 12pm 2pm 4pm 6pm 8pm 10pm 12am 2am 4am 6am Hourly breakdown
  5. 5. Increasing Size-On-DiskSize in GB Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  6. 6. Scaling andPostgreSQL Behavior
  7. 7. Size on Disk
  8. 8. Tuples, Indexes, Overhead
  9. 9. Table Size +Size of all combined Indexes Relations Indexes
  10. 10. Constraints • Available Memory • Disk Speed • IO Bus Speed
  11. 11. Keep it in memory.
  12. 12. Get Fast Disks & I/O.
  13. 13. Process Forking + Locks
  14. 14. Client Connections
  15. 15. One Connection perConcurrent Request
  16. 16. Apache+PHPOne connection per backend for each pg_connect
  17. 17. PythonOne connection per connection*
  18. 18. ODBCOne connection to Postgres per ODBC connection
  19. 19. Master Process Lock Stats Collector Contention? AutovacuumEach backend for a connected Wall Writer client has to check for locks Wall Writer Connection Client Connection Backend
  20. 20. Master ProcessNew Client Stats CollectorConnection? Autovacuum Wall Writer Access Share Access Exclusive Wall Writer Exclusive Share Connection Client Connection Backend Share Row Exclusive Share Update Connection Client Connection Row Share Backend Row Exclusive
  21. 21. Master Process Stats Collector Too many Autovacuumconnections? Wall Writer Wall Writer Slow performance Connection Client Connection Backend Connection Client Connection Backend ... Connection Client Connection Backend
  22. 22. 250 Apache Backends x1 Connection per Backend x 250 Servers = 62,500 Connections
  23. 23. Solvable Problems!
  24. 24. The Trailblazers
  25. 25. Solving Concurrency
  26. 26. pgBouncer
  27. 27. Session Pooling
  28. 28. Transactional Pooling
  29. 29. Statement Pooling
  30. 30. Connection Pooling Clients Clients Clients Hundreds Hundreds Hundreds pgBouncer Tens Tens Tens Postgres Postgres Postgres Server #1 Server #2 Server #3
  31. 31. Add Local Pooling Clients Clients Clients Hundreds Hundreds Hundreds Local pgBouncer Local pgBouncer Local pgBouncer Tens Tens Tens pgBouncer Tens Tens Tens Postgres Postgres Postgres Server #1 Server #2 Server #3
  32. 32. Easy to runUsage: pgbouncer [OPTION]... config.ini -d, --daemon Run in background (as a daemon) -R, --restart Do a online restart -q, --quiet Run quietly -v, --verbose Increase verbosity -u, --user=<username> Assume identity of <username> -V, --version Show version -h, --help Show this help screen and exit
  33. 33. userlist.txt“username” “password”“foo” “bar”
  34. 34. pgbouncer.ini
  35. 35. Specifying Connections[databases]; foodb over unix socketfoodb =; redirect bardb to bazdb on localhostbardb = host=localhost dbname=bazdb; access to dest database will go with single userforcedb = host=127.0.0.1 port=300 user=baz password=fooclient_encoding=UNICODE datestyle=ISO connect_query=SELECT1
  36. 36. Base Daemon Config[pgbouncer]logfile = pgbouncer.logpidfile = pgbouncer.pid; ip address or * which means all ip-slisten_addr = 127.0.0.1listen_port = 6432; unix socket is also used for -R.;unix_socket_dir = /tmp
  37. 37. Authentication; any, trust, plain, crypt, md5auth_type = trust#auth_file = 8.0/main/global/pg_authauth_file = etc/userlist.txtadmin_users = user2, someadmin, otheradminstats_users = stats, root
  38. 38. Stats Users? SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|VERSION SHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEMpgbouncer=# SHOW CLIENTS; type | user | database | state | addr | port | local_addr | local_port | connect_time------+-------+-----------+--------+-----------+-------+------------+------------+--------------------- C | stats | pgbouncer | active | 127.0.0.1 | 47229 | 127.0.0.1 | 6000 | 2011-09-13 17:55:46* Truncated columns for display purposes
  39. 39. psql 9.0+ Problem?psql -U stats -p 6432 pgbouncerpsql: ERROR:  Unknown startup parameterAdd to pgbouncer.ini:ignore_startup_parameters = application_name
  40. 40. Pooling Behaviorpool_mode = statementserver_check_query = select 1server_check_delay = 10max_client_conn = 1000default_pool_size = 20server_connect_timeout = 15server_lifetime = 1200server_idle_timeout = 60
  41. 41. Skytools
  42. 42. Scale-Out Reads Clients Clients Clients Clients pgBouncer Load BalancerRead Only Copy Read Only Copy Read Only Copy Read Only Copy Canonical Database
  43. 43. PGQ
  44. 44. The Ticker
  45. 45. ticker.ini[pgqadm]job_name = pgopen_tickerdb = dbname=pgopen# how often to run maintenance [seconds]maint_delay = 600# how often to check for activity [seconds]loop_delay = 0.1logfile = ~/Source/pgopen_skytools/%(job_name)s.logpidfile = ~/Source/pgopen_skytools/%(job_name)s.pid
  46. 46. Getting PGQ RunningSetup our ticker: pgqadm.py ticker.ini installRun the ticker daemon: pgqadm.py ticker.ini ticker -d
  47. 47. Londiste
  48. 48. replication.ini[londiste]job_name = pgopen_to_destinationprovider_db = dbname=pgopensubscriber_db = dbname=destination# it will be used as sql ident so no dots/spacespgq_queue_name = pgopenlogfile = ~/Source/pgopen_skytools/%(job_name)s.logpidfile = ~/Source/pgopen_skytools/%(job_name)s.pid
  49. 49. Install Londistelondiste.py replication.ini provider installlondiste.py replication.ini subscriber install
  50. 50. Start Replication Daemonlondiste.py replication.ini replay -d
  51. 51. DDL?
  52. 52. Add the Provider Tables and Sequenceslondiste.py replication.ini provider add public.auth_user
  53. 53. Add the Subscriber Tables and Sequenceslondiste.py replication.ini subscriber add public.auth_user
  54. 54. Great Success!
  55. 55. PL/Proxy
  56. 56. Scale-Out Reads & Writes plProxy Server A-F Server G-L Server M-R Server S-Z Server
  57. 57. How does it work?
  58. 58. Simple Remote ConnectionCREATE FUNCTION get_user_email(username text)RETURNS SETOF text AS $$ CONNECT dbname=remotedb; SELECT email FROM users WHERE username = $1;$$ LANGUAGE plproxy;
  59. 59. Sharded RequestCREATE FUNCTION get_user_email(username text)RETURNS SETOF text AS $$ CLUSTER “usercluster”; RUN ON hashtext(username);$$ LANGUAGE plproxy;
  60. 60. Sharding Setup• Need 3 Functions: • plproxy.get_cluster_partitions(cluster_name text) • plproxy.get_cluster_version(cluster_name text) • plproxy.get_cluster_config(in cluster_name text, out key text, out val text)
  61. 61. get_cluster_partitionsCREATE OR REPLACE FUNCTIONplproxy.get_cluster_partitions(cluster_name text)RETURNS SETOF text AS $$BEGIN IF cluster_name = usercluster THEN RETURN NEXT dbname=part00 host=127.0.0.1; RETURN NEXT dbname=part01 host=127.0.0.1; RETURN; END IF; RAISE EXCEPTION Unknown cluster;END;$$ LANGUAGE plpgsql;
  62. 62. get_cluster_versionCREATE OR REPLACE FUNCTIONplproxy.get_cluster_version(cluster_name text)RETURNS int4 AS $$BEGIN IF cluster_name = usercluster THEN RETURN 1; END IF; RAISE EXCEPTION Unknown cluster;END;$$ LANGUAGE plpgsql;
  63. 63. get_cluster_configCREATE OR REPLACE FUNCTION plproxy.get_cluster_config( in cluster_name text, out key text, out val text)RETURNS SETOF record AS $$BEGIN -- lets use same config for all clusters key := connection_lifetime; val := 30*60; -- 30m RETURN NEXT; RETURN;END;$$ LANGUAGE plpgsql;
  64. 64. get_cluster_config values• connection_lifetime• query_timeout• disable_binary• keepalive_idle• keepalive_interval• keepalive_count
  65. 65. SQL/MED
  66. 66. SQL/Med Cluster DefinitionCREATE SERVER a_cluster FOREIGN DATA WRAPPER plproxy OPTIONS ( connection_lifetime 1800, disable_binary 1, p0 dbname=part00 hostname=127.0.0.1, p1 dbname=part01 hostname=127.0.0.1, p2 dbname=part02 hostname=127.0.0.1, p3 dbname=part03 hostname=127.0.0.1 );
  67. 67. PLProxy + SQL/Med Behavior• PL/Proxy will prefer SQL/Med cluster definitions over the plproxy.get_* functions• PL/Proxy will fallback to plproxy.get_* functions if there are no SQL/Med clusters
  68. 68. SQL/MED User MappingCREATE USER MAPPING FOR bob SERVER a_cluster OPTIONS (user bob, password secret);CREATE USER MAPPING FOR public SERVER a_cluster OPTIONS (user plproxy, password foo);
  69. 69. plproxyrc• plpgsql based api for table based management of PL/Proxy• Used to manage complicated PL/Proxy infrastructure @myYearbook• BSD Licensed https://github.com/myYearbook/plproxyrc
  70. 70. “Server-to-Server” Postgres Postgres Postgres Server #1 Server #2 Server #3 pgBouncer
  71. 71. Complex PL/Proxy and pgBouncer Environment Clients Local pgBouncer Load Balancer Clients Local pgBouncer Clients Local pgBouncer pgBouncer pgBouncer Postgres plProxy Server plProxy Server Server #1 pgBouncer Postgres Load Balancer Server #3 pgBouncer Postgres Server #3
  72. 72. Other Tools and Methods?
  73. 73. Questions?

×