Harder, better, faster, stronger: PostgreSQL 9.1

11,974 views
11,821 views

Published on

Published in: Technology
6 Comments
7 Likes
Statistics
Notes
No Downloads
Views
Total views
11,974
On SlideShare
0
From Embeds
0
Number of Embeds
123
Actions
Shares
0
Downloads
113
Comments
6
Likes
7
Embeds 0
No embeds

No notes for slide

Harder, better, faster, stronger: PostgreSQL 9.1

  1. 1. Harder Better Faster Stronger PostgreSQL 9.1 Selena Deckelmann PostgreSQL Global Development GroupSo O mS eCC O on N fer 20 en 11 e c
  2. 2. 11 20 N O SCO
  3. 3. Say “Postgres.” c e 11 en 20 fer O on N eCC O mS So
  4. 4. c e 11 en 20 fer O on N eCC O mSSo
  5. 5. “World’s most advanced open source database.”So O mS eCC O on N fer 20 en 11 e c
  6. 6. c e 11 en 20 fer O on N eCC O mSSo
  7. 7. c e 11 en 20 fer O on N eCC O mSSo
  8. 8. c e 11 en 20 fer O on N eCC O mSSo
  9. 9. Postgres disrupts the commercial relational database industry.So O mS eCC O on N fer 20 en 11 e c
  10. 10. Disruptive forces Licensing “Enterprise” Web development Proprietary DBA Career PathSo O mS eCC O on N fer 20 en 11 e c
  11. 11. Licensing Drop-in replacement for custom application development. Cheap Data Warehousing Ecosystem!So O mS eCC O on N fer 20 en 11 e c
  12. 12. “Enterprise” Web development Postgres is a “Developers database” - Extension is encouraged! Database of choice for Django Oracle wants PHP developersSo O mS eCC O on N fer 20 en 11 e c
  13. 13. “Visit the Ruby and Python Topic for more information pertaining to Ruby/RoR, Python and other non-PHP scripting languages.”So O mS eCC O on N fer 20 en 11 e c
  14. 14. DBA Career path Companies can’t hire Postgres DBAs fast enough. Postgres is close enough to Oracle, that Oracle DBAs can jump in. What we do better: Data loading, automation, TUI and GUI admin toolsSo O mS eCC O on N fer 20 en 11 e c
  15. 15. And... Stability Support major releases for at least 5 years. Commits to HEAD are stable. Bugs reported are fixed in <48 hrs.So O mS eCC O on N fer 20 en 11 e c
  16. 16. 10 features Harder Better More secure & robust: We made us better: SE Postgres Writeable CTE Replication Tools Per-column collations Extensions Faster Stronger Performance: First in class: KNN Indexes Serializable Snapshot Isolation Unlogged Tables Foreign Data WrappersSo O mS e Synchronous Replication CC O on N fer 20 en 11 e c
  17. 17. Harder c e 11 en 20 fer O on N eCC O mS So
  18. 18. Harder Replication Tools WAL-based replication: File-based (default 16-MB files) Streaming (per-WAL record) (9.0) Asynchronous Warm standby Hot standby (9.0) Synchronous (9.1) new! Cascading replication (just committed... 9.2) new! http://www.pgcon.org/2010/schedule/attachments/149_PGCon2010-Built-in-Replication.pdfSo O mS eCC O on N fer 20 en 11 e c
  19. 19. Harder Replication Tools pg_stat_replication view shows all replicas and their statuses pg_basebackup single-command cloning over port 5432 http://www.postgresql.org/docs/9.1/static/app-pgbasebackup.html promote to master pg_ctl promote formerly only could do this by creating a trigger file new functions pg_last_xact_replay_timestamp() pg_xlog_replay_pause()So O mS eCC O on N fer 20 en 11 e c
  20. 20. Harder SE-Postgres SE-Linux security label support! Originally a NSA-funded Linux kernel extension For tablespace, schema, table, column, sequence, view and procedure object classes Will have integrated support in pgAdmin! http://rhaas.blogspot.com/2010/09/se-linux-for-postgresql-part-1.htmlSo O mS http://rhaas.blogspot.com/2010/09/se-linux-for-postgresql-part-2.html eCC O on N fer 20 http://wiki.postgresql.org/wiki/SEPostgreSQL_Documentation en 11 e c
  21. 21. Better Writeable CTE But first - What’s a CTE? Common Table Expressions A temporary table or VIEW that exists for just a single query. http://developer.postgresql.org/pgdocs/postgres/queries-with.html#QUERIES-WITH- MODIFYINGSo O mS eCC O on N fer 20 en 11 e c
  22. 22. Better c e 11 en 20 fer O on N eCC O mS So
  23. 23. Better Common Table Expression Example WITH RECURSIVE t1(a, b) AS ( VALUES(0,0) UNION ALL SELECT CASE CAST(b AS BOOLEAN) WHEN b % 3 = 0 THEN b WHEN b % 5 = 0 THEN b END, b+1 Euler Project, question 1: FROM t1 If we list all the natural numbers WHERE b < 1000 below 10 that are multiples of 3 or ) 5, we get 3, 5, 6 and 9. The sum of SELECT sum(a) FROM t1; these multiples is 23. Find the sum of all the multiples of 3 or 5 below 1000.So O mS eCC O on N fer 20 http://wiki.postgresql.org/wiki/Euler_Project,_Question_1 en 11 e c
  24. 24. Better Writeable CTE Makes maintaining partitions easier: WITH deleted_posts AS ( DELETE FROM posts WHERE created < now() - 6 months::INTERVAL RETURNING * ) SELECT user_id, count(*) FROM deleted_posts group BY 1; http://www.depesz.com/index.php/2011/03/16/waiting-for-9-1-writable-cte/So O mS eCC O http://xzilla.net/blog/2011/Mar/Upserting-via-Writeable-CTE.html on N fer 20 en 11 e c
  25. 25. Better Per-column collation Supports multiple languages in the same table, and have ORDER BY sort by language order (assuming you’ve loaded the correct collation library) BEFORE: $ select * from normal_polish order by some_text; some_text ----------- alfa car ćma coś lamaSo O mS eCC O on N fer 20 en 11 e c
  26. 26. Better Per-column collation AFTER: $ select * from normal_polish order by some_text; some_text ----------- alfa car coś ćma lamaSo O mS eCC O on N fer 20 en 11 e c
  27. 27. Better Extensions Postgres-specific package management for contrib/, PgFoundry projects, tools. Like Oracle "packages" End user view: http://facility9.com/2011/03/postgresql-extensions/ Docs: http://developer.postgresql.org/pgdocs/postgres/sql- createextension.htmlSo O mS eCC O on N fer 20 en 11 e c
  28. 28. ExtensionsBetter 11 en c e 20 fer O on N eCC O mS So
  29. 29. Faster c e 11 en 20 fer O on N eCC O mS So
  30. 30. Faster K-Nearest Neighbor Indexes Incredibly fast search for things that are "near" a particular object Operator is: <-> Datatypes supported: pg_trgm contrib and btree_gist datatypes (point) Wiki: http://wiki.postgresql.org/wiki/ Whats_new_in_PostgreSQL_9.1#K-Nearest-Neighbor_Indexing Developer: http://www.sai.msu.su/~megera/postgres/talks/ pgday-2010.pdfSo O mS eCC O on N fer 20 en 11 e c
  31. 31. Faster K-Nearest Neighbor Indexes $ create table test ( position point ); $ insert into test (position) select point( random() * 1000, random() * 1000) from generate_series(1,1000000); $ create index q on test using gist ( position ); $ select *, position <-> point(500,500) from test order by position <-> point(500,500) limit 10; $ explain analyze select *, position <-> point(500,500) FROM test ORDER BY position <-> point(500,500) limit 10; QUERY PLAN Limit (cost=0.00..0.77 rows=10 width=16) (actual time=0.164..0.475 rows=10 loops=1) -> Index Scan using q on test (cost=0.00..76512.60 rows=1000000 width=16) (actual time=0.163..0.473 rows=10 loops=1)So O Order By: ("position" <-> (500,500)::point) mS eCC O on N fer 20 en 11 e c
  32. 32. Faster Unlogged Tables Background: every write to a table in Postgres is essentially *two* writes because of the Write Ahead Log (basis for replication, and durability guarantees) Unlogged tables are FREED of this durability constraint! Could think of them as Global Temporary tables (see Haas post) http://www.depesz.com/index.php/2011/01/03/waiting-for-9-1-unlogged-tables/ http://rhaas.blogspot.com/2010/05/global-temporary-and-unlogged-tables.html Example: =$ perl -le my $x = "aaaaaaaa"; for (1..450000) {print $x++} | perl -pe s/.*/(047$&047)/;s/s*z/,/ if $.%3 | sed s/.*/insert into test (some_text) values &;/So O mS eCC O on N fer 20 en 11 e c
  33. 33. Stronger Serializable Snapshot Isolation PostgreSQLs is the first production implementation Includes: per transaction support Why use it? * No more blocking on table locks! * No more "select for update" * Simplifies programming Wiki: http://wiki.postgresql.org/wiki/SSI Developer: My notes from Kevins version of the talk: http://www.chesnok.com/ daily/2011/03/24/raw-notes-from-kevin-grittners-talk-on-ssi/So O mS eCC O on N fer 20 en 11 e c
  34. 34. Stronger c e 11 en 20 fer O on N eCC O mS So
  35. 35. Stronger Stronger Foreign Data Wrappers * Remote datasource access * Initially implemented TEXT, CSV data sources * Twitter: http://pgxn.org/dist/twitter_fdw/ * Google Summer of Code student is working on other data sources * Underway currently: Oracle and MySQL sources Awesome because: * 6-function API for implementing * Good for implementing imports, things which would otherwise fail if you just used COPY (imporoving COPY is a whole other rant...)So O mS eCC O on N fer 20 en 11 e c
  36. 36. Stronger Stronger Foreign Data Wrappers Not awesome because: * Nothing other than sequential scans are possible for query plans * Not that many data sources implemented yet Expect tons of FDWs to be implemented once we get 9.1 to production releaseSo O mS eCC O on N fer 20 en 11 e c
  37. 37. Stronger Syncronous Replication This is *the* most important use case for WAL-level replication. User: http://developer.postgresql.org/pgdocs/postgres/warm- standby.html#SYNCHRONOUS-REPLICATION (kind of terrible for a person trying to understand how it works) Somewhat better: http://wiki.postgresql.org/wiki/What %27s_new_in_PostgreSQL_9.1#Synchronous_replication_and_other_replic ation_features Wiki: http://wiki.postgresql.org/wiki/Synchronous_replicationSo O mS eCC O on N fer 20 en 11 e c
  38. 38. Stronger Syncronous Replication WAL stands for "write ahead log" and it is what ensures durability for all write operations * WAL-shipping 16-MB chunks of these records was the pre-9.0 method for built-in replication * 9.0 and later have Streaming rep - per WAL record (per-commit) shipping of changes * Logical changes to DB, NOT statement-based * All-or-nothing wrt databases and tablesSo O mS eCC O on N fer 20 en 11 e c
  39. 39. Stronger Syncronous Replication * Create a replication role (no longer a forced role creation like in 9.0) * pg_hba.conf (host based authorization) update to allow replication user to access system * Five key configuration options: wal_level, max_wal_senders, archive_mode, archive_command, synchronous_standby_names * Create a base backup that will be the slave * recovery.conf config: restore_command, standby_mode, primary_conninfo, trigger_fileSo O mS eCC O on N fer 20 en 11 e c
  40. 40. Terrifyingly live demo.So O mS eCC O on N fer 20 en 11 e c
  41. 41. Harder Better Faster Stronger Thanks to Josh Berkus http://www.pgexperts.com/document.html?id=50 Depesz http://www.depesz.comSo O mS eCC O on N fer 20 en 11 e c
  42. 42. Harder Better Faster Stronger Other Postgres talks: Pro PostgreSQL Robert Treat, 1:40pm Wednesday Unbreaking your Django Application Christophe Pettus, 9amSo O mS (tutorial happening NOW) eCC O on N fer 20 en 11 e c
  43. 43. Harder Better Faster Stronger Selena Deckelmann @selenamarie http://chesnok.com For hire! :)So O mS eCC O on N fer 20 en 11 e c

×