Version 8.4Holy Frijole,Thats a lot of features!                               February 2009                              ...
8.4: A Few Patches
8.4: A Few Patches 5 CommitFests
8.4: A Few Patches    5 CommitFests9 Months of Development
8.4: A Few Patches    5 CommitFests9 Months of DevelopmentOver 1600 GIT Updates
8.4: A Few Patches         5 CommitFests    9 Months of Development     Over 1600 GIT UpdatesMore Than 2 Dozen Major Featu...
Looks like some database projects still  know how to put out new version.
Some-but-not-all 8.4 Features●   Windowing Functions          ●   Unsigned Integers●   Common Table Expressions     ●   Bo...
SQL Features●   Windowing Functions●   Common Table Expressions●   array_agg●   Per-database Collations●   New data types ...
Windowing Functions●   Aggregate over part of the data    –   SQL 2008 standard    –   Great for BI, OLAP●   Functions:   ...
Windowing FunctionsSELECT   y,   m,   SUM(SUM(people)) OVER (PARTITION BY y ORDER BY m),   AVG(people)FROM(   SELECT     E...
Common Table Expressions●   Ability to create "named subqueries" for your    query.●   Best use: WITH RECURSIVE    –   rea...
Common Table ExpressionsWITH RECURSIVE subdepartment AS(  --  SELECT * FROM department WHERE id = A  UNION ALL  -- recursi...
array_agg●   History:    –   added Arrays in 7.4         ●   array_accum() aggregate example code    –   intarray contrib ...
Per-Database Collations●   Collations (ordering character sets) used to be    per installation●   Now they are per databas...
New Data Types●   Make migrating from other DBMSes easier●   CIText (in /contrib)    –   Case Insensitive Text    –   Full...
Better d in psql●   d is now multi-version compatible    –   dt etc. wont error if you connect an 8.4 client to an        ...
Add columns to VIEWs●   In the bad old days:    –   need to add another column to your VIEW?    –   have to drop it & recr...
Performance & Monitoring●   Parallel Restore●   Improved Hash Indexes●   pg_stat_user_functions●   pg_stat_statements●   M...
Parallel Restore●   In 8.3, we were single-threaded               pg_dump   dump     Restore                          file...
Parallel Restore●   In 8.4, Multi-core, Restore!                                    Restore                               ...
Improved Hash Indexes●   Our old hash indexes were slow and useless●   Improved hash indexes are fast!    –   use them for...
pg_stat_user_functions●   For each of your functions, see    –   # of times called    –   amount of time spent    –   amou...
pg_stat_statements
pg_stat_statements log   log    pgFouine       file
pg_stat_statementspostgres=# SELECT * FROM pg_stat_statements ORDER BY total_time DESCLIMIT 3;-[ RECORD 1 ]---------------...
More DTrace Probes* Probes to measure query time                                     * Probes to measure checkpoint stats ...
auto_explain●   misnamed; actually allows you to manually set specific    queries/sessions/functions to output explain pla...
More Performance Improvements●   Free Space Map is dynamically sized (no more    max_fsm_pages!)●   Visibility Map    –   ...
Stored Procedures●   Default Parameters●   Variadic Parameters●   New PL/pgSQL Statements●   PL/pythonU OUT Parameters
DEFAULT parametersCREATE OR REPLACE FUNCTI ONadder ) a i nt de f a ul t 4 0 ,       b i nt de f a ul t 2 (RETURNS i nt LAN...
VARIADIC parametersCREATE OR REPLACE FUNCTION  adder(VARIADIC v int[])  RETURNS int AS $$DECLARE s int; i int;BEGIN  s:=0;...
New PL/PgSQL Statements●   RETURNS TABLE    –   SQL-compliant alias for "SETOF"●   CASE statement    –   real switching lo...
PL/pythonU OUT Parameters●   You now can use IN, OUT and INOUT    parameters with PL/pythonU functions.●   Thats it!
Exotic Features●   SQL/MED Connection Manager●   Multi-column GIN Indexes●   Boyer-Moore String Searching
SQL/MED●   Foundation for connecting to external servers    –   Future of PL/proxy and DBconnect    –   Future of DBI-Conn...
Multi-Column GIN Indexes●   Bad Old Days: to do a single Full Text Search    index over several columns, you had to    con...
Boyer-Moore String Searching
Boyer-Moore String Searching   No, I dont know what it is either.         But we have it now.
Security●   Refactored SSL●   Improved pg_hba.conf●   Column-level Permissions●   SE-Postgres
Refactored SSL              by Magnus●   Proper certificate verification    –   Choose level, full verification is default...
pg_hba Improvements●   "crypt" is gone (insecure)●   «ident sameuser» => «ident»●   New format for options    –   name=val...
Column PermissionsREVOKE SELECT (col1, col2), INSERT (col1, col2)  ON tab1 FROM role2;●   Restrict access to sensitive col...
PostgreSQL Needs YOU
Many Patches == Lots of Testing●   Bug Testing    –   can you make 8.4 crash?●   Specification Testing    –   do the featu...
Many Patches == Lots of Testing1. Take a copy of your production applications2. Port them to 8.43. Report breakage and iss...
Contact Information●   Josh Berkus                                       ●    Upcoming events    –   josh@postgresql.org  ...
Upcoming SlideShare
Loading in …5
×

8.4 Upcoming Features

947 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

8.4 Upcoming Features

  1. 1. Version 8.4Holy Frijole,Thats a lot of features! February 2009 San Franciso Josh Berkus, PostgreSQL Core Team
  2. 2. 8.4: A Few Patches
  3. 3. 8.4: A Few Patches 5 CommitFests
  4. 4. 8.4: A Few Patches 5 CommitFests9 Months of Development
  5. 5. 8.4: A Few Patches 5 CommitFests9 Months of DevelopmentOver 1600 GIT Updates
  6. 6. 8.4: A Few Patches 5 CommitFests 9 Months of Development Over 1600 GIT UpdatesMore Than 2 Dozen Major Features
  7. 7. Looks like some database projects still know how to put out new version.
  8. 8. Some-but-not-all 8.4 Features● Windowing Functions ● Unsigned Integers● Common Table Expressions ● Boyer-Moore String Searching● Parallel Restore ● Improved Hash Indexes● CIText ● More DTrace probes● array_agg ● Default & Variadic parameters● auto_explain ● New PL/pgSQL statements● SQL/MED connection manager ● pg_stat_statements● Per-Database Collations ● pg_stat_functions● d commands improved ● SSL refactor● Multicolumn GIN indexes ● pg_hba improvements● Column-level permissions ● Performance improvements
  9. 9. SQL Features● Windowing Functions● Common Table Expressions● array_agg● Per-database Collations● New data types – Unsigned Integers – CIText● Improved d commands● Add columns to existing VIEWs
  10. 10. Windowing Functions● Aggregate over part of the data – SQL 2008 standard – Great for BI, OLAP● Functions: – row_number() – rank() – lead() – lag()● More from David Fetter later!
  11. 11. Windowing FunctionsSELECT y, m, SUM(SUM(people)) OVER (PARTITION BY y ORDER BY m), AVG(people)FROM( SELECT EXTRACT(YEAR FROM accident_date) AS y, EXTRACT(MONTH FROM accident_date) AS m, * FROM accident SELECT)s depname,GROUP BY y, m; empno, salary, rank() OVER (PARTITION BY depname ORDER BY salary) FROM empsalary;
  12. 12. Common Table Expressions● Ability to create "named subqueries" for your query.● Best use: WITH RECURSIVE – real recursive queries – "walk" trees with one query● more from David Fetter later
  13. 13. Common Table ExpressionsWITH RECURSIVE subdepartment AS( -- SELECT * FROM department WHERE id = A UNION ALL -- recursive term referring to "subdepartment" SELECT d.* FROM department AS d, subdepartmentAS sd WHERE d.id = sd.parent_department)SELECT * FROM subdepartment;
  14. 14. array_agg● History: – added Arrays in 7.4 ● array_accum() aggregate example code – intarray contrib module in 8.0 ● only ints, but very fast● array_agg() in 8.4: all arrays, fast C Code from Robert Haas, new contributor! –SELECT status, array_agg(username) FROM logins GROUP BY status;
  15. 15. Per-Database Collations● Collations (ordering character sets) used to be per installation● Now they are per database● Someday they will be per column● Google Summer of Code Project!CREATE DATABASE mydbCOLLATE sv_se.UTF-8CTYPE sv_se.UTF-8TEMPLATE template0
  16. 16. New Data Types● Make migrating from other DBMSes easier● CIText (in /contrib) – Case Insensitive Text – Full CI indexing, comparisons● Unsigned Integers (in pgFoundry) – migrate from MySQL, others
  17. 17. Better d in psql● d is now multi-version compatible – dt etc. wont error if you connect an 8.4 client to an 8.2 database● df for user functions only – dfS for system functions● ef to edit a funcion
  18. 18. Add columns to VIEWs● In the bad old days: – need to add another column to your VIEW? – have to drop it & recreate it – have to drop & recreate all dependancies – enter the World Of Pain● In 8.4: – ALTER VIEW lets you add columns – Cant rename or modify though
  19. 19. Performance & Monitoring● Parallel Restore● Improved Hash Indexes● pg_stat_user_functions● pg_stat_statements● More Dtrace probes● auto_explain● Other Performance Improvements
  20. 20. Parallel Restore● In 8.3, we were single-threaded pg_dump dump Restore file 8 Hours
  21. 21. Parallel Restore● In 8.4, Multi-core, Restore! Restore Restore Restore Restore pg_dump dump Restore file Restore Restore Restore 2 Hours
  22. 22. Improved Hash Indexes● Our old hash indexes were slow and useless● Improved hash indexes are fast! – use them for ID columns ● or other unique keys – not completely recovery-safe yet though ● dont switch over production DBs until 8.5● Google Summer of Code project!
  23. 23. pg_stat_user_functions● For each of your functions, see – # of times called – amount of time spent – amount of time spent excluding other functions
  24. 24. pg_stat_statements
  25. 25. pg_stat_statements log log pgFouine file
  26. 26. pg_stat_statementspostgres=# SELECT * FROM pg_stat_statements ORDER BY total_time DESCLIMIT 3;-[ RECORD 1 ]------------------------------------------------------------userid | 10dbid | 63781query | UPDATE branches SET bbalance = bbalance + $1 WHERE bid = $2;calls | 3000total_time | 20.716706rows | 3000-[ RECORD 2 ]------------------------------------------------------------userid | 10dbid | 63781query | UPDATE tellers SET tbalance = tbalance + $1 WHERE tid = $2;calls | 3000total_time | 17.1107649999999rows | 3000-[ RECORD 3 ]------------------------------------------------------------userid | 10dbid | 63781query | UPDATE accounts SET abalance = abalance + $1 WHERE aid = $2;calls | 3000total_time | 0.645601rows | 3000
  27. 27. More DTrace Probes* Probes to measure query time * Probes to measure checkpoint stats such as running time,query-parse-start (int, char *) buffers written, xlog files added, removed, recycled, etcquery-parse-done (int, char *)query-plan-start () checkpoint-start (int)query-plan-done () checkpoint-done (int, int, int, int, int)query-execute-start ()query-execute-done () * Probes to measure Idle in Transaction and client/networkquery-statement-start (int, char *) timequery-statement-done (int, char *) idle-transaction-start (int, int) idle-transaction-done ()* Probes to measure dirty buffer writes by the backend because * Probes to measure sort timebgwriter is not effective sort-start (int, int, int, int, int) sort-done (int, long)dirty-buffer-write-start (int, int, int, int)dirty-buffer-write-done (int, int, int, int) * Probes to determine whether or not the deadlock detector* Probes to measure physical writes from the shared buffer has found a deadlockbuffer-write-start (int, int, int, int)buffer-write-done (int, int, int, int, int) deadlock-found () deadlock-notfound (int)* Probes to measure reads of a relation from a particular bufferblock * Probes to measure reads/writes by block numbers andbuffer-read-start (int, int, int, int, int) relationsbuffer-read-done (int, int, int, int, int, int) smgr-read-start (int, int, int, int) smgr-read-end (int, int, int, int, int, int)* Probes to measure the effectiveness of buffer caching smgr-write-start (int, int, int, int)buffer-hit () smgr-write-end (int, int, int, int, int, int)buffer-miss ()* Probes to measure I/O time because wal_buffers is too smallwal-buffer-write-start ()wal-buffer-write-done ()
  28. 28. auto_explain● misnamed; actually allows you to manually set specific queries/sessions/functions to output explain plans to the log postgres=# LOAD auto_explain; postgres=# SET auto_explain.log_min_duration = 0; postgres=# SELECT count(*) FROM pg_class, pg_index WHERE oid = indrelid AND indisunique; This might produce log output such as: LOG: duration: 0.986 ms plan: Aggregate (cost=14.90..14.91 rows=1 width=0) -> Hash Join (cost=3.91..14.70 rows=81 width=0) Hash Cond: (pg_class.oid = pg_index.indrelid) -> Seq Scan on pg_class (cost=0.00..8.27 rows=227 width -> Hash (cost=2.90..2.90 rows=81 width=4) -> Seq Scan on pg_index (cost=0.00..2.90 rows=81●
  29. 29. More Performance Improvements● Free Space Map is dynamically sized (no more max_fsm_pages!)● Visibility Map – VACUUM only changed pages – Index-only Scans in 8.5● Less writing to pgstat file – plus you can move it
  30. 30. Stored Procedures● Default Parameters● Variadic Parameters● New PL/pgSQL Statements● PL/pythonU OUT Parameters
  31. 31. DEFAULT parametersCREATE OR REPLACE FUNCTI ONadder ) a i nt de f a ul t 4 0 , b i nt de f a ul t 2 (RETURNS i nt LANGUAGE sql AS sel ect $ 1 + $ 2 ;SELECT adder ) ( ;SELECT adder ) 1( ;SELECT adder ) 1, 2( ;
  32. 32. VARIADIC parametersCREATE OR REPLACE FUNCTION adder(VARIADIC v int[]) RETURNS int AS $$DECLARE s int; i int;BEGIN s:=0; FOR i IN SELECT generate_subscripts(v,1) LOOP s := s + i; END LOOP;RETURN s;END;$$ LANGUAGE plpgsql;SELECT adder(1);SELECT adder(1,2,3);SELECT adder(40,2);
  33. 33. New PL/PgSQL Statements● RETURNS TABLE – SQL-compliant alias for "SETOF"● CASE statement – real switching logicCASE WHEN x BETWEEN 0 AND 10 THEN msg := value is between zero and ten; WHEN x BETWEEN 11 AND 20 THEN msg := value is between eleven and twenty;END CASE;
  34. 34. PL/pythonU OUT Parameters● You now can use IN, OUT and INOUT parameters with PL/pythonU functions.● Thats it!
  35. 35. Exotic Features● SQL/MED Connection Manager● Multi-column GIN Indexes● Boyer-Moore String Searching
  36. 36. SQL/MED● Foundation for connecting to external servers – Future of PL/proxy and DBconnect – Future of DBI-ConnectCREATE FOREIGN DATA WRAPPER pgsql LIBRARY pgsql_fdw;CREATE SERVER foo FOREIGN DATA WRAPPER pgsql OPTIONS (host remotehost, dbname remotedb);CREATE USER MAPPING FOR PUBLIC SERVER foo OPTIONS (username bob, password secret);
  37. 37. Multi-Column GIN Indexes● Bad Old Days: to do a single Full Text Search index over several columns, you had to concatenate them.● New Goodness: you can now do a proper multicolumn index – and its faster!
  38. 38. Boyer-Moore String Searching
  39. 39. Boyer-Moore String Searching No, I dont know what it is either. But we have it now.
  40. 40. Security● Refactored SSL● Improved pg_hba.conf● Column-level Permissions● SE-Postgres
  41. 41. Refactored SSL by Magnus● Proper certificate verification – Choose level, full verification is default● Control over all key and certificate files● SSL certificate authentication – Trusted root certificate – Map «cn» value of certificate
  42. 42. pg_hba Improvements● "crypt" is gone (insecure)● «ident sameuser» => «ident»● New format for options – name=value for all options● usermaps for all external methods – with regexp support● Parsed on reload
  43. 43. Column PermissionsREVOKE SELECT (col1, col2), INSERT (col1, col2) ON tab1 FROM role2;● Restrict access to sensitive columns from unprivileged ROLEs – more fine-grained security – no longer need to use VIEWs to do this
  44. 44. PostgreSQL Needs YOU
  45. 45. Many Patches == Lots of Testing● Bug Testing – can you make 8.4 crash?● Specification Testing – do the features do what the docs say they do?● Performance Testing – is 8.4 really faster? How much?● Combinational Testing – what happens when you put several new features together?
  46. 46. Many Patches == Lots of Testing1. Take a copy of your production applications2. Port them to 8.43. Report breakage and issues4. Play with implementing new features Do It Now! Were counting on you!
  47. 47. Contact Information● Josh Berkus ● Upcoming events – josh@postgresql.org – SCALE 7, Los – http://it.toolbox.com/ Angeles, Feb. 20 blogs/database-soup – pgCon 2009, Ottawa, May 20 This talk is copyright 2009 Josh Berkus, and is licensed under the Creative Commons Attribution License

×