A Peek in the Elephant's Trunk
Upcoming SlideShare
Loading in...5
×
 

A Peek in the Elephant's Trunk

on

  • 1,014 views

Get a peek into the new features expected to be in PostgreSQL 9.4 when it is released later this year.

Get a peek into the new features expected to be in PostgreSQL 9.4 when it is released later this year.

Statistics

Views

Total Views
1,014
Views on SlideShare
989
Embed Views
25

Actions

Likes
1
Downloads
13
Comments
0

3 Embeds 25

http://www.scoop.it 18
https://twitter.com 6
https://uplift.blackboard.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    A Peek in the Elephant's Trunk A Peek in the Elephant's Trunk Presentation Transcript

    • © 2014 EnterpriseDB Corporation. 1 A Peek in the Elephants Trunk Dave Page | 2014-04-04
    • © 2014 EnterpriseDB Corporation. 2 •  PostgreSQL −  Core Team Member −  pgAdmin Project Lead −  Secretary, PostgreSQL Europe −  Vice-chair, PostgreSQL Community Association of Canada •  EnterpriseDB −  Chief Architect, Tools & Installers −  Tea boy Dave Page
    • © 2014 EnterpriseDB Corporation. 3 •  Discuss the idea on pgsql-hackers −  Ensure no-one else is already doing the same work −  Gain acceptance for the idea −  Get consensus for the design and implementation plan •  Post the patch to the mailing list −  Add it to commitfest.postgresql.org −  Discuss and implement improvements until everyone is happy −  Reviews and feedback generally come during the commitfest •  Breathe a sigh of relief as your work is committed −  Profit! (when the next major release is fully baked) Patch lifecycle
    • © 2014 EnterpriseDB Corporation. 4 •  Most developers re-focus on reviewing and committing •  Commitfests happen four times per year −  June, September, November, January −  Commitfests usually last a month −  Except January, which is normally longer Commitfests
    • © 2014 EnterpriseDB Corporation. 5 (as of 1st April) Statistics •  4 commitfests closed •  169 patches committed •  39 patches rejected •  24 patches awaiting review •  15 patches ready for a committer •  11 patches returned with feedback or waiting on the author •  341 patches in total (though some appear in multiple commitfests)!
    • © 2014 EnterpriseDB Corporation. 6 Hackers – feel free to chime in with additional details, comments or corrections!! So what’s new in 9.4?
    • © 2014 EnterpriseDB Corporation. 7 Performance
    • © 2014 EnterpriseDB Corporation. 8 •  XLogInsert scaling −  Improves the scaling of WAL insert insert operations by reducing lock contention. This allows for greater write scaling. −  Author: Heikki Linnakangas −  Reviewers: Andres Freund, Abhijit Menon-Sen −  Committer: Heikki Linnakangas Performance
    • © 2014 EnterpriseDB Corporation. 9 •  Reduce WAL for update operations −  Reduce the size of WAL records generated for UPDATE operations, leading to less WAL data, reduced I/O etc. −  Authors: Amit Kapila, Hari Babu −  Reviewers: Robert Haas, Heikki Linnakangas −  Committer: Heikki Linnakangas Performance
    • © 2014 EnterpriseDB Corporation. 10 •  pg_prewarm −  A contrib module that provides a pg_prewarm() SQL function that can be used to load parts of relations into the OS cache (prefetch or a sync read) or the PostgreSQL buffer cache. The user can specify the relation, fork, and optionally start & end blocks to load. −  Author: Robert Haas −  Reviewers: Álvaro Herrera, Amit Kapila, Jeff Janes, Gurjeet Singh, and others −  Committer: Robert Haas Performance
    • © 2014 EnterpriseDB Corporation. 11 psql (9.4devel) Type "help" for help. -- Install the extension pgbench=# CREATE EXTENSION pg_prewarm; CREATE EXTENSION -- Prewarm the pgbench_accounts table by loading the -- main fork (heap) into shared buffers pgbench=# SELECT pg_prewarm('pgbench_accounts'); pg_prewarm ------------ 163935 (1 row) pg_prewarm - Example
    • © 2014 EnterpriseDB Corporation. 12 Server Features
    • © 2014 EnterpriseDB Corporation. 13 •  Extensible external TOAST tuple support −  Infrastructure to allow future development of alternative compression schemes for out-of-line TOAST tuples. −  Author: Andres Freund −  Reviewers: Hitoshi Harada, Robert Haas −  Committer: Robert Haas Server Features
    • © 2014 EnterpriseDB Corporation. 14 •  JSONB −  This is a new JSON datatype that stores JSON documents in a binary format that doesn't need to be re-parsed when accessed. This allows for much faster querying and indexing. −  Authors: Oleg Bartunov, Teodor Sigaev, Alexander Korotkov, Peter Geoghegan and Andrew Dunstan −  Reviewer: Andres Freund −  Committer: Andrew Dunstan Server Features
    • © 2014 EnterpriseDB Corporation. 15 1 Using jsonb_hash_ops 2 Targeted index on a single key 3 Producing a count based on a search for a particular nested value https://plus.google.com/+ThomBrownUK/posts/1JizRBGPYBq JSONB - Thom’s quickie benchmark JSON JSONB MongoDB Table Size 1322MB 1375MB 1666MB Index Size 630MB 283MB1 N/A Targeted Index Size2 79MB 58MB1 N/A Speed3 18.759ms 1.111ms 2ms
    • © 2014 EnterpriseDB Corporation. 16 -- Create a GIN jsonb_hash_ops index CREATE INDEX docs_idx ON docs USING GIN (doc jsonb_hash_ops); -- Containment: Is one document contained within another? postgres=# SELECT postgres-# '{"product": "PostgreSQL", "version": 9.4}'::jsonb postgres-# @> postgres-# '{"version": 9.4}'::jsonb; ?column? ---------- t (1 row) JSONB - Examples
    • © 2014 EnterpriseDB Corporation. 17 System Administration
    • © 2014 EnterpriseDB Corporation. 18 •  ALTER SYSTEM −  Allows the user to edit the server configuration file via SQL. Changes are written to postgresql.auto.conf, which is read after postgresql.conf and will override settings found there. −  Author: Amit Kapila −  Reviewer: Hari Babu −  Committer: Tatsuo Ishii System Administration
    • © 2014 EnterpriseDB Corporation. 19 -- Check what work_mem is currently set to postgres=# SHOW work_mem; work_mem ---------- 8MB (1 row) -- Update the system config to set it to a higher value postgres=# ALTER SYSTEM SET work_mem = '16MB'; ALTER SYSTEM ALTER SYSTEM - Example
    • © 2014 EnterpriseDB Corporation. 20 -- Reload the server configuration postgres=# SELECT pg_reload_conf(); pg_reload_conf ---------------- t (1 row) -- Et voilà! postgres=# SHOW work_mem; work_mem ---------- 16MB (1 row) ALTER SYSTEM - Example
    • © 2014 EnterpriseDB Corporation. 21 viper:pgsql-dev dpage$ cat data/postgresql.auto.conf # Do not edit this file manually! # It will be overwritten by ALTER SYSTEM command. work_mem = '16MB' ALTER SYSTEM - Example
    • © 2014 EnterpriseDB Corporation. 22 •  ALTER TABLESPACE...MOVE ALL TO... −  Allows bulk moving of relations in tablespaces: ALTER TABLESPACE name MOVE { ALL | TABLES | INDEXES | MATERIALIZED VIEWS } [ OWNED BY role_name [, ...] ] TO new_tablespace [ NOWAIT ] NOWAIT causes it to bail out if locks cannot be obtained immediately. ALL means all to superusers. −  Author: Stephen Frost −  Reviewer: Robert Haas −  Committer: Stephen Frost System Administration
    • © 2014 EnterpriseDB Corporation. 23 •  Relocation of tablespaces in pg_base_backup −  Adds a new "-T olddir=newdir" option to pg_basebackup to allow tablespaces to be relocated easily. −  Author: Steeve Lennmark −  Reviewer: Peter Eisentraut −  Committer: Peter Eisentraut System Administration
    • © 2014 EnterpriseDB Corporation. 24 •  Base Backup throttling −  Adds a new "--max-rate=N" option to pg_basebackup that throttles the backup rate to N, where N is 32KB - 1024MB per second. Affects the data directory and "fetched" WAL only. Has no effect on streamed WAL. −  Authors: Antonin Houska, Álvaro Herrera −  Reviewers: Zoltán Böszörményi, Stefan Radomski, Andres Freund, Fujii Masao −  Committer: Álvaro Herrera System Administration
    • © 2014 EnterpriseDB Corporation. 25 Monitoring
    • © 2014 EnterpriseDB Corporation. 26 •  pg_stat_archiver statistics view −  A new view that shows stats about the archiver: number of WAL files archived, name of the last archived file, time of the last successful archive operation, number of failed attempts, name of the last file that failed, time of the last failure and the time at which the stats were last reset. −  Author: Gabriele Bartolini −  Reviewers: Vik Fearing, Michael Paquier, Fujii Masao −  Committer: Fujii Masao Monitoring
    • © 2014 EnterpriseDB Corporation. 27 •  Include planning time in EXPLAIN ANALYZE −  Simply adds the planning time to the output from EXPLAIN ANALYZE. −  Author: Andreas Karlsson −  Reviewer: Robert Haas −  Committer: Robert Haas Monitoring
    • © 2014 EnterpriseDB Corporation. 28 •  Improve EXPLAIN to show grouping columns in Agg/ Group nodes −  Includes the list of grouping columns (or expressions) in aggregate and group nodes in EXPLAIN output. −  Author: Kevin Grittner −  Committer: Tom Lane Monitoring
    • © 2014 EnterpriseDB Corporation. 29 -- Get the analysed query plan for a simple aggregate query pgbench=# EXPLAIN ANALYZE SELECT sum(abalance) FROM pgbench_accounts GROUP BY bid; QUERY PLAN --------------------------------------------------------------- HashAggregate (cost=313935.00..313936.00 rows=100 width=8) (actual time=2666.421..2666.436 rows=100 loops=1) Group Key: bid -> Seq Scan on pgbench_accounts (cost=0.00..263935.00 rows=10000000 width=8) (actual time=0.006..646.185 rows=10000000 loops=1) Planning time: 0.061 ms Total runtime: 2666.480 ms (5 rows) EXPLAIN/EXPLAIN ANALYZE - Example
    • © 2014 EnterpriseDB Corporation. 30 Replication
    • © 2014 EnterpriseDB Corporation. 31 •  Recovery target = immediate −  Adds a new recovery target option (immediate) that will cause recovery to complete as soon as a consistent state is reached. Previously you could recover to the end of the WAL (the default), a named restore point, a timestamp or a transaction ID. −  Author: MauMau −  Committer: Heikki Linnakangas Replication
    • © 2014 EnterpriseDB Corporation. 32 •  Time delayed standbys −  Adds a min_recovery_apply_delay option that can be used in recovery.conf files on standby servers to delay replay of WAL by the specified amount of time. This can be useful to recover from user errors etc, by halting replication after an error has been spotted, but before replay occurs on the standby. −  Author: Fabrízio de Royes Mello −  Reviewers: Mitsumasa KONDO, Jaime Casanova −  Committer: Simon Riggs Replication
    • © 2014 EnterpriseDB Corporation. 33 •  Logical decoding −  Allows logical changeset data to be extracted from the WAL stream and consumed via SQL statements or using the streaming replication protocol. −  Author: Andres Freund −  Reviewers: Kevin Grittner, Robert Haas, Tom Lane, Steve Singer −  Committer: Robert Haas Replication
    • © 2014 EnterpriseDB Corporation. 34 -- Create a replication slot pgbench=# SELECT * FROM pg_create_logical_replication_slot('pgbench_slot', 'test_decoding'); slotname | xlog_position --------------+--------------- pgbench_slot | 0/33D7B550 (1 row) -- Check for replication data pgbench=# SELECT * FROM pg_logical_slot_get_changes('pgbench_slot', NULL, NULL); Location | xid | data ----------+-----+------ (0 rows) Logical Decoding - Example
    • © 2014 EnterpriseDB Corporation. 35 -- Do some work pgbench=# INSERT INTO pgbench_branches VALUES (999, 0, NULL); INSERT 0 1 -- Get the replication data pgbench=# SELECT * FROM pg_logical_slot_get_changes('pgbench_slot', NULL, NULL); location | xid | data -----------+-------+------------------------------------------ 0/33D7B550 | 57844 | BEGIN 57844 0/33D7B550 | 57844 | table public.pgbench_branches: INSERT: bid[integer]:999 bbalance[integer]:0 filler[character]:null 0/33D7B620 | 57844 | COMMIT 57844 (3 rows) -- Drop the replication slot pgbench=# SELECT pg_drop_replication_slot('pgbench_slot'); pg_drop_replication_slot -------------------------- (1 row) Logical Decoding - Example
    • © 2014 EnterpriseDB Corporation. 36 SQL
    • © 2014 EnterpriseDB Corporation. 37 •  Refresh materialised views concurrently −  Allows materialised views to be refreshed without affecting concurrent reads, using transactional semantics. −  Author: Kevin Grittner −  Reviewer: Hitoshi Harada −  Committer: Kevin Grittner SQL
    • © 2014 EnterpriseDB Corporation. 38 •  WITH CHECK option for auto-updateable views −  At present you can insert a record into a view that would actually be filtered out by the view, but is valid in the underlying table. This adds options to allow views to validate data and reject rows that wouldn't be visible. These checks can be cascaded to underlying views as well. −  Author: Dean Rasheed −  Reviewer: Pavel Stehule −  Committer: Stephen Frost SQL
    • © 2014 EnterpriseDB Corporation. 39 •  UNNEST()/WITH ORDINALITY −  SQL standard syntax for adding a row number to unnest() and any other SRF call. −  Authors: Andrew Gierth, David Fetter −  Reviewers: Dean Rasheed, Jeevan Chalke −  Committer: Greg Stark SQL
    • © 2014 EnterpriseDB Corporation. 40 -- Un-nest an array, and add a row number postgres=# SELECT * FROM unnest('{a,b,c,d,e,f}'::text[]) WITH ORDINALITY; unnest | ordinality --------+------------ a | 1 b | 2 c | 3 d | 4 e | 5 f | 6 (6 rows) UNNEST/WITH ORDINALITY - Example
    • © 2014 EnterpriseDB Corporation. 41 •  WITHIN GROUP −  This patch introduces generic support for ordered-set and hypothetical-set aggregate functions, as well as implementations of the instances defined in SQL:2008 (percentile_cont(), percentile_disc(), rank(), dense_rank(), percent_rank(), cume_dist()). Also included are mode() (though it is not in the spec), as well as versions of percentile_cont() and percentile_disc() that can compute multiple percentile values in one pass over the data. −  Authors: Atri Sharma, Andrew Gierth −  Reviewers: Pavel Stehule, Vik Fearing −  Committer: Tom Lane SQL
    • © 2014 EnterpriseDB Corporation. 42 -- Create a test table postgres=# CREATE TABLE t AS SELECT generate_series(1, 20) v; SELECT 20 -- Find the most common value postgres=# SELECT mode() WITHIN GROUP (ORDER BY v) FROM t; mode ------ 1 (1 row) -- Insert another (duplicate) value postgres=# INSERT INTO t (v) VALUES (15); INSERT 0 1 -- Now find the most common value postgres=# SELECT mode() WITHIN GROUP (ORDER BY v) FROM t; mode ------ 15 (1 row) WITHIN GROUP – Example: mode()
    • © 2014 EnterpriseDB Corporation. 43 -- Create a test table postgres=# CREATE TABLE t AS SELECT generate_series(1, 20) v; SELECT 20 -- Discrete Percentile - Select the first value whose position -- in the ordering equals or exceeds the specified fraction postgres=# SELECT percentile_disc(0.4) WITHIN GROUP (ORDER BY v) AS q FROM t; q --- 8 (1 row) -- Continuous Percentile - Select a value corresponding to the -- specified fraction in the ordering, interpolating if -- necessary postgres=# SELECT percentile_cont(0.4) WITHIN GROUP (ORDER BY v) AS q FROM t; q ----- 8.6 (1 row) WITHIN GROUP – Example: percentile…()
    • © 2014 EnterpriseDB Corporation. 44 Security
    • © 2014 EnterpriseDB Corporation. 45 •  Remove Kerberos 5 authentication −  Does exactly what it says on the tin. Kerberos 5 authentication has been deprecated since 8.3 - users should use GSSAPI instead. −  Author: Magnus Hagander −  Committer: Magnus Hagander Security
    • © 2014 EnterpriseDB Corporation. 46 •  Various SSL/TLS improvements, including: −  Add TLS v1.1/1.2 support to the server −  Remove SSL v3 support. −  Change the default ciphersuite to HIGH:!aNULL −  HIGH: Contains only secure and well-researched algorithms. −  !aNULL: Needed to disable suites that do not authenticate server. −  Author: Marko Kreen −  Reviewers: Wim Lewis, Adrian Klaver −  Committers: Tom Lane, Peter Eisentratut, Noah Misch Security
    • © 2014 EnterpriseDB Corporation. 47 Miscellaneous
    • © 2014 EnterpriseDB Corporation. 48 •  Dynamic shared memory −  This does not make shared_buffers dynamically resizeable! What it does is allow the server to create new shared memory segments on the fly, to allow communication between processes. The driver for this is parallelism - e.g, for a backend to communicate with worker processes that it might spawn. −  Authors: Robert Haas, Amit Kapila −  Committer: Robert Haas Miscellaneous
    • © 2014 EnterpriseDB Corporation. 49 •  Dynamic background workers −  This change allows new background workers to be started on demand from a backend process, largely in preparation for future parallel query features. Background workers can be installed as extensions, and then started using an SQL function call (or from the backend code of course). The backend then signals the postmaster to start the required processes. −  Author: Robert Haas −  Reviewer: Michael Paquier −  Committer: Robert Haas Miscellaneous
    • © 2014 EnterpriseDB Corporation. 50 And one more thing…
    • © 2014 EnterpriseDB Corporation. 51 2014-03-15 22:35:15 EST LOG: loaded library "$libdir/plugins/plugin_debugger.so" 2014-03-15 22:36:22 EST LOG: loaded library "$libdir/plugins/plugin_debugger.so" 2014-03-15 22:36:26 EST LOG: loaded library "$libdir/plugins/plugin_debugger.so" 2014-03-15 22:36:27 EST LOG: loaded library "$libdir/plugins/plugin_debugger.so" Ever seen this?
    • © 2014 EnterpriseDB Corporation. 52 Lessen library-loading log level. author Jeff Davis <jdavis@postgresql.org> Sun, 24 Nov 2013 18:50:54 +0000 (10:50 -0800) committer Jeff Davis <jdavis@postgresql.org> Sun, 24 Nov 2013 18:50:54 +0000 (10:50 -0800) Previously, messages were emitted at the LOG level every time a backend preloaded a library. That was acceptable (though unnecessary) for shared_preload_libraries; but it was excessive for local_preload_libraries and session_preload_libraries. Reduce to DEBUG1. Also, there was logic in the EXEC_BACKEND case to avoid repeated messages for shared_preload_libraries by demoting them to DEBUG2. DEBUG1 seems more appropriate there, as well, so eliminate that special case. Peter Geoghegan. Thank you Peter!
    • © 2014 EnterpriseDB Corporation. 53 •  Depesz and Michael Paquier for the blogs •  Thom Brown for comparing MongoDB to PostgreSQL •  Developers for the awesome new features in 9.4 •  Reviewers for taking the time to test the patches •  Committers for taking responsibility for the patches Thank you…
    • © 2014 EnterpriseDB Corporation. 54