A Peek in the Elephant's Trunk

© 2014 EnterpriseDB Corporation. 1
A Peek in the Elephants Trunk
Dave Page | 2014-04-04

•  PostgreSQL
−  Core Team Member
−  pgAdmin Project Lead
−  Secretary, PostgreSQL Europe
−  Vice-chair, PostgreSQL Community Association of Canada
•  EnterpriseDB
−  Chief Architect, Tools & Installers
−  Tea boy
Dave Page

•  Discuss the idea on pgsql-hackers
−  Ensure no-one else is already doing the same work
−  Gain acceptance for the idea
−  Get consensus for the design and implementation plan
•  Post the patch to the mailing list
−  Add it to commitfest.postgresql.org
−  Discuss and implement improvements until everyone is happy
−  Reviews and feedback generally come during the commitfest
•  Breathe a sigh of relief as your work is committed
−  Profit! (when the next major release is fully baked)
Patch lifecycle

•  Most developers re-focus on reviewing and committing
•  Commitfests happen four times per year
−  June, September, November, January
−  Commitfests usually last a month
−  Except January, which is normally longer
Commitfests

(as of 1st April)
Statistics
•  4 commitfests closed
•  169 patches committed
•  39 patches rejected
•  24 patches awaiting review
•  15 patches ready for a committer
•  11 patches returned with feedback or waiting on the
author
•  341 patches in total (though some appear in multiple
commitfests)!

Hackers – feel free to chime in with additional details,
comments or corrections!!
So what’s new in 9.4?

Performance

•  XLogInsert scaling
−  Improves the scaling of WAL insert insert operations by
reducing lock contention. This allows for greater write scaling.
−  Author: Heikki Linnakangas
−  Reviewers: Andres Freund, Abhijit Menon-Sen
−  Committer: Heikki Linnakangas
Performance

•  Reduce WAL for update operations
−  Reduce the size of WAL records generated for UPDATE
operations, leading to less WAL data, reduced I/O etc.
−  Authors: Amit Kapila, Hari Babu
−  Reviewers: Robert Haas, Heikki Linnakangas
Performance

•  pg_prewarm
−  A contrib module that provides a pg_prewarm() SQL function
that can be used to load parts of relations into the OS cache
(prefetch or a sync read) or the PostgreSQL buffer cache. The
user can specify the relation, fork, and optionally start & end
blocks to load.
−  Author: Robert Haas
−  Reviewers: Álvaro Herrera, Amit Kapila, Jeff Janes, Gurjeet
Singh, and others
−  Committer: Robert Haas
Performance

psql (9.4devel)
Type "help" for help.
-- Install the extension
pgbench=# CREATE EXTENSION pg_prewarm;
CREATE EXTENSION
-- Prewarm the pgbench_accounts table by loading the
-- main fork (heap) into shared buffers
pgbench=# SELECT pg_prewarm('pgbench_accounts');
pg_prewarm
------------
163935
(1 row)
pg_prewarm - Example

Server Features

•  Extensible external TOAST tuple support
−  Infrastructure to allow future development of alternative
compression schemes for out-of-line TOAST tuples.
−  Author: Andres Freund
−  Reviewers: Hitoshi Harada, Robert Haas
Server Features

•  JSONB
−  This is a new JSON datatype that stores JSON documents in a
binary format that doesn't need to be re-parsed when
accessed. This allows for much faster querying and indexing.
−  Authors: Oleg Bartunov, Teodor Sigaev, Alexander Korotkov,
Peter Geoghegan and Andrew Dunstan
−  Reviewer: Andres Freund
−  Committer: Andrew Dunstan
Server Features

1 Using jsonb_hash_ops
2 Targeted index on a single key
3 Producing a count based on a search for a particular nested value
https://plus.google.com/+ThomBrownUK/posts/1JizRBGPYBq
JSONB - Thom’s quickie benchmark
JSON JSONB MongoDB
Table Size 1322MB 1375MB 1666MB
Index Size 630MB 283MB1 N/A
Targeted Index Size2 79MB 58MB1 N/A
Speed3 18.759ms 1.111ms 2ms

-- Create a GIN jsonb_hash_ops index
CREATE INDEX docs_idx ON docs USING GIN (doc jsonb_hash_ops);
-- Containment: Is one document contained within another?
postgres=# SELECT
postgres-# '{"product": "PostgreSQL", "version": 9.4}'::jsonb
postgres-# @>
postgres-# '{"version": 9.4}'::jsonb;
?column?
----------
t
(1 row)
JSONB - Examples

System Administration

•  ALTER SYSTEM
−  Allows the user to edit the server configuration file via SQL.
Changes are written to postgresql.auto.conf, which is read
after postgresql.conf and will override settings found there.
−  Author: Amit Kapila
−  Reviewer: Hari Babu
−  Committer: Tatsuo Ishii

-- Check what work_mem is currently set to
postgres=# SHOW work_mem;
work_mem
----------
8MB
(1 row)
-- Update the system config to set it to a higher value
postgres=# ALTER SYSTEM SET work_mem = '16MB';
ALTER SYSTEM
ALTER SYSTEM - Example

-- Reload the server configuration
postgres=# SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
-- Et voilà!
postgres=# SHOW work_mem;
work_mem
----------
16MB
(1 row)

viper:pgsql-dev dpage$ cat data/postgresql.auto.conf
# Do not edit this file manually!
# It will be overwritten by ALTER SYSTEM command.
work_mem = '16MB'

•  ALTER TABLESPACE...MOVE ALL TO...
−  Allows bulk moving of relations in tablespaces:
ALTER TABLESPACE name MOVE { ALL | TABLES |
INDEXES | MATERIALIZED VIEWS } [ OWNED BY
role_name [, ...] ] TO new_tablespace
[ NOWAIT ]
NOWAIT causes it to bail out if locks cannot be obtained
immediately. ALL means all to superusers.
−  Author: Stephen Frost
−  Reviewer: Robert Haas
−  Committer: Stephen Frost

•  Relocation of tablespaces in pg_base_backup
−  Adds a new "-T olddir=newdir" option to pg_basebackup to
allow tablespaces to be relocated easily.
−  Author: Steeve Lennmark
−  Reviewer: Peter Eisentraut
−  Committer: Peter Eisentraut

•  Base Backup throttling
−  Adds a new "--max-rate=N" option to pg_basebackup that
throttles the backup rate to N, where N is 32KB - 1024MB per
second. Affects the data directory and "fetched" WAL only.
Has no effect on streamed WAL.
−  Authors: Antonin Houska, Álvaro Herrera
−  Reviewers: Zoltán Böszörményi, Stefan Radomski, Andres
Freund, Fujii Masao
−  Committer: Álvaro Herrera

Monitoring

•  pg_stat_archiver statistics view
−  A new view that shows stats about the archiver: number of
WAL files archived, name of the last archived file, time of the
last successful archive operation, number of failed attempts,
name of the last file that failed, time of the last failure and the
time at which the stats were last reset.
−  Author: Gabriele Bartolini
−  Reviewers: Vik Fearing, Michael Paquier, Fujii Masao
−  Committer: Fujii Masao
Monitoring

•  Include planning time in EXPLAIN ANALYZE
−  Simply adds the planning time to the output from EXPLAIN
ANALYZE.
−  Author: Andreas Karlsson
−  Reviewer: Robert Haas
Monitoring

•  Improve EXPLAIN to show grouping columns in Agg/
Group nodes
−  Includes the list of grouping columns (or expressions) in
aggregate and group nodes in EXPLAIN output.
−  Author: Kevin Grittner
−  Committer: Tom Lane
Monitoring

-- Get the analysed query plan for a simple aggregate query
pgbench=# EXPLAIN ANALYZE SELECT sum(abalance) FROM
pgbench_accounts GROUP BY bid;
QUERY PLAN
---------------------------------------------------------------
HashAggregate (cost=313935.00..313936.00 rows=100 width=8)
(actual time=2666.421..2666.436 rows=100 loops=1)
Group Key: bid
-> Seq Scan on pgbench_accounts (cost=0.00..263935.00
rows=10000000 width=8) (actual time=0.006..646.185
rows=10000000 loops=1)
Planning time: 0.061 ms
Total runtime: 2666.480 ms
(5 rows)
EXPLAIN/EXPLAIN ANALYZE - Example

Replication

•  Recovery target = immediate
−  Adds a new recovery target option (immediate) that will cause
recovery to complete as soon as a consistent state is reached.
Previously you could recover to the end of the WAL (the
default), a named restore point, a timestamp or a transaction
ID.
−  Author: MauMau
Replication

•  Time delayed standbys
−  Adds a min_recovery_apply_delay option that can be used in
recovery.conf files on standby servers to delay replay of WAL
by the specified amount of time. This can be useful to recover
from user errors etc, by halting replication after an error has
been spotted, but before replay occurs on the standby.
−  Author: Fabrízio de Royes Mello
−  Reviewers: Mitsumasa KONDO, Jaime Casanova
−  Committer: Simon Riggs
Replication

•  Logical decoding
−  Allows logical changeset data to be extracted from the WAL
stream and consumed via SQL statements or using the
streaming replication protocol.
−  Author: Andres Freund
−  Reviewers: Kevin Grittner, Robert Haas, Tom Lane, Steve
Singer
Replication

-- Create a replication slot
pgbench=# SELECT * FROM
pg_create_logical_replication_slot('pgbench_slot',
'test_decoding');
slotname | xlog_position
--------------+---------------
pgbench_slot | 0/33D7B550
(1 row)
-- Check for replication data
pg_logical_slot_get_changes('pgbench_slot', NULL, NULL);
Location | xid | data
----------+-----+------
(0 rows)
Logical Decoding - Example

-- Do some work
pgbench=# INSERT INTO pgbench_branches VALUES (999, 0, NULL);
INSERT 0 1
-- Get the replication data
pg_logical_slot_get_changes('pgbench_slot', NULL, NULL);
location | xid | data
-----------+-------+------------------------------------------
0/33D7B550 | 57844 | BEGIN 57844
0/33D7B550 | 57844 | table public.pgbench_branches: INSERT:
bid[integer]:999 bbalance[integer]:0 filler[character]:null
0/33D7B620 | 57844 | COMMIT 57844
(3 rows)
-- Drop the replication slot
pgbench=# SELECT pg_drop_replication_slot('pgbench_slot');
pg_drop_replication_slot
--------------------------
(1 row)
Logical Decoding - Example

SQL

•  Refresh materialised views concurrently
−  Allows materialised views to be refreshed without affecting
concurrent reads, using transactional semantics.
−  Author: Kevin Grittner
−  Reviewer: Hitoshi Harada
−  Committer: Kevin Grittner
SQL

•  WITH CHECK option for auto-updateable views
−  At present you can insert a record into a view that would
actually be filtered out by the view, but is valid in the
underlying table. This adds options to allow views to validate
data and reject rows that wouldn't be visible. These checks
can be cascaded to underlying views as well.
−  Author: Dean Rasheed
−  Reviewer: Pavel Stehule
−  Committer: Stephen Frost
SQL

•  UNNEST()/WITH ORDINALITY
−  SQL standard syntax for adding a row number to unnest() and
any other SRF call.
−  Authors: Andrew Gierth, David Fetter
−  Reviewers: Dean Rasheed, Jeevan Chalke
−  Committer: Greg Stark
SQL

-- Un-nest an array, and add a row number
postgres=# SELECT * FROM unnest('{a,b,c,d,e,f}'::text[]) WITH
ORDINALITY;
unnest | ordinality
--------+------------
a | 1
b | 2
c | 3
d | 4
e | 5
f | 6
(6 rows)
UNNEST/WITH ORDINALITY - Example

•  WITHIN GROUP
−  This patch introduces generic support for ordered-set and
hypothetical-set aggregate functions, as well as
implementations of the instances defined in SQL:2008
(percentile_cont(), percentile_disc(), rank(), dense_rank(),
percent_rank(), cume_dist()). Also included are mode()
(though it is not in the spec), as well as versions of
percentile_cont() and percentile_disc() that can compute
multiple percentile values in one pass over the data.
−  Authors: Atri Sharma, Andrew Gierth
−  Reviewers: Pavel Stehule, Vik Fearing
−  Committer: Tom Lane
SQL

-- Create a test table
postgres=# CREATE TABLE t AS SELECT generate_series(1, 20) v;
SELECT 20
-- Find the most common value
postgres=# SELECT mode() WITHIN GROUP (ORDER BY v) FROM t;
mode
------
1
(1 row)
-- Insert another (duplicate) value
postgres=# INSERT INTO t (v) VALUES (15);
INSERT 0 1
-- Now find the most common value
postgres=# SELECT mode() WITHIN GROUP (ORDER BY v) FROM t;
mode
------
15
(1 row)
WITHIN GROUP – Example: mode()

-- Create a test table
postgres=# CREATE TABLE t AS SELECT generate_series(1, 20) v;
SELECT 20
-- Discrete Percentile - Select the first value whose position
-- in the ordering equals or exceeds the specified fraction
postgres=# SELECT percentile_disc(0.4) WITHIN GROUP (ORDER BY
v) AS q FROM t;
q
---
8
(1 row)
-- Continuous Percentile - Select a value corresponding to the
-- specified fraction in the ordering, interpolating if
-- necessary
postgres=# SELECT percentile_cont(0.4) WITHIN GROUP (ORDER BY
v) AS q FROM t;
q
-----
8.6
(1 row)
WITHIN GROUP – Example: percentile…()

Security

•  Remove Kerberos 5 authentication
−  Does exactly what it says on the tin. Kerberos 5 authentication
has been deprecated since 8.3 - users should use GSSAPI
instead.
−  Author: Magnus Hagander
−  Committer: Magnus Hagander
Security

•  Various SSL/TLS improvements, including:
−  Add TLS v1.1/1.2 support to the server
−  Remove SSL v3 support.
−  Change the default ciphersuite to HIGH:!aNULL
−  HIGH: Contains only secure and well-researched algorithms.
−  !aNULL: Needed to disable suites that do not authenticate server.
−  Author: Marko Kreen
−  Reviewers: Wim Lewis, Adrian Klaver
−  Committers: Tom Lane, Peter Eisentratut, Noah Misch
Security

Miscellaneous

•  Dynamic shared memory
−  This does not make shared_buffers dynamically resizeable!
What it does is allow the server to create new shared memory
segments on the fly, to allow communication between
processes. The driver for this is parallelism - e.g, for a backend
to communicate with worker processes that it might spawn.
−  Authors: Robert Haas, Amit Kapila
Miscellaneous

•  Dynamic background workers
−  This change allows new background workers to be started on
demand from a backend process, largely in preparation for
future parallel query features. Background workers can be
installed as extensions, and then started using an SQL
function call (or from the backend code of course). The
backend then signals the postmaster to start the required
processes.
−  Author: Robert Haas
−  Reviewer: Michael Paquier
Miscellaneous

And one more thing…

2014-03-15 22:35:15 EST LOG: loaded library "$libdir/plugins/plugin_debugger.so"
Ever seen this?

Lessen library-loading log level.
author Jeff Davis <jdavis@postgresql.org>
Sun, 24 Nov 2013 18:50:54 +0000 (10:50 -0800)
committer Jeff Davis <jdavis@postgresql.org>
Sun, 24 Nov 2013 18:50:54 +0000 (10:50 -0800)
Previously, messages were emitted at the LOG level every time a
backend preloaded a library. That was acceptable (though unnecessary)
for shared_preload_libraries; but it was excessive for
local_preload_libraries and session_preload_libraries. Reduce to
DEBUG1.
Also, there was logic in the EXEC_BACKEND case to avoid repeated
messages for shared_preload_libraries by demoting them to
DEBUG2. DEBUG1 seems more appropriate there, as well, so eliminate
that special case.
Peter Geoghegan.
Thank you Peter!

•  Depesz and Michael Paquier for the blogs
•  Thom Brown for comparing MongoDB to PostgreSQL
•  Developers for the awesome new features in 9.4
•  Reviewers for taking the time to test the patches
•  Committers for taking responsibility for the patches
Thank you…

A Peek in the Elephant's Trunk

More Related Content

What's hot

Similar to A Peek in the Elephant's Trunk

More from EDB

Recently uploaded

A Peek in the Elephant's Trunk