PostgreSQL goes eleven

1. PostgreSQL Goes to Eleven!
Joe Conway
joe.conway@crunchydata.com
mail@joeconway.com
Crunchy Data
March 22, 2018

2. Community
PostgreSQL 11
12 and Beyond
History
Development Process
PostgreSQL Community and Future Development
Community
History
Development Process
PostgreSQL 11
Patches committed
Work inprocess
12 and Beyond
Actively worked items
Under active discussion
Joe Conway PGConf APAC 2018 2/37

3. Community
PostgreSQL 11
12 and Beyond
History
Development Process
University of California Berkeley
Courtesy: Bruce Momjian
Joe Conway PGConf APAC 2018 3/37

4. Community
PostgreSQL 11
12 and Beyond
History
Development Process
History: INGRES
1974 – 1985: University INGRES
INteractive Graphics REtrieval System
Prototype of a relational DBMS
University of California at Berkeley
Prof. Stonebraker
Spawned commercial databases: Sybase, MSSQL, NonStop SQL, others
1980 – Present: Commercialization
Commercial success
Relational Technologies ⇒ Ingres Corp.
⇒ Computer Associates ⇒ Ingres Corp. ⇒ Actian
2006 – Present: Open Source
Joe Conway PGConf APAC 2018 4/37

5. Community
PostgreSQL 11
12 and Beyond
History
Development Process
History: POSTGRES
1986 – 1994: University POSTGRES
”Post Ingres”
University of California at Berkeley
Prof. Stonebraker
Prototype of an object-relational DBMS
POSTQUEL query language
Spawned commercial databases: Illustra ⇒ Informix, others
Joe Conway PGConf APAC 2018 5/37

6. Community
PostgreSQL 11
12 and Beyond
History
Development Process
History: PostgreSQL
1994 – 1995: Postgres95
University of California at Berkeley
Andrew Yu and Jolly Chen
Conversion to SQL
More liberal license
1996 – present: PostgreSQL
Open-source project
PostgreSQL Global Development Group
Spawned many derived products
Joe Conway PGConf APAC 2018 6/37

7. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Originators
Courtesy: Bruce Momjian
Joe Conway PGConf APAC 2018 7/37

8. Community
PostgreSQL 11
12 and Beyond
History
Development Process
First Core Team + Jolly and Andrew
Courtesy: Bruce Momjian Joe Conway PGConf APAC 2018 8/37

9. Community
PostgreSQL 11
12 and Beyond
History
Development Process
PostgreSQL 10 Year Anniversary Summit - 2006
Courtesy: Alvaro Herrera
Joe Conway PGConf APAC 2018 9/37

10. Community
PostgreSQL 11
12 and Beyond
History
Development Process
PostgreSQL Community
Diagram Courtesy of Lætitia Avrot
Joe Conway PGConf APAC 2018 10/37

11. Community
PostgreSQL 11
12 and Beyond
History
Development Process
PostgreSQL Development Cycle
Note: future dates are estimates
Joe Conway PGConf APAC 2018 11/37

12. Community
PostgreSQL 11
12 and Beyond
History
Development Process
PostgreSQL Versioning
Old Versioning scheme
Up to version 9.6
Three segments
Major X.Y (e.g. 9.6)
Minor X.Y.Z (e.g. 9.6.8)
New Versioning scheme
Starting with version 10
Only two segments
Major X (e.g. 10)
Minor X.Z (e.g. 10.3)
Joe Conway PGConf APAC 2018 12/37

13. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Commitfests
Commitfest App: https://commitfest.postgresql.org/
CF-entries:
Patch or set of patches
Implements some goal
Proposed for inclusion
One or more authors/reviewers
Examples: new feature, bugfix, refactoring, improved docs
Joe Conway PGConf APAC 2018 13/37

14. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Commitfests
Goals:
Patches do not get lost
Reviewers/committers can quickly find patches deserving attention
Single place tracking discussion and documenting state of CF-entries
Encourage:
people review other people’s patches
review patches proportionally to patches submitted
Joe Conway PGConf APAC 2018 14/37

15. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Commitfests
CF-entry States:
Needs Review
Waiting for Author
Ready for Committer
Committed
Rejected
Returned with Feedback
Moved to next CF
Joe Conway PGConf APAC 2018 15/37

16. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Commitfests
Process:
Start on certain date
submission deadline date is day before start date, AoE (UTC-12)
deadline not passed if, anywhere on earth, deadline date has not yet passed
must be in Needs Review or Ready for Committer state
CF manager: ensures CF-entries in appropriate state, moves process along
Reviewers analyze/test patches, provide feedback
Authors adapt entries per reviews/discussion
When satisfied, reviewer marks CF-entry Ready for Committer
Committer either commits or resets state
Joe Conway PGConf APAC 2018 16/37

17. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Commitfests
Process:
Waiting for Author too long ⇒ Returned with Feedback
At end of commitfest:
Needs Review ⇒ Moved to next CF
Waiting on Author ⇒ Returned with Feedback
Joe Conway PGConf APAC 2018 17/37

18. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Commitfests
Last Commitfest:
CF before the expected feature freeze date
Extra restrictions:
No nontrivial CF-entries unless previously submitted to earlier CF
CF-entries deemed unlikely to finish in last CF aggressively be moved early
Joe Conway PGConf APAC 2018 18/37

19. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Commitfests
Joe Conway PGConf APAC 2018 19/37

20. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Release Stats
Note: as of late Feb 2018
Joe Conway PGConf APAC 2018 20/37

21. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Release Stats
Note: as of late Feb 2018
Joe Conway PGConf APAC 2018 21/37

22. Community
PostgreSQL 11
12 and Beyond
History
Development Process
Release Stats
Note: as of late Feb 2018
Joe Conway PGConf APAC 2018 22/37

23. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed by the Numbers (CF1-CF3)
Bug Fixes - 59
Clients - 22
ECPG - 2
pg dump - 3
pg receivewal - 2
pgbench - 7
psql - 8
Code Comments - 3
Documentation - 16
Miscellaneous - 22
Monitoring and Control - 3
Joe Conway PGConf APAC 2018 23/37

24. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed by the Numbers (CF1-CF3)
Performance - 26
caching - 3
index - 3
memory - 2
miscellaneous - 4
parallel query - 6
partitioning - 2
plan/opt - 6
Procedural Languages - 4
Refactoring - 27
Replication and Recovery - 5
Joe Conway PGConf APAC 2018 24/37

25. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed by the Numbers (CF1-CF3)
Security - 1
Server Features - 13
authentication - 1
miscellaneous - 4
parallel query - 1
partitioning - 6
security - 1
SQL Commands - 4
System Administration - 5
Joe Conway PGConf APAC 2018 25/37

26. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed - Notable Features (as of 17 March)
Partitioning
Many bug fixes and miscellaneous improvements
Hash Partitioning
Default partition
Partition-wise JOIN for partitioned tables
pg_dump/pg_restore reload through parent
Local partitioned indexes
UPDATE moves rows between partitions
UNIQUE indexes on partitioned tables
Joe Conway PGConf APAC 2018 26/37

27. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed - Notable Features (as of 17 March)
Parallelization
Many bug fixes and miscellaneous improvements
Queries with InitPlans
CREATE TABLE AS queries
Prepared statements with generic plans.
Queries with Append plan nodes
Hash joins
Btree index builds
Joe Conway PGConf APAC 2018 27/37

28. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed - Notable Features (as of 17 March)
Performance
Pushdown LIMIT through subqueries to underlying sort, where possible
SET STATISTICS on expression indexes
Automatic ”prewarm” for pg_prewarm
Configurable WAL segment size at initdb time
Index-only Bitmap scans
Generational Memory Allocator
Improve performance of MemoryContext creation
Support huge pages on Windows
Push down UPDATE/DELETE joins to remote postgres fdw servers
Clone extended stats in CREATE TABLE (LIKE INCLUDING ALL)
Joe Conway PGConf APAC 2018 28/37

29. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed - Notable Features (as of 17 March)
Logical replication
Many bug fixes and miscellaneous improvements
Testing
Coverage Analysis improvements
Additional tests
Improved code coverage
Authentication
Custom search filters for LDAP auth
LDAPS support
libpq connection parameter scram_channel_binding
SCRAM channel bindings tls-unique and tls-server-end-point
Joe Conway PGConf APAC 2018 29/37

30. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed - Notable Features (as of 17 March)
Miscellaneous
Build with Visual Studio 2017
Arrays over domains
Domains over composite types
Fewer superuser checks, more GRANT-based
Convert documentation to DocBook XML
SQL procedures with transaction control
Transaction control in PL procedures
User-callable SHA-2 functions
pg_stat_statements 64 bit queryid
Window frame clauses now full SQL:2011 support
Allow external command for obtaining passphrases for SSL key files
Joe Conway PGConf APAC 2018 30/37

31. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Already Committed - Notable Features (as of 17 March)
pgbench
Allow non-ASCII characters in variable names
Add approximated Zipfian-distributed random generator
Add pow(), aka power(), function
Improve scripting language in pgbench
psql
Use PSQL_PAGER in preference to PAGER if set
gdesc command
Variables to track success/failure of SQL
Test for variable existence
Joe Conway PGConf APAC 2018 31/37

32. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Patches proposed
Partitioning
Faster partition pruning
Runtime partition pruning (for prepared statements)
Partition-wise aggregation
Allow updating partition key
Support INSERT .. ON CONFLICT
Tuple routing for foreign partitions
Remove redundant query checks, based on partition
Foreign keys on partitioned tables
Foreign key arrays
Joe Conway PGConf APAC 2018 32/37

33. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Patches proposed - continued
Planning
Aggregation push-down
Convert JOIN OR-clauses into UNIONs
Remove LEFT JOINs in more cases
Remove useless DISTINCT clauses
Indexes
Incremental sort
Predicate locking in Gist/Hash indexes
REINDEX CONCURRENTLY
Covering Indexes
Joe Conway PGConf APAC 2018 33/37

34. Community
PostgreSQL 11
12 and Beyond
Committed
Proposed
Patches proposed - continued
Administration
Allow VACUUM to use more than 1G
Make relation extension lock cheaper
Generic WAL compression
Online enabling of checksums
Verify Checksums during Basebackups
SQL
MERGE
SQL/JSON Standard (functions, JSON_TABLE, jsonpath)
SQL ASSERTION
Performance
Just-In-Time (JIT) Compilation expressions and tuple deform
ALTER TABLE ADD COLUMN fast DEFAULT
Joe Conway PGConf APAC 2018 34/37

35. Community
PostgreSQL 11
12 and Beyond
Actively Being Worked
Ongoing Discussions
Actively Being Worked
64bit transaction IDs
Parallelize more operations (VACUUM?)
More improvements in partitioning (automatic creation?)
Database Encryption
Logical Replication conflict handling
Synchronous logical replication
Chained transactions
Generated columns
Alternate to OpenSSL (e.g. GnuTLS) Support
Joe Conway PGConf APAC 2018 35/37

36. Community
PostgreSQL 11
12 and Beyond
Actively Being Worked
Ongoing Discussions
Ongoing Discussions
Pluggable Storage
Columnar Storage
Global Indexes
Index-Organized Tables
Autonomous Transactions
Cron-like Background Worker
Building more on logical replication
Using FDWs from parallel workers
Asynchronous Append with FDWs
CREATE VARIABLE
Joe Conway PGConf APAC 2018 36/37

37. Community
PostgreSQL 11
12 and Beyond
Actively Being Worked
Ongoing Discussions
Questions?
Thank You!
joe.conway@credativ.com
mail@joeconway.com
Joe Conway PGConf APAC 2018 37/37