These are the slides used by Dilip Kumar of EnterpriseDB for his presentation at pgDay Asia 2016, Singpaore. He talked about scalability and performance improvements in PostgreSQL v9.6, which is expected to be released in Dec/2016 - Jan/2017.

1. © 2013 EDB All rights reserved. 1
Scalability And Performance
Improvements In PostgreSQL 9.6
•
Dilip Kumar | 2016.03.17
PgDay Asia Singapore

2. Who am I ?
Dilip Kumar
 Currently working at EnterpriseDB
 Have worked to develop various features on PostgreSQL (for
internal projects) as well as on other In-House DB at Huawei.
 I have also contributed patches to community.
 Holds around 14 patents in my name in various DB technologies.
 I have presented a paper in PgCon 2015

3. 3
 Journey From 9.1 to 9.5
 What’s In 9.6 ?
 MVCC Scalability Improvement
 Clog Scalability
 Bulk Load Scalability
 Parallel Query
 Sorting Improvement
 Hash Lock Improvement
 Index Only Scan with partial Index
 Partial Sort
 Cache the Snapshot
 Buffer Header Spin Lock
Contents

4. Journey From 9.1 to 9.5

5. Whats’s In 9.6 ?
ProcArray Lock Contention Solved : Commited
Parallel Sequence Scan : Commited
Parallel NLJ and HJ : Commited
Clog Control Lock Issue : In Progress
Bulk Load Scalability : In Progress
Buffer Header Lock Issue : In progress
Hash Header Lock Contention : In Progress
Sorting Improvement using Quick Sort : In Progress
Checkpoint Continuous Flushing : In Progress
Index Only Scan with partial Index : In Progress
Caching the Snapshot : In Progress
Partial Sort : In Progress

6. MVCC Scalability Improvement
6
ProcArrayLock:
 Was the major contention point
reported in 9.5 and blocking Read
write workload to scale beyond 30
cores in TPCC.
 With Pgbench also After 30 cores
scalability was not linear.
1 8 16 32 64 128
0
5000
10000
15000
20000
25000
30000
pgbench -M prepared Median Of 30 mins of Runs Syncrhronous Commit=On
Head
Clients
TPS

7. MVCC Scalability Improvement
 Many Solutions were tried to
overcome this problem, like
CSN snapshot, Incremental
Snapshot.
 Finally Group clear Xid in
ProcArrayEnd Transaction
Successfully got committed for
9.6 version and Scaling is
almost linear upto 64 Clients in
64 thread machine.
1 8 16 32 64 128
0
5000
10000
15000
20000
25000
30000
35000
pgbench-MpreparedMedianOf30minsofRunsSyncrhronousCommit=On
Head
Patch
Clients
TPS

8. Clog Control Improvement
8
Clog Control Lock:
 Afer reducing ProcArrayLock contenton CLogControlLock become next
visible contenton Point.
 Contenton is mainly due to two reasons, one is that while writng the
transacton status in CLOG, it acquires EXCLUSIVE CLogControlLock which
contends with every other transacton which tries to access the CLOG for
checking transacton status and to reduce it. Second contenton is due to
the reason that when the CLOG page is not found in CLOG bufers, it needs
to acquire CLogControlLock in Exclusive mode which again contends with
shared lockers which tries to access the transacton status.
 Soluton Used for ClogControl Lock is Similar to the ProcArray Group Clear
XID.

9. Clog Control Improvement
9

10. Bulk Load Scalability
Relation Extension Lock:
 Currently Relation extension Lock is becoming bottleneck while extending
the relation in parallel.
 Both COPY and INSERT are Suffering From the same problem.
 Recently we are working on this various solutions are tries, and Currently
this is in Progress. (Lock Free Extension, Extend In multiple Blocks with
User Knob, Group Extend the Relation, Extend In multiple of Lock
Waiters).

11. Bulk Load Scalability
1 2 4 8 16 32 64
0
200
400
600
800
1000
1200
COPY10000 Record (4Bytes) Data Fits in Shared Buffers
Base
Patch
Clients
TPS
1 2 4 8 16 32 64
0
50
100
150
200
250
300
INSERT 1000 Records (1K) data doesn't fits In Shared Buffers
Base
Patch
Clients
TPS

12. Parallel Query
12
 Parallel Query is a Great Win for 9.6 this allows to run
single query in Parallel in multiple workers.
 Parallel Sequence Scan and Parallel Join are already
Committed and some are in progress like Parallel Index
Scan, Parallel Aggregate.
 Parallel Sequence Scan and Parallel Join has shown Great
Improvement for Single Query.

13. Parallel Query
13
Q3 Q4 Q5 Q7 Q8
0
2000
4000
6000
8000
10000
12000
14000
TPC-H Query with scale factor=5
Time in ms

14. Sorting Improvement
Using quicksort for every external sort:
This usage a quick sort instead of Replacement
Selection.
Replacing with quick sort which is cache
conscious algorithm, that improves the
performance significantly.

15. Sorting Improvement
1.7 3.5 7 14
0
50000
100000
150000
200000
250000
300000
350000
400000
Time Calculation for Using Quick Sort For All External Sort Work_mem= DEFAULT (4mb).
Head
Patch
Data Size (GB)
Time in (ms)

16. Hash Header Lock Contention
Hash Header mutex Contention:
 Postgres internal hash table is used for many key Operation.
 Heavy Weight Locks are managed in hash Tables.
 Buffer pool is managed by hash Tables.
 But whenever hash needs to allocate element from free list or
release to free list, there is one common lock and that become
main contention point.
 As part of this work this is converted to partitioned level freelist
now each partition have separate freelist and separate Locks.
 Freelist elements can be shared across partitions.

17. Hash Header Lock Contention
z
64 128
0
50000
100000
150000
200000
250000
300000
350000
PgBench: Scale Factor 300 Shared Buffer 512MB
Head
Patch
Cleints
TPS

18. Index only scan with partial index
What is Partial Index ?
 A partial index is an index built over a subset of a table; the subset is
defined by a conditional expression (called the predicate of the partial
index). The index contains entries for only those table rows that satisfy the
predicate.
 A major motivation for partial indexes is to avoid indexing common values.
Since a query searching for a common value (one that accounts for more
than a few percent of all the table rows) will not use the index anyway, there
is no point in keeping those rows in the index at all. This reduces the size of
the index, which will speed up queries that do use the index.

19. Index only scan with partial index
Problem:
 Currently partial indexes end up not using index only scans in most of the
cases.
 unless you include all columns from the index predicate to the index, the
planner will decide index only scans are not possible.
 Adding those columns which are not needed at runtime, will only increase
the index size.
Solution:
 Select the index only scan, if there are some key which are not really
required at runtime.

20. Partial Sort
 In PostgreSQL If all the sort keys are part of the index then
index scan will be selected for sorting.
 Otherwise scan all tuple from heap and Sort them
completely.
 As per this patch, mix both methods:
 get results from index in order which partially meets our
requirements
 do rest of work from heap.

21. Cache The Snapshot
 As per this Improvement whenever backend take a
snapshot its stored in a shared cache.
 Whenever any transaction is getting commited, saved
snapshot will be marked Invalid.
 When there is mix load of read and write that time if b/w
multiple read there is no write transaction, snapshot can be
reuse.
 When write transaction Invalidate the snapshot, First
transaction taking the snapshot will recalculate and update
the cached snapshot

22. Cache The Snapshot
64 88 128 256
0
5000
10000
15000
20000
25000
30000
35000
40000
Pgbench: Scale factor 300 Shared Buff=8GB
BASE
ONLY CLOG CHANGES
CLOG + SAVE SNAPSHOT
Clients
TPS

23. Buffer Header lock Improvement
Convert Pin/Unpin Spin Lock to Atomic Operation:
 Spin Lock inside the Pin and Unpin buffer is also one of the
bottleneck, While testing scalability in Big NUMA Machines.
 As per this patch Spin lock is converted to Atomic Operation and
Increasing/decreasing reference count is done using just one
ATOMIC operations.
 Other Operation which need to do more work other than just
updating state variable, that takes a Lock and Lock is using atomic
operation.

24. Buffer Header lock Improvement
1 8 16 32 64
0
100000
200000
300000
400000
500000
600000
700000
800000
pgbench read only test -M prepared S.F=1000 Shared Buffer=8GB
Head
Pached
Clients
TPS

25. Questions?
25

26. Thanks!
26