Pg_shardman provides PostgreSQL sharding via partitioning and foreign data wrappers (FDW), with high availability through logical replication and ACID transactions. It aims for horizontal scalability of reads and writes with an OLTP workload. Key features include hash-sharding data across nodes, replication for high availability, and distributed two-phase commit with a distributed snapshot manager to provide transactions across shards. It utilizes partitioning, FDW, and logical replication capabilities in PostgreSQL to provide a sharded cluster with transactions.

1. pg_shardman:
PostgreSQL sharding
via postgres_fdw,
pg_pathman and
logical replication.
Arseny Sher, Stas Kelvich
Postgres Professional

2. Read and write scalability
High availability
ACID transactions
What people typically expect from the cluster
2

3. CAP theorem: common myths
3

4. Informal statement: it is impossible to implement a read/write data object that provides
all three properties.
Consistency in CAP means linearizability
wow, so strict
Availability in CAP means that any node must give non-error answer to every
query.
... but execution can take arbitrary time
P in CAP means that the system continues operation after network failure
And in real life, we always want the system to continue operation after network
failure
CAP theorem: common myths
4

5. This combination of availability and consistency over the wide area is generally
considered impossible due to the CAP Theorem. We show how Spanner achieves this
combination and why it is consistent with CAP.
Eric Brewer. Spanner, TrueTime & The CAP Theorem. February 14, 2017
CAP theorem: conclusions
5

6. We aim for
Write (and read) horizontal scalability
Mainly OLTP workload with occasional analytical queries
Decent transactions
pg_shardman is PG 10 extension, PostgreSQL license, available at GitHub
Some features require patched Postgres
pg_shardman
6

7. pg_shardman is a compilation of several technologies.
Scalability: hash-sharding via partitioning and fdw
HA: logical replication
ACID: 2PC + distributed snapshot manager
pg_shardman foundations
7

8. Let’s go up from partitioning.
Because it’s like sharding, but inside one node.
Partitioning benefits
Sequential access to single (or a few) partitions instead of random access to huge
table
Effective cache usage when most frequently used data located in several partitions
...
Sharding
8

9. 9.6 and below:
Range and list partitioning, complex manual management
Not efficient
New declarative partitioning in 10:
+ Range and list partitioning with handy DDL
- No insertions to foreign partitions, no triggers on parent tables
- Updates moving tuples between partitions are not supported
pg_pathman extension:
Hash and range partitioning
Planning and execution optimizations
FDW support
Partitioning in PostgreSQL
9

10. Partitioning in PostgreSQL
10

11. FDW (foreign data wrappers) mechanism in PG gives access to external sources of
data. postgres_fdw extension allows querying one PG instance from another.
Going beyond one node: FDW
11

12. Since 9.6 postgres_fdw can push-down joins.
Since 10 postgres_fdw can push-down aggregates and more kinds of joins.
explain (analyze, costs off) select count(*)
from remote.customer
group by country_code;
QUERY PLAN
--------------------------------------------------------------
Foreign Scan (actual time=353.786..353.896 rows=100 loops=1)
Relations: Aggregate on (remote.customer)
postgres_fdw optimizations
12

13. Currently parallel foreign scans are not supported :(
... and limitations
13

14. partitioning + postgres_fdw => sharding
14

15. partitioning + postgres_fdw => sharding
15

16. pg_shardman supports only distribution by hash
It splits the load evenly
Currently it is impossible to change number of shards, it should be chosen
beforehand wisely
Too little shards will balance poorly after of nodes addition/removal
Too many shards bring overhead, especially for replication
~10 shards per node looks like adequate baseline
Another common approach for resharding is consistent hashing
Data distribution schemas
16

17. Possible schemas of replication
per-node, using streaming (physical) replication of PostgreSQL
High availability
17

18. 1
1
Taken from citus docs
Per-node replication in Citus MX
18

19. per-node, using streaming (physical) replication of PostgreSQL
Requires 2x nodes, or 2х PG instances per node.
Possible schemas of replication
19

20. per-node, using streaming (physical) replication of PostgreSQL
Requires 2x nodes, or 2х PG instances per node.
per-shard, using logical replication
Possible schemas of replication
20

21. Logical replication – new in PostgreSQL 10
21

22. Logical replication – new in PostgreSQL 10
22

23. Replicas in pg_shardman
23

24. Synchronous replication:
We don’t lose transactions reported as committed
Write it blocked if replica doesn’t respond
Slower
Currently we can reliably failover only if we have 1 replica per shard
Asynchronous replication:
Last committed transactions might be lost
Writes don’t block
Faster
Synchronous, asynchronous replication and
availability
24

25. Node addition with seamless rebalance
25

26. Node failover
26

27. We designate one special node ’sharlord’.
It holds tables with metadata.
Metadata can be synchronously replicated somewhere to change shardlord in case
of failure.
Currently shardlord can’t hold usual data itself.
How to manage this zoo
27

28. select shardman.add_node(’port=5433’);
select shardman.add_node(’port=5434’);
Example
28

29. select shardman.add_node(’port=5433’);
select shardman.add_node(’port=5434’);
create table pgbench_accounts (aid int not null, bid int, abalance int,
filler char(84));
select shardman.create_hash_partitions(’pgbench_accounts’,’aid’, 30, 1);
Example
29

30. [local]:5432 ars@ars:5434=# table shardman.partitions;
part_name | node_id | relation
---------------------+---------+------------------
pgbench_accounts_0 | 1 | pgbench_accounts
pgbench_accounts_1 | 2 | pgbench_accounts
pgbench_accounts_2 | 3 | pgbench_accounts
...
Example
30

31. [local]:5432 ars@ars:5434=# table shardman.replicas;
part_name | node_id | relation
---------------------+---------+------------------
pgbench_accounts_0 | 2 | pgbench_accounts
pgbench_accounts_1 | 3 | pgbench_accounts
pgbench_accounts_2 | 1 | pgbench_accounts
...
Example
31

32. Distributed transactions:
Distributed atomicity
Distributed isolation
Profit! (distributed)
Transactions in shardman
32

33. All reliable distributed systems are alike each unreliable is unreliable in its own way.
Kyle Kingsbury and Leo Tolstoy.
Transactions in shardman
33

34. Distributed transactions:
Atomicity: 2PC
Isolation: Clock-SI
Transactions in shardman
34

35. Transactions in shardman: 2PC
35

36. Two-phase commit is the anti-availability protocol.
P. Helland. ACM Queue, Vol. 14, Issue 2, March-April 2016.
Transactions in shardman: 2PC
36

37. Transactions in shardman: 2PC
37

38. Transactions in shardman: 2PC
38

39. Transactions in shardman: 2PC
39

40. Transactions in shardman: 2PC
40

41. So what we can do about it?
Make 2PC fail-recovery tolerant: X3PC, Paxos Commit
Back-up partitions!
Transactions in shardman: 2PC
41

42. Transactions in shardman: 2PC
42

43. Spanner mitigates this by having each member be a Paxos group, thus ensuring each
2PC “member” is highly available even if some of its Paxos participants are down.
Eric Brewer.
Transactions in shardman: 2PC
43

44. Profit? Not yet!
Transactions in shardman: isolation
44

45. Transactions in shardman: isolation
45

46. postgres_fdw.use_twophase = on
BEGIN;
UPDATE holders SET horns -= 1 WHERE holders.id = $id1;
UPDATE holders SET horns += 1 WHERE holders.id = $id2;
COMMIT;
SELECT sum(horns_count) FROM holders;
-> 1
-> -2
-> 0
Transactions in shardman: isolation
46

47. MVCC in two sentences:
UPDATE/DELETE create new tuple version, without in-place override
Each tx gets current database version at start (xid, csn,timestamp) and able to see
only appropriate versions.
acc1
ver 10: {1, 0}
ver 20: {1, 2}
ver 30: {1, 4}
––––– snapshot = 34 –––––
ver 40: {1, 2}
Transactions in shardman: isolation
47

48. BEGIN
Transactions in shardman: isolation
48

49. Do some serious stuff
Transactions in shardman: isolation
49

50. COMMIT
Transactions in shardman: isolation
50

51. BEGIN
Transactions in shardman: isolation
51

52. Do some serious web scale stuff
Transactions in shardman: isolation
52

53. COMMIT
Transactions in shardman: isolation
53

54. Transactions in shardman: Clock Skew
54

55. Clock-SI slightly changes visibility rules:
version = timestamp
Visibility’: Waits if tuple came from future. (Do not allow time-travel paradoxes!)
Visibility”: Waits if tuple already prepared(P) but not yet commited(C).
Commit’: Receives local versions from partitions on Prepare and Commits with
maximal version.
Transactions in shardman: isolation
55

56. 0 2 4 6 8 10 12 14
nodes
0
10000
20000
30000
40000
50000
TPS
pgbench -N on ec2 c3.2xlarge, client is oblivious about keys distribution
single node, no shardman
pg_shardman, no replication
pg_shardman, redundancy 1, async replication
Some benchmarks
56

57. pg_shardman with docs is available at github.com/postgrespro/pg_shardman
Report issues on GitHub
Some features require patched postgres
github.com/postgrespro/postgres_cluster/tree/pg_shardman
2PC and distributed snapshot manager
COPY FROM to sharded tables additionaly needs patched pg_pathman
We appreciate feedback!
57