There's no shortage of physical replication solutions for PostgreSQL, they scale horizontally and provide high read availability. But where they fall short is write availability, which leads many users to consider PostgreSQL logical replication. Existing solutions have a single point of failure or are dependent on a forked, vendor-provided PostgreSQL extension making reliable, enterprise-class logical replication hard to come by. Furthermore, these solutions put limits on scaling PostgreSQL. By combining Kafka, an open source event streaming system with PostgreSQL, customers can get a fault tolerant, scalable logical replication service. Learn how EDB Replicate leverages Kafka for high write availability needed for today's demanding consumers who expect their applications to be always available and won't tolerate latency.

1. Overcoming Write
Availability Challenges of
PostgreSQL
Bhavani Rao, John Dalton
Hosted by: Molly Halsey

2. ● Slides and recording will be available in next 48 hours
● Submit questions via Zoom – will be answering at end
Welcome

3. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.3
Agenda
● High write availability
● Challenges with logical replication
● Solution
● Features and benefits
● Use cases
● Next steps

4. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.4
High Availability Read Availability Write Availability
What is write availability?
• Redundant DB and infra for
tolerating failures
• Multiple levels of
redundancy for higher avail
HA architecture tailored to
applications that have a high
read to write ratio
HA architecture designed for
applications with a
high write to read ratio

5. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.5
Replication to active writable standby
Write availability
• Fast failover to secondary database
• Application writing only to one database
• Typically deployed within one region
• Often required for financial applications
M2M1
App
Replication
Connects to M2
when M1 fails

6. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.6
• Provider: grounded in PostgreSQL,
replication, or other DBMS tech
• License: commercial or open source
• Support: self, community, or vendor
• Deployment: private, public, or multi-
cloud, single or multi-region
Database replication considerations
Many factors to consider
• Ecosystem: other database and systems in
your enterprise infrastructure
• Architecture: centralized or distributed
• Change capture mechanism: invasive or
extension
• Replication method: physical or logical, batch
or continuous

7. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.7
• Fine-grained control over replicating changes to
data objects, from selected rows to full database
• Node can be on different versions of PostgreSQL
• Bi-directional replication, supports writes to
multiple database nodes
• Interoperability with other database technologies
using ANSI SQL standards
Replication methods
Logical
• Binary data in write ahead logs (WAL) from all
schemas in active DB sent to replicas byte for byte
• Each node on same major PostgreSQL version
• Application writes to active database are
replicated to read only replicas
• Asynchronous and synchronous replication
supported
Physical
Best for Write AvailabilityBest for Read Availability

8. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.8
Challenges with logical replication tools
• Service must be co-located on same node as PostgreSQL server
• Has single point of failure
• Dependent on vendor provided or forked PostgreSQL
• Limited to DML only changes
• Difficult to scale horizontally

9. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.9
• Vertical scalability is always limited
• Horizontal scalability enables resiliency
• Multiple redundant managed nodes
enable service availability in case of a
node failure
• Scale replication node to workload
independent of DB node workload
Scalability and availability
Distributing computing architecture as enabler
Horizontal
Increase number
of instances
Vertical
Increase resource
size for node

10. Solution

11. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.11
Logical replication with EDB Replicate
WHY
HOW
WHAT
Protects data availability for mission critical applications to mitigate business risk
EDB Replicate is built with Kafka to help eliminate single points of failure and provide high write-
availability
Provides a scalable, fault tolerant logical data replication for single- and multi-master replication use
cases

12. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.12
How EDB Replicate works
• Each node can publish and subscribe to changes
• Logical decoding of WAL identifies SQL level changes
published via Kafka
• If a node fails another will take over
• Failed database resumes from last sync point, in case of
conflicts last update wins
• Anti-affinity deployment pattern is used, i.e., EDB
Replicate not co-located on database node
Availability
Zone 1
Availability
Zone 3
Availability
Zone 2
App App App
EDB
Replicate 1
EDB
Replicate 2
EDB
Replicate 3
EDB Replicate 3
takes over on
failure at EDB
Replicate 2
Cloud Service Provider

13. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.13
Software Architecture
Apache Kafka Fault-tolerant distributed streaming
replication
Apache ZooKeeper Distributed configuration management
Apache Avro Compressed data serialization and
schema management
Logical Decoding Logical change data capture component
of Postgres database server
Producer Streams from database to Kafka topic
Consumer Streams from Kafka topic to database
DB
App CLI
Logical
Decoding
EDB Replicate
Consumer
Producer
ZooKeeperController
Kafka Broker
Topic
Avro
REST API

14. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.14
Apache Kafka
Open source stream-processing platform
• De facto infrastructure standard for capturing and publishing events
• Used to integrate enterprise processes and workflows
• Kafka collects the ‘digital exhaust’ of an enterprise for ML and continuous intelligence
• Distributed architecture provides fault tolerance

15. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.15
Scalability and flexibility
Feature Benefit
• Horizontally scalable - add replication nodes (Kafka
brokers) with no physical machine limits
• Distributed - replication service is not co-located with
the production database
• Supports community PostgreSQL and EDB Postgres
Advanced Server
• Support different major versions of EDB Postgres
Advanced Server or PostgreSQL in the same cluster
• Increase processing horsepower to match replication
workload without degrading database performance
• Replication processes do not compete for the same
resources as the PostgreSQL instance
• Avoid vendor lock-in by using community PostgreSQL
• Supports rolling upgrades with minimal/zero downtime

16. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.16
Fault tolerance and performance
Feature Benefit
• Fault-tolerant - resistant to replication
node/machine failure within a cluster
• Supports bidirectional replication; writes on any
node are replicated to other PostgreSQL nodes
• End-to-end data compression with encryption -
Avro data storage format and Kafka data
compression
• No single point of failure. No need to retain WAL
logs when the replication node fails
• Enables complex architectures with multiple
masters, manages conflicts for data consistency
• Reduces message size sent between cluster nodes,
thereby reducing data latency

17. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.17
Improves usability
Feature Benefit
• Add new DBs without seeding from production
• Row-level filters
• Multiple publications per DBs in same cluster
• Email alerts based on thresholds, monitoring
and centralized event tracking
• High data availability without performance impact
• Replication can be fine grained and integrate data
from multiple DBs
• Changes from multiple database table groupings
can be replicated between DBs
• Simplified maintenance of large clusters

18. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.18
• Encrypted database connection
• SCRAM and certificate-based authentication
• ACL-based access control
• PostgreSQL 9.6, 10, 11, 12
• RHEL 7 & 8 on x86-64 as RPMs
• Bundled Kafka with compression options
Additional capabilities
Environmental
• CLI configuration management
• Insert, delete and row level
update-update conflicts managed
• Multiple publications per database
• Row level filtering
• Cascading replication
• Offline snapshots
Functional

19. Recommended
use cases

20. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.20
APAC EMEANA
• Data availability for applications operating
across multiple geographic regions
• Updates must be to local region data only, ie,
data model supports natural sharding
• All regions have complete data set
• Logistics applications are examples where
data may be needed in multiple regions
Geo-Distributed
Minimize application latency for users
M3M2M1
App App App
Replication

21. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.21
• For multiple applications with common
reference data
• Application of record maintains data in
master database
• Replicas are read only for reference data
used by applications
• Inventory applications are an example where
product catalog reference data is needed at a
retail store
Data Distribution
Accurate and consistent shared reference data
R2MR1
App1 App2 App3
Replication

22. Who is EDB?

23. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.23
EDB supercharges PostgreSQL
Privately Held
Major PostgreSQL
community leader
Over 5000
customers
The largest provider of open-source PostgreSQL
based software, services and support
Founded in
2004
Over 10 years of
consecutive quarterly
subscription growth
500+
employees
16 offices
worldwide

24. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.24 CONFIDENTIAL
News: EDB Acquires 2ndQuadrant
Now the largest dedicated provider of PostgreSQL products and solutions worldwide
Brings together world’s top PostgreSQL experts, combining two best-in-class technical teams
+

25. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.25
• Replication mitigates business risk by protecting data availability
for mission critical applications
• Modern PostgreSQL data replication must support the evolving
multi-cloud needs of enterprises
• EDB Replicate is designed for enterprise-class logical replication
Summary

26. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.26
Learn More
Other resources
EDB Replicate
General Availability late October
Postgres Pulse EDB Youtube
Channel
Contact EDB
info@edbpostgres.com