Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Grant McAlister – Senior Principal Engineer - RD...
RDS PostgreSQL Universe
CLIENTS
RDS
PostgreSQL
Aurora
PostgreSQL
EBS
Aurora
Storage
Postgres 9.6—same extensions
Backup/Re...
RDS PostgreSQL – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
RDS PostgreSQL – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
RDS PostgreSQL – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
cname update
Primary Update
RDS PostgreSQL – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
cname update
RDS PostgreSQL – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
cname update
16TB – 40,000 IOPS
RDS PostgreSQL – Read Replicas
Sync
Replication
Multi-AZ
Async Replication
RDS PostgreSQL – Read Replicas
Async Replication
RDS PostgreSQL – Read Replicas
Async Replication
AZ-1 AZ-2 AZ-3
Aurora - Storage and Replicas
RW
Application Application
Write log
records
AZ-1 AZ-2 AZ-3
Aurora - Storage and Replicas
RW
Application Application
Write log
records
Read
blocks
AZ-1 AZ-2 AZ-3
Aurora - Storage and Replicas
RW
Application Application
Write log
records
Read
blocks
AZ-1 AZ-2 AZ-3
Aurora - Storage and Replicas
RW
Application Application
Write log
records
Read
blocks
RO
Application
AZ-1 AZ-2 AZ-3
Aurora - Storage and Replicas
RW
Application
RO
Application
Async
Invalidation
& Update
Asyn...
RO
Application
AZ-1 AZ-2 AZ-3
Aurora - Storage and Replicas
RW
Application
RO
Application
Async
Invalidation
& Update
Asyn...
RO
Application
AZ-1 AZ-2 AZ-3
Aurora - Storage and Replicas
RW
Application
RO
Application
Async
Invalidation
& Update
Asyn...
Issues with 6 of 6 Replication
Location 1
Location 2
Location 3
Start
Issues with 6 of 6 Replication
Location 1
Location 2
Location 3
Start Finish
6
10
21
31
7
12
28
123
0
20
40
60
80
100
120
140
50 90 99.9 99.99
Latency(ms)
Percentile
High Concurrency Sync Write Test
...
Amazon Aurora Timing Example
Location 1
Location 2
Location 3
Start
Amazon Aurora Timing Example
Location 1
Location 2
Location 3
Start Finish
Only need 4/6
sync writes
Amazon Aurora Timing Example
Location 1
Location 2
Location 3
Start Finish
Only need 4/6
sync writes
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
Queued Work
Storage
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
Queued Work
Storage
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
Queued Work
Storage
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
Queued Work
Storage
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
Queued Work
Storage
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
Queued Work
Storage
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
A Queued Work
Storage
B
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
A
Queued Work
Storage
B C D E
2 ...
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
A
Queued Work
Storage
B C D E
4 ...
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
A
Queued Work
Storage
B C D E
6 ...
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
WAL
Block in
Memory
Aurora
Storage
Amazon S3
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6;
Full
Block
WAL
Block in
Memory
Aurora...
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6;
Full
Block
WAL
Block in
Memory
Aurora...
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6;
Checkpoint
Datafile
Full
Block
WAL
Bl...
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6;
Checkpoint
Datafile
Full
Block
WAL
Ar...
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6; update t set y = 6;
Checkpoint
Datafi...
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6; update t set y = 6;
Checkpoint
Datafi...
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6; update t set y = 6;
Checkpoint
Datafi...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Aurora Gives >2x Lower Response Times
0.0...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Aurora Recovers Up to 97%Faster
3 GiB Red...
Versions and Features
PostgreSQL Versions
RDS supports multiple major versions
Latest minor versions available
• 9.6.5 – Aurora Postgres 9.6.3
•...
Major version upgrade
Prod
9.5
Backup
Major version upgrade
Prod
9.5
pg_upgrade
Backup
Major version upgrade
Prod
9.5
pg_upgrade
Backup Backup
Major version upgrade
Prod
9.5
Prod
9.6
pg_upgrade
Backup Backup
No PITR
Major version upgrade
Prod
9.5
Test
9.5
pg_upgrade
Restore to a test instance
Major version upgrade
Prod
9.5
Test
9.5
Test
9.6
pg_upgrade
Restore to a test instance
Application
Testing
Major version upgrade
Prod
9.5
Prod
9.6
pg_upgrade
Backup Backup
No PITR
Test
9.5
Test
9.6
pg_upgrade
Restore to a test in...
Extension & Module Additions
rds-postgres-extensions-request@amazon.com
9.3 Original - 32
9.3 Current - 35
9.4 Current - 3...
New PostgreSQL Extensions Supported
Extensions Description
pgrouting Provides geospatial routing functionality for PostGIS...
log_fdw
set log_destination to csvlog
postgres=> create extension log_fdw;
postgres=> CREATE SERVER log_fdw_server FOREIGN...
log_fdw - continued
can be done without csv
postgres=> select
create_foreign_table_for_log_file('pg_log','log_fdw_server',...
pg_hint_plan
Add to shared_preload_libraries
• pg_hint_plan.debug_print
• pg_hint_plan.enable_hint
• pg_hint_plan.enable_h...
pg_hint_plan - example
postgres=> EXPLAIN SELECT * FROM pgbench_branches b
postgres-> JOIN pgbench_accounts a ON b.bid = a...
auto_explain (9.6.3+ only)
Verify auto_explain is in shared_preload_libraries
Set following values:
• auto_explain.log_min...
Security
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest...
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest...
Unencrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Unencrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
SnapshotDB
Instance
Snapshot
Share with account
Unencrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
SnapshotDB
Instance
Snapshot
Share with accoun...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Encryption at Rest
Default
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Share with account
Encryption at Rest
D...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Share with account
Encryption at Rest
C...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
SnapshotDB
Instance
Snapshot
Share with account
...
PostgreSQL Audit : pgaudit (9.6.3+)
CREATE ROLE rds_pgaudit;
Add pgaudit to shared_preload_libraries in parameter group
SE...
HIPAA-eligible service & FedRAMP
• RDS PostgreSQL & Aurora PostgreSQL are HIPAA-eligible services
• https://aws.amazon.com...
Replication
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS Snaps...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS Snaps...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
Postg...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
Postg...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
Postg...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
Postg...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
Postg...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
Postg...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas— RDS PostgreSQL
PostgreSQL
RW
EBS
Postg...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Amazon Aurora
Aurora
RW
Aurora Storage
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Amazon Aurora
Aurora
RW
PostgreSQL
RO
A...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Amazon Aurora
Aurora
RW
PostgreSQL
RO
u...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Amazon Aurora
Aurora
RW
PostgreSQL
RO
u...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Amazon Aurora
Aurora
RW
PostgreSQL
RO
u...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Amazon Aurora
Aurora
RW
PostgreSQL
RO
u...
RDS PostgreSQL - Cross Region Replicas
AZ1 AZ2
US-EAST-1
RDS PostgreSQL - Cross Region Replicas
AZ1 AZ2
US-EAST-1
RDS PostgreSQL - Cross Region Replicas
AZ1 AZ2 AZ1
Async Replication
US-EAST-1 EU-WEST-1
Data movement & Migration
Migration to RDS & Aurora PostgreSQL
Methods
• PostgreSQL - pg_dump / pg_restore
• AWS Data Migration Service (DMS) & Sche...
Schema Conversion Tool - SCT
Downloadable tool (Windows, Mac, Linux Desktop)
Source Database Target Database on Amazon RDS...
SCT - Analysis
SCT - Detailed
Customer
Premises,
EC2, RDS
Application Users
RDS &
Aurora
PostgreSQL
VPN
AWS DMS—Logical Replication
Customer
Premises,
EC2, RDS
Application Users
RDS &
Aurora
PostgreSQL
VPN
AWS DMS—Logical Replication
Start a replication ...
Customer
Premises,
EC2, RDS
Application Users
RDS &
Aurora
PostgreSQL
VPN
AWS DMS—Logical Replication
Start a replication ...
Customer
Premises,
EC2, RDS
Application Users
RDS &
Aurora
PostgreSQL
VPN
AWS DMS—Logical Replication
Start a replication ...
Customer
Premises,
EC2, RDS
Application Users
RDS &
Aurora
PostgreSQL
VPN
AWS DMS—Logical Replication
Start a replication ...
Customer
Premises,
EC2, RDS
Application Users
RDS &
Aurora
PostgreSQL
VPN
AWS DMS—Logical Replication
Start a replication ...
Customer
Premises,
EC2, RDS
Application Users
RDS &
Aurora
PostgreSQL
VPN
AWS DMS—Logical Replication
Start a replication ...
Migration—Snapshot Import
Migration—Snapshot Import
Migration—Snapshot Import
Snapshot
Migration—Snapshot Import
Snapshot
Migration—Snapshot Import
Migration—Snapshot Import
Migration—Snapshot Import
Logical Replication Support – RDS PostgreSQL
• Supported with 9.6.1+, 9.5.4+ and 9.4.9+
• Set rds.logical_replication para...
Logical Decoding Space Usage
CloudWatch – Slot usage for WAL
Logical Replication Support - Example
RDS
Postgres
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
DMS
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
EC2
Postgres
On Premise
Postgres
...
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
EC2
Postgres
On Premise
Postgres
...
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
EC2
Postgres
On Premise
Postgres
...
Vacuuming
Vacuum parameters
Will auto vacuum when
• autovacuum_vacuum_threshold +
autovacuum_vacuum_scale_factor * pgclass.reltuples...
RDS autovacuum logging (9.4.5+)
log_autovacuum_min_duration = 5000 (i.e. 5 secs)
rds.force_autovacuum_logging_level = LOG
...
RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2)
pg_stat_activity
BEFORE
usename | query
----------+-----------------------...
RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2)
pg_stat_activity
BEFORE
usename | query
----------+-----------------------...
RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2)
pg_stat_activity
BEFORE
usename | query
----------+-----------------------...
CloudWatch Metric
Aurora - Intelligent Vacuum Prefetch
PostgreSQL
Aurora PostgreSQL
Aurora - Intelligent Vacuum Prefetch
PostgreSQL
Aurora PostgreSQL
Aurora - Intelligent Vacuum Prefetch
PostgreSQL
Aurora PostgreSQL
Aurora - Intelligent Vacuum Prefetch
PostgreSQL
Aurora PostgreSQL
Aurora - Intelligent Vacuum Prefetch
PostgreSQL
Aurora PostgreSQL
Submit
Batch I/O
Aurora - Intelligent Vacuum Prefetch
PostgreSQL
Aurora PostgreSQL
Submit
Batch I/O
402 seconds
163 seconds
Performance Monitoring
Enhanced Operating System (OS) metrics
1-60 second granularity
cpuUtilization
• guest
• irq
• system
• wait
• idl:
• user
...
Enhanced Monitoring - Process List
OS metrics
Performance Insights
Performance Insights—Zoom
Performance Insights—Bad Query
Thank you!
Questions?
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1

Share

Deep dive into the Rds PostgreSQL Universe Austin 2017

Download to read offline

A deep dive into the two RDS PostgreSQL offerings, RDS PostgreSQL and Aurora PostgreSQL. Covering what is common between the engines, what is different and updates that we have done over the past year.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Deep dive into the Rds PostgreSQL Universe Austin 2017

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Grant McAlister – Senior Principal Engineer - RDS December 2017 Deep Dive into the RDS PostgreSQL Universe
  2. 2. RDS PostgreSQL Universe CLIENTS RDS PostgreSQL Aurora PostgreSQL EBS Aurora Storage Postgres 9.6—same extensions Backup/Recovery - PITR High Availability & Durability Secure Read Replicas Cross Region Snapshots Scale Compute – Online Scale Storage Cross Region Replication Outbound Logical Replication
  3. 3. RDS PostgreSQL – Multi-AZ Physical Synchronous Replication AZ1 AZ2
  4. 4. RDS PostgreSQL – Multi-AZ Physical Synchronous Replication AZ1 AZ2
  5. 5. RDS PostgreSQL – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update Primary Update
  6. 6. RDS PostgreSQL – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update
  7. 7. RDS PostgreSQL – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update 16TB – 40,000 IOPS
  8. 8. RDS PostgreSQL – Read Replicas Sync Replication Multi-AZ Async Replication
  9. 9. RDS PostgreSQL – Read Replicas Async Replication
  10. 10. RDS PostgreSQL – Read Replicas Async Replication
  11. 11. AZ-1 AZ-2 AZ-3 Aurora - Storage and Replicas RW Application Application Write log records
  12. 12. AZ-1 AZ-2 AZ-3 Aurora - Storage and Replicas RW Application Application Write log records Read blocks
  13. 13. AZ-1 AZ-2 AZ-3 Aurora - Storage and Replicas RW Application Application Write log records Read blocks
  14. 14. AZ-1 AZ-2 AZ-3 Aurora - Storage and Replicas RW Application Application Write log records Read blocks
  15. 15. RO Application AZ-1 AZ-2 AZ-3 Aurora - Storage and Replicas RW Application RO Application Async Invalidation & Update Async Invalidation & Update Write log records Read blocks
  16. 16. RO Application AZ-1 AZ-2 AZ-3 Aurora - Storage and Replicas RW Application RO Application Async Invalidation & Update Async Invalidation & Update Write log records Read blocks
  17. 17. RO Application AZ-1 AZ-2 AZ-3 Aurora - Storage and Replicas RW Application RO Application Async Invalidation & Update Async Invalidation & Update Write log records Read blocks RW Automatic Scalable Storage to 64TB
  18. 18. Issues with 6 of 6 Replication Location 1 Location 2 Location 3 Start
  19. 19. Issues with 6 of 6 Replication Location 1 Location 2 Location 3 Start Finish
  20. 20. 6 10 21 31 7 12 28 123 0 20 40 60 80 100 120 140 50 90 99.9 99.99 Latency(ms) Percentile High Concurrency Sync Write Test 2 Node (4 copy) 3 Node (6 Copy) Cost of Additional Synchronous Replicas
  21. 21. Amazon Aurora Timing Example Location 1 Location 2 Location 3 Start
  22. 22. Amazon Aurora Timing Example Location 1 Location 2 Location 3 Start Finish Only need 4/6 sync writes
  23. 23. Amazon Aurora Timing Example Location 1 Location 2 Location 3 Start Finish Only need 4/6 sync writes
  24. 24. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage Queued Work Storage
  25. 25. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage Queued Work Storage
  26. 26. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage Queued Work Storage
  27. 27. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage Queued Work Storage
  28. 28. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage Queued Work Storage
  29. 29. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage Queued Work Storage
  30. 30. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage A Queued Work Storage B
  31. 31. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage A Queued Work Storage B C D E 2 2 1 0 1 A B C D E Durability Tracking
  32. 32. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage A Queued Work Storage B C D E 4 3 4 2 4 A B C D E Durability Tracking
  33. 33. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage A Queued Work Storage B C D E 6 5 6 3 5 A B C D E Durability Tracking
  34. 34. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora WAL Block in Memory Aurora Storage Amazon S3
  35. 35. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; Full Block WAL Block in Memory Aurora Storage Amazon S3
  36. 36. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; Full Block WAL Block in Memory Aurora Storage Amazon S3
  37. 37. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; Checkpoint Datafile Full Block WAL Block in Memory Aurora Storage Amazon S3
  38. 38. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; Checkpoint Datafile Full Block WAL Archive Block in Memory Aurora Storage Amazon S3
  39. 39. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; update t set y = 6; Checkpoint Datafile Full Block WAL Archive Block in Memory Aurora Storage Amazon S3
  40. 40. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; update t set y = 6; Checkpoint Datafile Full Block WAL Archive Block in Memory Aurora Storage Amazon S3
  41. 41. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; update t set y = 6; Checkpoint Datafile Full Block WAL Archive Block in Memory Aurora Storage no checkpoint = no FPW Amazon S3
  42. 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Aurora Gives >2x Lower Response Times 0.00 100.00 200.00 300.00 400.00 500.00 600.00 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 responsetime,ms minute sysbench response time (p95), 30 GiB, 1024 clients PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)
  43. 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Aurora Recovers Up to 97%Faster 3 GiB Redo Recovered in 19 seconds 10 GiB Redo Recovered in 50 seconds 30 GiB Redo Recovered in 123 seconds 0 20 40 60 80 100 120 140 160 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 RecoveryTimeinSeconds(lessisbetter) Writes / Second (more is better) RECOVERY TIME FROM CRASH UNDER LOAD Bubble size represents redo log, which must be recovered As PostgreSQL throughput goes up, so does log size and crash recovery time Amazon Aurora has no redo. Recovered in 3 seconds while maintaining significantly greater throughput.
  44. 44. Versions and Features
  45. 45. PostgreSQL Versions RDS supports multiple major versions Latest minor versions available • 9.6.5 – Aurora Postgres 9.6.3 • 9.5.9 • 9.4.14 • 9.3.19 Next major release - PostgreSQL 10
  46. 46. Major version upgrade Prod 9.5 Backup
  47. 47. Major version upgrade Prod 9.5 pg_upgrade Backup
  48. 48. Major version upgrade Prod 9.5 pg_upgrade Backup Backup
  49. 49. Major version upgrade Prod 9.5 Prod 9.6 pg_upgrade Backup Backup No PITR
  50. 50. Major version upgrade Prod 9.5 Test 9.5 pg_upgrade Restore to a test instance
  51. 51. Major version upgrade Prod 9.5 Test 9.5 Test 9.6 pg_upgrade Restore to a test instance Application Testing
  52. 52. Major version upgrade Prod 9.5 Prod 9.6 pg_upgrade Backup Backup No PITR Test 9.5 Test 9.6 pg_upgrade Restore to a test instance Application Testing
  53. 53. Extension & Module Additions rds-postgres-extensions-request@amazon.com 9.3 Original - 32 9.3 Current - 35 9.4 Current - 39 9.5 Current - 46 Future - ??? 9.6 Current - 57
  54. 54. New PostgreSQL Extensions Supported Extensions Description pgrouting Provides geospatial routing functionality for PostGIS postgresql-hll HyperLogLog data type support pg_repack Remove bloat from tables and indexes in version 9.6.3 pgaudit Provide detailed session and object audit logging in versions 9.6.3 and 9.5.7 auto_explain Log execution plans of slow statements automatically in versions 9.6.3 and 9.5.7 pg_hint_plan Provides control of execution plans by using hint phrases log_fdw Extension to query your database engine logs within the database pg_freespacemap Examine free space map decoder_raw Output plugin to generates raw queries for logical replication changes wal2json Output plugin for logical decoding in versions 9.6.3+ and 9.5.7+
  55. 55. log_fdw set log_destination to csvlog postgres=> create extension log_fdw; postgres=> CREATE SERVER log_fdw_server FOREIGN DATA WRAPPER log_fdw; postgres=> select * from list_postgres_log_files(); file_name | file_size_bytes ----------------------------------+----------------- postgresql.log.2017-03-28-17.csv | 2068 postgres.log | 617 postgres=> select create_foreign_table_for_log_file('pg_csv_log','log_fdw_server','postgresql.log.2017-03-28-17.csv'); postgres=> select log_time, message from pg_csv_log where message like 'connection%'; log_time | message ----------------------------+-------------------------------------------------------------------------------- 2017-03-28 17:50:01.862+00 | connection received: host=ec2-54-174-205.compute-1.amazonaws.com port=45626 2017-03-28 17:50:01.868+00 | connection authorized: user=mike database=postgres
  56. 56. log_fdw - continued can be done without csv postgres=> select create_foreign_table_for_log_file('pg_log','log_fdw_server','postgresql.log.2017-03-28-17'); postgres=> select log_entry from pg_log where log_entry like '%connection%'; log_entry ----------------------------------------------------------------------------------------------------------------------------- ----------------------- 2017-03-28 17:50:01 UTC:ec2-54-174.compute-1.amazonaws.com(45626):[unknown]@[unknown]:[20434]:LOG: received: host=ec2-54-174-205..amazonaws.com 2017-03-28 17:50:01 UTC:ec2-54-174.compute-1.amazonaws.com(45626):mike@postgres:[20434]:LOG: connection authorized: user=mike database=postgres 2017-03-28 17:57:44 UTC:ec2-54-174.compute-1.amazonaws.com(45626):mike@postgres:[20434]:ERROR: column "connection" does not exist at character 143
  57. 57. pg_hint_plan Add to shared_preload_libraries • pg_hint_plan.debug_print • pg_hint_plan.enable_hint • pg_hint_plan.enable_hint_table • pg_hint_plan.message_level • pg_hint_plan.parse_messages
  58. 58. pg_hint_plan - example postgres=> EXPLAIN SELECT * FROM pgbench_branches b postgres-> JOIN pgbench_accounts a ON b.bid = a.bid ORDER BY a.aid; QUERY PLAN ------------------------------------------------------------------------------------------- Sort (cost=15943073.17..15993073.17 rows=20000000 width=465) Sort Key: a.aid -> Hash Join (cost=5.50..802874.50 rows=20000000 width=465) Hash Cond: (a.bid = b.bid) -> Seq Scan on pgbench_accounts a (cost=0.00..527869.00 rows=20000000 width=97) -> Hash (cost=3.00..3.00 rows=200 width=364) -> Seq Scan on pgbench_branches b (cost=0.00..3.00 rows=200 width=364) postgres=> /*+ NestLoop(a b) */ postgres-> EXPLAIN SELECT * FROM pgbench_branches b postgres-> JOIN pgbench_accounts a ON b.bid = a.bid ORDER BY a.aid; QUERY PLAN ------------------------------------------------------------------------------------------------------------------- Nested Loop (cost=0.58..44297240.44 rows=20000000 width=465) -> Index Scan using pgbench_accounts_pkey on pgbench_accounts a (cost=0.44..847232.44 rows=20000000 width=97) -> Index Scan using pgbench_branches_pkey on pgbench_branches b (cost=0.14..2.16 rows=1 width=364) Index Cond: (bid = a.bid)
  59. 59. auto_explain (9.6.3+ only) Verify auto_explain is in shared_preload_libraries Set following values: • auto_explain.log_min_duration = 5000 • auto_explain.log_nested_statements = on
  60. 60. Security
  61. 61. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest
  62. 62. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest ssl_mode=disable
  63. 63. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest ssl_mode=disable rds.force_ssl=1 (default 0)
  64. 64. Unencrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account
  65. 65. Unencrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account SnapshotDB Instance Snapshot Share with account
  66. 66. Unencrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account SnapshotDB Instance Snapshot Share with account Share to Public
  67. 67. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account
  68. 68. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Encryption at Rest Default
  69. 69. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Share with account Encryption at Rest Default
  70. 70. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Share with account Encryption at Rest Custom Key Add external account
  71. 71. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account SnapshotDB Instance Snapshot Share with account Encryption at Rest Custom Key Add external account
  72. 72. PostgreSQL Audit : pgaudit (9.6.3+) CREATE ROLE rds_pgaudit; Add pgaudit to shared_preload_libraries in parameter group SET pgaudit.role = rds_pgaudit; CREATE EXTENSION pgaudit; For tables to be enabled for auditing: GRANT SELECT ON table1 TO rds_pgaudit; Database logs will show entry as follows: 2017-06-12 19:09:49 UTC:…:pgadmin@postgres:[11701]:LOG: AUDIT: OBJECT,1,1,READ,SELECT,TABLE,public.t1,select * from t1; ...
  73. 73. HIPAA-eligible service & FedRAMP • RDS PostgreSQL & Aurora PostgreSQL are HIPAA-eligible services • https://aws.amazon.com/compliance/hipaa-compliance/ • RDS PostgreSQL - FedRAMP in AWS GovCloud (US) region • https://aws.amazon.com/compliance/fedramp/
  74. 74. Replication
  75. 75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS
  76. 76. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS Snapshot
  77. 77. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS Snapshot EBS
  78. 78. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS PostgreSQL RO EBS
  79. 79. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS PostgreSQL RO EBS Catchup
  80. 80. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS PostgreSQL RO EBS update
  81. 81. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS PostgreSQL RO EBS update
  82. 82. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS PostgreSQL RO EBS update Async Replication
  83. 83. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS PostgreSQL RO EBS update Async Replication
  84. 84. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas— RDS PostgreSQL PostgreSQL RW EBS PostgreSQL RO EBS update Async Replication
  85. 85. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Amazon Aurora Aurora RW Aurora Storage
  86. 86. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Amazon Aurora Aurora RW PostgreSQL RO Aurora Storage
  87. 87. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Amazon Aurora Aurora RW PostgreSQL RO update Aurora Storage
  88. 88. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Amazon Aurora Aurora RW PostgreSQL RO update Aurora Storage
  89. 89. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Amazon Aurora Aurora RW PostgreSQL RO update Async Replication Aurora Storage
  90. 90. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Amazon Aurora Aurora RW PostgreSQL RO update Async Replication Aurora Storage update in memory
  91. 91. RDS PostgreSQL - Cross Region Replicas AZ1 AZ2 US-EAST-1
  92. 92. RDS PostgreSQL - Cross Region Replicas AZ1 AZ2 US-EAST-1
  93. 93. RDS PostgreSQL - Cross Region Replicas AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1
  94. 94. Data movement & Migration
  95. 95. Migration to RDS & Aurora PostgreSQL Methods • PostgreSQL - pg_dump / pg_restore • AWS Data Migration Service (DMS) & Schema Conversion Tool (SCT) • RDS PostgreSQL - Snapshot Import
  96. 96. Schema Conversion Tool - SCT Downloadable tool (Windows, Mac, Linux Desktop) Source Database Target Database on Amazon RDS Microsoft SQL Server Amazon Aurora, MySQL, PostgreSQL MySQL PostgreSQL Oracle Amazon Aurora, MySQL, PostgreSQL PostgreSQL Amazon Aurora, MySQL
  97. 97. SCT - Analysis
  98. 98. SCT - Detailed
  99. 99. Customer Premises, EC2, RDS Application Users RDS & Aurora PostgreSQL VPN AWS DMS—Logical Replication
  100. 100. Customer Premises, EC2, RDS Application Users RDS & Aurora PostgreSQL VPN AWS DMS—Logical Replication Start a replication instance AWS Database Migration Service
  101. 101. Customer Premises, EC2, RDS Application Users RDS & Aurora PostgreSQL VPN AWS DMS—Logical Replication Start a replication instance Connect to source and target databases AWS Database Migration Service
  102. 102. Customer Premises, EC2, RDS Application Users RDS & Aurora PostgreSQL VPN AWS DMS—Logical Replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases AWS Database Migration Service
  103. 103. Customer Premises, EC2, RDS Application Users RDS & Aurora PostgreSQL VPN AWS DMS—Logical Replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data AWS Database Migration Service
  104. 104. Customer Premises, EC2, RDS Application Users RDS & Aurora PostgreSQL VPN AWS DMS—Logical Replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync AWS Database Migration Service
  105. 105. Customer Premises, EC2, RDS Application Users RDS & Aurora PostgreSQL VPN AWS DMS—Logical Replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync Switch applications over to the target at your convenience AWS Database Migration Service
  106. 106. Migration—Snapshot Import
  107. 107. Migration—Snapshot Import
  108. 108. Migration—Snapshot Import Snapshot
  109. 109. Migration—Snapshot Import Snapshot
  110. 110. Migration—Snapshot Import
  111. 111. Migration—Snapshot Import
  112. 112. Migration—Snapshot Import
  113. 113. Logical Replication Support – RDS PostgreSQL • Supported with 9.6.1+, 9.5.4+ and 9.4.9+ • Set rds.logical_replication parameter to 1 • As user who has rds_replication & rds_superuser role SELECT * FROM pg_create_logical_replication_slot('test_slot', 'test_decoding'); pg_recvlogical -d postgres --slot test_slot -U master --host $rds_hostname -f - --start • Added support for Event Triggers
  114. 114. Logical Decoding Space Usage
  115. 115. CloudWatch – Slot usage for WAL
  116. 116. Logical Replication Support - Example RDS Postgres
  117. 117. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift DMS
  118. 118. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS
  119. 119. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS EC2 Oracle S3 (new)
  120. 120. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS EC2 Oracle Custom Logical Handler S3 (new) NoSQL DB
  121. 121. Vacuuming
  122. 122. Vacuum parameters Will auto vacuum when • autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor * pgclass.reltuples How hard auto vacuum works • autovacuum_max_workers • autovacuum_nap_time • autovacuum_cost_limit • autovacuum_cost_delay
  123. 123. RDS autovacuum logging (9.4.5+) log_autovacuum_min_duration = 5000 (i.e. 5 secs) rds.force_autovacuum_logging_level = LOG …[14638]:ERROR: canceling autovacuum task …[14638]:CONTEXT: automatic vacuum of table "postgres.public.pgbench_tellers" …[14638]:LOG: skipping vacuum of "pgbench_branches" --- lock not available
  124. 124. RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2) pg_stat_activity BEFORE usename | query ----------+------------------------------------------------------------- rdsadmin | <insufficient privilege> rdsadmin | <insufficient privilege> gtest | SELECT c FROM sbtest27 WHERE id BETWEEN 392582 AND 392582+4 gtest | select usename, query from pg_stat_activity NOW usename | query ----------+---------------------------------------------- rdsadmin | <insufficient privilege> gtest | select usename, query from pg_stat_activity gtest | COMMIT rdsadmin | autovacuum: ANALYZE public.sbtest16
  125. 125. RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2) pg_stat_activity BEFORE usename | query ----------+------------------------------------------------------------- rdsadmin | <insufficient privilege> rdsadmin | <insufficient privilege> gtest | SELECT c FROM sbtest27 WHERE id BETWEEN 392582 AND 392582+4 gtest | select usename, query from pg_stat_activity NOW usename | query ----------+---------------------------------------------- rdsadmin | <insufficient privilege> gtest | select usename, query from pg_stat_activity gtest | COMMIT rdsadmin | autovacuum: ANALYZE public.sbtest16
  126. 126. RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2) pg_stat_activity BEFORE usename | query ----------+------------------------------------------------------------- rdsadmin | <insufficient privilege> rdsadmin | <insufficient privilege> gtest | SELECT c FROM sbtest27 WHERE id BETWEEN 392582 AND 392582+4 gtest | select usename, query from pg_stat_activity NOW usename | query ----------+---------------------------------------------- rdsadmin | <insufficient privilege> gtest | select usename, query from pg_stat_activity gtest | COMMIT rdsadmin | autovacuum: ANALYZE public.sbtest16
  127. 127. CloudWatch Metric
  128. 128. Aurora - Intelligent Vacuum Prefetch PostgreSQL Aurora PostgreSQL
  129. 129. Aurora - Intelligent Vacuum Prefetch PostgreSQL Aurora PostgreSQL
  130. 130. Aurora - Intelligent Vacuum Prefetch PostgreSQL Aurora PostgreSQL
  131. 131. Aurora - Intelligent Vacuum Prefetch PostgreSQL Aurora PostgreSQL
  132. 132. Aurora - Intelligent Vacuum Prefetch PostgreSQL Aurora PostgreSQL Submit Batch I/O
  133. 133. Aurora - Intelligent Vacuum Prefetch PostgreSQL Aurora PostgreSQL Submit Batch I/O 402 seconds 163 seconds
  134. 134. Performance Monitoring
  135. 135. Enhanced Operating System (OS) metrics 1-60 second granularity cpuUtilization • guest • irq • system • wait • idl: • user • total • steal • nice diskIO • writeKbPS • readIOsPS • await • readKbPS • rrqmPS • util • avgQueueLen • tps • readKb • writeKb • avgReqSz • wrqmPS • writeIOsPS memory • writeback • cached • free • inactive • dirty • mapped • active • total • slab • buffers • pageTable • Hugepages swap • cached • total • free tasks • sleeping • zombie • running • stopped • total • blocked fileSys • used • usedFiles • usedFilePercent • maxFiles • total • usedPercent loadAverageMinute • fifteen • five • one uptime processList • name • cpuTime • parentID • memoryUsedPct • cpuUsedPct • id • rss • vss
  136. 136. Enhanced Monitoring - Process List
  137. 137. OS metrics
  138. 138. Performance Insights
  139. 139. Performance Insights—Zoom
  140. 140. Performance Insights—Bad Query
  141. 141. Thank you! Questions?
  • lidoogi

    Mar. 8, 2018

A deep dive into the two RDS PostgreSQL offerings, RDS PostgreSQL and Aurora PostgreSQL. Covering what is common between the engines, what is different and updates that we have done over the past year.

Views

Total views

681

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

37

Shares

0

Comments

0

Likes

1

×