Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Grant McAlister – Senior Principal Engineer - RD...
RDS Version Updates
Support for 9.5 (default)
New Minor Releases
• 9.5.4
• 9.4.9
• 9.3.14
Extension Support Additions
recent ip4r, pg_buffercache, pgstattuple
9.5 address_standardizer, address_standardizer_us,
hs...
Extension Support Additions
recent ip4r, pg_buffercache, pgstattuple
9.5 address_standardizer, address_standardizer_us,
hs...
9.5 Parameter Changes - Checkpointing
checkpoint_segments=16 checkpoint_timeout=5 min
min_wal_size=256MB & max_wal_size=2G...
9.5 Parameter Changes - Checkpointing
checkpoint_segments=16 checkpoint_timeout=5 min
Checkpoint after 5 min or 16x16 (256...
9.5 Parameter Changes - Checkpointing
checkpoint_segments=16 checkpoint_timeout=5 min
Checkpoint after 5 min or 16x16 (256...
9.5 Parameter Changes - Checkpointing
checkpoint_segments=16 checkpoint_timeout=5 min
Checkpoint after 5 min or 16x16 (256...
9.5 RDS Parameter Default Improvement
rds_superuser_reserved_connections
9.4 Defaults to 0
9.5 Defaults to 2
max_connectio...
max_connections
-
100
200
300
400
500
600
700
800
900
1,000
t2.micro t2.small t2.medium t2.large m3.medium m3.large
Connec...
max_connections
-
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
Connections
Old New
All Version Default Parameter Changes
maintenance_work_mem
Before
9.3 = 16MB
9.4 = 64MB
Now
9.3/9.4/9.5 = GREATEST({DBInst...
maintenance_work_mem
-
100
200
300
400
500
600
maintenance_work_mem(MB)
Ver 9.3 old Ver 9.4 old Current
maintenance_work_mem
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
maintenance_work_mem(MB)
Ver 9.3 old Ver 9.4 ol...
Major version upgrade
Prod
9.4
Major version upgrade
Prod
9.4
Backup
Major version upgrade
Prod
9.4
pg_upgrade
Backup
Major version upgrade
Prod
9.4
pg_upgrade
Backup Backup
Major version upgrade
Prod
9.4
Prod
9.5
pg_upgrade
Backup Backup
Major version upgrade
Prod
9.4
Prod
9.5
pg_upgrade
Backup Backup
No PITR
Major version upgrade
Prod
9.4
Test
9.4
Restore to a test instance
Major version upgrade
Prod
9.4
Test
9.4
pg_upgrade
Restore to a test instance
Major version upgrade
Prod
9.4
Test
9.4
Test
9.5
pg_upgrade
Restore to a test instance
Major version upgrade
Prod
9.4
Test
9.4
Test
9.5
pg_upgrade
Restore to a test instance
Application
Testing
Major version upgrade
Prod
9.4
Test
9.4
Test
9.5
pg_upgrade
Restore to a test instance
Application
Testing
Major version upgrade
Prod
9.4
Prod
9.5
pg_upgrade
Backup Backup
No PITR
Test
9.4
Test
9.5
pg_upgrade
Restore to a test in...
Security
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
Log Backups
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
Log Backups
Security Group
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest...
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest...
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest...
Forcing SSL on all connections
DB
Instance
Snapshot
Application
Host
SSL
Log Backups
Security Group
VPC
Encryption at Rest...
Unencrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
SnapshotDB
Instance
Snapshot
Share with accoun...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Encryption at Rest
Default
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Share with account
Encryption at Rest
D...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Share with account
Encryption at Rest
D...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Share with account
Encryption at Rest
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Share with account
Encryption at Rest
C...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Share with account
Encryption at Rest
C...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
Snapshot
Snapshot
Share with account
Encryption ...
Encrypted Snapshot Sharing
DB
Instance
Snapshot
Prod Account
Test Account
SnapshotDB
Instance
Snapshot
Share with account
...
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
40,000
45,000
2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Thr...
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Threads
Trans...
Data movement
Logical Replication Support
• Supported with 9.5.4 and 9.4.9
• Set rds.logical_replication parameter to 1
• As user who ha...
Logical Decoding Space Usage
Move data to the same or different database engine
Keep your apps running during the migration
Start your first migration ...
Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Keep your apps running during the migration
Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Keep your apps running during the...
Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target data...
Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target data...
Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target data...
Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target data...
Customer
Premises
Application Users
EC2
or
RDS
Internet
VPN
Start a replication instance
Connect to source and target data...
AWS Database Migration Service - PostgreSQL
• Source - on premise or EC2 PostgreSQL (9.4+)
RDS (9.4.9 or 9.5.4)
• Destinat...
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
DMS
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
DMS
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
On Premise
Postgres
DMS
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
EC2
Postgres
On Premise
Postgres
...
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
EC2
Postgres
On Premise
Postgres
...
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
EC2
Postgres
On Premise
Postgres
...
Logical Replication Support - Example
RDS
Postgres
RDS
Postgres
Logical Replica
Redshift
EC2
Postgres
On Premise
Postgres
...
Schema Conversion Tool - SCT
Downloadable tool (Windows, Mac, Linux Desktop)
Source Database Target Database on Amazon RDS...
SCT - Analysis
SCT - Detailed
Vacuum parameters
Will auto vacuum when
• autovacuum_vacuum_threshold +
autovacuum_vacuum_scale_factor * pgclass.reltuples...
RDS autovacuum logging (9.4.5+)
log_autovacuum_min_duration = 5000 (i.e. 5 secs)
rds.force_autovacuum_logging_level = LOG
...
RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2)
pg_stat_activity
BEFORE
usename | query
----------+-----------------------...
CloudWatch Metric
Scale and availability
M4 Instance Class – pgbench read only
0
2000
4000
6000
8000
10000
12000
14000
1 2 4 8 16
TransactionsperSecond(TPS)
Thread...
Enhanced Operating System (OS) metrics
1-60 second granularity
cpuUtilization
• guest
• irq
• system
• wait
• idl:
• user
...
Process List
OS metrics
shared_buffers parameter
244GB RAM
PG processes
shared_buffers parameter
244GB RAM
PG processes
shared_buffers1/4
shared_buffers parameter
244GB RAM
PG processes
shared_buffers
Linux
pagecache
1/4
shared_buffers parameter
244GB RAM
PG processes
shared_buffers
Linux
pagecache
select of data – check for buffer in shared...
shared_buffers parameter
244GB RAM
PG processes
shared_buffers
Linux
pagecache
select of data – check for buffer in shared...
shared_buffers parameter
244GB RAM
PG processes
shared_buffers
Linux
pagecache
select of data – check for buffer in shared...
shared_buffers parameter
244GB RAM
PG processes
shared_buffers
Linux
pagecache
select of data – check for buffer in shared...
shared_buffers parameter
244GB RAM
PG processes
shared_buffers
Linux
pagecache
select of data – check for buffer in shared...
shared_buffers parameter
244GB RAM
PG processes
shared_buffers
Linux
pagecache
select of data – check for buffer in shared...
0
2,000
4,000
6,000
8,000
10,000
12,000
3% 6% 13% 25% 50% 75%
transactionspersecond(TPS)
shared_buffers as a percentage of...
0
2,000
4,000
6,000
8,000
10,000
12,000
13% 25% 50% 75%
transactionspersecond(TPS)
shared_buffers as a percentage of syste...
Stats on RAMDISK
• Set rds.pg_stat_ramdisk_size in MB’s
• Creates a RAM disk and sets stats_temp_directory to
use it.
• Re...
Availability – Read and Write – Multi-AZ
AZ1 AZ2
Availability – Read and Write – Multi-AZ
AZ1 AZ2
Availability – Read and Write – Multi-AZ
AZ1 AZ2
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
Primary Update
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
cname update
Primary Update
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
cname update
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
cname update
Availability – Read and Write – Multi-AZ
Physical
Synchronous
Replication
AZ1 AZ2
DNS
cname update
Read Replicas = Availability
Sync
Replication
Multi-AZ
Read Replicas = Availability
Sync
Replication
Multi-AZ
Async Replication
Read Replicas = Availability
Sync
Replication
Multi-AZ
Async Replication
Read Replicas = Availability
Sync
Replication
Multi-AZ
Async Replication
Read Replicas = Availability
Async Replication
Read Replicas = Availability
Async Replication
Read Replicas = Availability
Async Replication
Read Replicas = Availability
Async Replication
Read Replica promotion
AZ1 AZ2 AZ3
Read Replica promotion
AZ1 AZ2 AZ3
Read Replica promotion
AZ1 AZ2 AZ3
Cross Region Replicas – DR & Moves
AZ1 AZ2
US-EAST-1
Cross Region Replicas – DR & Moves
AZ1 AZ2 AZ1
Async Replication
US-EAST-1 EU-WEST-1
Cross Region Replicas – DR & Moves
AZ1 AZ2 AZ1
Async Replication
US-EAST-1 EU-WEST-1
Cross Region Replicas – DR & Moves
AZ1 AZ2 AZ1
Async Replication
US-EAST-1 EU-WEST-1
Cross Region Replicas – DR & Moves
AZ1 AZ2 AZ1
Async Replication
US-EAST-1 EU-WEST-1
Cross Region Replicas – DR & Moves
AZ1 AZ2 AZ1
Async Replication
US-EAST-1 EU-WEST-1
AZ2
Cross Region Replicas – DR & Moves
AZ1
US-EAST-1 EU-WEST-1
AZ2
Cross Region Replicas – Reduce Latency
AZ1 AZ2
US-EAST-1
Cross Region Replicas – Reduce Latency
AZ1 AZ2 AZ1
Async Replication
US-EAST-1 EU-WEST-1
Replication – In Region
Replication – In Region
xlog1
Replication – In Region
xlog1
xlog2
xlog3
xlog99
xlog1
Replication – In Region
xlog2
xlog3
xlog99
xlog1
Replication – In Region
xlog2
xlog3
xlog99
xlog1
xlog1
Replication – In Region
xlog2
xlog3
xlog99
xlog1
Replication – Cross Region & Slots
Replication – Cross Region & Slots
xlog1
Replication – Cross Region & Slots
xlog1
xlog2
xlog3
xlog98
xlog4
xlog99
Replication – Cross Region & Slots
xlog1
xlog2
xlog3
xlog98
xlog4
xlog99
Replication – Cross Region & Slots
xlog1
xlog2
xlog3
xlog98
xlog4
xlog99
Replication – Cross Region & Slots
Promote
Replication – Cross Region & Slots
Replication – Cross Region & Slots
Delete
Replication – Cross Region & Slots
Replication – Cross Region & Slots
Replication – Cross Region & Slots
max_standby_streaming_delay = -1
Replication – Cross Region & Slots
max_standby_streaming_delay = -1
Replication – Cross Region & Slots
CloudWatch – Replication Lag
CloudWatch – Slot usage for WAL
Burst mode: GP2 and T2
T2 – Amazon EC2 instance with burst capability
• Base performance + burst
• Earn credits per hour w...
T2 – CPU credits
Burst mode: what’s new
db.t2.large
• 60 CPU Initial Credit
• 36 CPU Credit earned per hour
• Base Performance – 60%
• 8 GB...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
TransactionsperSecond(TPS...
AWS Database BLOG
https://aws.amazon.com/blogs/database/
Thank you!
Questions?
Upcoming SlideShare
Loading in …5
×

Amazon RDS for PostgreSQL - Postgres Open 2016 - New Features and Lessons Learned

885 views

Published on

Presentation from Postgres Open 2016 in Dallas (Sept 2016) - Covers new RDS features introduced over the last year and lessons learned operating a large fleet of PostgreSQL.

Published in: Technology
  • Login to see the comments

Amazon RDS for PostgreSQL - Postgres Open 2016 - New Features and Lessons Learned

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Grant McAlister – Senior Principal Engineer - RDS September 2016 Amazon RDS for PostgreSQL New Features and Lessons Learned
  2. 2. RDS Version Updates Support for 9.5 (default) New Minor Releases • 9.5.4 • 9.4.9 • 9.3.14
  3. 3. Extension Support Additions recent ip4r, pg_buffercache, pgstattuple 9.5 address_standardizer, address_standardizer_us, hstore_plperl, tsm_system_rows, tsm_system_time
  4. 4. Extension Support Additions recent ip4r, pg_buffercache, pgstattuple 9.5 address_standardizer, address_standardizer_us, hstore_plperl, tsm_system_rows, tsm_system_time rds-postgres-extensions-request@amazon.com 9.3 Original - 32 9.3 Current - 35 9.4 Current - 39 9.5 Current - 44 Future - ???
  5. 5. 9.5 Parameter Changes - Checkpointing checkpoint_segments=16 checkpoint_timeout=5 min min_wal_size=256MB & max_wal_size=2GB checkpoint_timeout=5 min
  6. 6. 9.5 Parameter Changes - Checkpointing checkpoint_segments=16 checkpoint_timeout=5 min Checkpoint after 5 min or 16x16 (256MB) min_wal_size=256MB & max_wal_size=2GB checkpoint_timeout=5 min
  7. 7. 9.5 Parameter Changes - Checkpointing checkpoint_segments=16 checkpoint_timeout=5 min Checkpoint after 5 min or 16x16 (256MB) min_wal_size=256MB & max_wal_size=2GB checkpoint_timeout=5 min
  8. 8. 9.5 Parameter Changes - Checkpointing checkpoint_segments=16 checkpoint_timeout=5 min Checkpoint after 5 min or 16x16 (256MB) min_wal_size=256MB & max_wal_size=2GB checkpoint_timeout=5 min Checkpoint after 5 min or 2GB
  9. 9. 9.5 RDS Parameter Default Improvement rds_superuser_reserved_connections 9.4 Defaults to 0 9.5 Defaults to 2 max_connections 9.3/9.4 {DBInstanceClassMemory/31457280} 9.5 LEAST({DBInstanceClassMemory/9531392},5000) Higher values for smaller instances but stops at 5000 connections on large instances
  10. 10. max_connections - 100 200 300 400 500 600 700 800 900 1,000 t2.micro t2.small t2.medium t2.large m3.medium m3.large Connections Old New
  11. 11. max_connections - 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 Connections Old New
  12. 12. All Version Default Parameter Changes maintenance_work_mem Before 9.3 = 16MB 9.4 = 64MB Now 9.3/9.4/9.5 = GREATEST({DBInstanceClassMemory/63963136*1024},65536) Minimum of 64MB but now scales with instance size Only applies to default parameter groups and newly create custom groups
  13. 13. maintenance_work_mem - 100 200 300 400 500 600 maintenance_work_mem(MB) Ver 9.3 old Ver 9.4 old Current
  14. 14. maintenance_work_mem - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 maintenance_work_mem(MB) Ver 9.3 old Ver 9.4 old Current
  15. 15. Major version upgrade Prod 9.4
  16. 16. Major version upgrade Prod 9.4 Backup
  17. 17. Major version upgrade Prod 9.4 pg_upgrade Backup
  18. 18. Major version upgrade Prod 9.4 pg_upgrade Backup Backup
  19. 19. Major version upgrade Prod 9.4 Prod 9.5 pg_upgrade Backup Backup
  20. 20. Major version upgrade Prod 9.4 Prod 9.5 pg_upgrade Backup Backup No PITR
  21. 21. Major version upgrade Prod 9.4 Test 9.4 Restore to a test instance
  22. 22. Major version upgrade Prod 9.4 Test 9.4 pg_upgrade Restore to a test instance
  23. 23. Major version upgrade Prod 9.4 Test 9.4 Test 9.5 pg_upgrade Restore to a test instance
  24. 24. Major version upgrade Prod 9.4 Test 9.4 Test 9.5 pg_upgrade Restore to a test instance Application Testing
  25. 25. Major version upgrade Prod 9.4 Test 9.4 Test 9.5 pg_upgrade Restore to a test instance Application Testing
  26. 26. Major version upgrade Prod 9.4 Prod 9.5 pg_upgrade Backup Backup No PITR Test 9.4 Test 9.5 pg_upgrade Restore to a test instance Application Testing
  27. 27. Security
  28. 28. Forcing SSL on all connections DB Instance Snapshot Application Host Log Backups
  29. 29. Forcing SSL on all connections DB Instance Snapshot Application Host Log Backups Security Group
  30. 30. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group
  31. 31. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC
  32. 32. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest
  33. 33. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest ssl_mode=disable
  34. 34. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest ssl_mode=disable
  35. 35. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest ssl_mode=disable rds.force_ssl=1 (default 0)
  36. 36. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest ssl_mode=disable rds.force_ssl=1 (default 0)
  37. 37. Unencrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account SnapshotDB Instance Snapshot Share with account Share to Public
  38. 38. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account
  39. 39. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Encryption at Rest Default
  40. 40. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Share with account Encryption at Rest Default
  41. 41. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Share with account Encryption at Rest Default
  42. 42. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Share with account Encryption at Rest
  43. 43. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Share with account Encryption at Rest Custom Key
  44. 44. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Share with account Encryption at Rest Custom Key Add external account
  45. 45. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account Snapshot Snapshot Share with account Encryption at Rest Custom Key Add external account
  46. 46. Encrypted Snapshot Sharing DB Instance Snapshot Prod Account Test Account SnapshotDB Instance Snapshot Share with account Encryption at Rest Custom Key Add external account
  47. 47. 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000 2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Threads TransactionsPerSecond(TPS) PG Bench - Read Only - In Memory Regular Encrypted Encryption at rest overhead No measureable overhead
  48. 48. 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Threads TransactionsPerSecond(TPS) PG Bench - Read & Write Regular Encrypted Encryption at rest overhead 5 to 10% Overhead on heavy write
  49. 49. Data movement
  50. 50. Logical Replication Support • Supported with 9.5.4 and 9.4.9 • Set rds.logical_replication parameter to 1 • As user who has rds_replication & rds_superuser role SELECT * FROM pg_create_logical_replication_slot('test_slot', 'test_decoding'); pg_recvlogical -d postgres --slot test_slot -U master --host $rds_hostname -f - --start • Added support for Event Triggers
  51. 51. Logical Decoding Space Usage
  52. 52. Move data to the same or different database engine Keep your apps running during the migration Start your first migration in 10 minutes or less Replicate within, to, or from AWS EC2 or RDS AWS Database Migration Service (DMS)
  53. 53. Customer Premises Application Users EC2 or RDS Internet VPN Keep your apps running during the migration
  54. 54. Customer Premises Application Users EC2 or RDS Internet VPN Start a replication instance Keep your apps running during the migration AWS Database Migration Service
  55. 55. Customer Premises Application Users EC2 or RDS Internet VPN Start a replication instance Connect to source and target databases Keep your apps running during the migration AWS Database Migration Service
  56. 56. Customer Premises Application Users EC2 or RDS Internet VPN Start a replication instance Connect to source and target databases Select tables, schemas, or databases Keep your apps running during the migration AWS Database Migration Service
  57. 57. Customer Premises Application Users EC2 or RDS Internet VPN Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Keep your apps running during the migration AWS Database Migration Service
  58. 58. Customer Premises Application Users EC2 or RDS Internet VPN Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync Keep your apps running during the migration AWS Database Migration Service
  59. 59. Customer Premises Application Users EC2 or RDS Internet VPN Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync Switch applications over to the target at your convenience Keep your apps running during the migration AWS Database Migration Service
  60. 60. AWS Database Migration Service - PostgreSQL • Source - on premise or EC2 PostgreSQL (9.4+) RDS (9.4.9 or 9.5.4) • Destination can be EC2 or RDS • Initial bulk copy via consistent select • Uses PostgreSQL logical replication support to provide change data capture https://aws.amazon.com/dms/
  61. 61. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica DMS
  62. 62. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift DMS
  63. 63. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift On Premise Postgres DMS
  64. 64. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS
  65. 65. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS EC2 Oracle
  66. 66. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS EC2 Oracle Custom Logical Handler
  67. 67. Logical Replication Support - Example RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS EC2 Oracle Custom Logical Handler NoSQL DB
  68. 68. Schema Conversion Tool - SCT Downloadable tool (Windows, Mac, Linux Desktop) Source Database Target Database on Amazon RDS Microsoft SQL Server Amazon Aurora, MySQL, PostgreSQL MySQL PostgreSQL Oracle Amazon Aurora, MySQL, PostgreSQL PostgreSQL Amazon Aurora, MySQL
  69. 69. SCT - Analysis
  70. 70. SCT - Detailed
  71. 71. Vacuum parameters Will auto vacuum when • autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor * pgclass.reltuples How hard auto vacuum works • autovacuum_max_workers • autovacuum_nap_time • autovacuum_cost_limit • autovacuum_cost_delay
  72. 72. RDS autovacuum logging (9.4.5+) log_autovacuum_min_duration = 5000 (i.e. 5 secs) rds.force_autovacuum_logging_level = LOG …[14638]:ERROR: canceling autovacuum task …[14638]:CONTEXT: automatic vacuum of table "postgres.public.pgbench_tellers" …[14638]:LOG: skipping vacuum of "pgbench_branches" --- lock not available http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.Comm onDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Autovacuum
  73. 73. RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2) pg_stat_activity BEFORE usename | query ----------+------------------------------------------------------------- rdsadmin | <insufficient privilege> rdsadmin | <insufficient privilege> gtest | SELECT c FROM sbtest27 WHERE id BETWEEN 392582 AND 392582+4 gtest | select usename, query from pg_stat_activity NOW usename | query ----------+---------------------------------------------- rdsadmin | <insufficient privilege> gtest | select usename, query from pg_stat_activity gtest | COMMIT rdsadmin | autovacuum: ANALYZE public.sbtest16
  74. 74. CloudWatch Metric
  75. 75. Scale and availability
  76. 76. M4 Instance Class – pgbench read only 0 2000 4000 6000 8000 10000 12000 14000 1 2 4 8 16 TransactionsperSecond(TPS) Threads db.m3.large db.m4.large 46% Better Price/Performance 37% TPS Increase $0.390 $0.365
  77. 77. Enhanced Operating System (OS) metrics 1-60 second granularity cpuUtilization • guest • irq • system • wait • idl: • user • total • steal • nice diskIO • writeKbPS • readIOsPS • await • readKbPS • rrqmPS • util • avgQueueLen • tps • readKb • writeKb • avgReqSz • wrqmPS • writeIOsPS memory • writeback • cached • free • inactive • dirty • mapped • active • total • slab • buffers • pageTable • Hugepages swap • cached • total • free tasks • sleeping • zombie • running • stopped • total • blocked fileSys • used • usedFiles • usedFilePercent • maxFiles • total • usedPercent loadAverageMinute • fifteen • five • one uptime processList • name • cpuTime • parentID • memoryUsedPct • cpuUsedPct • id • rss • vss
  78. 78. Process List
  79. 79. OS metrics
  80. 80. shared_buffers parameter 244GB RAM PG processes
  81. 81. shared_buffers parameter 244GB RAM PG processes shared_buffers1/4
  82. 82. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache 1/4
  83. 83. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache select of data – check for buffer in shared_buffers 1/4
  84. 84. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache select of data – check for buffer in shared_buffers 1/4
  85. 85. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache select of data – check for buffer in shared_buffers if not in shared_buffers load from pagecache/disk 1/4
  86. 86. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache select of data – check for buffer in shared_buffers if not in shared_buffers load from pagecache/disk EBS 1/4
  87. 87. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache select of data – check for buffer in shared_buffers if not in shared_buffers load from pagecache/disk EBS 1/4
  88. 88. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache select of data – check for buffer in shared_buffers if not in shared_buffers load from pagecache/disk EBS 1/4 shared_buffers = working set size
  89. 89. 0 2,000 4,000 6,000 8,000 10,000 12,000 3% 6% 13% 25% 50% 75% transactionspersecond(TPS) shared_buffers as a percentage of system memory pgbench write workload on r3.8xlarge working set = 10% of memory 25 threads 50 threads 100 threads 200 threads 400 threads 800 threads
  90. 90. 0 2,000 4,000 6,000 8,000 10,000 12,000 13% 25% 50% 75% transactionspersecond(TPS) shared_buffers as a percentage of system memory pgbench write workload on r3.8xlarge working set = 50% of memory 25 threads 50 threads 100 threads 200 threads 400 threads 800 threads
  91. 91. Stats on RAMDISK • Set rds.pg_stat_ramdisk_size in MB’s • Creates a RAM disk and sets stats_temp_directory to use it. • Reduces IOPS • Good for instances with many tables/indexes and databases.
  92. 92. Availability – Read and Write – Multi-AZ AZ1 AZ2
  93. 93. Availability – Read and Write – Multi-AZ AZ1 AZ2
  94. 94. Availability – Read and Write – Multi-AZ AZ1 AZ2
  95. 95. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2
  96. 96. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2
  97. 97. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2
  98. 98. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS
  99. 99. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS Primary Update
  100. 100. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update Primary Update
  101. 101. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update
  102. 102. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update
  103. 103. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update
  104. 104. Read Replicas = Availability Sync Replication Multi-AZ
  105. 105. Read Replicas = Availability Sync Replication Multi-AZ Async Replication
  106. 106. Read Replicas = Availability Sync Replication Multi-AZ Async Replication
  107. 107. Read Replicas = Availability Sync Replication Multi-AZ Async Replication
  108. 108. Read Replicas = Availability Async Replication
  109. 109. Read Replicas = Availability Async Replication
  110. 110. Read Replicas = Availability Async Replication
  111. 111. Read Replicas = Availability Async Replication
  112. 112. Read Replica promotion AZ1 AZ2 AZ3
  113. 113. Read Replica promotion AZ1 AZ2 AZ3
  114. 114. Read Replica promotion AZ1 AZ2 AZ3
  115. 115. Cross Region Replicas – DR & Moves AZ1 AZ2 US-EAST-1
  116. 116. Cross Region Replicas – DR & Moves AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1
  117. 117. Cross Region Replicas – DR & Moves AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1
  118. 118. Cross Region Replicas – DR & Moves AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1
  119. 119. Cross Region Replicas – DR & Moves AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1
  120. 120. Cross Region Replicas – DR & Moves AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1 AZ2
  121. 121. Cross Region Replicas – DR & Moves AZ1 US-EAST-1 EU-WEST-1 AZ2
  122. 122. Cross Region Replicas – Reduce Latency AZ1 AZ2 US-EAST-1
  123. 123. Cross Region Replicas – Reduce Latency AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1
  124. 124. Replication – In Region
  125. 125. Replication – In Region xlog1
  126. 126. Replication – In Region xlog1 xlog2 xlog3 xlog99 xlog1
  127. 127. Replication – In Region xlog2 xlog3 xlog99 xlog1
  128. 128. Replication – In Region xlog2 xlog3 xlog99 xlog1 xlog1
  129. 129. Replication – In Region xlog2 xlog3 xlog99 xlog1
  130. 130. Replication – Cross Region & Slots
  131. 131. Replication – Cross Region & Slots xlog1
  132. 132. Replication – Cross Region & Slots xlog1 xlog2 xlog3 xlog98 xlog4 xlog99
  133. 133. Replication – Cross Region & Slots xlog1 xlog2 xlog3 xlog98 xlog4 xlog99
  134. 134. Replication – Cross Region & Slots xlog1 xlog2 xlog3 xlog98 xlog4 xlog99
  135. 135. Replication – Cross Region & Slots Promote
  136. 136. Replication – Cross Region & Slots
  137. 137. Replication – Cross Region & Slots Delete
  138. 138. Replication – Cross Region & Slots
  139. 139. Replication – Cross Region & Slots
  140. 140. Replication – Cross Region & Slots max_standby_streaming_delay = -1
  141. 141. Replication – Cross Region & Slots max_standby_streaming_delay = -1
  142. 142. Replication – Cross Region & Slots
  143. 143. CloudWatch – Replication Lag
  144. 144. CloudWatch – Slot usage for WAL
  145. 145. Burst mode: GP2 and T2 T2 – Amazon EC2 instance with burst capability • Base performance + burst • Earn credits per hour when below base performance • Can store up to 24 hours worth of credits • Amazon CloudWatch metrics to see credits and usage GP2 – SSD-based Amazon EBS storage • 3 IOPS per GB base performance • Earn credits when usage below base • Burst to 3000+ IOPS
  146. 146. T2 – CPU credits
  147. 147. Burst mode: what’s new db.t2.large • 60 CPU Initial Credit • 36 CPU Credit earned per hour • Base Performance – 60% • 8 GB RAM • Increased IO bandwidth • Encryption at rest support
  148. 148. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard Burst mode vs. classic vs. Provisioned IOPS $0.58 per hour
  149. 149. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS Burst mode vs. classic vs. Provisioned IOPS $0.58 per hour $0.40 per hour
  150. 150. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS Burst mode vs. classic vs. Provisioned IOPS $0.58 per hour $0.40 per hour $0.50 per hour
  151. 151. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.40 per hour $0.50 per hour
  152. 152. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.40 per hour $0.50 per hour
  153. 153. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 db.t2.medium + 1TB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.23 per hour $0.40 per hour $0.50 per hour
  154. 154. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 db.t2.medium + 1TB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.23 per hour $0.40 per hour $0.50 per hour
  155. 155. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 db.t2.medium + 1TB gp2 db.t2.large + 1TB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.23 per hour $0.40 per hour $0.50 per hour $0.30 per hour
  156. 156. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 db.t2.medium + 1TB gp2 db.t2.large + 1TB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.23 per hour $0.40 per hour $0.50 per hour $0.30 per hour
  157. 157. AWS Database BLOG https://aws.amazon.com/blogs/database/
  158. 158. Thank you! Questions?

×