Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:Invent 2017

1,674 views

Published on

Amazon Aurora is a fully-managed relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. The initial launch of Amazon Aurora delivered these benefits for MySQL. We have now added PostgreSQL compatibility to Amazon Aurora. In this session, Amazon Aurora experts discuss best practices to maximize the benefits of the Amazon Aurora PostgreSQL-compatible edition in your environment.

Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:Invent 2017

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Deep Dive on Amazon Aurora PostgreSQL - Compatible Edition G r a n t M c A l i s t e r — S e n i o r P r i n c i p a l E n g i n e e r D A T 4 0 2 N o v e m b e r 2 9 , 2 0 1 7 D A T 4 0 2
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. RDS PostgreSQL Universe
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. RDS PostgreSQL Universe CLIENTS RDS PostgreSQL Aurora PostgreSQL EBS Aurora Storage Postgres 9.6—same extensions Backup/Recovery - PITR High Availability & Durability Secure Read Replicas Cross Region Snapshots Scale Compute – Online Scale Storage Cross Region Replication Outbound Logical Replication
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Base Architecture
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. RO Application AZ-1 AZ-2 AZ-3 Aurora Storage and Replicas RW Application RO Application Async Invalidation & Update Async Invalidation & Update Write log records Read blocks RW Automatic Scalable Storage to 64TB
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Log-Based Storage
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Concurrency—Remove Log Buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage A Queued Work Storage B C D E 0 0 0 0 0 A B C D E 2 2 1 0 1 A B C D E 4 3 4 2 4 A B C D E 6 5 6 3 5 A B C D E Durability Tracking
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Checkpoints & Full-Page Writes in PostgreSQL Block in Memory PostgreSQL update t set y = 6; Checkpoint Datafile Full Block WAL Archive 4K 4K 8K During crash recovery, PostgreSQL uses the FPW block in the WAL to replace the bad checkpointed block
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Aurora PostgreSQL—Writing Less Block in Memory PostgreSQL Aurora update t set y = 6; update t set y = 6; Checkpoint Datafile Full Block WAL Archive Block in Memory Aurora Storage no checkpoint = no FPW Amazon S3
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1- 200 1- 100 101- 200 Insert 124 1-50 51- 100 1-25 26- 50 51- 75 76- 100 101- 150 151- 200 101- 125 126- 150 151- 175 176- 200 Insert 99 1- 200 101- 200 101- 150 101- 125 4 blocks loaded 1- 100 51- 100 76- 100 3 blocks loaded Insert a Random Value into a B-Tree 151- 200 151- 200 2 blocks loaded Insert 161 At least 3 FPW
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Insert Test Test Table • UUID PK—Random • ID int—Right Lean Sequence • VARCHAR(100)—Random • VARCHAR(50)—Small Set of Words • INT—Random • INT—Random (smaller set) • BOOLEAN—Random (50/50) • BOOLEAN—Somewhat Random (75/25) • Timestamp—Right Lean
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 0 5,000 10,000 15,000 20,000 25,000 30,000 1 31 61 91 121 151 181 211 241 271 InsertsPerSecond Minutes Insert Workload—PostgreSQL 9.6 BASE 16GB Max WAL Aurora PostgreSQL
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Update Test Test Table • UUID PK—Random • ID int—Right Lean Sequence • VARCHAR(100)—Random • VARCHAR(50)—Small Set of Words • INT—Random • INT—Random (smaller set) • BOOLEAN—Random (50/50) • BOOLEAN—Somewhat Random (75/25) • Timestamp—Right Lean UPDATE #1 UPDATE #2
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3,729 4,871 17,158 - 5,000 10,000 15,000 20,000 TPS(2UpdatesperTransaction) Update Workload—PostgreSQL 9.6 BASE 16GB Max WAL Aurora PostgreSQL
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Aurora Recovers Up to 97%Faster 3 GiB Redo Recovered in 19 seconds 10 GiB Redo Recovered in 50 seconds 30 GiB Redo Recovered in 123 seconds 0 20 40 60 80 100 120 140 160 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 RecoveryTimeinSeconds(lessisbetter) Writes / Second (more is better) RECOVERY TIME FROM CRASH UNDER LOAD Bubble size represents redo log, which must be recovered As PostgreSQL throughput goes up, so does log size and crash recovery time Amazon Aurora has no redo. Recovered in 3 seconds while maintaining significantly greater throughput.
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Durability—4 of 6 Quorum
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Issues with 6 of 6 Replication Location 1 Location 2 Location 3 Start Finish
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 6 10 21 31 7 12 28 123 0 20 40 60 80 100 120 140 50 90 99.9 99.99 Latency(ms) Percentile High Concurrency Sync Write Test 2 Node (4 copy) 3 Node (6 Copy) Cost of Additional Synchronous Replicas
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Aurora Timing Example Location 1 Location 2 Location 3 Start Finish Only need 4/6 sync writes
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Aurora Gives >2x Lower Response Times 0.00 100.00 200.00 300.00 400.00 500.00 600.00 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 responsetime,ms minute sysbench response time (p95), 30 GiB, 1024 clients PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas and Clones
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—PostgreSQL PostgreSQL RW EBS Snapshot PostgreSQL RO EBS update Async Replication Catchup
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Amazon Aurora Aurora RW PostgreSQL RO update Async Replication Aurora Storage update in memory
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pgbench Benchmark Async Replication and apply PostgreSQL Aurora PostgreSQL RW RO accounts tellers branches history accounts tellers branches history Async Replication and memory update RW RO accounts tellers branches history accounts
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Backfill on PostgreSQL pgbench RW 8K tps on Primary – RO 200k tps on replica backfill pgbench_history
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replicas—Backfill on Amazon Aurora pgbench RW 8K tps on Primary – RO 200k tps on replica backfill pgbench_history
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Aurora Storage RO Application AZ-1 AZ-2 AZ-3 Fast Clones RW Application RW Developers Reporting Async Invalidation & Update Clone update Primary Storage Clone Storage
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Caching
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Caching Changes—No Double Buffering 488GB RAM PG + OS processes shared_buffers Linux pagecache select of data – check for block in shared_buffers if not in shared_buffers load from pagecache/disk EBS 1/4 duplicate buffers PG + OS processes shared_buffers PostgreSQL Aurora PostgreSQL Aurora Storage select of data – check for block in shared_buffers or load from Aurora storage 3/4 Survivable Cache
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Caching Changes—No Double Buffering 689,068 417,496 334,691 682,931 - 100,000 200,000 300,000 400,000 500,000 600,000 700,000 800,000 TransactionsPerSecond(tps) pgbench read only - scale 22,000 - r4.16xlarge Aurora 75% Cache PostgreSQL 25% Cache PostgreSQL 10% Cache PostgreSQL 75% Cache 1.6x 2.0x 18K read iops no reads heavy double buffering no double buffering no survivable cache Approx 350GB working set
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vacuuming
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why Vacuuming Matters tuple1 tuple2 tuple3 tuple4 tuple5 tuple6 tuple1tuple1 tuple2 tuple2 tuple7 tuple8 tuple9 tuple10 Run Vacuumtuple3 tuple4 tuple3 tuple4 tuple1 tuple2 tuple3 tuple4 tuple5 tuple6 tuple1tuple1 tuple2 tuple2 tuple7 tuple8 tuple9 tuple10 No Vacuumtuple3 tuple4 tuple3 tuple4 more blocks = more Cache misses, non HOT updates and more FPW
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4,000 5,000 6,000 7,000 8,000 9,000 10,000 1 61 121 181 241 301 361 421 481 541 601 661 721 781 841 901 961 1021 1081 1141 1201 1261 TPS Minutes Updates—No Vacuum Running transaction id wrap around
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Intelligent Vacuum Prefetch PostgreSQL Aurora PostgreSQL Submit Batch I/O
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vacuuming in Memory—Insert Only Workload HEAP PAGES 1- 200 1- 100 101- 200 1-50 51- 100 1-25 26- 50 51- 75 76- 100 101- 150 151- 200 101- 125 126- 150 151- 175 176- 200 1- 201 101- 201 151- 201 176- 201 1- 202 101- 202 151- 202 176- 202 No long running transactions
  36. 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vacuum Freeze Not in Memory—PostgreSQL Block in Memory Checkpoint Datafile Full Block WAL Archive Not Frozen VACUUM
  37. 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vacuum Freeze in Memory—PostgreSQL Block in Memory Checkpoint Datafile WAL Archive Frozen
  38. 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vacuum in Memory, Cont. • alter table X set (vacuum settings) • Increase checkpoint_timeout • Manual Test • Vacuum in Memory before checkpoint – 72 seconds • Vacuum in Memory after checkpoint – 152 seconds • Vacuum not in Memory – 402 seconds – 163 seconds
  39. 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring Performance
  40. 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enhanced Monitoring—Process List
  41. 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance Insights
  42. 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance Insights—Zoom
  43. 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance Insights—Bad Query
  44. 44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration
  45. 45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration to Aurora PostgreSQL Methods • AWS Data Migration Service (DMS) • PostgreSQL - pg_dump / pg_restore • RDS PostgreSQL - Snapshot Import
  46. 46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Customer Premises, EC2, RDS Application Users Aurora PostgreSQLVPN AWS DMS—Logical Replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync Switch applications over to the target at your convenience AWS Database Migration Service
  47. 47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration—Snapshot Import Snapshot
  48. 48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! D e e p D i v e o n A m a z o n A u r o r a P o s t g r e S Q L — D A T 4 0 2 G r a n t M c A l i s t e r — S e n i o r P r i n c i p a l E n g i n e e r

×