re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations

2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Deep dive into Amazon Aurora and its innovations D A T 3 2 6 Grant McAlister Senior Principal Engineer AWS

3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Designed for unparalleled high performance and availability at global scale with full MySQL and PostgreSQL compatibility at 1/10th the cost of commercial databases  5x throughput of standard MySQL and 3x of standard PostgreSQL  Scale out up to 15 read replicas  Decoupled storage and compute enabling cost optimization  Fast database cloning  Distributed, dynamically scaling storage subsystem Performance & scalability  6 copies of data across 3 AZs (customers pays for 1)  Automatic, continuous, incremental backups with point- in-time recovery (PITR)  Fault-tolerant, self-healing, auto- scaling storage  Global Database for disaster recovery Availability & durability  Network isolation  Encryption at rest/in transit  Supports multiple secure authentication mechanisms and audit controls Highly secure  Automates time-consuming management of administration tasks like hardware provisioning, database setup, patching, and backups  Serverless configuration options Fully managed

5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora storage and replicas RW Application Application Write log records Aurora storage Availability Zone 3 Availability Zone 2 Availability Zone 1 6 5 4 3 2 1

6. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora storage and replicas RW Application Application Write log records Read blocks Aurora storage Availability Zone 3 Availability Zone 2 Availability Zone 1

11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Aurora storage and replicas RW Application RO Application Async Invalidation & Update Async invalidation & update Write log records Read blocks Aurora storage RO RO RO RO Availability Zone 3 Availability Zone 2 Availability Zone 1 db.r6i.4xlarge db.serverless db.r6g.4xlarge

14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Aurora storage and replicas RW Application RO Application Async Invalidation & Update Write log records Read blocks RW Aurora storage RO RO RO RO Availability Zone 3 Availability Zone 2 Availability Zone 1 db.r6i.4xlarge db.serverless db.r6g.4xlarge

15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Aurora storage and replicas RW Application RO Application Async Invalidation & Update Write log records Read blocks RW Aurora storage RO RO RO RO Availability Zone 3 Availability Zone 2 Availability Zone 1 db.r6i.4xlarge db.serverless db.r6g.4xlarge

16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Aurora storage and replicas RW Application RO Application Async Invalidation & Update Write log records Read blocks RW Aurora storage RO RO RO RO Availability Zone 3 Availability Zone 2 Availability Zone 1 db.r6i.4xlarge db.serverless db.r6g.4xlarge AWS JDBC

17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Aurora storage RO Application RW Application RO Application primary DB cluster secondary DB cluster

18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Aurora storage RO Application RW Application RO Application Replication servers Aurora storage primary DB cluster secondary DB cluster

23. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Aurora storage RO Application RW Application RO Application Replication servers Aurora storage DR primary DB cluster secondary DB cluster

24. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO primary DB cluster secondary DB cluster

25. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO primary DB cluster secondary DB cluster

26. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO Application Application RO Application RO primary DB cluster secondary DB cluster

32. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Aurora storage RO Application RW Application RO Application Aurora storage RO Application Application RO Application RO RW primary DB cluster secondary DB cluster

33. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Managed planned failover Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO Application Application RO Application RO primary DB cluster secondary DB cluster

34. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Managed planned failover Aurora storage RO Application Application RO Application Replication servers Aurora storage RO Application Application RO Application RO primary DB cluster secondary DB cluster RO

35. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Managed planned failover Aurora storage RO Application Application RO Application Replication servers Aurora storage RO Application Application RO Application RO primary DB cluster secondary DB cluster RO verify

36. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Managed planned failover Aurora storage RO Application Application RO Application Aurora storage RO Application Application RO Application RO RW RO primary DB cluster

37. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Managed planned failover Aurora storage RO Application Application RO Application Aurora storage RO Application Application RO Application RO RW RO Replication servers Replication agents primary DB cluster secondary DB cluster

38. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Global Database Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RO Application RW Application RO Application Replication servers Region B Availability Zone 3 Availability Zone 1 Availability Zone 2 Replication agents Aurora storage R O Applicatio n Applicatio n R O Applicatio n R O Region C Availability Zone 3 Availability Zone 1 Availability Zone 2 Replication agents Aurora storage R O Applicatio n Applicatio n R O Region D Availability Zone 3 Availability Zone 1 Availability Zone 2 Replication agents Aurora storage db.serverless

39. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Write Forwarding Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO Application Application RO Application RO

40. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Write Forwarding Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO Application Application RO Application RO Application

42. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Write Forwarding Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO Application Application RO Application RO Application TUNNEL

44. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication agents Region B Region A Availability Zone 3 Availability Zone 1 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 2 Amazon Aurora Global Database Write Forwarding Aurora storage RO Application RW Application RO Application Replication servers Aurora storage RO Application Application RO Application RO Application --enable-global-write-forwarding

49. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora Block in memory t-v1 Aurora storage Block in memory PostgreSQL t-v1 t-v2 t-v2 Full block WAL update t set y = 6 Amazon Simple Storage Service (Amazon S3)

50. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora Block in memory t-v1 Aurora storage Block in memory PostgreSQL t-v1 t-v2 t-v3 t-v2 Full block WAL update t set y = 6 Amazon Simple Storage Service (Amazon S3)

51. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora Block in memory t-v1 Aurora storage Block in memory PostgreSQL t-v1 t-v2 t-v3 t-v2 Full block t-v3 WAL update t set y = 6 Amazon Simple Storage Service (Amazon S3)

52. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora Block in memory t-v1 Aurora storage Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive update t set y = 6 Amazon Simple Storage Service (Amazon S3)

53. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora Block in memory t-v1 Aurora storage Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive 4K 4K 8K update t set y = 6 Amazon Simple Storage Service (Amazon S3)

56. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora Block in memory t-v1 Aurora storage Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive 4K 4K 8K update t set y = 6 Amazon Simple Storage Service (Amazon S3) recovery in minutes

57. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora update t set y = 6 Block in memory t-v1 Aurora storage Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive 4K 4K 8K update t set y = 6 Amazon Simple Storage Service (Amazon S3) recovery in minutes

58. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora update t set y = 6 Block in memory t-v1 t-v2 Aurora storage t-v2 Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive 4K 4K 8K update t set y = 6 Amazon Simple Storage Service (Amazon S3) recovery in minutes

59. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora update t set y = 6 Block in memory t-v1 t-v2 t-v3 Aurora storage t-v2 Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive 4K 4K 8K update t set y = 6 Amazon Simple Storage Service (Amazon S3) recovery in minutes

60. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora update t set y = 6 Block in memory t-v1 t-v2 t-v3 Aurora storage t-v2 t-v3 Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive 4K 4K 8K update t set y = 6 Amazon Simple Storage Service (Amazon S3) recovery in minutes

61. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL: Writing less Aurora update t set y = 6 Block in memory t-v1 t-v2 t-v3 Aurora storage t-v2 t-v3 No engine checkpoint = no FPW Block in memory PostgreSQL t-v1 t-v2 t-v3 Checkpoint Datafile t-v2 Full block t-v3 WAL Archive 4K 4K 8K update t set y = 6 Amazon Simple Storage Service (Amazon S3) recovery in minutes continuous & parallel coalesce recovery in seconds

62. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora Block in memory row1 Aurora storage Block in memory MySQL row1 row2 row2 log insert Amazon Simple Storage Service (Amazon S3)

63. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora Block in memory row1 Aurora storage Block in memory MySQL row1 row2 Checkpoint row2 log insert Amazon Simple Storage Service (Amazon S3)

64. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora Block in memory row1 Aurora storage Block in memory MySQL row1 row2 Checkpoint row2 Full block log insert Amazon Simple Storage Service (Amazon S3) doublewrite buffer

65. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora Block in memory row1 Aurora storage Block in memory MySQL row1 row2 Checkpoint Datafile row2 Full block log Archive insert Amazon Simple Storage Service (Amazon S3) doublewrite buffer

66. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora Block in memory row1 Aurora storage Block in memory MySQL row1 row2 Checkpoint Datafile row2 Full block log Archive 4K 4K 16K insert Amazon Simple Storage Service (Amazon S3) 4K 4K doublewrite buffer

67. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora Block in memory row1 Aurora storage Block in memory MySQL row1 row2 Checkpoint Datafile row2 Full block log Archive 4K 4K 16K insert Amazon Simple Storage Service (Amazon S3) 4K 4K doublewrite buffer

68. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora Block in memory row1 Aurora storage Block in memory MySQL row1 row2 Checkpoint Datafile row2 Full block log Archive 4K 4K 16K insert Amazon Simple Storage Service (Amazon S3) recovery in minutes 4K 4K doublewrite buffer

69. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora insert Block in memory row1 Aurora storage Block in memory MySQL row1 row2 Checkpoint Datafile row2 Full block log Archive 4K 4K 16K insert Amazon Simple Storage Service (Amazon S3) recovery in minutes 4K 4K doublewrite buffer

70. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL: Writing less Aurora insert Block in memory row1 row2 Aurora storage row2 No engine checkpoint = no doublewrite buffer Block in memory MySQL row1 row2 Checkpoint Datafile row2 Full block log Archive 4K 4K 16K insert Amazon Simple Storage Service (Amazon S3) recovery in minutes continuous & parallel coalesce recovery in seconds 4K 4K doublewrite buffer

82. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora RW Storage Node Incoming queue Data blocks Update queue Hot log Peer storage nodes Coalesce Amazon S3 A C B A B C B C A

86. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Management – Dynamic resizing new partitions every hour drop existing create new 2 hour spike drop existing create new drop the spike used space inside the db used storage space

87. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Management – Dynamic resizing new partitions every hour drop existing create new 2 hour spike drop existing create new drop the spike used space inside the db used storage space 2X extra storage costs

88. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Management – Dynamic resizing new partitions every hour drop existing create new 2 hour spike drop existing create new drop the spike

89. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Management – Dynamic resizing new partitions every hour drop existing create new 2 hour spike drop existing create new drop the spike used space inside the db used storage space

90. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availability Zone 2 Availability Zone 1 Availability Zone 3 RO Application Fast clones RW Application Reporting application Read blocks Aurora storage Primary storage

91. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availability Zone 2 Availability Zone 1 Availability Zone 3 RO Application Fast clones RW Application RW Reporting application Read blocks Aurora storage Primary storage Clone storage Clone

98. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availability Zone 2 Availability Zone 1 Availability Zone 3 RO Application Fast clones RW Application RW Reporting application Write log records Read blocks Aurora storage Primary storage Clone storage Clone

102. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fast clone example 0 5000 10000 15000 20000 25000 0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 Transactions per second (TPS) Minutes PGBench RW Scale 10K - Target Rate 20K TPS Main Database Clone Database

107. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Export to S3 via clone Amazon Aurora Primary(R/W) Snapshot Aurora storage Amazon Simple Storage Service (Amazon S3) Aurora storage Amazon Aurora Primary Snapshot

109. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Export to S3 via clone Amazon Aurora Primary(R/W) Aurora storage Amazon Simple Storage Service (Amazon S3) Amazon Aurora CLONE

110. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Export to S3 via clone Amazon Aurora Primary(R/W) Aurora storage Amazon Simple Storage Service (Amazon S3) Amazon Aurora CLONE

111. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Export to S3 via clone Amazon Aurora Primary(R/W) Aurora storage Amazon Simple Storage Service (Amazon S3) Amazon Aurora CLONE parallel export – Aurora MySQL

113. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora MySQL Updates • Support for major version upgrade to MySQL 3 with MySQL 8.0 Compatibility • IPv6 support • Active Directory support • Parallel Export to S3 • Enhanced binlog

114. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. enhanced binlog Binlog Enhanced Binlog • greatly reduced overhead for enabling binlog • reduced cost to read binlogs for cdc

116. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL Updates • Support for PostgreSQL14 – new extensions tds_fdw & lo • Multi Major Version Upgrade • SCRAM auth enforcement • IPv6 support • Improved consistency of minor version & patch upgrade times • Improved availability of read replicas • Logical Replication cache

117. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT AWS Database Migration Service (AWS DMS)

118. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora write wal log (needed for logical decoding) write transaction log Users / Applications INSERT AWS Database Migration Service (AWS DMS)

119. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT AWS Database Migration Service (AWS DMS)

120. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT logical decoding AWS Database Migration Service (AWS DMS) read wal log (needed for logical decoding)

121. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT logical decoding AWS Database Migration Service (AWS DMS) INSERT read wal log (needed for logical decoding)

122. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT logical decoding AWS Database Migration Service (AWS DMS)

123. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT logical decoding AWS Database Migration Service (AWS DMS) wal log cache

124. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT logical decoding AWS Database Migration Service (AWS DMS) wal log cache cache read

125. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL – Logical replication cache Aurora storage Amazon Aurora Users / Applications INSERT logical decoding AWS Database Migration Service (AWS DMS) INSERT wal log cache cache read

127. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Trusted Language Extensions (TLE) Builders create new libraries using TLE framework Customers choose TLE extensions for their apps DBAs control who can install and manage TLE extensions Available on Aurora and RDS PostgreSQL 14.5 and higher Application Extension code is installed by TLE A M A Z O N A U R O R A W I T H P O S T G R E S Q L C O M P A T I B I L I T Y , A M A Z O N R D S F O R P O S T G R E S Q L DBA C U S T O M E R B U I L D E R TLE Code PostgreSQL Open Source Project - Source code: https://github.com/aws/pg_tle - Apache 2.0 licensed

128. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Safely extend PostgreSQL capabilities T L E P R O V I D E S S A F E O P E R A T I O N O F P O S T G R E S Q L E X T E N S I O N S I N P R O D U C T I O N Enforces use of PostgreSQL trusted languages TLE API provides safe access to PostgreSQL internals Supports languages that have C-like performance Removes need for C expertise to create a safe extension Build and use extensions on your timeline Reduces DBA certification burden

129. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. How it works Build functions using a trusted PostgreSQL language Install functions using TLE PostgreSQL users can start using the extension using "CREATE EXTENSION" Collect feedback for making new versions of software Any "trusted” PostgreSQL procedural language can be used in a TLE JavaScript Perl Tcl PL/pgSQL Coming soon: Rust

131. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Serverless On-demand and automatically scaling configuration Automatically scales capacity based on application needs Simple pay-per-use pricing per second Scales instantly to support demanding applications Worry-free database capacity management

132. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instant, in-place scaling • Scales in place in under a second by adding more CPU and memory resources and billed by the second • No impact due to scaling even when running hundreds of thousands of transactions AWS Lambda Amazon Aurora

147. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Buffer pool resizing Buffer pool Access frequency Storage volume Reads Default memory allocation: 75% for buffer pool and 25% for heap Buffer pool size scaled along with capacity Parameters automatically adjusted: MySQL: innodb_buffer_pool_size PostgreSQL: shared_buffers Buffer pool scaled down through a combination of least frequently used (LFU) and least recently used (LRU) algorithms

148. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Buffer pool resizing Buffer pool Access frequency Storage volume Default memory allocation: 75% for buffer pool and 25% for heap Buffer pool size scaled along with capacity Parameters automatically adjusted: MySQL: innodb_buffer_pool_size PostgreSQL: shared_buffers Buffer pool scaled down through a combination of least frequently used (LFU) and least recently used (LRU) algorithms

149. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Buffer pool resizing Buffer pool Access frequency Storage volume Page read Page read Default memory allocation: 75% for buffer pool and 25% for heap Buffer pool size scaled along with capacity Parameters automatically adjusted: MySQL: innodb_buffer_pool_size PostgreSQL: shared_buffers Buffer pool scaled down through a combination of least frequently used (LFU) and least recently used (LRU) algorithms

150. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Buffer pool resizing Buffer pool Access frequency Storage volume Page read Default memory allocation: 75% for buffer pool and 25% for heap Buffer pool size scaled along with capacity Parameters automatically adjusted: MySQL: innodb_buffer_pool_size PostgreSQL: shared_buffers Buffer pool scaled down through a combination of least frequently used (LFU) and least recently used (LRU) algorithms

151. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Buffer pool resizing Buffer pool Access frequency Storage volume Evict cold pages Default memory allocation: 75% for buffer pool and 25% for heap Buffer pool size scaled along with capacity Parameters automatically adjusted: MySQL: innodb_buffer_pool_size PostgreSQL: shared_buffers Buffer pool scaled down through a combination of least frequently used (LFU) and least recently used (LRU) algorithms

154. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Buffer pool resizing Buffer pool Access frequency Storage volume Shrink memory Default memory allocation: 75% for buffer pool and 25% for heap Buffer pool size scaled along with capacity Parameters automatically adjusted: MySQL: innodb_buffer_pool_size PostgreSQL: shared_buffers Buffer pool scaled down through a combination of least frequently used (LFU) and least recently used (LRU) algorithms

155. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Buffer pool resizing Buffer pool Access frequency Storage volume Shrink memory Default memory allocation: 75% for buffer pool and 25% for heap Buffer pool size scaled along with capacity Parameters automatically adjusted: MySQL: innodb_buffer_pool_size PostgreSQL: shared_buffers Buffer pool scaled down through a combination of least frequently used (LFU) and least recently used (LRU) algorithms

157. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DevOps Guru for RDS A new Machine Learning (ML)-powered capability for Amazon RDS that automatically detects and diagnoses database performance and operational issues, enabling you to resolve bottlenecks in minutes

158. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. DevOps Guru for RDS • Finds DB performance anomalies • Analyzes the anomaly • Highlights: • Prevalent wait events • Prevalent SQL statements • Other anomalous metrics • Recommends next steps LOCKS 78% SELECT NAME FROM CUSTOMERS; SELECT ITEM FROM F; MEMORY What to do about locking issues…

160. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Introducing Amazon GuardDuty RDS Protection Protect your data stored in Amazon RDS databases (starting with Amazon Aurora) by identifying suspicious login attempts to mitigate threats before they escalate Continuously monitor all Aurora databases across your organization with a single click Use machine learning to reduce noise and avoid alert fatigue

161. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon GuardDuty RDS Protection: How does it work? Aurora Metrics ingestion Detect suspicious activity GuardDuty Amazon GuardDuty RDS Protection Explain + understand Remediate + GuardDuty RDS Protection

164. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. What GuardDuty RDS Protection can detect Amazon Aurora Brute Force Password Spraying/Guessing pgAdmin Stolen Credentials Virtual Private Cloud (VPC) ? Scanning

165. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. When GuardDuty RDS Protection detects a threat you can quickly take action Amazon Aurora Virtual Private Cloud (VPC) Security group

168. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. When GuardDuty RDS Protection detects a threat you can quickly take action Amazon Aurora Virtual Private Cloud (VPC) Security group X

173. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) create-blue-green-deployment RO RO

174. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO Target mycluster-green-x1234 Aurora MySQL 2.10.2 (5.7) • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO

175. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO Target mycluster-green-x1234 Aurora MySQL 3.02.2 (8.0)

176. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO Target mycluster-green-x1234 Aurora MySQL 3.02.2 (8.0)

177. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO Target mycluster-green-x1234 Aurora MySQL 3.02.2 (8.0) AVAILABLE

178. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO switchover-blue-green-deployment Target mycluster-green-x1234 Aurora MySQL 3.02.2 (8.0) AVAILABLE

179. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO switchover-blue-green-deployment Target mycluster-green-x1234 Aurora MySQL 3.02.2 (8.0) SWITCHOVER_IN_PROGRESS

184. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO switchover-blue-green-deployment SWITCHOVER_COMPLETED Target mycluster Aurora MySQL 3.02.2 (8.0) Source mycluster-old1 Aurora MySQL 2.10.2 (5.7)

185. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO switchover-blue-green-deployment Target mycluster Aurora MySQL 3.02.2 (8.0) Source mycluster-old1 Aurora MySQL 2.10.2 (5.7) delete-blue-green-deployment

186. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Aurora storage RW Users / Applications db cluster endpoint RO Source mycluster Aurora MySQL 2.10.2 (5.7) Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO RO RO switchover-blue-green-deployment Target mycluster Aurora MySQL 3.02.2 (8.0) Source mycluster-old1 Aurora MySQL 2.10.2 (5.7) delete-blue-green-deployment customer verification

187. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora Blue/Green Deployments Region Availability Zone 3 Availability Zone 1 Availability Zone 2 Users / Applications db cluster endpoint Aurora storage RO RW RO • Major/Minor Upgrades • Schema Changes • Static Parameter Changes • Maintenance Updates create-blue-green-deployment RO RO switchover-blue-green-deployment delete-blue-green-deployment Target mycluster Aurora MySQL 3.02.2 (8.0)

189. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Transaction analytics at scale Requires building and managing complex data pipelines Amazon Aurora Amazon Redshift Transactional Applications Analytics Applications DMS AWS Glue Amazon S3 Analytics Analyst Amazon EMR Amazon S3 Analyst Data Engineer

190. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Introducing: Amazon Aurora zero-ETL integration with Amazon Redshift An easy and secure way to enable near real-time analytics on petabytes of transactional data Amazon Aurora Amazon Redshift Transactional Applications Analytics Applications Analytics Analyst Analyst Data Engineer Zero-ETL integration

191. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Designed for simplicity and performance Aurora storage Amazon Redshift Amazon Aurora Redshift storage • Easy creation of Redshift integration target • Automated data seeding and continuous replication at storage layer • Run analytics along side ingestion • Monitor progress, lag, and performance from Redshift system tables and CloudWatch Client Application Zero-ETL Integration

192. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Operational analytics across many databases Amazon Aurora Amazon Aurora Amazon Aurora Amazon Redshift Zero-ETL Recommendation Engine 74 Amazon Aurora databases 200K-300K daily transaction <10 seconds lag 200TB 30B rows on largest table 70+ columns ML and analytics services Amazon Redshift ML BI and analytics apps Connect apps to analyze and visualize your data SQL Amazon Redshift ML ML and analytics services Analyze open standards- based data formats Amazon Redshift integration for Apache Spark

193. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Integration at the storage layer Aurora storage Amazon Redshift Amazon Aurora MySQL Redshift storage Transaction log Change data capture (CDC) log

194. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Integration at the storage layer Aurora storage Amazon Redshift Amazon Aurora MySQL Redshift storage Transaction log Change data capture (CDC) log Seed data parallel direct export

195. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Integration at the storage layer Aurora storage Amazon Redshift Amazon Aurora MySQL Redshift storage CDC Streaming Change data capture (CDC) log CDC from storage Replicate to Redshift Transaction log enhanced binlog

196. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Related sessions DAT221 Breakout PostgreSQL Trusted Language Extensions DAT222 Breakout Blue/Green Deployments DAT328 Breakout Amazon Aurora Zero-ETL to Amazon Redshift SEC218 Breakout Amazon GuardDuty RDS Protection

re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations

Similar to re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations (20)

More from Grant McAlister

More from Grant McAlister (11)

Recently uploaded

Recently uploaded (20)

re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations

Editor's Notes