What's new in Amazon Aurora - ADB203 - Atlanta AWS Summit

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’s new in Amazon Aurora
A D B 2 0 3
Steve Abraham
Principal database specialist SA
AWS

Amazon Relational Database Service
(Amazon RDS)
Choice Value Innovation

Amazon RDS
Choice ofopen sourceand commercialdatabases
Amazon RDS platform
Open-source engines Commercial engines
Advanced monitoring
Routine maintenance
Push-button scaling
Automatic failover
Backup and recovery
X-region replication
Isolation and security
Industry compliance
Automated patching
Cloud-native engine

Aurora…
Enterprise database at open-sourceprice
Delivered as a managed service
Aurora
Speed and availability of high-end commercial databases
Simplicity and cost-effectiveness of open-source databases
Drop-in compatibility with MySQL and PostgreSQL
Simple pay-as-you-go pricing

Aurora innovations
Reimagining databases forthe cloud
Automate administrative tasks – Fully managed service
Scale-out, distributed, multi-tenant design
Service-oriented architecture leveraging AWS services

Scale-out, distributed architecture
Purpose-built log-structured distributed
storage system designed for databases
Storage volume is striped across
hundreds of storage nodes distributed over
three different Availability Zones
Six copies of data, two copies
in each Availability Zone to protect
against Availability Zone+1 failures
Shared storage volume
Storage nodes with SSDs
Availability
Zone 1
SQL
Transactions
Caching
Availability
Zone 2
SQL
Transactions
Caching
Availability
Zone 3
SQL
Transactions
Caching

Leveraging AWS services
Invoke Lambda events from stored
procedures/triggers
Load data from Amazon S3; store
snapshots and backups in
Amazon S3
AWS Lambda
function
Amazon
Simple Storage
Service
(Amazon S3)
AWS Identity
and Access
Management (IAM)
Amazon
CloudWatch
Use IAM roles to manage database
access control
Upload systems metrics
and audit logs to CloudWatch

Automate administrative tasks
Automatic failover
Backup and recovery
Isolation and security
Industry compliance
Push-button scaling
Automated patching
Advanced monitoring
Routine maintenance
Takes care of your time-consuming database-management tasks,
freeing you to focus on your applications and business
You AWS
Schema design
Query construction
Query optimization

Security management
Encryption to secure data at rest using customer-
managed keys
• AES-256; hardware accelerated
• All blocks on disk and in Amazon S3 are encrypted
• Key management via AWS Key Management Service
Encrypted cross-region replication, snapshot
copy; SSL to secure data in transit
Advanced auditing and logging without
any performance impact
Database-activity monitoring
Database
engine
*NEW*
Customer master key(s)
Data key 1
Storage
node
Storage
node
Storage
node
Storage
node
Data key 1 Data key 1 Data key 1

Industry certifications
Aurora gives each database instance IP
firewall protection
Aurora offers transparent encryption at
rest and SSL protection for data in
transit
Amazon Virtual Private Cloud lets you
isolate and control network
configuration and connect securely to
your IT infrastructure
IAM provides resource-level permission
controls
*New* *New* *New*

Aurora customer adoption
Fastest growing service in AWS history
Aurora is used by 3/4 of the top 100 AWS customers

Who are moving to Aurora and why?
Customers using
open-source engines
• Higher performance – Up to 5x
• Better availability and durability
• Reduces cost – Up to 60%
• Easy migration; no application change
Customers using
commercial engines
• One-tenth of the cost; no licenses
• Integration with cloud ecosystem
• Comparable performance and availability
• Migration tooling and services

Amazon Aurora migration options
Source database From where Recommended option
Amazon RDS
Amazon Elastic Cloud
Compute (Amazon EC2),
on premises
Amazon EC2, on premises, Amazon
RDS
Console-based automated
snapshot ingestion and catch-up
via binlog replication
Binary snapshot ingestion
through Amazon S3 and catch-
up via binlog replication
Schema conversion using AWS
Schema Conversion Tool and
data migration via AWS
Database Migration Service
(AWS DMS)

What’s new in AWS DMS and AWS SCT?
Workload qualification
Allows customers to estimate and manage the efforts required to migrate Oracle
and SQL Server workloads to Aurora
Migration playbook
Step-by-step instructions on how to migrate from Oracle and SQL Server to Aurora
Schema conversion
Automation from Oracle and SQL Server to Aurora is over 90%
Native start point
Customers can use engine-native utilities to copy data, such as PG dump and restore, and
replicate the changes using AWS DMS

Aurora manageability
Serverless, data API, advanced monitoring

Aurora Serverless use cases
Infrequently used applications
(e.g., low-volume blog site)
Applications with variable load – Peaks
of activity that are hard to predict (e.g.,
news site)
Development or test databases not
needed on nights or weekends
Consolidated fleets of multi-tenant
Software-as-a-Service applications

Aurora Serverless
Starts up on demand, shuts down when
not in use
Scales up and down automatically
No application impact when scaling
Pay per second, 1-minute minimum
Warm pool
of instances
Application
Database storage
Scalable database capacity
Request routers

Scale up and down with load
1
2
4
8
16
32
64
128
0
500
1000
1500
2000
2500
3000
1
12
23
34
45
56
67
78
89
100
111
122
133
144
155
166
177
188
199
210
221
232
243
254
265
276
287
298
309
320
331
342
353
364
375
386
397
408
419
430
441
452
463
474
485
496
507
518
529
540
551
562
573
584
595
606
617
628
639
650
661
672
683
694
705
716
727
TPS ACU

What’s new in Aurora Serverless
New regions
Seoul, Singapore, Sydney, Mumbai, London, Northern California, Paris, Frankfurt, Canada Central
Compliance and audit
FedRAMP, HIPPA, PCI, SOC, ISO, and HITRUST; publish logs to Amazon CloudWatch
Preview
Support for Aurora PostgreSQL
Preview
Support for REST Data API

Amazon RDS Data API for serverless applications
Millions of Internet of
Things (IoT)/
mobile devices
Data API fleet
API
Endpoint
Aurora
Serverless
Access through simple web interface
• Public endpoint addressable from anywhere
• No client configuration required
• No persistent connections required
Ideal for serverless applications (Lambda)
Ideal for lightweight applications (IoT)

Performance insights
Dashboard showing database load
▪ Easy – for example, drag and drop
▪ Powerful – drill down using zoom-in
Identifies source of bottlenecks
▪ Sort by top SQL
▪ Slice by host, user, wait events
Adjustable time frame
▪ Hour, day, week , month
▪ Up to 2 years of data; 7 days free
Max vCPU
CPU bottleneck
SQL w/ high CPU

CloudWatch
consumer
Database activity monitoring
Search: Look for specific events across log files
Metrics: Measure activity in your Aurora database cluster
• Continuously monitor activity in your database clusters by sending these audit logs to CloudWatch logs
• Export to Amazon S3 for long-term archival; analyze logs using Amazon Athena; visualize logs with Amazon
QuickSight
Visualizations: Create activity dashboards
Alarms: Get notified or take actions
Aurora
CloudWatch
Third-party
consumer
Amazon
Kinesis

Aurora performance
5x faster than MySQL; 3x faster than PostgreSQL

Write and read throughput
Aurora MySQLis 5x faster than MySQL
0
50,000
100,000
150,000
200,000
250,000
MySQL 5.6 MySQL 5.7 MySQL 8.0
Aurora 5.6 Aurora 5.7
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
MySQL 5.6 MySQL 5.7 MySQL 8.0
Aurora 5.6 Aurora 5.7
Write throughput Read throughput
Using Sysbench with 250 tables and 200,000 rows per table on R4.16XL

Copy in
Copy in
Vacuum
Vacuum
Index build
Index build
0 500 1000 1500 2000 2500 3000 3500 4000
PostgreSQL
Amazon Aurora
Runtime (seconds)
pgbench initialization, scale 10,000 (150 GiB)
86% reduction in vacuum time
Bulk data-load performance
Aurora PostgreSQLloads data 2x fasterthan PostgreSQL

Bulk data-load performance
Aurora MySQLloads data 2.5x fasterthan MySQL
Data loading
Data loading
Index build
Index build
0 100 200 300 400 500 600 700 800
MySQL
Amazon
Aurora
Runtime (sec.)
10 Sysbench tables, 10-MM rows per each

0.00
100.00
200.00
300.00
400.00
500.00
600.00
700.00
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Responsetime,ms
Sysbench response time (p95), 30 GiB, 1,024 clients
PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)
Performance variability under load
AuroraPostgreSQLis ~10x moreconsistent than PostgreSQL

Sysbench write-only workload with 250 tables and 200,000 initial rows per table
0
500
1,000
1,500
2,000
2,500
0 100 200 300 400 500 600
Time from start of run (sec.)
Write response time (ms) Amazon Aurora MySQL
Performance variability under load
Aurora MySQLis ~25x moreconsistent than MySQL

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
0
50
100
150
200
250
2015 2016 2017 2018
Max write throughput – up 100%
0
100
200
300
400
500
600
700
800
2015 2016 2017 2018
Max read throughput – up 42%
Launched with R3.8xl
32 cores, 256-GB memory
Now support R4.16xl
R5.24xl coming soon
Besides many performance optimizations, we are also upgrading hardware platform
Performance improvement over time
AuroraMySQL:2015–2018

How did we achieve this?
Do less work
• Do fewer I/Os
• Minimize network packets
• Cache prior results
• Offload the database engine
Be more efficient
• Process asynchronously
• Reduce latency path
• Use lock-free data structures
• Batch operations together
• Databases are all about I/O
• Network-attached storage is all about packets/second
• High-throughput processing is all about context switches

Aurora I/O profile
MySQL with Replica Aurora
Amazon EBS mirrorAmazon EBS mirror
Availability Zone 1 Availability Zone 2
Amazon EBS
Amazon Elastic Block
Store (Amazon EBS)
Primary
instance
Replica
instance
1
2
3
4
5
Amazon
S3
MySQL I/O profile for 30-minute Sysbench run
780,000 transactions
7,388,000 I/Os per million transactions (excludes mirroring, standby)
Average of 7.4 I/Os per transaction
Availability Zone 1 Availability Zone 3
Primary
instance
Availability Zone 2
Replica
instance
Async 4/6 Quorum
Distributed writes
Replica
instance
Amazon
S3
Aurora I/O profile for 30-min. Sysbench run
27,378K transactions – 35x more
0.95 I/Os per transaction (6x amplification) – 7.7x less
Binlog Data Double-writeLog From files

Aurora lock management
MySQL lock manager Aurora lock manager
Scan
Delete
Insert
Scan
Scan
Delete
Scan
Insert
Insert
Delete
Insert
Scan
Same locking semantics as MySQL
Concurrent access to lock chains
Multiple scanners in individual lock chains
Lock-free deadlock detection
Needed to support many concurrent sessions, high-update throughput

Asynchronous key
prefetch (AKP)
▪ Works for queries using Batched
Key Access (BKA) join algorithm
and Multi-Range Read (MRR)
optimization
▪ Performs a secondary to primary
index lookup during JOIN evaluation
▪ Uses background thread
to asynchronously load pages
into memory
Latency improvement factor vs. BKA join algorithm. Decision support
benchmark, R3.8xlarge.
14.57
-
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
Cold buffer
AKP used in queries 2, 5, 8, 9, 11, 13, 17, 18, 19, 20, 21

Batched scans
Standard MySQL processes rows one-at-a-time,
resulting in high overhead due to
• Repeated function calls
• Locking and latching
• Cursor store and restore
• InnoDB to MySQL format conversion
Aurora scan tuples from
the InnoDB buffer pool in batches for
• Table full scans
• Index full scans
• Index range scans Latency improvement factor vs. BKA join algorithm. Decision support
benchmark, R3.8xlarge.
1.78X
-
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00

Parallel query processing
Aurora storage has thousands of CPUs
• Opportunity to push down and parallelize query
processing
• Moving processing close to data reduces network
traffic and latency
However, there are significant challenges
• Data is not range partitioned – require full scans
• Data may be in-flight
• Read views may not allow viewing most recent data
• Not all functions can be pushed down
Database node
Storage nodes
Push down
predicates
Aggregate
results

Well-known decision support benchmark
0x
20x
40x
60x
80x
100x
120x
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22
Query response time reduction
▪ Peak speed up ~120x
▪ > 10x speedup: 8 of 22 queries
We were able to test Aurora’s parallel query feature and the performance gains were very
good. To be specific, we were able to reduce the instance type from r3.8xlarge to
r3.2xlarge. For this use case, parallel query was a great win for us.
Jyoti Shandil, Cloud data architect
Performance results

What about availability
“Performance matters only if your database is up”

Six-way replicated storage
Survives catastrophic failures
• Six copies across three Availability
Zones
• Four out of six write quorum;
three out of six read quorum
• Peer-to-peer replication
for repairs
• Volume striped across hundreds
of storage nodes
SQL
Transaction
Availability Zone 1 Availability Zone 2 Availability Zone 3
Caching
SQL
Transaction
Caching

Read replica and custom endpoint
Master
Read
replica
Read
replica
Read
replica
Shared distributed storage volume
Reader endpoint #1 Reader endpoint #2
Up to 15 promotable read replicas across multiple Availability Zones
Redo-log-based replication leads to low replica lag—typically < 10 ms
Custom reader endpoint with configurable failover order

Instant crash recovery
Traditional database
Have to replay logs since
the last checkpoint
Typically 5 minutes between checkpoints
Single-threaded in MySQL; requires a large
number of disk accesses
Aurora
Underlying storage replays redo records on
demand as part of a disk read
Parallel, distributed, asynchronous
No replay for startup
Checkpointed data Redo log
Crash at T0 requires
a re-application of the
SQL in the redo log since
last checkpoint
T0 T0
Crash at T0 will result in redo logs being applied to
each segment on demand, in parallel,
asynchronously

Continuous availability with multimaster
Master Master Master Master
Shared distributed-storage volume
Application #1 Application #2

Scaling write workload
Optimistic conflict management
There are many “oases” of consistency in
Aurora
Database nodes know transaction orders
from that node
Storage nodes know transactions orders
applied at that node
Only have conflicts when data changed at
both multiple database nodes and
multiple storage nodes
Much less coordination required
Near linear throughput scaling for
workloads with no or low conflict
Lower commit latency for workloads with
low conflict

Transactions with no conflict
▪ Transactions T1 and T2 from Blue and
Orange masters update different tables
(pages) – Table 1 and Table 2
▪ No logical or physical conflicts; no
coordination required
▪ Near-linear scaling with number of masters
for this type of sharded or partitioned
workloads
1 1 1 1 11 2 2 2 2 22
Page1 Page2
Master
Master
Begin transaction
(T1)
Update (Table1)
Commit (T1)
Begin transaction
(T2)
Update (Table2)
Commit (T2)
Table 1 Table 2

Conflict resolution
1 1 11
Page1 Page2
Master
Master
Begin transaction
(T1)
Update (Table1)
Commit (T1)
Begin transaction
(T2)
Update (Table1)
Rollback (T2)
Table 1 Table 2
▪ Transactions T1 and T2 from Blue and Orange
masters update the same table (page) – Table 1
▪ Transaction T1 from Blue master achieves
quorum – Transaction T1 is committed
▪ Transaction T2 from Orange master fails to
achieve quorum – Transaction T2 is rolled back
▪ Storage nodes act as arbitrator for conflict
resolution
▪ Logical conflicts are handled through Multi-
Version Concurrency Control
X
2 2

Aurora multimaster: Scaling and availability
0
10000
20000
30000
40000
50000
60000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99 101103105107109111113115117119121123125
Aggregatedthroughput
Time in minutes
Sysbench workload on 4 R4.XL nodes
Adding a node Adding a node Node going down Node recovering

Global replication: Logical
Faster disaster recovery and enhanceddata locality
▪ Promote read-replica to a master
for faster recovery
in the event of disaster
▪ Bring data close to your customer’s
applications
in different regions
▪ Promote to a master
for easy migration

High throughput: Up to 150,000 writes per second – Negligible performance impact
Low replica lag: < 1 second cross-country replica lag under heavy load
Fast recovery: < 1 minute to accept full read-write workloads after region failure
Global replication: Physical
MR R
Region 1
Shared storage
R
Region 2
Shared storage
Replication
fleet
Replication
fleet

Global replication performance
Logical vs. physical replication
Logical replication Physical replication
0
100
200
300
400
500
600
0
50,000
100,000
150,000
200,000
250,000
seconds
QPS
QPS
Lag
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
0
50,000
100,000
150,000
200,000
250,000
seconds
QPS
QPS
Lag

Database backtrack
t0 t1 t2
t0 t1
t2
t3 t4
t3
t4
Rewind to t1
Rewind to t3
Invisible Invisible
Backtrack brings the database to a point in time without requiring restore from backups
• Backtracking from an unintentional Data Manipulation Language or Data Definition Language operation
• Backtrack is not destructive; you can backtrack multiple times to find the right point in time
• Also useful for quality assurance (rewind your database between test runs)

What's new in Amazon Aurora - ADB203 - Atlanta AWS Summit

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to What's new in Amazon Aurora - ADB203 - Atlanta AWS Summit

Similar to What's new in Amazon Aurora - ADB203 - Atlanta AWS Summit (20)

More from Amazon Web Services

More from Amazon Web Services (20)

What's new in Amazon Aurora - ADB203 - Atlanta AWS Summit