Amazon Aurora: Database Week SF

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Amazon Aurora
Mahesh Pakala
Partner Solutions Architect
pakalam@amazon.com
Arabinda Pani
DB Specialist Solutions Architect
arabindp@amazon.com

What is Amazon Aurora ……
D atab ase r e im ag in e d fo r th e clo u d
R Speed and availability of high-end commercial databases
R Simplicity and cost-effectiveness of open source databases
R Drop-in compatibility with MySQL and PostgreSQL
R Simple pay as you go pricing
Delivered as a managed service

Re-imagining relational database
Automate administrative tasks – fully managed service
1
2
3
Scale-out and distributed design
Service-oriented architecture leveraging AWS services

Scale-out, distributed architecture
Master Replica Replica Replica
Availability
Zone 1
Shared storage volume
Availability
Zone 2
Availability
Zone 3
Storage nodes with SSDs
§ Purpose-built log-structured distributed
storage system designed for databases
§ Storage volume is striped across
hundreds of storage nodes distributed
over 3 different availability zones
§ Six copies of data, two copies in each
availability zone to protect against
AZ+1 failures
§ Plan to apply same principles to other
layers of the stack
SQL
Transactions
Caching
SQL
Transactions
Caching
SQL
Transactions
Caching

Leveraging cloud ecosystem
Lambda
S3
IAM
CloudWatch
Invoke Lambda events from stored procedures/triggers.
Load data from S3, store snapshots and backups in S3.
Use IAM roles to manage database access control.
Upload systems metrics and audit logs to CloudWatch.

Automate administrative tasks
• Schema design
• Query construction
• Query optimization
Automatic fail-over
Backup & recovery
Isolation & security
Industry compliance
Push-button scaling
Automated patching
Advanced monitoring
Routine maintenance
Takes care of your time-consuming database management tasks, freeing
you to focus on your applications and business
You
AWS

Aurora is used by ¾ of the top
100 AWS customers
Aurora customer adoption
Fastest growing service in AWS history

Who is moving to Aurora and why?
Customers using
commercial engines
Customers using MySQL
and PostgreSQL engines
§ Higher performance – up to 5x
§ Better availability and durability
§ Reduces cost – up to 60%
§ Easy migration; no application change
§ One tenth of the cost; no licenses
§ Integration with cloud ecosystem
§ Comparable performance and availability
§ Migration tooling and services

Amazon Aurora is fast …
5x more throughput than MySQL
3x more throughput than PostgreSQL

Do fewer I/Os
Minimize network packets
Cache prior results
Offload the database engine
DO LESS WORK
Process asynchronously
Reduce latency path
Use lock-free data structures
Batch operations together
BE MORE EFFICIENT
How did we achieve this?
DATABASES ARE ALL ABOUT I/O
NETWORK-ATTACHED STORAGE IS ALL ABOUT PACKETS/SECOND
HIGH-THROUGHPUT PROCESSING IS ALL ABOUT CONTEXT SWITCHES

BINLOG DATA DOUBLE-WRITELOG FRM FILES
TYPE OF WRITE
MYSQL WITH REPLICA
EBS mirrorEBS mirror
AZ 1 AZ 2
Amazon S3
EBS
Amazon Elastic
Block Store (EBS)
Primary
Instance
Replica
Instance
1
2
3
4
5
AZ 1 AZ 3
Primary
Instance
Amazon S3
AZ 2
Replica
Instance
ASYNC
4/6 QUORUM
DISTRIBUTED
WRITES
Replica
Instance
AMAZON AURORA
780K transactions
7,388K I/Os per million txns (excludes mirroring, standby)
Average 7.4 I/Os per transaction
MySQL I/O profile for 30 min Sysbench run
27,378K transactions 35X MORE
0.95 I/Os per transaction (6X amplification) 7.7X LESS
Aurora IO profile for 30 min Sysbench run
Aurora I/O profile

Scan
Delete
Aurora lock management
Scan
Delete
Insert
Scan Scan
Insert
Delete
Scan
Insert
Insert
MySQL lock manager Aurora lock manager
§ Same locking semantics as MySQL
§ Concurrent access to lock chains
Needed to support many concurrent sessions, high update throughput
§ Multiple scanners in individual lock chains
§ Lock-free deadlock detection

New performance enhancements
► Smart selector
► Logical read ahead
► Read views
► Hash joins
► Parallel query
Read performance
Write performance
Meta-data access
► NUMA aware scheduler
► Latch-free lock manager
► Instant schema update
► B-Tree concurrency
► Catalog concurrency
► Faster index build

What about availability?
“Performance only matters if your database is up”

• Six copies across three availability zones
• 4 out 6 write quorum; 3 out of 6 read quorum
• Peer-to-peer replication for repairs
• Volume striped across hundreds of storage nodes
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
Read and write availabilityRead availability
6-way replicated storage
S u rv iv e s cat ast ro p h ic fail u re s

Up to 15 promotable read replicas
Master
Read
Replica
Read
Replica
Read
Replica
Shared distributed storage volume
Reader end-point
► Up to 15 promotable read replicas across multiple availability zones
► Re-do log based replication leads to low replica lag – typically < 10ms
► Reader end-point with load balancing and auto-scaling

Cross-region read replicas
Fast e r disast e r r e co v e r y an d e n h an ce d dat a l o cal it y
• Promote read-replica to a
master for faster recovery in
the event of disaster
• Bring data close to your
customer’s applications in
different regions
• Promote to a master for easy
migration

Availability is about more than HW failures
• You also incur availability disruptions when you
1. Patch your database software – Zero Down Time Patch
2. Perform large scale database reorganizations – Fast Cloning
3. DBA errors requiring database restores – Online Point-in-time Restore

Database backtrack
• Backtrack brings the database to a point in time without requiring restore from backups
• Backtracking from an unintentional DML or DDL operation
• Backtrack is not destructive. You can backtrack multiple times to find the right point in time
t0 t1 t2
t0 t1
t2
t3 t4
t3
t4
Rewind to t1
Rewind to t3
Invisible Invisible

Fast database cloning
• Create a copy of a database without
duplicate storage costs
• Creation of a clone is nearly instantaneous –
we don’t copy data
• Data copy happens only on write – when
original and cloned volume data differ
Typical use cases:
• Clone a production DB to run tests
• Reorganize a database
• Save a point in time snapshot for analysis
without impacting production system.
Production database
Clone Clone
Clone
Dev/test
applications
Benchmarks
Production
applications
Production
applications

DevTest environments
Master Replica
Clone
Master Replica
Production Account Development Account
Cross-account snapshot
Test on prod data
• Create a clone of multi-terabyte DB clusters in minutes. Use for diagnosis and pre-prod
testing
• Restore a production DB cluster in a development accounts for development, staging, or
partnering with other teams

Performance Insights
• Dashboard showing
• Load on Database
– Easy
– Powerful
• Identifies source of
bottlenecks
– Top SQL
• Adjustable time frame
– Hour, day, week , month
– Up to 35 days of data
Max CPU

Hybrid operations and migration
Amazon Aurora
Corporate Data Center
Replication
Analytics
Prod apps
Provider of an online marketplace for short-term rentals replicates on-premise
production data into Aurora for read-only analytics workloads.
§ Seamless replicate to Aurora: Full compatibility with MySQL and PostgreSQL
§ Data replicated in minutes – run analytics on near real-time data
§ Provides a clean waypoint towards full write/read migration
MySQL

Higher performance, lower Cost
§ Fewer instances needed
§ Smaller instances can be used
Safe.com lowered their bill by 40% by switching from sharded
MySQL to a single Aurora instance.
Double Down Interactive (gaming) lowered their bill by 67%
while also achieving better latencies (most queries ran faster)
and lower CPU utilization.
§ No need to pre-provision storage
§ No additional storage for read replicas

Higher Performance, Lower Cost
Our application usage had grown exponentially over the last year. We were looking for
horizontal scaling of our database to address the increased load. Amazon Aurora’s
relatively low replication lag has helped us handle current load and positions us well for
future growth.
“
7x 10x
Database connections CPU utilization
2x
Response time
“

Aurora now offers MySQL 5.7 compatibility
Aurora MySQL 1.x = MySQL 5.6 compatible
Aurora MySQL 2.x = MySQL 5.7 compatible
• Available now
o JSON support
o Generated Columns
o Spatial indexes (already supported in Aurora 5.6)
o General performance improvements
• Coming later
o GTID (MySQL 8.0)
o Multi-source replication
o Selective replication

Pop-up Loft
Amazon Aurora Serverless

Serverless Computing + Relational Databases
• Starts up on demand, shuts down when not in use?
• Seamlessly scales with no instances to manage?
• Is billed per second for the database capacity you use?
• Is compatible with existing database applications?
What if you could have a relational database that:
Introducing Amazon Aurora Serverless:
On-demand, auto-scaling database for applications with variable workloads
Now available for Aurora MySQL!

Aurora Serverless Use Cases
• Infrequently used applications (e.g. low-volume blog
site)
• Applications with variable load - peaks of activity
that are hard to predict (e.g. news site)
• Development or test databases not needed on
nights or weekends
• Consolidated fleets of multi-tenant SaaS
applications

Serverless Endpoint Provisioning
• When you provision a database,
Aurora Serverless:
– Creates an Aurora storage volume
– Provisions proxy endpoint in your
VPC for the application
connection
– Configures network load
balancing behind proxy
– Initializes request routers to route
database traffic
– Provisions initial capacity
DATABASE STORAGE
APPLICATION
CUSTOMER VPC
VPC PROXY
ENDPOINTS
VPC
ENDPOINTS
NETWORK LOAD BALANCER
REQUEST
ROUTERS
INITIAL
CAPACITY

REQUEST
ROUTER
Scaling Up and Down
• Aurora monitors the application
load on the database (CPU,
memory, connections)
• When scaling thresholds are
reached, instance auto-scales up
or down (usually under 5
seconds)
• Scaling operations are
transparent to the application –
connections and session state are
transferred to new instance
• Buffer hot pages transferred
• You can configure minimum and
maximum capacity for scaling
• Scale manually via Web Service
API DATABASE STORAGE
CURRENT
INSTANCE
WARM POOL
NEW
INSTANCE
PROXY ENDPOINT
APPLICATION

Scaling to and from Zero
• If desired, instances are
removed after user-defined
period of inactivity
• While database is stopped,
you only pay for storage
• A database request triggers
instance provisioning (usually
around 20 seconds)
• Scaling to Zero is more
computationally expensive as
there is no running server
from which to copy the buffer
cache
• Connections are disconnected
when scaled to 0
WARM POOL
INSTANCE
DATABASE STORAGE
REQUEST
ROUTER
APPLICATION
PROXY ENDPOINT

Pop-up Loft
Amazon Aurora Multi-Master

Single-region
single-master
• 1 read-write instance
• 15 read-only replicas
Single-region
multi-master
• All database instances are read-write
Single-region
single-master
• 1 read-write instance
• 15 read-only replicas
Single-region
multi-master
• All database instances are read-write
• Now in preview for Aurora for MySQL
Multi-region
multi-master
• Active-active replication across regions
• Coming soon

Single-region Multi-Master: Use cases
Continuous write
availability
• Mission-critical workload needing continuous write
availability (e.g., reservation system)
• Sharded application that want to limit blast radius of an
instance failure (e.g., ecommerce)
• Write-intensive application needing more than 200K
writes/sec of r4.16xlarge (e.g., gaming)
• Applications in hyper-growth businesses needing to future-
proof DB platform choice
Write scale-out

Continuous write availability
SHARED STORAGE
M1 M2 M_n
App
Instance level
end-point
Instance level
end-point
Instance level
end-point

SHARED STORAGE
M1 M2 M_n
M1 failure;
transactions abort
App
Other nodes continue
to operate as-before
Connections
terminated

SHARED STORAGE
M1 M2 M_nM1 recovers
independently
App
to operate as-before
Redistribute connections from
failed node to other nodes

SHARED STORAGE
M1 M2 M_n
M1 recovery
complete
App
to operate
Add new
connections

SHARED STORAGE
M1 M2 M_n
M1 recovery
complete
App
to operate
Add new
connections
AZ1 AZ2 AZ3
Uninterrupted read-write availability to the multi-master cluster
even through AZ failure

Write scale-out in Aurora – conflict resolution w/ distrib. ledger
P1 P2
Aurora instance Aurora instance
Aurora storage
“Blue”
Master
“Orange”
Master
2 Commit (BT1) -
4 Commit (OT1) -
The storage nodes know transaction
orders applied at that node
3 Write (OT1, P1) OT1
Blue Master transaction = BT1
Orange Master transaction = OT1
BT1 OT1 Implied txn
sequence
1 Write (BT1, P1) BT1
Operation

P1 P2
Aurora storage
“Blue”
Master
“Orange”
Master
4 Commit (BT1) -
The storage nodes know transaction
orders applied at that node
2 Write (OT1, P1) BT1 à OT1
BT1 OT1 Implied txn
sequence
Operation
3 Conflict BT1 à OT1x

On Conflict transaction is rolled back
without waiting for commit
P1 P2
Aurora storage
“Blue”
Master
“Orange”
Master
2 Write (OT1, P2) BT1, OT1
3 Write (BT1, P2) OT1 à BT1
6 Commit (OT1) OT1
BT1 OT1
5 Write (OT1, P1) OT1
Implied txn
sequence
Operation
OT1 à BT14 Conflict x
x x

Multi-region Multi-Master
with cross-region physical replication
REGION 1 REGION 2
HEAD NODES HEAD NODES
MULTI-AZ STORAGE VOLUME MULTI-AZ STORAGE VOLUME
LOCAL PARTITION REMOTE PARTITION
Cross-region replication happens at in the storage layer,
and based on physical redo (no binlog)

Multi-region Multi-Master
Write accepted locally
Optimistic concurrency control – no distributed lock
manager, no chatty lock management protocol
REGION 1 REGION 2
HEAD NODES HEAD NODES
MULTI-AZ STORAGE VOLUME MULTI-AZ STORAGE VOLUME
LOCAL PARTITION LOCAL PARTITIONREMOTE PARTITION REMOTE PARTITION
Near-linear performance scaling when there is no or
low levels of conflicts

Pop-up Loft
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS

Amazon Aurora: Database Week SF

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Amazon Aurora: Database Week SF

Similar to Amazon Aurora: Database Week SF (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Amazon Aurora: Database Week SF