More Related Content Similar to Amazon Aurora (20) More from Amazon Web Services (20) Amazon Aurora1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Aurora
Mahesh Pakala
Partner Solutions Architect
pakalam@amazon.com
Arabinda Pani
DB Specialist Solutions Architect
arabindp@amazon.com
2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
What is Amazon Aurora ……
Datab ase re imag ine d for the clou d
Speed and availability of high-end commercial databases
Simplicity and cost-effectiveness of open source databases
Drop-in compatibility with MySQL and PostgreSQL
Simple pay as you go pricing
Delivered as a managed service
3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Re-imagining relational database
Automate administrative tasks – fully managed service
1
2
3
Scale-out and distributed design
Service-oriented architecture leveraging AWS services
4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Scale-out, distributed architecture
Master Replica Replica Replica
Availability
Zone 1
Shared storage volume
Availability
Zone 2
Availability
Zone 3
Storage nodes with SSDs
Purpose-built log-structured distributed
storage system designed for databases
Storage volume is striped across
hundreds of storage nodes distributed
over 3 different availability zones
Six copies of data, two copies in each
availability zone to protect against
AZ+1 failures
Plan to apply same principles to other
layers of the stack
SQL
Transactions
Caching
SQL
Transactions
Caching
SQL
Transactions
Caching
5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Leveraging cloud ecosystem
Lambda
S3
IAM
CloudWatch
Invoke Lambda events from stored procedures/triggers.
Load data from S3, store snapshots and backups in S3.
Use IAM roles to manage database access control.
Upload systems metrics and audit logs to CloudWatch.
6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Automate administrative tasks
• Schema design
• Query construction
• Query optimization
Automatic fail-over
Backup & recovery
Isolation & security
Industry compliance
Push-button scaling
Automated patching
Advanced monitoring
Routine maintenance
Takes care of your time-consuming database management tasks, freeing
you to focus on your applications and business
You
AWS
7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Aurora is used by ¾ of the top
100 AWS customers
Aurora customer adoption
Fastest growing service in AWS history
8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Who is moving to Aurora and why?
Customers using
commercial engines
Customers using MySQL
and PostgreSQL engines
Higher performance – up to 5x
Better availability and durability
Reduces cost – up to 60%
Easy migration; no application change
One tenth of the cost; no licenses
Integration with cloud ecosystem
Comparable performance and availability
Migration tooling and services
9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Aurora is fast …
5x more throughput than MySQL
3x more throughput than PostgreSQL
10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do fewer I/Os
Minimize network packets
Cache prior results
Offload the database engine
DO LESS WORK
Process asynchronously
Reduce latency path
Use lock-free data structures
Batch operations together
BE MORE EFFICIENT
How did we achieve this?
DATABASES ARE ALL ABOUT I/O
NETWORK-ATTACHED STORAGE IS ALL ABOUT PACKETS/SECOND
HIGH-THROUGHPUT PROCESSING IS ALL ABOUT CONTEXT SWITCHES
11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
BINLOG DATA DOUBLE-WRITELOG FRM FILES
TYPE OF WRITE
MYSQL WITH REPLICA
EBS mirrorEBS mirror
AZ 1 AZ 2
Amazon S3
EBS
Amazon Elastic
Block Store (EBS)
Primary
Instance
Replica
Instance
1
2
3
4
5
AZ 1 AZ 3
Primary
Instance
Amazon S3
AZ 2
Replica
Instance
ASYNC
4/6 QUORUM
DISTRIBUTED
WRITES
Replica
Instance
AMAZON AURORA
780K transactions
7,388K I/Os per million txns (excludes mirroring, standby)
Average 7.4 I/Os per transaction
MySQL I/O profile for 30 min Sysbench run
27,378K transactions 35X MORE
0.95 I/Os per transaction (6X amplification) 7.7X LESS
Aurora IO profile for 30 min Sysbench run
Aurora I/O profile
12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Scan
Delete
Aurora lock management
Scan
Delete
Insert
Scan Scan
Insert
Delete
Scan
Insert
Insert
MySQL lock manager Aurora lock manager
Same locking semantics as MySQL
Concurrent access to lock chains
Needed to support many concurrent sessions, high update throughput
Multiple scanners in individual lock chains
Lock-free deadlock detection
13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
New performance enhancements
► Smart selector
► Logical read ahead
► Read views
► Hash joins
► Parallel query
Read performance
Write performance
Meta-data access
► NUMA aware scheduler
► Latch-free lock manager
► Instant schema update
► B-Tree concurrency
► Catalog concurrency
► Faster index build
14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
What about availability?
“Performance only matters if your database is up”
15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Six copies across three availability zones
• 4 out 6 write quorum; 3 out of 6 read quorum
• Peer-to-peer replication for repairs
• Volume striped across hundreds of storage nodes
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
Read and write availabilityRead availability
6-way replicated storage
Su rvive s catastrop hic failu re s
16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Up to 15 promotable read replicas
Master
Read
Replica
Read
Replica
Read
Replica
Shared distributed storage volume
Reader end-point
► Up to 15 promotable read replicas across multiple availability zones
► Re-do log based replication leads to low replica lag – typically < 10ms
► Reader end-point with load balancing and auto-scaling
17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Cross-region read replicas
Faste r disaste r re cove ry and e nhance d data locality
• Promote read-replica to a
master for faster recovery in
the event of disaster
• Bring data close to your
customer’s applications in
different regions
• Promote to a master for easy
migration
18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Availability is about more than HW failures
• You also incur availability disruptions when you
1. Patch your database software – Zero Down Time Patch
2. Perform large scale database reorganizations – Fast Cloning
3. DBA errors requiring database restores – Online Point-in-time Restore
19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Database backtrack
• Backtrack brings the database to a point in time without requiring restore from backups
• Backtracking from an unintentional DML or DDL operation
• Backtrack is not destructive. You can backtrack multiple times to find the right point in time
t0 t1 t2
t0 t1
t2
t3 t4
t3
t4
Rewind to t1
Rewind to t3
Invisible Invisible
20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Fast database cloning
• Create a copy of a database without
duplicate storage costs
• Creation of a clone is nearly instantaneous –
we don’t copy data
• Data copy happens only on write – when
original and cloned volume data differ
Typical use cases:
• Clone a production DB to run tests
• Reorganize a database
• Save a point in time snapshot for analysis
without impacting production system.
Production database
Clone Clone
Clone
Dev/test
applications
Benchmarks
Production
applications
Production
applications
21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
DevTest environments
Master Replica
Shared storage volume
Clone
Shared storage volume
Master Replica
Production Account Development Account
Cross-account snapshot
Test on prod data
• Create a clone of multi-terabyte DB clusters in minutes. Use for diagnosis and pre-prod
testing
• Restore a production DB cluster in a development accounts for development, staging, or
partnering with other teams
22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Performance Insights
• Dashboard showing
• Load on Database
– Easy
– Powerful
• Identifies source of
bottlenecks
– Top SQL
• Adjustable time frame
– Hour, day, week , month
– Up to 35 days of data
Max CPU
23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Hybrid operations and migration
Amazon Aurora
Corporate Data Center
Replication
Analytics
Prod apps
Provider of an online marketplace for short-term rentals replicates on-premise
production data into Aurora for read-only analytics workloads.
Seamless replicate to Aurora: Full compatibility with MySQL and PostgreSQL
Data replicated in minutes – run analytics on near real-time data
Provides a clean waypoint towards full write/read migration
MySQL
24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Higher performance, lower Cost
Fewer instances needed
Smaller instances can be used
Safe.com lowered their bill by 40% by switching from sharded
MySQL to a single Aurora instance.
Double Down Interactive (gaming) lowered their bill by 67%
while also achieving better latencies (most queries ran faster)
and lower CPU utilization.
No need to pre-provision storage
No additional storage for read replicas
25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Higher Performance, Lower Cost
Our application usage had grown exponentially over the last year. We were looking for
horizontal scaling of our database to address the increased load. Amazon Aurora’s
relatively low replication lag has helped us handle current load and positions us well for
future growth.
“
7x 10x
Database connections CPU utilization
2x
Response time
“
26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Aurora now offers MySQL 5.7 compatibility
Aurora MySQL 1.x = MySQL 5.6 compatible
Aurora MySQL 2.x = MySQL 5.7 compatible
• Available now
o JSON support
o Generated Columns
o Spatial indexes (already supported in Aurora 5.6)
o General performance improvements
• Coming later
o GTID (MySQL 8.0)
o Multi-source replication
o Selective replication
27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Aurora Serverless
28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Serverless Computing + Relational Databases
• Starts up on demand, shuts down when not in use?
• Seamlessly scales with no instances to manage?
• Is billed per second for the database capacity you use?
• Is compatible with existing database applications?
What if you could have a relational database that:
Introducing Amazon Aurora Serverless:
On-demand, auto-scaling database for applications with variable workloads
Now available for Aurora MySQL!
29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Aurora Serverless Use Cases
• Infrequently used applications (e.g. low-volume blog
site)
• Applications with variable load - peaks of activity
that are hard to predict (e.g. news site)
• Development or test databases not needed on
nights or weekends
• Consolidated fleets of multi-tenant SaaS
applications
30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Serverless Endpoint Provisioning
• When you provision a database,
Aurora Serverless:
– Creates an Aurora storage volume
– Provisions proxy endpoint in your
VPC for the application
connection
– Configures network load
balancing behind proxy
– Initializes request routers to route
database traffic
– Provisions initial capacity
DATABASE STORAGE
APPLICATION
CUSTOMER VPC
VPC PROXY
ENDPOINTS
VPC
ENDPOINTS
NETWORK LOAD BALANCER
REQUEST
ROUTERS
INITIAL
CAPACITY
31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
REQUEST
ROUTER
Scaling Up and Down
• Aurora monitors the application
load on the database (CPU,
memory, connections)
• When scaling thresholds are
reached, instance auto-scales up
or down (usually under 5
seconds)
• Scaling operations are
transparent to the application –
connections and session state are
transferred to new instance
• Buffer hot pages transferred
• You can configure minimum and
maximum capacity for scaling
• Scale manually via Web Service
API DATABASE STORAGE
CURRENT
INSTANCE
WARM POOL
NEW
INSTANCE
PROXY ENDPOINT
APPLICATION
32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Scaling to and from Zero
• If desired, instances are
removed after user-defined
period of inactivity
• While database is stopped,
you only pay for storage
• A database request triggers
instance provisioning (usually
around 20 seconds)
• Scaling to Zero is more
computationally expensive as
there is no running server
from which to copy the buffer
cache
• Connections are disconnected
when scaled to 0
WARM POOL
INSTANCE
DATABASE STORAGE
REQUEST
ROUTER
APPLICATION
PROXY ENDPOINT
33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Aurora Multi-Master
34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Single-region
single-master
• 1 read-write instance
• 15 read-only replicas
Single-region
multi-master
• All database instances are read-write
Single-region
single-master
• 1 read-write instance
• 15 read-only replicas
Single-region
multi-master
• All database instances are read-write
• Now in preview for Aurora for MySQL
Multi-region
multi-master
• Active-active replication across regions
• Coming soon
35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Single-region Multi-Master: Use cases
Continuous write
availability
• Mission-critical workload needing continuous write
availability (e.g., reservation system)
• Sharded application that want to limit blast radius of an
instance failure (e.g., ecommerce)
• Write-intensive application needing more than 200K
writes/sec of r4.16xlarge (e.g., gaming)
• Applications in hyper-growth businesses needing to future-
proof DB platform choice
Write scale-out
36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Continuous write availability
SHARED STORAGE
M1 M2 M_n
App
Instance level
end-point
Instance level
end-point
Instance level
end-point
37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Continuous write availability
SHARED STORAGE
M1 M2 M_n
M1 failure;
transactions abort
App
Other nodes continue
to operate as-before
Connections
terminated
38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Continuous write availability
SHARED STORAGE
M1 M2 M_nM1 recovers
independently
App
Other nodes continue
to operate as-before
Redistribute connections from
failed node to other nodes
39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Continuous write availability
SHARED STORAGE
M1 M2 M_n
M1 recovery
complete
App
Other nodes continue
to operate
Add new
connections
40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Continuous write availability
SHARED STORAGE
M1 M2 M_n
M1 recovery
complete
App
Other nodes continue
to operate
Add new
connections
AZ1 AZ2 AZ3
Uninterrupted read-write availability to the multi-master cluster
even through AZ failure
41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Write scale-out in Aurora – conflict resolution w/ distrib. ledger
P1 P2
Aurora instance Aurora instance
Aurora storage
“Blue”
Master
“Orange”
Master
2 Commit (BT1) -
4 Commit (OT1) -
The storage nodes know transaction
orders applied at that node
3 Write (OT1, P1) OT1
Blue Master transaction = BT1
Orange Master transaction = OT1
BT1 OT1 Implied txn
sequence
1 Write (BT1, P1) BT1
Operation
42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Write scale-out in Aurora – conflict resolution w/ distrib. ledger
P1 P2
Aurora instance Aurora instance
Aurora storage
“Blue”
Master
“Orange”
Master
4 Commit (BT1) -
The storage nodes know transaction
orders applied at that node
2 Write (OT1, P1) BT1 OT1
Blue Master transaction = BT1
Orange Master transaction = OT1
BT1 OT1 Implied txn
sequence
1 Write (BT1, P1) BT1
Operation
3 Conflict BT1 OT1x
43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Write scale-out in Aurora – conflict resolution w/ distrib. ledger
On Conflict transaction is rolled back
without waiting for commit
P1 P2
Aurora instance Aurora instance
Aurora storage
“Blue”
Master
“Orange”
Master
2 Write (OT1, P2) BT1, OT1
3 Write (BT1, P2) OT1 BT1
6 Commit (OT1) OT1✅
Blue Master transaction = BT1
Orange Master transaction = OT1
BT1 OT1
5 Write (OT1, P1) OT1
Implied txn
sequence
1 Write (BT1, P1) BT1
Operation
OT1 BT14 Conflict x
x x
44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Multi-region Multi-Master
with cross-region physical replication
REGION 1 REGION 2
HEAD NODES HEAD NODES
MULTI-AZ STORAGE VOLUME MULTI-AZ STORAGE VOLUME
LOCAL PARTITION REMOTE PARTITION
Cross-region replication happens at in the storage layer,
and based on physical redo (no binlog)
45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Multi-region Multi-Master
Write accepted locally
Optimistic concurrency control – no distributed lock
manager, no chatty lock management protocol
REGION 1 REGION 2
HEAD NODES HEAD NODES
MULTI-AZ STORAGE VOLUME MULTI-AZ STORAGE VOLUME
LOCAL PARTITION LOCAL PARTITIONREMOTE PARTITION REMOTE PARTITION
Near-linear performance scaling when there is no or
low levels of conflicts
46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS