SlideShare a Scribd company logo
1 of 77
S U M M I T
• Madrid
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Bases de datos relacionales escalables con
Amazon Aurora
Javier Ramirez
@supercoco9
Tech Evangelist
AWS
• C 1
Ignasi Nogués
@inogues
CEO
Clickedu
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Aurora: Enterprise database at open source price
Fastest growing service in AWS history. Aurora is used by ¾ of the top 100 AWS customers
Which type of
phone are you
carrying today?
If you got over your phone from the
2000s…
why are you still using databases that
follow the design patterns of the 80s?
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Local
Storage
SQL
Transactions
Caching
Logging
Compute
Traditional Database Architecture
• Monolithic stack in a Single box
• Large blast radius
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Quick recap: Database internals contd…
• Recovery
Tc
Tf
Tx1
Tx2
Tx3
Tx4
checkpoint system failure
 Tx1 can be ignored as checkpoint has been taken after its commit
 Tx2 and Tx3 redone using REDO procedure
 Tx4 is undone using REDO/UNDO procedure
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Traditional Distributed Database stack
Storage
Application Application Application
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Storage StorageStorage Storage
 Same Monolithic stack  Distributed consensus
algorithms perform poorly
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora: Scale-out, Distributed architecture
• Push Log applicator to Storage
Master Replica Replica Replica
Master
Shared storage volume
Replica Replica
SQL
Transactions
Caching
SQL
Transactions
Caching
SQL
Transactions
Caching
AZ1 AZ2 AZ3
 Write performance
 Read scale out
 AZ + 1 failure tolerance
• 4/6 Write Quorum & Local tracking
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Read replica and custom endpoint
Master
Read
replica
Read
replica
Read
replica
Shared distributed storage volume
Reader endpoint #1 Reader endpoint #2
Up to 15 promotable read replicas across multiple availability zones
Re-do log based replication leads to low replica lag – typically < 10ms
Custom reader end-point with configurable failover order
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
• Partition volume into 𝑛 fixed size segments
• Replicate each segment 6-ways into a Protection Group (PG)
• A single PG failing is enough to fail the entire volume
• Trade-off between likelihood of faults and time to repair
• If segments are too small then failures are more likely
• If segments are too big then repairs take too long
• Choose the biggest size that lets us repair “fast enough”
• We currently picked a segment size of 10GB as we can repair a 10GB
segment in ~10 seconds on a 10Gbps link
Segmented storage
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Continuous backup
• Take periodic snapshot of each segment in parallel; stream the redo logs to Amazon S3
• Backup happens continuously without performance or availability impact
Segment snapshot Log records
Recovery point
Segment 1
Segment 2
Segment 3
Time
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Instant crash redo recovery
Traditional database
• Replay logs since the last checkpoint
• Slow replay in the single thread
Amazon Aurora
• At restore, retrieve the appropriate segment
snapshots and log streams to storage nodes
Checkpointed Data Redo Log
Crash at T0 requires
a re-application of the
SQL in the redo log since
last checkpoint
T0 T0
Crash at T0 will result in redo logs being
applied to each segment on demand, in
parallel, asynchronously
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Database backtrack
• Backtrack brings the database to a point in time without requiring restore from backups
• Backtracking from an unintentional DML or DDL operation
• Backtrack is not destructive. You can backtrack multiple times to find the right point in time
t0 t1 t2
t0 t1
t2
t3 t4
t3
t4
Rewind to t1
Rewind to t3
Invisible Invisible
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Fast Database Cloning
• Create a copy of a database without duplicate storage
costs
• Creation of a clone is instantaneous since it doesn’t
require deep copy
• Data copy happens only on write – when original
and cloned volume data differ
– Typical use cases:
• Clone a production DB to run tests
• Reorganize a database
• Save a point in time snapshot for analysis without
impacting production system Production database
Clone Clone
Clone
Dev/test
applications
Benchmarks
Production
applications
Production
applications
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Page
1
Page
2
Page
3
Page
4
Page
5
Source database Cloned database
As databases diverge, new pages are added appropriately to each database
while still referencing pages common to both databases
Page
1
Page
2
Page
3
Page
4
Page
5
Page
6
Page
1
Page
3
Page
5
Page
3
Page
5
Protection group 1
Page
2
Page
4
Page
6
Page
2
Protection group 2
Shared distributed storage system
Database Cloning: How does it work?
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Write and read throughput
Aurora MySQL is 5x faster than MySQL
0
50,000
100,000
150,000
200,000
250,000
MySQL 5.6 MySQL 5.7 MySQL 8.0
Aurora 5.6 Aurora 5.7
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
MySQL 5.6 MySQL 5.7 MySQL 8.0
Aurora 5.6 Aurora 5.7
Write throughput Read throughput
Using Sysbench with 250 tables and 200,000 rows per table on R4.16XL
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Copy In
Copy In
Vacuum
Vacuum
Index Build
Index Build
0 500 1000 1500 2000 2500 3000 3500 4000
PostgreSQL
Amazon Aurora
Runtime (seconds)
pgbench initialization, scale 10000 (150 GiB)
86% reduction in vacuum time
Bulk data load performance
Aurora PostgreSQL loads data 2x faster than PostgreSQL
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Bulk data load performance
Aurora MySQL loads data 2.5x faster than MySQL
Data loading
Data loading
Index build
Index build
0 100 200 300 400 500 600 700 800
MySQL
Amazon
Aurora
Runtime (sec.)
10 Sysbench Tables, 10MM rows per each
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
0.00
100.00
200.00
300.00
400.00
500.00
600.00
700.00
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Responsetime,ms
SYSBENCH RESPONSE TIME (p95), 30 GiB, 1024 CLIENTS
PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)
Performance variability under load
Aurora PostgreSQL is ~10x more consistent than PostgreSQL
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Sysbench write-only workload with 250 tables and 200,000 initial rows per table
0
500
1,000
1,500
2,000
2,500
0 100 200 300 400 500 600
Time from start of run (sec.)
Write response time (ms.) Amazon Aurora MySQL
Performance variability under load
Aurora MySQLis ~25x moreconsistent than MySQL
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
0
50
100
150
200
250
2015 2016 2017 2018
Max write throughput – up 100%
0
100
200
300
400
500
600
700
800
2015 2016 2017 2018
Max read throughput – up 42%
Launched with R3.8xl
32 cores, 256-GB memory
Now support R4.16xl
64 cores, 512-GB memory
R5.24xl coming soon
96 cores, 768-GB memory
Besides many performance optimizations, we are also upgrading HW platform
Performance improvement over time
Aurora MySQL –2015-2018
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How did we achieve this?
• Do less work
• Do fewer I/Os
• Minimize network packets
• Cache prior results
• Offload the database engine
• Be more efficient
• Process asynchronously
• Reduce latency path
• Use lock-free data structures
• Batch operations together
• Databases are all about I/O
• Network-attached storage is all about packets/second
• High-throughput processing is all about context switches
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
BINLOG DATA DOUBLE-WRITELOG FRM FILES
TYPE OF WRITE
MYSQL WITH REPLICA
EBS mirrorEBS mirror
AZ 1 AZ 2
EBS
Amazon Elastic
Block Store (EBS)
Primary
Instance
Replica
Instance
1
2
3
4
5
AZ 1 AZ 3
Primary
Instance
AZ 2
Replica
Instance
ASYNC
4/6 QUORUM
DISTRIBUTED
WRITES
Replica
Instance
Amazon S3
AMAZON AURORA
0.78MM transactions
7.4 I/Os per transaction
MySQL I/O profile for 30 min Sysbench run
27MM transactions 35X MORE
0.95 I/Os per transaction 7.7X LESS
Aurora IO profile for 30 min Sysbench run
MySQL vs. Aurora I/O profile
Amazon S3
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Driving down query latency
Asynchronous Key
Prefetch
Hash joinsBatched scans
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Performance Improvement
TPC –C benchmark
0x
2x
4x
6x
8x
10x
12x
14x
16x
18x
20x
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10Q11Q12Q13Q14Q15Q16Q17Q18Q19Q20Q21Q22
Query response time reduction
 Peak speed up ~18x
 >2x speedup: 10 of 22 queries
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Intelligent Vacuum prefetch
PostgreSQL
Aurora PostgreSQL
Submit
Batch I/O
up to
256 blocks
402 seconds
163 seconds
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Parallel query processing
• Aurora storage has thousands of CPUs
• Opportunity to push down and parallelize query
processing
• Moving processing close to data reduces network
traffic and latency
• However, there are significant challenges
• Data is not range partitioned – require full scans
• Data may be in-flight
• Read views may not allow viewing most recent data
• Not all functions can be pushed down
Database Node
Storage nodes
Push down
predicates
Aggregate
results
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Database node processing
• Query Optimizer produces PQ Plan and creates PQ
context based on leaf page discovery.
• PQ request is sent to storage node along with PQ
context.
• Storage node produces:
• Partial results streams with processed stable rows.
• Raw stream of unprocessed rows with pending undos.
• Head node aggregates these data streams to produce
final results. STORAGE NODES
OPTIMIZER
EXECUTOR
INNODB
NETWORK STORAGE DRIVER
AGGREGATOR
APPLICATION
PARTIAL
RESULTS
STREAM
RESULTS
IN-FLIGHT
DATA
PQ CONTEXT
PQ PLAN
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Aurora PQ summary
• Performance
• 120x lower latencies on TPCH-like benchmarks with Improved I/O performance and reduced CPU usage on the head node.
• High Concurrency
• Run both OLTP and light OLAP workloads simultaneously and efficiently.
• Cost Effective
• PQ comes at no extra cost. Can run on your live data. Potentially reduced effort and data duplication in your ETL pipeline.
• Quiet Tenant
• Reduced chance of evicting frequently used pages from the buffer pool that are used by OLTP workload.
• Ecosystem
• Get Aurora goodies such as PiTR, Continuous backup, Fast Cloning with PQ.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Parallel Query - Performance results
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TPC-C benchmark
0x
20x
40x
60x
80x
100x
120x
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10Q11Q12Q13Q14Q15Q16Q17Q18Q19Q20Q21Q22
Query response time reduction
 Peak speed up ~120x
 >10x speedup: 8 of 22 queries
We were able to test Aurora’s parallel query feature and the performance gains
were very good. To be specific, we were able to reduce the instance type from
r3.8xlarge to r3.2xlarge. For this use case, parallel query was a great win for us.
Jyoti Shandil, Cloud Data Architect
Performance results
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Global replication: Logical
Faster disaster recovery and enhanced data locality
 Promote read-replica to a master
for faster recovery
in the event of disaster
 Bring data close to your customer’s
applications
in different regions
 Promote to a master
for easy migration
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Global physical replication
• Primary region • Secondary region
1
ASYNC 4/6 QUORUM
Continuous
backup
AZ 1
Primary
Instance
Amazon
S3
AZ 2
Replica
Instance
AZ 3
Replica
Instance
Replication
Server
Replication Fleet
Storage Fleet
11
4
AZ 1
Replica
Instance
AZ 2 AZ 3
ASYNC 4/6 QUORUM
Continuous
backup
Amazon
S3
Replica
Instance
Replica
Instance
Replication
Agent
Replication Fleet
Storage Fleet
3
3
2
① Primary instance sends log records in parallel to storage nodes, replica
instances and replication server
② Replication server streams log records to Replication Agent in secondary
region
③ Replication agent sends log records in parallel to storage nodes, and replica
instances
④ Replication server pulls log records from storage nodes to catch up after
outages
High throughput: Up to 150K writes/sec – negligible
performance impact
Low replica lag: < 1 sec cross-region replica lag under heavy
load
Fast recovery: < 1 min to accept full read-write workloads after
region failure
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Global replication performance
Logical replication Physical replication
0
100
200
300
400
500
600
0
50,000
100,000
150,000
200,000
250,000
seconds
QPS
QPS Lag
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
0
50,000
100,000
150,000
200,000
250,000
seconds
QPS
QPS Lag
Logical vs. physical replication
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Master
Replica
Orange Master Blue Master
SQL
Transactions
Caching
SQL
Transactions
Caching
Aurora Multi-Master Architecture
Shared Storage Volume
 No Pessimistic Locking
 No Global Ordering
 No Global Commit-Coordination
Replica
• Membership
• Heartbeat
• Replication
• Metadata
Cluster Services
1
1
1
1 1
1
2
2
2
2 2
2
3 3
3 3
3 3
1 3?
T1 T2
AZ1
AZ2
AZ3
Decoupled
Decoupled
Decoupled
 Decoupled System
 Microservices Architecture
2
 Optimistic Conflict Resolution
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scaling write workload
Optimistic conflict management
There are many “oases” of
consistency in Aurora
Database nodes know transaction
orders from that node
Storage nodes know transactions
orders applied at that node
Only have conflicts when data
changed at both multiple database
nodes AND multiple storage nodes
Much less coordination required
Near linear throughput scaling for
workloads with no or low conflict
Lower commit latency for workloads
with low conflict
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora Multi-Master: Scaling and availability
0
10000
20000
30000
40000
50000
60000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99 101103105107109111113115117119121123125
AggregatedThroughput
Time in minutes
Sysbench workload on 4 R4.XL nodes
Adding a node Adding a node Node going down Node recovering
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Aurora Serverless
 Starts up on demand, shuts down
when not in use
 Scales up/down automatically
 No application impact when scaling
 Pay per second, 1 minute minimum
WARM POOL
OF INSTANCES
APPLICATION
AURORA STORAGE
AURORA
REQUEST ROUTER
DATABASE END-POINT
AURORA STORAGE
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scale up and down with load
1
2
4
8
16
32
64
128
0
500
1000
1500
2000
2500
3000
1
12
23
34
45
56
67
78
89
100
111
122
133
144
155
166
177
188
199
210
221
232
243
254
265
276
287
298
309
320
331
342
353
364
375
386
397
408
419
430
441
452
463
474
485
496
507
518
529
540
551
562
573
584
595
606
617
628
639
650
661
672
683
694
705
716
727
TPS ACU
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Simpler experience: Less to worry about
• No CPU credits to monitor
• No commitment to particular availability zone
• No migrations between instance type generations
• No DB instance reservations to manage
• No instances to manage
• Encryption at REST is always enabled
• No need to manually suspend and resume database
• No DNS propagation delays
• No maintenance window
• No old database versions to upgrade
Clickedu + AWS: Better together
Learning and
administration
activities happen in
and outside
schools. Data needs
to be accessible from
anywhere, on any
device.
Education trends
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
School challenges
• Hard to access to the right
information for each user
• Curriculum, operations and data
related to performance is
unorganized or kept in separate
systems
• Teachers, students, parents and
administrators need access
wherever they are
• Need to connect data from all
parts of the school ecosystem
Not enough time Hard to find data Need mobile access
CLASSROOM ADMINISTRATION ACADEMIC COMMUNICATION
QUALITY TOOLS 100% CLOUD EASY LMS AND CONNECTABLE EDUCATIVE CONTENTS
The MIS for K-12: an educative E.R.P.
functionality
Administration Controls
• Reception Notices
• Room Booking
• Inventory Management
• Staff attendance and cover
module
• Documentation
Management
• Grades
• Payments
• EFQM & ISO quality
Services
• Extracurricular activities
• Transport and catering
• Special education
• Flipped classroom
• Individual timetables
• Student documentation
• Pupil Census
• General reporting
• Workforce census
Communication
• SMS & Email
• Centralized Communication
• Internal Messenger
• Contacts & Calendars
• Integrate with Google Mail
and Microsoft 365
• Parent Portal
• Behaviour Management
• Survey, newsletter, website
creation
LMS & Assessments
• Complete student profiles
• Collaborative Learning
• Recommendations
• Forums
• Projects
• Submission Box
• Self-Assessment Tests
• Rubrics
• Grades and assessment
• Official grade
documentation
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Why AWS?
• Private cloud never was enough but most of the
time was so much
• Problems of free space on File System
• Huge use of data, no scalable database systems
• There were frequent errors doing things “by
hand”
• Security
• Complaints because we had to stop our servers
• It was difficult to test on real time
• No global servers
Global – AWS helps on going around the
planet
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
“Unlimited” &
elastic
Autoscaling
Advantages
• Use of CloudWatch
• Alarm System with CloudWatch
• SES use for emails
• S3 with versioning and separate files from
code
• Environments to test new versions are easy
to create
Other uses of AWS
•AI with the use of
Sagemaker
•Integrate voice
systems (LEX; Polly)
•Integrate Alexa &
Chime
RDS Advantage
• 45 days of RDS snapshots
• Autoscaling of RDS with Aurora
• No stops on the new versions
• Data Encryption & access control
RDS evolution
PRIVATE CLOUD MYSQL
4 Cluster MASTER-SLAVE (1-1)
db.r3.2xlarge
AWS - AURORA
MASTER + 4 SLAVE
db.r3.2xlarge
PRIVATE CLOUD MYSQL
2 Clusters MASTER-SLAVE (1-1)
db.r3.2xlarge
AWS MYSQL
4 Clusters MASTER-SLAVE
db.r3.xlarge
AWS- AURORA
MASTER + 2 SLAVE IN
AUTOSCALING MODEL
db.r3.2xlarge
2013 2014 2015 2017 2019
PRIVATE CLOUD MYSQL
4 Cluster M-S (1-2)
db.r3.2xlarge + db.r3.xlarge
Statistics (by minutes), one MySQL in 2014
Migration
• Lots of Planning
• Partner help
• We connect an AWS RDS
as a slave to the previous
private cloud
• Low downtime, rather
because of the DNS than
to the switch from SLAVE
to MASTER
Aurora Migration
• Lots of Planning
• Well advised
• Hard work reviewing the platform to be sure
all was compatible but 0 issues found.
• All process tested before
• First cluster so easy to migrate (automated
with AWS service, less than 1 hour)
• Hard to find a screen to do the change for the
other 3 clusters:
– Downtime expected of 16 hours
– Had to be summer on Spain/Andorra/UK; we found
also a suitable week for LATAM (Chile & Colombia)
• Transfer from other clusters made with dumps
to be sure all was right.
– Problems with rights and users
• Finally the transfer was 8 hours for cluster
250
320
400
500
600
700
280
358
448
560
672
1047
308
394
493
616
739
1243
0
200
400
600
800
1,000
1,200
1,400
PRI PRI PRI AWS OD AWS RESERVA AURORA SC
2013 2014 2015 2016 2017 2019
Plattform evolution
NÚMERO BASES DATOS ESPACIO (en TB) IOPS (en millones)
Data in time
Annual cost €51,260.64
€73,781.28
€85,041.60
€101,094.06
€57,423.70
€44,410.65
€38,645.35
€-
€20,000.00
€40,000.00
€60,000.00
€80,000.00
€100,000.00
€120,000.00
1
2013 PRI 2014 PRI 2015 PRI 2016 AWS OD 2017 AWS RESERVA 2017 AURORA 2019 AURORA SC
Performance
Key idea: writer with reasonable CPU
Key idea: peak connections
Performance (2)
Key idea: autoscaling
Performance (3)
Key idea: replica MS with really low latency
Next step
Serverless
Aurora ?
Thanks!!!!
Ignasi Nogués
inogues@clickedu.net
@inogues
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Aurora migration options
Source database From where Recommended option
RDS
EC2, on premises
EC2, on premises, RDS
Console-based automated
snapshot ingestion and catch up
via binlog replication.
Binary snapshot ingestion
through S3 and catch up via
binlog replication.
Schema conversion using SCT
and data migration via DMS.
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Publications
• Amazon Aurora: Design Considerations for High Throughput Cloud-Native
Relational Databases. In SIGMOD 2017
• Amazon Aurora: On Avoiding Distributed Consensus for I/Os, Commits, and
Membership Changes. In SIGMOD 2018
• https://aws.amazon.com/blogs/database/amazon-aurora-under-the-hood-quorum-
and-correlated-failure/
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Related breakouts
• How to choose the right database
– David Sanz Gil
• Migrating business critical applications to AWS
– Jose Maria Fuster Millan
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Javier Ramirez
@supercoco9
Ignasi Nogués
@inogues
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

More Related Content

What's hot

Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftAmazon Web Services
 
Building Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS GlueBuilding Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS GlueAmazon Web Services
 
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...Amazon Web Services
 
Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28
Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28
Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28Amazon Web Services
 
Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018
Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018
Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018Amazon Web Services
 
What's new in Amazon Aurora - ADB207 - New York AWS Summit
What's new in Amazon Aurora - ADB207 - New York AWS SummitWhat's new in Amazon Aurora - ADB207 - New York AWS Summit
What's new in Amazon Aurora - ADB207 - New York AWS SummitAmazon Web Services
 
Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftAmazon Web Services
 
Loading Data into Amazon Redshift
Loading Data into Amazon RedshiftLoading Data into Amazon Redshift
Loading Data into Amazon RedshiftAmazon Web Services
 
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Web Services
 
Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR
Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMRCost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR
Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMRProvectus
 
Building High Performance Apps with In-memory Data
Building High Performance Apps with In-memory DataBuilding High Performance Apps with In-memory Data
Building High Performance Apps with In-memory DataAmazon Web Services
 
Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018
Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018
Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018Amazon Web Services
 
Loading Data into Redshift with Lab
Loading Data into Redshift with LabLoading Data into Redshift with Lab
Loading Data into Redshift with LabAmazon Web Services
 
Building a modern data platform on AWS. Utrecht AWS Dev Day
Building a modern data platform on AWS. Utrecht AWS Dev DayBuilding a modern data platform on AWS. Utrecht AWS Dev Day
Building a modern data platform on AWS. Utrecht AWS Dev Dayjavier ramirez
 
Building a global database with MongoDB Atlas - DEM16-S - New York AWS Summit
Building a global database with MongoDB Atlas - DEM16-S - New York AWS SummitBuilding a global database with MongoDB Atlas - DEM16-S - New York AWS Summit
Building a global database with MongoDB Atlas - DEM16-S - New York AWS SummitAmazon Web Services
 
Databases - Choosing the right Database on AWS
Databases - Choosing the right Database on AWSDatabases - Choosing the right Database on AWS
Databases - Choosing the right Database on AWSAmazon Web Services
 
What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018
What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018
What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018Amazon Web Services
 

What's hot (20)

Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon Redshift
 
Building Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS GlueBuilding Serverless ETL Pipelines with AWS Glue
Building Serverless ETL Pipelines with AWS Glue
 
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
 
Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28
Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28
Best practices for optimizing your EC2 costs with Spot Instances | AWS Floor28
 
Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018
Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018
Build on Amazon Aurora with MySQL Compatibility (DAT348-R4) - AWS re:Invent 2018
 
What's new in Amazon Aurora - ADB207 - New York AWS Summit
What's new in Amazon Aurora - ADB207 - New York AWS SummitWhat's new in Amazon Aurora - ADB207 - New York AWS Summit
What's new in Amazon Aurora - ADB207 - New York AWS Summit
 
Data Warehousing with Amazon Redshift
Data Warehousing with Amazon RedshiftData Warehousing with Amazon Redshift
Data Warehousing with Amazon Redshift
 
Loading Data into Amazon Redshift
Loading Data into Amazon RedshiftLoading Data into Amazon Redshift
Loading Data into Amazon Redshift
 
How Amazon uses AWS Analytics
How Amazon uses AWS AnalyticsHow Amazon uses AWS Analytics
How Amazon uses AWS Analytics
 
Using Data Lakes
Using Data LakesUsing Data Lakes
Using Data Lakes
 
Using Data Lakes
Using Data LakesUsing Data Lakes
Using Data Lakes
 
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
 
Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR
Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMRCost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR
Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR
 
Building High Performance Apps with In-memory Data
Building High Performance Apps with In-memory DataBuilding High Performance Apps with In-memory Data
Building High Performance Apps with In-memory Data
 
Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018
Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018
Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018
 
Loading Data into Redshift with Lab
Loading Data into Redshift with LabLoading Data into Redshift with Lab
Loading Data into Redshift with Lab
 
Building a modern data platform on AWS. Utrecht AWS Dev Day
Building a modern data platform on AWS. Utrecht AWS Dev DayBuilding a modern data platform on AWS. Utrecht AWS Dev Day
Building a modern data platform on AWS. Utrecht AWS Dev Day
 
Building a global database with MongoDB Atlas - DEM16-S - New York AWS Summit
Building a global database with MongoDB Atlas - DEM16-S - New York AWS SummitBuilding a global database with MongoDB Atlas - DEM16-S - New York AWS Summit
Building a global database with MongoDB Atlas - DEM16-S - New York AWS Summit
 
Databases - Choosing the right Database on AWS
Databases - Choosing the right Database on AWSDatabases - Choosing the right Database on AWS
Databases - Choosing the right Database on AWS
 
What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018
What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018
What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018
 

Similar to Scalable Relational Databases with Amazon Aurora. Madrid Summit 2019

What's new in Amazon Aurora - ADB203 - Chicago AWS Summit
What's new in Amazon Aurora - ADB203 - Chicago AWS SummitWhat's new in Amazon Aurora - ADB203 - Chicago AWS Summit
What's new in Amazon Aurora - ADB203 - Chicago AWS SummitAmazon Web Services
 
Scale Up and Modernize Your Database with Amazon Relational Database Service ...
Scale Up and Modernize Your Database with Amazon Relational Database Service ...Scale Up and Modernize Your Database with Amazon Relational Database Service ...
Scale Up and Modernize Your Database with Amazon Relational Database Service ...Amazon Web Services
 
Amazon Aurora - Rajeev Chakrabarti
Amazon Aurora - Rajeev ChakrabartiAmazon Aurora - Rajeev Chakrabarti
Amazon Aurora - Rajeev ChakrabartiAmazon Web Services
 
Amazon Aurora Getting started Guide -level 0
Amazon Aurora Getting started Guide -level 0Amazon Aurora Getting started Guide -level 0
Amazon Aurora Getting started Guide -level 0kartraj
 
Accelerate database development and testing with Amazon Aurora - ADB208 - New...
Accelerate database development and testing with Amazon Aurora - ADB208 - New...Accelerate database development and testing with Amazon Aurora - ADB208 - New...
Accelerate database development and testing with Amazon Aurora - ADB208 - New...Amazon Web Services
 
What's new in Amazon RDS - ADB206 - New York AWS Summit
What's new in Amazon RDS - ADB206 - New York AWS SummitWhat's new in Amazon RDS - ADB206 - New York AWS Summit
What's new in Amazon RDS - ADB206 - New York AWS SummitAmazon Web Services
 
What’s new in Amazon RDS - ADB207 - Chicago AWS Summit
What’s new in Amazon RDS - ADB207 - Chicago AWS SummitWhat’s new in Amazon RDS - ADB207 - Chicago AWS Summit
What’s new in Amazon RDS - ADB207 - Chicago AWS SummitAmazon Web Services
 
Amazon Aurora and AWS Database Migration Service
Amazon Aurora and AWS Database Migration ServiceAmazon Aurora and AWS Database Migration Service
Amazon Aurora and AWS Database Migration ServiceAmazon Web Services
 
Amazon Aurora, funzionalità e best practice per la migrazione di database su AWS
Amazon Aurora, funzionalità e best practice per la migrazione di database su AWSAmazon Aurora, funzionalità e best practice per la migrazione di database su AWS
Amazon Aurora, funzionalità e best practice per la migrazione di database su AWSAmazon Web Services
 
AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)
AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)
AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)Amazon Web Services
 
(DAT312) Using Amazon Aurora for Enterprise Workloads
(DAT312) Using Amazon Aurora for Enterprise Workloads(DAT312) Using Amazon Aurora for Enterprise Workloads
(DAT312) Using Amazon Aurora for Enterprise WorkloadsAmazon Web Services
 
Amazon Aurora (MySQL, Postgres)
Amazon Aurora (MySQL, Postgres)Amazon Aurora (MySQL, Postgres)
Amazon Aurora (MySQL, Postgres)AWS Germany
 
Build data-driven, high-performance, internet-scale applications with AWS dat...
Build data-driven, high-performance, internet-scale applications with AWS dat...Build data-driven, high-performance, internet-scale applications with AWS dat...
Build data-driven, high-performance, internet-scale applications with AWS dat...Amazon Web Services
 
AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)
AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)
AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)Amazon Web Services
 
Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)
Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)
Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)Amazon Web Services
 

Similar to Scalable Relational Databases with Amazon Aurora. Madrid Summit 2019 (20)

What's new in Amazon Aurora - ADB203 - Chicago AWS Summit
What's new in Amazon Aurora - ADB203 - Chicago AWS SummitWhat's new in Amazon Aurora - ADB203 - Chicago AWS Summit
What's new in Amazon Aurora - ADB203 - Chicago AWS Summit
 
Scale Up and Modernize Your Database with Amazon Relational Database Service ...
Scale Up and Modernize Your Database with Amazon Relational Database Service ...Scale Up and Modernize Your Database with Amazon Relational Database Service ...
Scale Up and Modernize Your Database with Amazon Relational Database Service ...
 
Amazon Aurora
Amazon AuroraAmazon Aurora
Amazon Aurora
 
Amazon Aurora
Amazon AuroraAmazon Aurora
Amazon Aurora
 
Amazon Aurora - Rajeev Chakrabarti
Amazon Aurora - Rajeev ChakrabartiAmazon Aurora - Rajeev Chakrabarti
Amazon Aurora - Rajeev Chakrabarti
 
Amazon Aurora
Amazon AuroraAmazon Aurora
Amazon Aurora
 
Amazon Aurora Getting started Guide -level 0
Amazon Aurora Getting started Guide -level 0Amazon Aurora Getting started Guide -level 0
Amazon Aurora Getting started Guide -level 0
 
Accelerate database development and testing with Amazon Aurora - ADB208 - New...
Accelerate database development and testing with Amazon Aurora - ADB208 - New...Accelerate database development and testing with Amazon Aurora - ADB208 - New...
Accelerate database development and testing with Amazon Aurora - ADB208 - New...
 
Amazon Aurora_Deep Dive
Amazon Aurora_Deep DiveAmazon Aurora_Deep Dive
Amazon Aurora_Deep Dive
 
What's new in Amazon RDS - ADB206 - New York AWS Summit
What's new in Amazon RDS - ADB206 - New York AWS SummitWhat's new in Amazon RDS - ADB206 - New York AWS Summit
What's new in Amazon RDS - ADB206 - New York AWS Summit
 
What’s new in Amazon RDS - ADB207 - Chicago AWS Summit
What’s new in Amazon RDS - ADB207 - Chicago AWS SummitWhat’s new in Amazon RDS - ADB207 - Chicago AWS Summit
What’s new in Amazon RDS - ADB207 - Chicago AWS Summit
 
Amazon Aurora and AWS Database Migration Service
Amazon Aurora and AWS Database Migration ServiceAmazon Aurora and AWS Database Migration Service
Amazon Aurora and AWS Database Migration Service
 
Amazon Aurora
Amazon AuroraAmazon Aurora
Amazon Aurora
 
Amazon Aurora, funzionalità e best practice per la migrazione di database su AWS
Amazon Aurora, funzionalità e best practice per la migrazione di database su AWSAmazon Aurora, funzionalità e best practice per la migrazione di database su AWS
Amazon Aurora, funzionalità e best practice per la migrazione di database su AWS
 
AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)
AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)
AWS re:Invent 2016: Amazon Aurora Deep Dive (GPST402)
 
(DAT312) Using Amazon Aurora for Enterprise Workloads
(DAT312) Using Amazon Aurora for Enterprise Workloads(DAT312) Using Amazon Aurora for Enterprise Workloads
(DAT312) Using Amazon Aurora for Enterprise Workloads
 
Amazon Aurora (MySQL, Postgres)
Amazon Aurora (MySQL, Postgres)Amazon Aurora (MySQL, Postgres)
Amazon Aurora (MySQL, Postgres)
 
Build data-driven, high-performance, internet-scale applications with AWS dat...
Build data-driven, high-performance, internet-scale applications with AWS dat...Build data-driven, high-performance, internet-scale applications with AWS dat...
Build data-driven, high-performance, internet-scale applications with AWS dat...
 
AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)
AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)
AWS re:Invent 2016: Getting Started with Amazon Aurora (DAT203)
 
Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)
Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)
Amazon Redshift 與 Amazon Redshift Spectrum 幫您建立現代化資料倉儲 (Level 300)
 

More from javier ramirez

¿Se puede vivir del open source? T3chfest
¿Se puede vivir del open source? T3chfest¿Se puede vivir del open source? T3chfest
¿Se puede vivir del open source? T3chfestjavier ramirez
 
QuestDB: The building blocks of a fast open-source time-series database
QuestDB: The building blocks of a fast open-source time-series databaseQuestDB: The building blocks of a fast open-source time-series database
QuestDB: The building blocks of a fast open-source time-series databasejavier ramirez
 
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez
 
Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...
Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...
Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...javier ramirez
 
Deduplicating and analysing time-series data with Apache Beam and QuestDB
Deduplicating and analysing time-series data with Apache Beam and QuestDBDeduplicating and analysing time-series data with Apache Beam and QuestDB
Deduplicating and analysing time-series data with Apache Beam and QuestDBjavier ramirez
 
Your Database Cannot Do this (well)
Your Database Cannot Do this (well)Your Database Cannot Do this (well)
Your Database Cannot Do this (well)javier ramirez
 
Your Timestamps Deserve Better than a Generic Database
Your Timestamps Deserve Better than a Generic DatabaseYour Timestamps Deserve Better than a Generic Database
Your Timestamps Deserve Better than a Generic Databasejavier ramirez
 
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...javier ramirez
 
QuestDB-Community-Call-20220728
QuestDB-Community-Call-20220728QuestDB-Community-Call-20220728
QuestDB-Community-Call-20220728javier ramirez
 
Processing and analysing streaming data with Python. Pycon Italy 2022
Processing and analysing streaming  data with Python. Pycon Italy 2022Processing and analysing streaming  data with Python. Pycon Italy 2022
Processing and analysing streaming data with Python. Pycon Italy 2022javier ramirez
 
QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...javier ramirez
 
Servicios e infraestructura de AWS y la próxima región en Aragón
Servicios e infraestructura de AWS y la próxima región en AragónServicios e infraestructura de AWS y la próxima región en Aragón
Servicios e infraestructura de AWS y la próxima región en Aragónjavier ramirez
 
Primeros pasos en desarrollo serverless
Primeros pasos en desarrollo serverlessPrimeros pasos en desarrollo serverless
Primeros pasos en desarrollo serverlessjavier ramirez
 
How AWS is reinventing the cloud
How AWS is reinventing the cloudHow AWS is reinventing the cloud
How AWS is reinventing the cloudjavier ramirez
 
Analitica de datos en tiempo real con Apache Flink y Apache BEAM
Analitica de datos en tiempo real con Apache Flink y Apache BEAMAnalitica de datos en tiempo real con Apache Flink y Apache BEAM
Analitica de datos en tiempo real con Apache Flink y Apache BEAMjavier ramirez
 
Getting started with streaming analytics
Getting started with streaming analyticsGetting started with streaming analytics
Getting started with streaming analyticsjavier ramirez
 
Getting started with streaming analytics: Setting up a pipeline
Getting started with streaming analytics: Setting up a pipelineGetting started with streaming analytics: Setting up a pipeline
Getting started with streaming analytics: Setting up a pipelinejavier ramirez
 
Getting started with streaming analytics: Deep Dive
Getting started with streaming analytics: Deep DiveGetting started with streaming analytics: Deep Dive
Getting started with streaming analytics: Deep Divejavier ramirez
 
Getting started with streaming analytics: streaming basics (1 of 3)
Getting started with streaming analytics: streaming basics (1 of 3)Getting started with streaming analytics: streaming basics (1 of 3)
Getting started with streaming analytics: streaming basics (1 of 3)javier ramirez
 
Monitorización de seguridad y detección de amenazas con AWS
Monitorización de seguridad y detección de amenazas con AWSMonitorización de seguridad y detección de amenazas con AWS
Monitorización de seguridad y detección de amenazas con AWSjavier ramirez
 

More from javier ramirez (20)

¿Se puede vivir del open source? T3chfest
¿Se puede vivir del open source? T3chfest¿Se puede vivir del open source? T3chfest
¿Se puede vivir del open source? T3chfest
 
QuestDB: The building blocks of a fast open-source time-series database
QuestDB: The building blocks of a fast open-source time-series databaseQuestDB: The building blocks of a fast open-source time-series database
QuestDB: The building blocks of a fast open-source time-series database
 
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
 
Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...
Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...
Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...
 
Deduplicating and analysing time-series data with Apache Beam and QuestDB
Deduplicating and analysing time-series data with Apache Beam and QuestDBDeduplicating and analysing time-series data with Apache Beam and QuestDB
Deduplicating and analysing time-series data with Apache Beam and QuestDB
 
Your Database Cannot Do this (well)
Your Database Cannot Do this (well)Your Database Cannot Do this (well)
Your Database Cannot Do this (well)
 
Your Timestamps Deserve Better than a Generic Database
Your Timestamps Deserve Better than a Generic DatabaseYour Timestamps Deserve Better than a Generic Database
Your Timestamps Deserve Better than a Generic Database
 
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...
 
QuestDB-Community-Call-20220728
QuestDB-Community-Call-20220728QuestDB-Community-Call-20220728
QuestDB-Community-Call-20220728
 
Processing and analysing streaming data with Python. Pycon Italy 2022
Processing and analysing streaming  data with Python. Pycon Italy 2022Processing and analysing streaming  data with Python. Pycon Italy 2022
Processing and analysing streaming data with Python. Pycon Italy 2022
 
QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...
 
Servicios e infraestructura de AWS y la próxima región en Aragón
Servicios e infraestructura de AWS y la próxima región en AragónServicios e infraestructura de AWS y la próxima región en Aragón
Servicios e infraestructura de AWS y la próxima región en Aragón
 
Primeros pasos en desarrollo serverless
Primeros pasos en desarrollo serverlessPrimeros pasos en desarrollo serverless
Primeros pasos en desarrollo serverless
 
How AWS is reinventing the cloud
How AWS is reinventing the cloudHow AWS is reinventing the cloud
How AWS is reinventing the cloud
 
Analitica de datos en tiempo real con Apache Flink y Apache BEAM
Analitica de datos en tiempo real con Apache Flink y Apache BEAMAnalitica de datos en tiempo real con Apache Flink y Apache BEAM
Analitica de datos en tiempo real con Apache Flink y Apache BEAM
 
Getting started with streaming analytics
Getting started with streaming analyticsGetting started with streaming analytics
Getting started with streaming analytics
 
Getting started with streaming analytics: Setting up a pipeline
Getting started with streaming analytics: Setting up a pipelineGetting started with streaming analytics: Setting up a pipeline
Getting started with streaming analytics: Setting up a pipeline
 
Getting started with streaming analytics: Deep Dive
Getting started with streaming analytics: Deep DiveGetting started with streaming analytics: Deep Dive
Getting started with streaming analytics: Deep Dive
 
Getting started with streaming analytics: streaming basics (1 of 3)
Getting started with streaming analytics: streaming basics (1 of 3)Getting started with streaming analytics: streaming basics (1 of 3)
Getting started with streaming analytics: streaming basics (1 of 3)
 
Monitorización de seguridad y detección de amenazas con AWS
Monitorización de seguridad y detección de amenazas con AWSMonitorización de seguridad y detección de amenazas con AWS
Monitorización de seguridad y detección de amenazas con AWS
 

Recently uploaded

dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.pptdokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.pptSonatrach
 
PKS-TGC-1084-630 - Stage 1 Proposal.pptx
PKS-TGC-1084-630 - Stage 1 Proposal.pptxPKS-TGC-1084-630 - Stage 1 Proposal.pptx
PKS-TGC-1084-630 - Stage 1 Proposal.pptxPramod Kumar Srivastava
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfMarinCaroMartnezBerg
 
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptxEMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptxthyngster
 
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Callshivangimorya083
 
04242024_CCC TUG_Joins and Relationships
04242024_CCC TUG_Joins and Relationships04242024_CCC TUG_Joins and Relationships
04242024_CCC TUG_Joins and Relationshipsccctableauusergroup
 
{Pooja: 9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...
{Pooja:  9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...{Pooja:  9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...
{Pooja: 9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...Pooja Nehwal
 
RA-11058_IRR-COMPRESS Do 198 series of 1998
RA-11058_IRR-COMPRESS Do 198 series of 1998RA-11058_IRR-COMPRESS Do 198 series of 1998
RA-11058_IRR-COMPRESS Do 198 series of 1998YohFuh
 
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfMarket Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfRachmat Ramadhan H
 
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfSchema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfLars Albertsson
 
Call Girls In Mahipalpur O9654467111 Escorts Service
Call Girls In Mahipalpur O9654467111  Escorts ServiceCall Girls In Mahipalpur O9654467111  Escorts Service
Call Girls In Mahipalpur O9654467111 Escorts ServiceSapana Sha
 
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...Suhani Kapoor
 
Ukraine War presentation: KNOW THE BASICS
Ukraine War presentation: KNOW THE BASICSUkraine War presentation: KNOW THE BASICS
Ukraine War presentation: KNOW THE BASICSAishani27
 
Industrialised data - the key to AI success.pdf
Industrialised data - the key to AI success.pdfIndustrialised data - the key to AI success.pdf
Industrialised data - the key to AI success.pdfLars Albertsson
 
VIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service Amravati
VIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service AmravatiVIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service Amravati
VIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service AmravatiSuhani Kapoor
 
Invezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signalsInvezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signalsInvezz1
 
定制英国白金汉大学毕业证(UCB毕业证书) 成绩单原版一比一
定制英国白金汉大学毕业证(UCB毕业证书)																			成绩单原版一比一定制英国白金汉大学毕业证(UCB毕业证书)																			成绩单原版一比一
定制英国白金汉大学毕业证(UCB毕业证书) 成绩单原版一比一ffjhghh
 
Unveiling Insights: The Role of a Data Analyst
Unveiling Insights: The Role of a Data AnalystUnveiling Insights: The Role of a Data Analyst
Unveiling Insights: The Role of a Data AnalystSamantha Rae Coolbeth
 

Recently uploaded (20)

dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.pptdokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
 
PKS-TGC-1084-630 - Stage 1 Proposal.pptx
PKS-TGC-1084-630 - Stage 1 Proposal.pptxPKS-TGC-1084-630 - Stage 1 Proposal.pptx
PKS-TGC-1084-630 - Stage 1 Proposal.pptx
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdf
 
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptxEMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx
 
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
 
E-Commerce Order PredictionShraddha Kamble.pptx
E-Commerce Order PredictionShraddha Kamble.pptxE-Commerce Order PredictionShraddha Kamble.pptx
E-Commerce Order PredictionShraddha Kamble.pptx
 
04242024_CCC TUG_Joins and Relationships
04242024_CCC TUG_Joins and Relationships04242024_CCC TUG_Joins and Relationships
04242024_CCC TUG_Joins and Relationships
 
{Pooja: 9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...
{Pooja:  9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...{Pooja:  9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...
{Pooja: 9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...
 
꧁❤ Aerocity Call Girls Service Aerocity Delhi ❤꧂ 9999965857 ☎️ Hard And Sexy ...
꧁❤ Aerocity Call Girls Service Aerocity Delhi ❤꧂ 9999965857 ☎️ Hard And Sexy ...꧁❤ Aerocity Call Girls Service Aerocity Delhi ❤꧂ 9999965857 ☎️ Hard And Sexy ...
꧁❤ Aerocity Call Girls Service Aerocity Delhi ❤꧂ 9999965857 ☎️ Hard And Sexy ...
 
RA-11058_IRR-COMPRESS Do 198 series of 1998
RA-11058_IRR-COMPRESS Do 198 series of 1998RA-11058_IRR-COMPRESS Do 198 series of 1998
RA-11058_IRR-COMPRESS Do 198 series of 1998
 
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfMarket Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
 
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfSchema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdf
 
Call Girls In Mahipalpur O9654467111 Escorts Service
Call Girls In Mahipalpur O9654467111  Escorts ServiceCall Girls In Mahipalpur O9654467111  Escorts Service
Call Girls In Mahipalpur O9654467111 Escorts Service
 
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
 
Ukraine War presentation: KNOW THE BASICS
Ukraine War presentation: KNOW THE BASICSUkraine War presentation: KNOW THE BASICS
Ukraine War presentation: KNOW THE BASICS
 
Industrialised data - the key to AI success.pdf
Industrialised data - the key to AI success.pdfIndustrialised data - the key to AI success.pdf
Industrialised data - the key to AI success.pdf
 
VIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service Amravati
VIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service AmravatiVIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service Amravati
VIP Call Girls in Amravati Aarohi 8250192130 Independent Escort Service Amravati
 
Invezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signalsInvezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signals
 
定制英国白金汉大学毕业证(UCB毕业证书) 成绩单原版一比一
定制英国白金汉大学毕业证(UCB毕业证书)																			成绩单原版一比一定制英国白金汉大学毕业证(UCB毕业证书)																			成绩单原版一比一
定制英国白金汉大学毕业证(UCB毕业证书) 成绩单原版一比一
 
Unveiling Insights: The Role of a Data Analyst
Unveiling Insights: The Role of a Data AnalystUnveiling Insights: The Role of a Data Analyst
Unveiling Insights: The Role of a Data Analyst
 

Scalable Relational Databases with Amazon Aurora. Madrid Summit 2019

  • 1. S U M M I T • Madrid
  • 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Bases de datos relacionales escalables con Amazon Aurora Javier Ramirez @supercoco9 Tech Evangelist AWS • C 1 Ignasi Nogués @inogues CEO Clickedu
  • 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Amazon Aurora: Enterprise database at open source price Fastest growing service in AWS history. Aurora is used by ¾ of the top 100 AWS customers
  • 4. Which type of phone are you carrying today?
  • 5. If you got over your phone from the 2000s… why are you still using databases that follow the design patterns of the 80s?
  • 6. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Local Storage SQL Transactions Caching Logging Compute Traditional Database Architecture • Monolithic stack in a Single box • Large blast radius
  • 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Quick recap: Database internals contd… • Recovery Tc Tf Tx1 Tx2 Tx3 Tx4 checkpoint system failure  Tx1 can be ignored as checkpoint has been taken after its commit  Tx2 and Tx3 redone using REDO procedure  Tx4 is undone using REDO/UNDO procedure
  • 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Traditional Distributed Database stack Storage Application Application Application SQL Transactions Caching Logging SQL Transactions Caching Logging SQL Transactions Caching Logging SQL Transactions Caching Logging SQL Transactions Caching Logging SQL Transactions Caching Logging Storage StorageStorage Storage  Same Monolithic stack  Distributed consensus algorithms perform poorly
  • 10. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Aurora: Scale-out, Distributed architecture • Push Log applicator to Storage Master Replica Replica Replica Master Shared storage volume Replica Replica SQL Transactions Caching SQL Transactions Caching SQL Transactions Caching AZ1 AZ2 AZ3  Write performance  Read scale out  AZ + 1 failure tolerance • 4/6 Write Quorum & Local tracking
  • 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Read replica and custom endpoint Master Read replica Read replica Read replica Shared distributed storage volume Reader endpoint #1 Reader endpoint #2 Up to 15 promotable read replicas across multiple availability zones Re-do log based replication leads to low replica lag – typically < 10ms Custom reader end-point with configurable failover order
  • 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T • Partition volume into 𝑛 fixed size segments • Replicate each segment 6-ways into a Protection Group (PG) • A single PG failing is enough to fail the entire volume • Trade-off between likelihood of faults and time to repair • If segments are too small then failures are more likely • If segments are too big then repairs take too long • Choose the biggest size that lets us repair “fast enough” • We currently picked a segment size of 10GB as we can repair a 10GB segment in ~10 seconds on a 10Gbps link Segmented storage
  • 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Continuous backup • Take periodic snapshot of each segment in parallel; stream the redo logs to Amazon S3 • Backup happens continuously without performance or availability impact Segment snapshot Log records Recovery point Segment 1 Segment 2 Segment 3 Time
  • 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Instant crash redo recovery Traditional database • Replay logs since the last checkpoint • Slow replay in the single thread Amazon Aurora • At restore, retrieve the appropriate segment snapshots and log streams to storage nodes Checkpointed Data Redo Log Crash at T0 requires a re-application of the SQL in the redo log since last checkpoint T0 T0 Crash at T0 will result in redo logs being applied to each segment on demand, in parallel, asynchronously
  • 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Database backtrack • Backtrack brings the database to a point in time without requiring restore from backups • Backtracking from an unintentional DML or DDL operation • Backtrack is not destructive. You can backtrack multiple times to find the right point in time t0 t1 t2 t0 t1 t2 t3 t4 t3 t4 Rewind to t1 Rewind to t3 Invisible Invisible
  • 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Fast Database Cloning • Create a copy of a database without duplicate storage costs • Creation of a clone is instantaneous since it doesn’t require deep copy • Data copy happens only on write – when original and cloned volume data differ – Typical use cases: • Clone a production DB to run tests • Reorganize a database • Save a point in time snapshot for analysis without impacting production system Production database Clone Clone Clone Dev/test applications Benchmarks Production applications Production applications
  • 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Page 1 Page 2 Page 3 Page 4 Page 5 Source database Cloned database As databases diverge, new pages are added appropriately to each database while still referencing pages common to both databases Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 1 Page 3 Page 5 Page 3 Page 5 Protection group 1 Page 2 Page 4 Page 6 Page 2 Protection group 2 Shared distributed storage system Database Cloning: How does it work?
  • 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Write and read throughput Aurora MySQL is 5x faster than MySQL 0 50,000 100,000 150,000 200,000 250,000 MySQL 5.6 MySQL 5.7 MySQL 8.0 Aurora 5.6 Aurora 5.7 0 100,000 200,000 300,000 400,000 500,000 600,000 700,000 800,000 MySQL 5.6 MySQL 5.7 MySQL 8.0 Aurora 5.6 Aurora 5.7 Write throughput Read throughput Using Sysbench with 250 tables and 200,000 rows per table on R4.16XL
  • 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Copy In Copy In Vacuum Vacuum Index Build Index Build 0 500 1000 1500 2000 2500 3000 3500 4000 PostgreSQL Amazon Aurora Runtime (seconds) pgbench initialization, scale 10000 (150 GiB) 86% reduction in vacuum time Bulk data load performance Aurora PostgreSQL loads data 2x faster than PostgreSQL
  • 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Bulk data load performance Aurora MySQL loads data 2.5x faster than MySQL Data loading Data loading Index build Index build 0 100 200 300 400 500 600 700 800 MySQL Amazon Aurora Runtime (sec.) 10 Sysbench Tables, 10MM rows per each
  • 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 0.00 100.00 200.00 300.00 400.00 500.00 600.00 700.00 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Responsetime,ms SYSBENCH RESPONSE TIME (p95), 30 GiB, 1024 CLIENTS PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup) Performance variability under load Aurora PostgreSQL is ~10x more consistent than PostgreSQL
  • 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Sysbench write-only workload with 250 tables and 200,000 initial rows per table 0 500 1,000 1,500 2,000 2,500 0 100 200 300 400 500 600 Time from start of run (sec.) Write response time (ms.) Amazon Aurora MySQL Performance variability under load Aurora MySQLis ~25x moreconsistent than MySQL
  • 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 0 50 100 150 200 250 2015 2016 2017 2018 Max write throughput – up 100% 0 100 200 300 400 500 600 700 800 2015 2016 2017 2018 Max read throughput – up 42% Launched with R3.8xl 32 cores, 256-GB memory Now support R4.16xl 64 cores, 512-GB memory R5.24xl coming soon 96 cores, 768-GB memory Besides many performance optimizations, we are also upgrading HW platform Performance improvement over time Aurora MySQL –2015-2018
  • 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How did we achieve this? • Do less work • Do fewer I/Os • Minimize network packets • Cache prior results • Offload the database engine • Be more efficient • Process asynchronously • Reduce latency path • Use lock-free data structures • Batch operations together • Databases are all about I/O • Network-attached storage is all about packets/second • High-throughput processing is all about context switches
  • 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T BINLOG DATA DOUBLE-WRITELOG FRM FILES TYPE OF WRITE MYSQL WITH REPLICA EBS mirrorEBS mirror AZ 1 AZ 2 EBS Amazon Elastic Block Store (EBS) Primary Instance Replica Instance 1 2 3 4 5 AZ 1 AZ 3 Primary Instance AZ 2 Replica Instance ASYNC 4/6 QUORUM DISTRIBUTED WRITES Replica Instance Amazon S3 AMAZON AURORA 0.78MM transactions 7.4 I/Os per transaction MySQL I/O profile for 30 min Sysbench run 27MM transactions 35X MORE 0.95 I/Os per transaction 7.7X LESS Aurora IO profile for 30 min Sysbench run MySQL vs. Aurora I/O profile Amazon S3
  • 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Driving down query latency Asynchronous Key Prefetch Hash joinsBatched scans
  • 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Performance Improvement TPC –C benchmark 0x 2x 4x 6x 8x 10x 12x 14x 16x 18x 20x Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10Q11Q12Q13Q14Q15Q16Q17Q18Q19Q20Q21Q22 Query response time reduction  Peak speed up ~18x  >2x speedup: 10 of 22 queries
  • 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Intelligent Vacuum prefetch PostgreSQL Aurora PostgreSQL Submit Batch I/O up to 256 blocks 402 seconds 163 seconds
  • 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Parallel query processing • Aurora storage has thousands of CPUs • Opportunity to push down and parallelize query processing • Moving processing close to data reduces network traffic and latency • However, there are significant challenges • Data is not range partitioned – require full scans • Data may be in-flight • Read views may not allow viewing most recent data • Not all functions can be pushed down Database Node Storage nodes Push down predicates Aggregate results
  • 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Database node processing • Query Optimizer produces PQ Plan and creates PQ context based on leaf page discovery. • PQ request is sent to storage node along with PQ context. • Storage node produces: • Partial results streams with processed stable rows. • Raw stream of unprocessed rows with pending undos. • Head node aggregates these data streams to produce final results. STORAGE NODES OPTIMIZER EXECUTOR INNODB NETWORK STORAGE DRIVER AGGREGATOR APPLICATION PARTIAL RESULTS STREAM RESULTS IN-FLIGHT DATA PQ CONTEXT PQ PLAN
  • 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Amazon Aurora PQ summary • Performance • 120x lower latencies on TPCH-like benchmarks with Improved I/O performance and reduced CPU usage on the head node. • High Concurrency • Run both OLTP and light OLAP workloads simultaneously and efficiently. • Cost Effective • PQ comes at no extra cost. Can run on your live data. Potentially reduced effort and data duplication in your ETL pipeline. • Quiet Tenant • Reduced chance of evicting frequently used pages from the buffer pool that are used by OLTP workload. • Ecosystem • Get Aurora goodies such as PiTR, Continuous backup, Fast Cloning with PQ.
  • 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Parallel Query - Performance results
  • 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T TPC-C benchmark 0x 20x 40x 60x 80x 100x 120x Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10Q11Q12Q13Q14Q15Q16Q17Q18Q19Q20Q21Q22 Query response time reduction  Peak speed up ~120x  >10x speedup: 8 of 22 queries We were able to test Aurora’s parallel query feature and the performance gains were very good. To be specific, we were able to reduce the instance type from r3.8xlarge to r3.2xlarge. For this use case, parallel query was a great win for us. Jyoti Shandil, Cloud Data Architect Performance results
  • 35. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Global replication: Logical Faster disaster recovery and enhanced data locality  Promote read-replica to a master for faster recovery in the event of disaster  Bring data close to your customer’s applications in different regions  Promote to a master for easy migration
  • 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Global physical replication • Primary region • Secondary region 1 ASYNC 4/6 QUORUM Continuous backup AZ 1 Primary Instance Amazon S3 AZ 2 Replica Instance AZ 3 Replica Instance Replication Server Replication Fleet Storage Fleet 11 4 AZ 1 Replica Instance AZ 2 AZ 3 ASYNC 4/6 QUORUM Continuous backup Amazon S3 Replica Instance Replica Instance Replication Agent Replication Fleet Storage Fleet 3 3 2 ① Primary instance sends log records in parallel to storage nodes, replica instances and replication server ② Replication server streams log records to Replication Agent in secondary region ③ Replication agent sends log records in parallel to storage nodes, and replica instances ④ Replication server pulls log records from storage nodes to catch up after outages High throughput: Up to 150K writes/sec – negligible performance impact Low replica lag: < 1 sec cross-region replica lag under heavy load Fast recovery: < 1 min to accept full read-write workloads after region failure
  • 38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Global replication performance Logical replication Physical replication 0 100 200 300 400 500 600 0 50,000 100,000 150,000 200,000 250,000 seconds QPS QPS Lag 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 0 50,000 100,000 150,000 200,000 250,000 seconds QPS QPS Lag Logical vs. physical replication
  • 39. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Master Replica Orange Master Blue Master SQL Transactions Caching SQL Transactions Caching Aurora Multi-Master Architecture Shared Storage Volume  No Pessimistic Locking  No Global Ordering  No Global Commit-Coordination Replica • Membership • Heartbeat • Replication • Metadata Cluster Services 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 1 3? T1 T2 AZ1 AZ2 AZ3 Decoupled Decoupled Decoupled  Decoupled System  Microservices Architecture 2  Optimistic Conflict Resolution
  • 41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Scaling write workload Optimistic conflict management There are many “oases” of consistency in Aurora Database nodes know transaction orders from that node Storage nodes know transactions orders applied at that node Only have conflicts when data changed at both multiple database nodes AND multiple storage nodes Much less coordination required Near linear throughput scaling for workloads with no or low conflict Lower commit latency for workloads with low conflict
  • 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Aurora Multi-Master: Scaling and availability 0 10000 20000 30000 40000 50000 60000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99 101103105107109111113115117119121123125 AggregatedThroughput Time in minutes Sysbench workload on 4 R4.XL nodes Adding a node Adding a node Node going down Node recovering
  • 43. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Aurora Serverless  Starts up on demand, shuts down when not in use  Scales up/down automatically  No application impact when scaling  Pay per second, 1 minute minimum WARM POOL OF INSTANCES APPLICATION AURORA STORAGE AURORA REQUEST ROUTER DATABASE END-POINT AURORA STORAGE
  • 45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Scale up and down with load 1 2 4 8 16 32 64 128 0 500 1000 1500 2000 2500 3000 1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177 188 199 210 221 232 243 254 265 276 287 298 309 320 331 342 353 364 375 386 397 408 419 430 441 452 463 474 485 496 507 518 529 540 551 562 573 584 595 606 617 628 639 650 661 672 683 694 705 716 727 TPS ACU
  • 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Simpler experience: Less to worry about • No CPU credits to monitor • No commitment to particular availability zone • No migrations between instance type generations • No DB instance reservations to manage • No instances to manage • Encryption at REST is always enabled • No need to manually suspend and resume database • No DNS propagation delays • No maintenance window • No old database versions to upgrade
  • 47. Clickedu + AWS: Better together
  • 48. Learning and administration activities happen in and outside schools. Data needs to be accessible from anywhere, on any device. Education trends
  • 49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T School challenges • Hard to access to the right information for each user • Curriculum, operations and data related to performance is unorganized or kept in separate systems • Teachers, students, parents and administrators need access wherever they are • Need to connect data from all parts of the school ecosystem Not enough time Hard to find data Need mobile access
  • 50. CLASSROOM ADMINISTRATION ACADEMIC COMMUNICATION QUALITY TOOLS 100% CLOUD EASY LMS AND CONNECTABLE EDUCATIVE CONTENTS The MIS for K-12: an educative E.R.P.
  • 51. functionality Administration Controls • Reception Notices • Room Booking • Inventory Management • Staff attendance and cover module • Documentation Management • Grades • Payments • EFQM & ISO quality Services • Extracurricular activities • Transport and catering • Special education • Flipped classroom • Individual timetables • Student documentation • Pupil Census • General reporting • Workforce census Communication • SMS & Email • Centralized Communication • Internal Messenger • Contacts & Calendars • Integrate with Google Mail and Microsoft 365 • Parent Portal • Behaviour Management • Survey, newsletter, website creation LMS & Assessments • Complete student profiles • Collaborative Learning • Recommendations • Forums • Projects • Submission Box • Self-Assessment Tests • Rubrics • Grades and assessment • Official grade documentation
  • 52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  • 53. Why AWS? • Private cloud never was enough but most of the time was so much • Problems of free space on File System • Huge use of data, no scalable database systems • There were frequent errors doing things “by hand” • Security • Complaints because we had to stop our servers • It was difficult to test on real time • No global servers
  • 54. Global – AWS helps on going around the planet
  • 55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T “Unlimited” & elastic
  • 57. Advantages • Use of CloudWatch • Alarm System with CloudWatch • SES use for emails • S3 with versioning and separate files from code • Environments to test new versions are easy to create
  • 58. Other uses of AWS •AI with the use of Sagemaker •Integrate voice systems (LEX; Polly) •Integrate Alexa & Chime
  • 59.
  • 60. RDS Advantage • 45 days of RDS snapshots • Autoscaling of RDS with Aurora • No stops on the new versions • Data Encryption & access control
  • 61. RDS evolution PRIVATE CLOUD MYSQL 4 Cluster MASTER-SLAVE (1-1) db.r3.2xlarge AWS - AURORA MASTER + 4 SLAVE db.r3.2xlarge PRIVATE CLOUD MYSQL 2 Clusters MASTER-SLAVE (1-1) db.r3.2xlarge AWS MYSQL 4 Clusters MASTER-SLAVE db.r3.xlarge AWS- AURORA MASTER + 2 SLAVE IN AUTOSCALING MODEL db.r3.2xlarge 2013 2014 2015 2017 2019 PRIVATE CLOUD MYSQL 4 Cluster M-S (1-2) db.r3.2xlarge + db.r3.xlarge
  • 62. Statistics (by minutes), one MySQL in 2014
  • 63. Migration • Lots of Planning • Partner help • We connect an AWS RDS as a slave to the previous private cloud • Low downtime, rather because of the DNS than to the switch from SLAVE to MASTER
  • 64. Aurora Migration • Lots of Planning • Well advised • Hard work reviewing the platform to be sure all was compatible but 0 issues found. • All process tested before • First cluster so easy to migrate (automated with AWS service, less than 1 hour) • Hard to find a screen to do the change for the other 3 clusters: – Downtime expected of 16 hours – Had to be summer on Spain/Andorra/UK; we found also a suitable week for LATAM (Chile & Colombia) • Transfer from other clusters made with dumps to be sure all was right. – Problems with rights and users • Finally the transfer was 8 hours for cluster
  • 65. 250 320 400 500 600 700 280 358 448 560 672 1047 308 394 493 616 739 1243 0 200 400 600 800 1,000 1,200 1,400 PRI PRI PRI AWS OD AWS RESERVA AURORA SC 2013 2014 2015 2016 2017 2019 Plattform evolution NÚMERO BASES DATOS ESPACIO (en TB) IOPS (en millones) Data in time
  • 67. Performance Key idea: writer with reasonable CPU Key idea: peak connections
  • 69. Performance (3) Key idea: replica MS with really low latency
  • 72. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Amazon Aurora migration options Source database From where Recommended option RDS EC2, on premises EC2, on premises, RDS Console-based automated snapshot ingestion and catch up via binlog replication. Binary snapshot ingestion through S3 and catch up via binlog replication. Schema conversion using SCT and data migration via DMS.
  • 73. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 74. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Publications • Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases. In SIGMOD 2017 • Amazon Aurora: On Avoiding Distributed Consensus for I/Os, Commits, and Membership Changes. In SIGMOD 2018 • https://aws.amazon.com/blogs/database/amazon-aurora-under-the-hood-quorum- and-correlated-failure/
  • 75. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Related breakouts • How to choose the right database – David Sanz Gil • Migrating business critical applications to AWS – Jose Maria Fuster Millan
  • 76. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Javier Ramirez @supercoco9 Ignasi Nogués @inogues
  • 77. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Editor's Notes

  1. (20 sec) You know, when databases first came out, it looked something like this. Monolithic architecture in a single box. With local storage, we were trading availability and durability to get better performance
  2. (30 seconds) And then we added more such boxes. As you can see, it’s the same SQL stack everywhere. Nothing changed! Moreover, we need heavy-weight distributed consensus for data replication and they perform poorly because of multiple phases, multiple rounds, sync points etc.
  3. (2.5 minutes) With Aurora, we did two big contributions: We pushed down Log applicator down to the storage => that allowed us to construct pages from the logs themselves. This is really cool because we don’t have to write full pages anymore. So unlike traditional databases which write both logs and pages, we just have to write logs. This means we have significantly less network IO, fundamentally less work on the engine - you don't need checkpointing anymore, you don't need flushing of pages or cache eviction any more. Instead of heavy weight distributed consensus for data replication. We use 4/6 write Quorum & Local tracking. The reason we can avoid distributed consensus is because we exploit monotonically increasing Log Sequence Number (LSN) by the Master that allows us to order the writes. And so SN's just accept the writes. There is no voting involved. We are going to see both these things in action. As a result, YOU get significantly better write performance YOU get Read scale out because they share the same storage with the master. YOU get AZ+1 failure tolerance => Aurora stores 6 copies, two copies per AZ. Even in the presence of background radiation, an entire AZ goes down on top, Aurora can handle it. No problem. YOU get an Instant database redo recovery because we don't have to explicitly do anything at startup other than doing some math to find out the point at which we crashed. Overall: You NO longer have to make a trade-off between performance, availability, and durability.
  4. We continuously back your data to S3. How does it work? Aurora divides the database into 10GB segments. And so we basically take snapshot of those segments and stream any delta redo logs to S3. On restore, we get those snapshots and apply delta log stream on top in parallel. This all is happening on the storage, has no performance impact on the database nodes.
  5. In traditional databases, we have to replay logs since the last checkpoint in a single thread. With Aurora, these redo logs are already being applied to each segment in parallel, ASYNCHRONOUSLY. For startup - we don’t have to do anything other than doing some math to find out the point at the time of the crash.
  6. Now, there are times when we accidentally delete a table or forgot a where clause in the delete statement. Backtrack allows to quick get the data back w/o fully restoring from backups. Relatively quick operation. Couple of minutes vs hours. In this example, you first backtracked to t1 and let it run some trx… note that you can actually backtrack back and forth to find out the right point, its not a destructive operation.
  7. (1 minute) Lets see what benefits we got out of this. Here is how the IO profile looks like for Aurora and MySQL. 1. On the left, we have MySQL on EBS. Thing to note is that it has to replicate all kinds of data. With Aurora, on the right, we ONLY have to replicate log records. As a result we do 7.7X less IO and 35X more work, despite 6X amplification due to 6 copies. 2. The other thing to note is that steps 1, 3, 4 on the left are synchronous and leads to jitter but with Aurora we use 4/6 quorum, that 's much more resilient to tail latency. And we will see that in a second, why that matters for your applications.
  8. (45 sec) So much about OLTP, lets talk about your OLAP queries. Here are some of the optimizations we did for OLAP... Batched scans – the idea is to scans tuples in batches from the innodb buffer pool to avoid latching pages again/again, traversing again/again and allows to use JIT optimizations. Mainly for in-memory workloads. Hash joins – improves equi-join performance. Build one side and scan through the other to probe the hash table. Lots of complexity with skew and duplicates – have to minimize the #passes, not to mention when to choose hash joins over other join operators like index join or nested loop joins. Async. Key Prefetch - prefetches pages in memory for index joins using BKA, quite useful for non-equi joins or some equi joins (if one side is small and the big side has a high-cardinality index on the joining column). For Out of cache workloads.
  9. (15 seconds) We ran TPCH like workload. Here you can the performance improvement from all those improvements. And as you can see, roughly half the queries are >2x better, with a peak speed up of roughly 18x.
  10. (30 sec) Processing closer to data significantly reduces the data transfer between HN and SN. As a result, there is a significant less impact on the OLTP performance because of buffer pool pollution and reduction in network traffic. We use 150GB just to show the impact because 8xl buffer pool is around 150GB something. And so if we bring in pages for OLAP queries, it will evict pages needed by OLTP queries.
  11. (2.5 min) With Aurora MM, there is no pessimistic locking, no explicit global ordering, no global commit-coordination. Aurora architecture is based on 3 techniques: First, it uses optimistic-conflict-resolution in the storage. To understand this better, lets say Orange Master runs T1 and Blue Master runs T2. Now, if T1 and T2 modifies different pages, there is no conflict and hence no sync required. However, if T1 and T2 both touches P2, then one of them wins and the other one have to retry based on the quorum. And as you can see, that doesn’t require any heavy-weight consensus protocol. Again we rely on quorum and local tracking with partitioned, monotonic LSN sequencing, from individual database nodes, to order the writes. Since logging layer is pushed down, Aurora decouples the Transaction layer from the logging layer, and this allows Aurora to separate physical conflict on the pages from the logical conflict between the transactions. Trx conflicts are handled through MVCC, and physical conflicts through OCR. Moreover, there is no direct coupling in the storage partitions or, with-in the database nodes in the cluster. Microservices architecture – there are independent, minimal and resilient services running in the cluster that are needed for async coordination. And any of them temporarily going down doesn’t NOT impact the whole cluster. Net-net: Aurora only coordinates when it has to coordinate. Lets see this in action.
  12. The cost of Clickedu for a school has two parts: Initial deployment (one-time fee) Annual service fee (for teachers and students) Every school is made up differently, and we work with you to create an agreement that meets the needs of your school. Ask about our ‘end of school year’ offer and related deployment fee discounts that may apply to you.