Organizations today are looking to free themselves from the constraints of on-premises commercial databases and leverage the power of cloud-native and open-source systems. Amazon Aurora is a MySQL- and PostgreSQL-compatible relational database that is built for the cloud, with the speed, reliability, and availability of commercial databases at one-tenth the cost. In this session, we provide an overview of Aurora and its features. We talk about the latest advances in migration tooling and automation, and we explain how many of the common legacy features of Oracle and SQL Server map to modern cloud variants. We also hear from Dow Jones about its migration journey to the cloud.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating Your Oracle & SQL Server Databases
to Amazon Aurora
John Winford
Global Lead – Database Freedom
Amazon Web Services
D A T 3 1 8
Scott Canham
Distinguished Software
Engineer
Dow Jones
Steve Stevenson
Distinguished Software
Engineer
Dow Jones

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
What is Amazon Aurora?
Modern improvements on legacy features
Migration tooling and automation
Dow Jones’ cloud journey
Questions

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Tuesday, November 27
DAT204 – What’s New in Amazon Aurora
10:00 – 11:00 | Venetian, Level 2, Venetian E
13:00 – 14:00 | Bellagio, Level 1, Grand Ballroom
Wednesday, November 28
DAT207 – Migrating Databases to the Cloud with AWS Database Migration Service
14:30 – 15:30 | Venetian, Level 2, Titian 2204
Thursday, November 29
DAT307 – Running Oracle Databases on Amazon RDS and Migrating to PostgreSQL (Workshop)
11:30 – 13:45 | Venetian, Level 2, Venetian H

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Aurora
MySQL and PostgreSQL compatible relational database built for the cloud
Performance and availability of commercial-grade databases at 1/10th the cost
Performance
and scalability
Availability
and durability
Highly secure Fully managed
5x throughput of standard
MySQL and 3x of standard
PostgreSQL; scale-out up to
15 read replicas
Fault-tolerant, self-healing
storage; six copies of data
across three AZs; continuous
backup to S3
Network isolation,
encryption at
rest/transit
Managed by Amazon RDS:
no hardware provisioning,
software patching, setup,
configuration, or backups

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traditional relational databases are hard to scale
Multiple layers of
functionality all in a
monolithic stack
SQL
Transactions
Caching
Logging
Storage

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reimagining the relational database
What if you were inventing the database today?
You would break apart the stack
You would build something that:
 Lets layers scale out independently
 Is self-healing
 Leverages distributed services

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scale out, distributed architecture
Master Replica
Availability
Zone 1
Shared storage volume
Availability
Zone 2
Availability
Zone 3
Storage nodes with SSDs
 Purpose-built log-structured
distributed storage system designed
for databases
 Storage volume is striped across
hundreds of storage nodes distributed
over three different Availability Zones
(AZs)
 Six copies of data, two copies in each
AZ to protect against AZ + one failures
SQL
Transactions
Caching
SQL
Transactions
Caching
SQL
Transactions
Caching

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Up to 15 read replicas across three AZs
Auto-scale new read replicas
Seamless recovery from read replica failures
Availability
Zone 1
Scale out read performance
Availability
Zone 2
Availability
Zone 3
Amazon Aurora—High performance
Scale out to millions of reads per second

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Aurora is fast …
Five times more throughput than MySQL
Three times more throughput than PostgreSQL

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Write performance Read performance
MySQL SysBench results
R3.8XL: 32 cores / 244 GB RAM
5x faster than Amazon RDS MySQL 5.6 & 5.7
Five times higher throughput than stock MySQL
based on industry standard benchmarks
0
25,000
50,000
75,000
100,000
125,000
150,000
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
Aurora MySQL 5.6 MySQL 5.7

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pgbench: Amazon Aurora is up to three times faster
Running the standard pgbench benchmark, Amazon Aurora delivers 1.6x the
peak throughput of PostgreSQL and 2.9x at high client counts
0
5
10
15
20
25
30
35
40
45
128 256 512 768 1024 1280 1536 1792 2048
Throughput(tps,thousands)
Number of clients
pgbench tpcb-like throughput, 150 GiB
PostgreSQL (Single AZ) Amazon Aurora (Three AZs)
2.9x

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sysbench: Amazon Aurora is two to five times faster
0
20
40
60
80
100
120
140
256 512 768 1024 1280 1536 1792 2048 2305 2560
writes/second,thousands
Number of clients
sysbench write-only 30 GiB
PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)
2.2x 5.3x
Running the standard sysbench benchmark, Amazon Aurora delivers more than
two times the absolute peak of PostgreSQL and five times at high client counts

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Aurora is four times faster at large scale
Scales from 1.8x to 4.4x better as database grows from 10 GiB to 100 GiB
74
49
30
136 134
131
0
20
40
60
80
100
120
140
160
10 GiB 30 GiB 100 GiB
writes/second,inthousands
Database size
sysbench write-only
PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)
4.4x
1.8x 2.8x

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Do fewer I/Os
Minimize network packets
Cache prior results
Offload the database engine
Do less work
Process asynchronously
Reduce latency path
Use lock-free data structures
Batch operations together
Be more efficient
How did we achieve this?
Databases are all about I/O
Network-attached storage is all about packets/second
High-throughput processing is all about context switches

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What about availability?

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Six copies across three Availability Zones
Four out of six write quorum; three out of six read quorum
Peer-to-peer replication for repairs
Volume striped across hundreds of storage nodes
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
Read and write availabilityRead availability
Six-way replicated storage
Survives catastrophic failures

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Aurora Recovers Up to 97% Faster
3 GiB Redo
Recovered in 19 seconds
10 GiB Redo
Recovered in 50 seconds
30 GiB Redo
Recovered in 123 seconds
0
20
40
60
80
100
120
140
160
0 20,000 40,000 60,000 80,000 100,000 120,000 140,000
RecoveryTimeinSeconds(lessisbetter)
Writes per Second (more is better)
Recovery time from crash under load
Bubble size represents redo log, which must be recovered
As PostgreSQL
throughput goes up, so
does log size and crash
recovery time
Amazon Aurora has no redo.
Recovered in three seconds while
maintaining significantly greater
throughput.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-region read replicas
Faster disaster recovery and enhanced data locality
Promote read-replica to a
master for faster recovery in the
event of disaster
Bring data close to your
customer’s applications in
different regions
Promote to a master for easy
migration

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Performance insights
Dashboard showing
Load on database
• Easy
• Powerful
Identifies source of bottlenecks
• Top SQL
Adjustable time frame
• Hour, day, week, month
• Up to 35 days of data
Max CPU

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Starts up on demand, shuts down when not in use
Automatically scales with no instances to manage
Pay per second for the database capacity you use
Aurora Serverless
On-demand, auto-scaling database for applications with variable workloads
Warm capacity
pool
Application

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Compatibility
• Aurora with MySQL
compatibility
• Can run Aurora with MySQL engine
versions 5.6 and 5.7
• Actually a fork of MySQL code, not
emulating
• Aurora with PostgreSQL
compatibility
• Can run Aurora with PostgreSQL
engine versions 9.6.3, 9.6.6, 9.6.8,
9.6.9 and 10.4
• Actually running PostgreSQL code,
not emulating

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Aurora customer adoption
Aurora is used by
three-fourths of the
top 100 AWS
customers
Fastest growing
service in AWS
history

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud offers an opportunity to do things better
Discover
cost savings
Better resource
efficiency
Increased
operational
resilience
Improved business
agility

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud native alternatives to legacy features
Item Oracle SQL Server Aurora
High availability RAC Always On Native architecture
Large objects BLOBs and CLOBs BLOBs and FILESTREAM Amazon S3
Disaster recovery Data Guard Log shipping Read replica
Database rewind Flashback database Backtrack
Email utl_mail Database mail AWS Lambda function
Database jobs dbms_scheduler SQL Server Agent Lambda function
Queuing Advanced queuing SQL Server Service Broker Amazon Simple Queue
Service (Amazon SQS)
Data redundancy Replication, mirroring Replication, mirroring Native architecture (six
times)
High IOPS More disks and controllers More disks and controller Native architecture
Encryption Oracle Advanced Security TDE (Enterprise) Native architecture
Survivable caches Service orientated buffer
cache (RAC)
In-memory OLTP Native architecture

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The SQL 2008 EOS countdown is on!
July 9, 2019
SQL Server 2008 and 2008 R2
End of support

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are AWS Database Migration Service (AWS
DMS) and AWS Schema Conversion Tool?
AWS DMS easily and securely migrates and/or replicate
your databases and data warehouses to AWS
AWS SCT converts your commercial database and data
warehouse schemas to open-source engines or AWS-native
services, such as Amazon Aurora and Amazon Redshift

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
More than 100,000 databases migrated with AWS DMS

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When to use AWS DMS and AWS SCT?
Modernize Migrate Replicate

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS SCT
• Modernize your data warehouse
Convert your Oracle, SQL Server, Netezza,
Greenplum, Vertica, or Teradata to Amazon
Redshift
Modernize
Amazon
Aurora
Amazon
Redshift
• Modernize your database
Convert your Oracle, SQL Server, or Db2 LUW to
PostgreSQL, MySQL, or Amazon Aurora
MySQL
PostgreSQL

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS
Migrate
• Migrate business-critical applications
• Migrate data warehouse to Amazon
Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Archive old data
• Migrate from NoSQL to SQL, SQL to
NoSQL, or NoSQL to NoSQL
Amazon RDS
Amazon
Redshift
Amazon
Aurora
Amazon
DynamoDB
Amazon S3

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS – Endpoint Support
Amazon
Redshift
Amazon
DynamoDB
Amazon S3
Amazon S3
Amazon
Aurora
Amazon
Aurora
Oracle SQL Server Netezza
Greenplum Vertica Teradata
AWS Snowball
Edge
MongoDB Cassandra
Amazon
Elasticsearch
Amazon Kinesis

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customer
premises
Application users
AWS
Internet
VPN
 Start a replication instance
 Connect to source and target
databases
 Select tables, schemas, or
databases
 Let AWS DMS create tables,
load data, and keep them in
sync
 Switch applications over to
the target at your
convenience
Keep your apps running during the migration
AWS
DMS

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Market data at Dow Jones
Trade/quote pricing and supporting data of financial instruments,
helping customers to make informed investment decisions
Financial instruments
Stocks, indexes, mutual funds, exchange-traded funds, bonds, futures,
options, currencies, cryptocurrencies (bitcoins), interest and loan rates,
IPOs, others

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Market data at Dow Jones

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Market database
~1.5 TB
800+ tables
100 rows to one billion rows
800+ GB dedicated to pricing history

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Market database
On-Premises Data Center B
Publisher
SubscriberSubscriberSubscriber
On-Premises Data Center A
Principal
Publisher
SubscriberSubscriberSubscriber
Remote
Distributor
Subscriber Subscriber

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Moving to the cloud—Why Aurora?
Directive to move majority of compute and data to the cloud
MS SQL licensing expense
Scalability
Managed
Reliability

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Migrate schema
• Migrate data
• Migrate applications and service
Where to start?
On Prem Amazon Aurora
Application users

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating schema
• Choosing the right tools
• Schema Conversion Tool
• MySQL Workbench migration tools
• Applying required manual changes
• Some hints within procedures or functions
• Dynamic SQL
• Referential integrity
• Partition tables
• MS SQL - partition views
• MySQL - partition tables
databasename.schema.tablename
versus
new database for each schema
schema.tablename
(Screenshot to come)

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SCT report example

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating data
• Data migration service
• Need production quality data
• No downtime
• Complete in reasonable amount of time
• Change data capture
• Capture all new data
• Keep destination in sync after migration
• Use until we are ready to cut over
DMS

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration details
• Large amounts of data
• Audit tables more than one billion rows
• 1.5 TB data to migrate
• 800 tables
DMS

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The big question

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Time to completion
SUN MON TUE WED THU FRI SAT

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Modify tasks
• Spread tables across tasks
• 12 Tables in parallel per task
• Balance tasks
• Analyze data and balanced per task
• Instance sizing
• Storage
• Network
• CPU
Profile and improve

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key performance gains
Data loading – daily processing of files with over 1 million lines
Before
On-prem dedicated box
Taking longer over time
~26.5 hours (daily?)
After
EC2 instance only when needed
Faster DB – faster loading
~6.5 hours
Application improvements
No longer a need for local SQL DB cache
Allows for easier autoscaling

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary
Schema transition – Pick the right tool
We used MySQL Workbench
Have also used SCT for other projects – always improving
DMS Tasks
Spread migration workload and right size
Clean/remove unnecessary data
Take advantage of native features
Right size DMS instances

54. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
John Winford
winfordj@amazon.com

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.