In this introductory session, we cover how to convert and migrate your relational databases, non-relational databases, and data warehouses to the cloud. AWS Database Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT) have been used to migrate tens of thousands of databases across the world. This includes homogeneous migrations, such as PostgreSQL to PostgreSQL, and heterogeneous migrations between different database engines, such as Oracle or SQL Server to Amazon Aurora, Amazon DynamoDB and Amazon Redshift. Learn how to quickly and securely migrate your data and procedural code, enjoy flexibility and cost savings, and minimize the downtime of your applications.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Database Migration Service and Schema Conversion Tool
Ed Murray
Manager AWS Migration Technology Team
SRV215
Migrating Databases to the Cloud:
Introduction to AWS DMS

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How can I get to the cloud?
How will my on-premises data migrate to the cloud?
How can I make it transparent to my users?
Afterwards, how will on-premises and cloud data interact?
How can I integrate my data assets within AWS?
Can I get help moving off of commercial databases?
How can I move data to my data lake?

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration used to be cost + complexity + time
Commercial data migration and replication software
Complex to set up and manage
Application downtime
Database-engine-specific application code

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are DMS and SCT?
AWS Database Migration Service (DMS) easily and securely migrates
and/or replicate your databases and data warehouses to AWS
AWS Schema Conversion Tool (SCT) converts your commercial database and
data warehouse schemas to open-source engines or AWS-native services,
such as Amazon Aurora and Redshift

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When to use DMS and SCT?
Modernize Migrate Replicate

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When to use SCT?
Modernize
Modernize your database tier
Modernize and Migrate your Data
Warehouse to Amazon Redshift
Amazon Aurora
Amazon Redshift

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SCT helps with converting tables, views, and code
Sequences
User-defined types
Synonyms
Packages
Stored procedures
Functions
Triggers
Schemas
Tables
Indexes
Views
Sort and distribution keys

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SCT Migration Assessment Report
• Assessment of migration
compatibility of source databases
with open-source database
engines – RDS MySQL, RDS
PostgreSQL and Aurora
• Recommends best target engine
• Provides details level of efforts to
complete migration

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Old World to Aurora Migration Playbooks
• Topic-by-topic overview of Oracle to Aurora PostgreSQL migrations and “hands-on” best
practices
• How to migrate from proprietary features and the different database objects
• Migration best practices
• SQL Server to Aurora MySQL – Q2
• Oracle to Aurora MySQL – Q3
SCT DMS Playbook
Schema Data Best Practices

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SCT Data Extractors
Extract Data from your data warehouse and migrate to Amazon Redshift
• Extracts data through local migration agents
• Data is optimized for Redshift and saved in local files
• Files are loaded to an Amazon S3 bucket (through network or Amazon Snowball)
and then to Amazon Redshift
Amazon RedshiftAWS SCT S3 Bucket

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When to use DMS*?
Migrate
• Migrate business-critical applications
• Migrate from Classic to VPC
• Migrate data warehouse to Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Archive old data
• Migrate from NoSQL to SQL, SQL to
NoSQL or NoSQL to NoSQL
Targets:
Amazon Dynamo
DB
Amazon Redshift
Amazon S3
Amazon Aurora
*DMS is a HIPAA certified service
Amazon S3
Sources:

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration Validation
Pre-migration Assessment
• Checks migration task settings prior to
launch
• Alerts for potential issues, such as
unsupported data types
Post-migration Assessment
• Validates that all data selected for
migration migrated properly

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS + Snowball
Common use cases –
• Migrate large databases (over 5TB)
• Migrate many databases at once
• Migrate over slow network
• Push vs. Pull

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When to use DMS?
Replicate • Create cross-regions Read Replicas
• Run your analytics in the cloud
• Keep your dev/test and production
environment sync

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security and Privacy Highlights – DMS
Security and privacy as top priority
• Certified service – ISO 27001, ISO 9001, SOC, PCI, HIPAA, FedRAMP
• Supports all AWS security features, such as security groups
• Supports fine-grained access controls via Identity and Access Management (IAM)
Ability to encrypt migrating data at multiple points
• Key Management Service (KMS) to encrypt data in underlying disk, or mask important
information
• SSL to support end-to-end encryption for connections to all sources and targets via SSL

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why use DMS and SCT?
Secure
Cost Effective
Remove Barriers
to Entry
Allow DB
Freedom
Keep a Leg in
the Cloud
Easy to Use, but
Sophisticated…
Near-Zero
Downtime

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does it work?

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database migration process
Source DB or DW
AWS SCT
Native Tool
Destination DB or DW
Step 2: Move your data
Source DB or DW
AWS SCT
Destination DB or DW
AWS DMS
Step 1: Convert or Copy your Schema

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customer
premises
Application users
AWS
Internet
VPN
Keep your apps running during the migration
Start a replication instance
Connect to source and target databases
Select tables, schemas, or databases
 Let AWS DMS create tables,
load data, and keep them in
sync
 Switch applications over to
the target at your convenience
AWS
DMS

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ option for high availability
Customer
premises
or AWS
AWS
Internet
VPN
AWS DMS
AWS DMS

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Load is table by table
Replication instance
Source Target

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Change data capture (CDC) and apply
Replication instanceSource Target
Update
t1 t2
t1
t2
Transactions
Change apply after bulk load

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source
Target
What else can I do?
Source
Source
Replication instance

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What else can I do? Target
Source Target
Target
Replication instance

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source
Take it all—or not
Target
Replication instance

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Homogenous or heterogeneous
SQL Server MySQL
Oracle Oracle
Oracle S3

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When To Use The Database Migration Service
1. Heterogeneous Migrations
2. Moving data to S3, Redshift, or DynamoDB
3. Homogeneous Migrations: When native tools aren’t sufficient
4. In conjunction with native tools or other methods
5. For ongoing replication to support all of the above use cases

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When Not to Use Database Migration Service
1. Homogeneous Migration: If native tools will work
2. If you have another tool that works better

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration Best Practices

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Full Load
• Load multiple Tables in Parallel
• Remove bottlenecks on the target
• Use Multiple Tasks
• Use Proper Replication Instance or Multiple Replication Servers
• Improve LOB performance
• Use Row filters.
On going Replication (CDC)
By default, AWS DMS processes changes in a transactional mode
Use batch apply if you can afford temporary lapses in transactional integrity
Use proper Indexes on target during CDC
➢ Pause after full load (apply secondary indexes)
➢ Pause after cached changes (enable foreign keys etc.)
Optimizing Full Load/CDC – Rules of Thumb

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS Preflight check

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS Preflight check, Sample Report

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS Data Validation
• Data validation is optional.
• DMS compares each row in the source with its corresponding row at the
target
• To enable data validation, set the EnableValidation setting to true.
• ThreadCount : The number of execution threads that AWS DMS uses
during validation.
• The default value for ThreadCount is 5.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS Validation

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Validation Statistics
ValidationState - Can have the following values:
• Pending records - Some records in the table are waiting for validation.
• Mismatched records - For more information, check the awsdms_validation_failures
table on the target endpoint.
• Suspended records - Some records in the table can't be validated.
• No primary key - The table can't be validated because it had no primary key.
• Table error - The table is in an error state and some data wasn't migrated.
• Validated - All rows in the table are validated.
• Error - The table can't be validated because of an unexpected error.
ValidationPending – Number of records migrated but not yet validated.
ValidationSuspended - Number of records that AWS DMS can't compare.
ValidationFailed - Number of records that failed the data validation phase.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Validation Troubleshooting
Table at the target endpoint: awsdms_validation_failures_v1.

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Validation Limitations
1. Table must have PK or unique.
2. PK columns can't be of type CLOB, BLOB, or BYTE.
3. Filtering currently not supported.
4. Consolidation not supported.
5. Modification of target outside DMS may compromise results.
6. Validation stops if > 1,000 failed or suspended records.
7. Data validation generates additional queries against the source and
target databases.

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DMS Log Management

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recommendations:
• A select * on the table can be resource intensive.
• Transaction Consistency Timeout( coming soon)
• Take note of the target table prep mode option
• Take note of the LOB settings to avoid LOB truncations during
full load
• Enable logging
• Use table mappings to select the right tables to migrate

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our Customers

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our Customers

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use DMS to securely continuously (CDC)
replicate mission-critical data from various
sources to Redshift. Use the API to fully
automate the process.
DMS reduced the time it took to migrate
over 35 million users to Aurora MySQL
by 40%.
Migrated data from on-premises Oracle RAC
to RDS for PostgreSQL. Today replicates data
to Redshift. Use SCT for database
modernization and DMS to split the data and
keep all systems in full sync.
Verizon is targeting a mass migration off of
Oracle to PostgreSQL. One project expects
a potential savings of $1.75M.
Trimble Leveraged SCT to determine an
appropriate target engine. Reduced
infrastructure costs to less than one-fourth
of their privately hosted infrastructure
The US Department
of Veterans Affairs used
DMS to modernize and securely migrate a
20 years Oracle database to RDS with no
downtime.
DMS Customer Experience

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The fine print

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sources for AWS Database Migration Service
Customers use the following databases as a source for data migration using AWS DMS:
On-premises and Amazon EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2014
• MySQL 5.5–5.7
• MariaDB 10, 10.1, 10.2
• PostgreSQL 9.4–9.6, 10
• SAP ASE 15.7+
• Azure DB
• Db2 LUW 9.7,10.1, 10.5, 11.1
RDS instance databases:
• Oracle Database 11g–12c
• Microsoft SQL Server 2008R2–2014
• MySQL versions 5.5–5.7
• MariaDB 10, 10.1, 10.2
• PostgreSQL 9.4–9.6, 10
• Amazon Aurora (MySQL-compatible data source)
• Amazon Aurora (PostgreSQL-compatible data source) *No CDC

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Targets for AWS Database Migration Service
Customers can use the following databases as a target for data replication using AWS
DMS:
On-premises and EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.6, 10
• SAP ASE 15.7+
RDS instance databases:
• Oracle Database 11g–12c
• Microsoft SQL Server 2008 R2 - 2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.6, 10
• Amazon Aurora (MySQL and PostgreSQL -compatible data sources)
• Amazon Redshift

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Database Migration service pricing
T2 for developing and periodic data migration tasks
C4 for large databases and minimizing time
T2 pricing starts at $0.018 per hour for T2.micro
C4 pricing starts at $0.154 per hour for C4.large
50 GB GP2 storage included with T2 instances
100 GB GP2 storage included with C4 instances
Data transfer inbound and within AZ is free
Data transfer across AZs starts at $0.01 per GB

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resources available to customers - DMS
Getting Started Guide: Review technical documentation.
Features and benefits: Highlights DMS features.
Pricing: Prices for replication instances, storage, and data transfer.
Support: Post your questions to our Support forum.
AWS SDK: Java-based API for creating and managing data migration tasks.
AWS Command Line Interface: Start and stop replication tasks with simple commands.

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resources available to customers - SCT
User Guide: Review technical docs at
aws.amazon.com/documentation/SchemaConversionTool/
or choose the Download button.
Download area: Get installation files for the Schema Conversion Tool.
Support forums: Ask questions and review how-to guides.
https://forums.aws.amazon.com/forum.jspa?forumID=208.

49. Submit Session Feedback
1. Tap the Schedule icon.
2. Select the session you
attended.
3. Tap Session Evaluation to
submit your feedback.

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.