In this introductory session, we cover how to convert and migrate your relational databases, non-relational databases, and data warehouses to the cloud. AWS Database Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT) have been used to migrate tens of thousands of databases across the world, including homogeneous migrations, such as PostgreSQL to PostgreSQL, and heterogeneous migrations between different database engines, such as Oracle or SQL Server to Amazon Aurora, Amazon DynamoDB, and Amazon Redshift. Learn how to quickly and securely migrate your data and procedural code, enjoy flexibility and cost savings, and minimize the downtime of your applications.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ramya Kaushik
Database Engineer, Amazon
SRV215
Migrating Databases to the Cloud:
Introduction to AWS DMS

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How can I get to the cloud?
How will my on-premises data migrate to the cloud?
How can I make it transparent to my users?
Afterwards, how will on-premises and cloud data interact?
How can I integrate my data assets within AWS?
Can I get help moving off of commercial databases?
How can I move data to my data lake?

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration used to be cost + complexity + time
Commercial data migration and replication software
Complex to set up and manage
Application downtime
Database engine-specific application code

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What are AWS DMS and AWS SCT?
AWS Database Migration Service (AWS DMS) easily and securely
migrates and replicates your databases and data warehouses to AWS
AWS Schema Conversion Tool (AWS SCT) converts your commercial
database and data warehouse schemas to open-source engines or AWS-
native services, such as Amazon Aurora and Amazon Redshift

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use AWS DMS and AWS SCT?
Modernize Migrate Replicate

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use AWS SCT?
Modernize
Modernize your database tier
Modernize and migrate your data
warehouse to Amazon Redshift
Amazon Aurora
Amazon Redshift

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT Migration Assessment
Connect AWS SCT to
source and target
databases
Run assessment report
Read executive summary
Follow detailed
instructions

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Components of the AWS SCT console
1. Source schema
2. Action items
3. Target schema
4. Schema element details
5. Edit window

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT helps with converting tables, views, and code
Sequences
User-defined types
Synonyms
Packages
Stored procedures
Functions
Triggers
Schemas
Tables
Indexes
Views
Sort and distribution keys

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT data extractors
Extract data from your data warehouse and migrate to Amazon Redshift
• Extracts data through local migration agents
• Data is optimized for Amazon Redshift and saved in local files
• Files are loaded to an Amazon S3 bucket (through network or AWS Snowball)
and then to Amazon Redshift
Amazon RedshiftAWS SCT S3 bucket

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use AWS DMS*?
Migrate
• Migrate business-critical applications
• Migrate from Classic to VPC
• Migrate data warehouse to Amazon
Redshift
• Upgrade to a minor version
• Consolidate shards into Amazon
Aurora
• Archive old data
• Migrate from NoSQL to SQL, SQL to
NoSQL, or NoSQL to NoSQL
Targets:
Amazon Dynamo
DB
Amazon Redshift
Amazon S3
Amazon Aurora
*AWS DMS is a HIPAA-certified service
Amazon S3
Sources:

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration validation
Pre-migration assessment
• Checks migration task settings prior to
launch
• Alerts for potential issues, such as
unsupported data types
Post-migration assessment
• Validates that all data selected for
migration migrated properly

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS DMS + AWS Snowball
Common use cases –
• Migrate large databases (over 5 TB)
• Migrate many databases at once
• Migrate over slow network
• Push vs. pull

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use AWS DMS?
Replicate • Create cross-region read replicas
• Run your analytics in the cloud
• Keep your dev / test and production
environment sync

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security and privacy highlights – AWS DMS
Security and privacy as top priority
• Certified service – ISO 27001, ISO 9001, SOC, PCI,
HIPAA, FedRAMP
• Supports all AWS security features, such as security
groups
• Supports fine-grained access controls via AWS
Identity and Access Management (IAM)
Ability to encrypt migrating data at multiple points
• AWS Key Management Service (AWS KMS) to encrypt
data in underlying disk, or mask important
information
• SSL to support end-to-end encryption for connections
to all sources and targets via SSL
Supports all migration use cases
• Secure source to secure target
• Secure source to non-secure target
• Non-secure source to secure target
• Non-secure source to non-secure target

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why use AWS DMS and AWS SCT?
Secure
Cost effective
Remove barriers
to entry
Allow DB
freedom
Keep a leg in
the cloud
Easy to use, but
Sophisticated …
Near-zero
downtime

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does it work?

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration process
Step 1: Convert or copy your schema
Source DB or DW
AWS SCT
Native tool
Destination DB or DW
Step 2: Move your data
Source DB or DW
AWS SCT
Destination DB or DW
AWS DMS

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer
premises
Application users
AWS
Internet
VPN
Start a replication instance
Connect to source and target
databases
Select tables, schemas, or databases
 Let AWS DMS create tables,
load data, and keep them in
sync
 Switch applications over to
the target at your convenience
Keep your apps running during the migration
AWS
DMS

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multi-AZ option for high availability
Customer
premises
or AWS
AWS
Internet
VPN
AWS DMS
AWS DMS

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load is table by table
Replication instance
Source Target

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Change data capture (CDC) and apply
Replication instanceSource Target
Update
t1 t2
t1
t2
Transactions
Change apply after bulk load

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source
Target
What else can I do?
Source
Source
Replication instance

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What else can I do? Target
Source Target
Target
Replication instance

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source
Take it all – or not
Target
Replication instance

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Homogenous or heterogeneous
SQL Server MySQL
Oracle Oracle
Oracle Amazon
S3

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use AWS DMS
1. Heterogeneous migrations
2. Moving data to Amazon S3, Amazon Redshift, or Amazon
DynamoDB
3. Homogeneous migrations: When native tools aren’t
sufficient
4. In conjunction with native tools or other methods
5. For ongoing replication to support all of the above use cases

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Homogeneous migration: If native tools will work
2. If you have another tool that works better
When not to use AWS DMS

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration best practices

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Full load
• Load multiple tables in parallel
• Remove bottlenecks on the target
• Use multiple tasks
• Use proper replication instance or multiple replication servers
• Improve LOB performance
• Use tow filters
Ongoing replication (CDC)
By default, AWS DMS processes changes in a transactional mode
Use batch apply if you can afford temporary lapses in transactional integrity
Use proper indexes on target during CDC
➢ Pause after full load (apply secondary indexes)
➢ Pause after cached changes (enable foreign keys, etc.)
Optimizing full load/CDC – rules of thumb

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS preflight check

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS preflight check, sample report

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS data validation
• Data validation is optional.
• AWS DMS compares each row in the source with its corresponding row at
the target.
• To enable data validation, set the EnableValidation to true.
• ThreadCount : The number of execution threads that AWS DMS uses
during validation.
• The default value for ThreadCount is 5.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS validation

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Validation statistics
ValidationState - Can have the following values:
• Pending records - Some records in the table are waiting for validation.
• Mismatched records - For more information, check the awsdms_validation_failures
table on the target endpoint.
• Suspended records - Some records in the table can't be validated.
• No primary key - The table can't be validated because it had no primary key.
• Table error - The table is in an error state and some data wasn't migrated.
• Validated - All rows in the table are validated.
• Error - The table can't be validated because of an unexpected error.
ValidationPending - Number of records migrated but not yet validated.
ValidationSuspended - Number of records that AWS DMS can't compare.
ValidationFailed - Number of records that failed the data validation phase.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data validation troubleshooting
Table at the target endpoint: awsdms_validation_failures_v1.

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data validation limitations
1. Table must have PK or unique.
2. PK columns can't be of type CLOB, BLOB, or BYTE.
3. Filtering currently not supported.
4. Consolidation not supported.
5. Modification of target outside AWS DMS may compromise results.
6. Validation stops if > 1,000 failed or suspended records.
7. Data validation generates additional queries against the source and target
databases.

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS log management

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recommendations
• A select * on the table can be resource-intensive.
• Transaction Consistency Timeout (coming soon)
• Take note of the target table prep mode option
• Take note of the LOB settings to avoid LOB truncations during
full load
• Enable logging
• Use table mappings to select the right tables to migrate

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our customers

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our customers

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use DMS to securely continuously (CDC)
replicate mission-critical data from various
sources to Redshift. Use the API to fully
automate the process.
DMS reduced the time it took to migrate
over 35 million users to Aurora MySQL
by 40%.
Migrated data from on-premises Oracle RAC
to RDS for PostgreSQL. Today replicates data
to Redshift. Use SCT for database
modernization and DMS to split the data and
keep all systems in full sync.
Verizon is targeting a mass migration off of
Oracle to PostgreSQL. One project expects
a potential savings of $1.75M.
Trimble Leveraged SCT to determine an
appropriate target engine. Reduced
infrastructure costs to less than one-fourth
of their privately hosted infrastructure
The US Department
of Veterans Affairs used
DMS to modernize and securely migrate a
20 years Oracle database to RDS with no
downtime.
AWS DMS customer experience

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partners

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS database migration partners

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The fine print

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sources for AWS DMS
Customers use the following databases as a source for data migration using AWS DMS:
On-premises and Amazon EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2014
• MySQL 5.5–5.7
• MariaDB 10, 10.1, 10.2
• PostgreSQL 9.4–9.6, 10
• SAP ASE 15.7+
• Azure DB
• Db2 LUW 9.7,10.1, 10.5, 11.1
RDS instance databases:
• Oracle Database 11g–12c
• Microsoft SQL Server 2008R2–2014
• MySQL versions 5.5–5.7
• MariaDB 10, 10.1, 10.2
• PostgreSQL 9.4–9.6, 10
• Amazon Aurora (MySQL-compatible data source)
• Amazon Aurora (PostgreSQL-compatible data source) *No CDC

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Targets for AWS DMS
Customers can use the following databases as a target for data replication using AWS
DMS
On-premises and EC2 instance databases
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.6, 10
• SAP ASE 15.7+
RDS instance databases
• Oracle Database 11g–12c
• Microsoft SQL Server 2008 R2 - 2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.6, 10
• Amazon Aurora (MySQL and PostgreSQL -compatible data sources)
• Amazon Redshift

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS
T2 for developing and periodic data migration tasks
C4 for large databases and minimizing time
T2 pricing starts at $0.018 per hour for T2.micro
C4 pricing starts at $0.154 per hour for C4.large
50 GB GP2 storage included with T2 instances
100 GB GP2 storage included with C4 instances
Data transfer inbound and within AZ is free
Data transfer across AZs starts at $0.01 per GB

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resources available to customers – AWS DMS
Getting Started Guide: Technical documentation
Features and benefits: Highlights AWS DMS features
Pricing: Prices for replication instances, storage, and data transfer
Support: Post your questions to our Support forum
AWS SDK: Java-based API for creating and managing data migration tasks
AWS Command Line Interface: Start-and-stop replication tasks with simple commands

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resources available to customers – AWS SCT
User Guide: Review technical documentation at
aws.amazon.com/documentation/SchemaConversionTool/
You can also choose to Download.
Download area: Get installation files for the AWS SCT.
Support forums: Ask questions, and review how-to guides.
https://forums.aws.amazon.com/forum.jspa?forumID=208.

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Submit session feedback
1. Tap the Schedule icon.
2. Select the session you attended.
3. Tap Session Evaluation to submit
your feedback.

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!