Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Migrating Databases to the Cloud: Introduction to AWS DMS - SRV215 - Toronto AWS Summit

278 views

Published on

In this introductory session, we cover how to convert and migrate your relational databases, non-relational databases, and data warehouses to the cloud. AWS Database Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT) have been used to migrate tens of thousands of databases across the world. This includes homogeneous migrations, such as PostgreSQL to PostgreSQL, and heterogeneous migrations between different database engines, such as Oracle or SQL Server to Amazon Aurora, Amazon DynamoDB and Amazon Redshift. Learn how to quickly and securely migrate your data and procedural code, enjoy flexibility and cost savings, and minimize the downtime of your applications.

  • Be the first to comment

  • Be the first to like this

Migrating Databases to the Cloud: Introduction to AWS DMS - SRV215 - Toronto AWS Summit

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Database Migration Service and Schema Conversion Tool Ed Murray Manager AWS Migration Technology Team SRV215 Migrating Databases to the Cloud: Introduction to AWS DMS
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How can I get to the cloud? How will my on-premises data migrate to the cloud? How can I make it transparent to my users? Afterwards, how will on-premises and cloud data interact? How can I integrate my data assets within AWS? Can I get help moving off of commercial databases? How can I move data to my data lake?
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migration used to be cost + complexity + time Commercial data migration and replication software Complex to set up and manage Application downtime Database-engine-specific application code
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What are DMS and SCT? AWS Database Migration Service (DMS) easily and securely migrates and/or replicate your databases and data warehouses to AWS AWS Schema Conversion Tool (SCT) converts your commercial database and data warehouse schemas to open-source engines or AWS-native services, such as Amazon Aurora and Redshift
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. When to use DMS and SCT? Modernize Migrate Replicate
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. When to use SCT? Modernize Modernize your database tier Modernize and Migrate your Data Warehouse to Amazon Redshift Amazon Aurora Amazon Redshift
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. SCT helps with converting tables, views, and code Sequences User-defined types Synonyms Packages Stored procedures Functions Triggers Schemas Tables Indexes Views Sort and distribution keys
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. SCT Migration Assessment Report • Assessment of migration compatibility of source databases with open-source database engines – RDS MySQL, RDS PostgreSQL and Aurora • Recommends best target engine • Provides details level of efforts to complete migration
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Old World to Aurora Migration Playbooks • Topic-by-topic overview of Oracle to Aurora PostgreSQL migrations and “hands-on” best practices • How to migrate from proprietary features and the different database objects • Migration best practices • SQL Server to Aurora MySQL – Q2 • Oracle to Aurora MySQL – Q3 SCT DMS Playbook Schema Data Best Practices
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. SCT Data Extractors Extract Data from your data warehouse and migrate to Amazon Redshift • Extracts data through local migration agents • Data is optimized for Redshift and saved in local files • Files are loaded to an Amazon S3 bucket (through network or Amazon Snowball) and then to Amazon Redshift Amazon RedshiftAWS SCT S3 Bucket
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. When to use DMS*? Migrate • Migrate business-critical applications • Migrate from Classic to VPC • Migrate data warehouse to Redshift • Upgrade to a minor version • Consolidate shards into Aurora • Archive old data • Migrate from NoSQL to SQL, SQL to NoSQL or NoSQL to NoSQL Targets: Amazon Dynamo DB Amazon Redshift Amazon S3 Amazon Aurora *DMS is a HIPAA certified service Amazon S3 Sources:
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migration Validation Pre-migration Assessment • Checks migration task settings prior to launch • Alerts for potential issues, such as unsupported data types Post-migration Assessment • Validates that all data selected for migration migrated properly
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMS + Snowball Common use cases – • Migrate large databases (over 5TB) • Migrate many databases at once • Migrate over slow network • Push vs. Pull
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. When to use DMS? Replicate • Create cross-regions Read Replicas • Run your analytics in the cloud • Keep your dev/test and production environment sync
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Security and Privacy Highlights – DMS Security and privacy as top priority • Certified service – ISO 27001, ISO 9001, SOC, PCI, HIPAA, FedRAMP • Supports all AWS security features, such as security groups • Supports fine-grained access controls via Identity and Access Management (IAM) Ability to encrypt migrating data at multiple points • Key Management Service (KMS) to encrypt data in underlying disk, or mask important information • SSL to support end-to-end encryption for connections to all sources and targets via SSL
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why use DMS and SCT? Secure Cost Effective Remove Barriers to Entry Allow DB Freedom Keep a Leg in the Cloud Easy to Use, but Sophisticated… Near-Zero Downtime
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How does it work?
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Database migration process Source DB or DW AWS SCT Native Tool Destination DB or DW Step 2: Move your data Source DB or DW AWS SCT Destination DB or DW AWS DMS Step 1: Convert or Copy your Schema
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer premises Application users AWS Internet VPN Keep your apps running during the migration Start a replication instance Connect to source and target databases Select tables, schemas, or databases  Let AWS DMS create tables, load data, and keep them in sync  Switch applications over to the target at your convenience AWS DMS
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-AZ option for high availability Customer premises or AWS AWS Internet VPN AWS DMS AWS DMS
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Load is table by table Replication instance Source Target
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Change data capture (CDC) and apply Replication instanceSource Target Update t1 t2 t1 t2 Transactions Change apply after bulk load
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Source Target What else can I do? Source Source Replication instance
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What else can I do? Target Source Target Target Replication instance
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Source Take it all—or not Target Replication instance
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Homogenous or heterogeneous SQL Server MySQL Oracle Oracle Oracle S3
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. When To Use The Database Migration Service 1. Heterogeneous Migrations 2. Moving data to S3, Redshift, or DynamoDB 3. Homogeneous Migrations: When native tools aren’t sufficient 4. In conjunction with native tools or other methods 5. For ongoing replication to support all of the above use cases
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. When Not to Use Database Migration Service 1. Homogeneous Migration: If native tools will work 2. If you have another tool that works better
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migration Best Practices
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Full Load • Load multiple Tables in Parallel • Remove bottlenecks on the target • Use Multiple Tasks • Use Proper Replication Instance or Multiple Replication Servers • Improve LOB performance • Use Row filters. On going Replication (CDC) By default, AWS DMS processes changes in a transactional mode Use batch apply if you can afford temporary lapses in transactional integrity Use proper Indexes on target during CDC ➢ Pause after full load (apply secondary indexes) ➢ Pause after cached changes (enable foreign keys etc.) Optimizing Full Load/CDC – Rules of Thumb
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMS Preflight check
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMS Preflight check, Sample Report
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMS Data Validation • Data validation is optional. • DMS compares each row in the source with its corresponding row at the target • To enable data validation, set the EnableValidation setting to true. • ThreadCount : The number of execution threads that AWS DMS uses during validation. • The default value for ThreadCount is 5.
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMS Validation
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Validation Statistics ValidationState - Can have the following values: • Pending records - Some records in the table are waiting for validation. • Mismatched records - For more information, check the awsdms_validation_failures table on the target endpoint. • Suspended records - Some records in the table can't be validated. • No primary key - The table can't be validated because it had no primary key. • Table error - The table is in an error state and some data wasn't migrated. • Validated - All rows in the table are validated. • Error - The table can't be validated because of an unexpected error. ValidationPending – Number of records migrated but not yet validated. ValidationSuspended - Number of records that AWS DMS can't compare. ValidationFailed - Number of records that failed the data validation phase.
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Validation Troubleshooting Table at the target endpoint: awsdms_validation_failures_v1.
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Validation Limitations 1. Table must have PK or unique. 2. PK columns can't be of type CLOB, BLOB, or BYTE. 3. Filtering currently not supported. 4. Consolidation not supported. 5. Modification of target outside DMS may compromise results. 6. Validation stops if > 1,000 failed or suspended records. 7. Data validation generates additional queries against the source and target databases.
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMS Log Management
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Recommendations: • A select * on the table can be resource intensive. • Transaction Consistency Timeout( coming soon) • Take note of the target table prep mode option • Take note of the LOB settings to avoid LOB truncations during full load • Enable logging • Use table mappings to select the right tables to migrate
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our Customers
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our Customers
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use DMS to securely continuously (CDC) replicate mission-critical data from various sources to Redshift. Use the API to fully automate the process. DMS reduced the time it took to migrate over 35 million users to Aurora MySQL by 40%. Migrated data from on-premises Oracle RAC to RDS for PostgreSQL. Today replicates data to Redshift. Use SCT for database modernization and DMS to split the data and keep all systems in full sync. Verizon is targeting a mass migration off of Oracle to PostgreSQL. One project expects a potential savings of $1.75M. Trimble Leveraged SCT to determine an appropriate target engine. Reduced infrastructure costs to less than one-fourth of their privately hosted infrastructure The US Department of Veterans Affairs used DMS to modernize and securely migrate a 20 years Oracle database to RDS with no downtime. DMS Customer Experience
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The fine print
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sources for AWS Database Migration Service Customers use the following databases as a source for data migration using AWS DMS: On-premises and Amazon EC2 instance databases: • Oracle Database 10g–12c • Microsoft SQL Server 2005–2014 • MySQL 5.5–5.7 • MariaDB 10, 10.1, 10.2 • PostgreSQL 9.4–9.6, 10 • SAP ASE 15.7+ • Azure DB • Db2 LUW 9.7,10.1, 10.5, 11.1 RDS instance databases: • Oracle Database 11g–12c • Microsoft SQL Server 2008R2–2014 • MySQL versions 5.5–5.7 • MariaDB 10, 10.1, 10.2 • PostgreSQL 9.4–9.6, 10 • Amazon Aurora (MySQL-compatible data source) • Amazon Aurora (PostgreSQL-compatible data source) *No CDC
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Targets for AWS Database Migration Service Customers can use the following databases as a target for data replication using AWS DMS: On-premises and EC2 instance databases: • Oracle Database 10g–12c • Microsoft SQL Server 2005–2014 • MySQL 5.5–5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.3–9.6, 10 • SAP ASE 15.7+ RDS instance databases: • Oracle Database 11g–12c • Microsoft SQL Server 2008 R2 - 2014 • MySQL 5.5–5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.3–9.6, 10 • Amazon Aurora (MySQL and PostgreSQL -compatible data sources) • Amazon Redshift
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Database Migration service pricing T2 for developing and periodic data migration tasks C4 for large databases and minimizing time T2 pricing starts at $0.018 per hour for T2.micro C4 pricing starts at $0.154 per hour for C4.large 50 GB GP2 storage included with T2 instances 100 GB GP2 storage included with C4 instances Data transfer inbound and within AZ is free Data transfer across AZs starts at $0.01 per GB
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Resources available to customers - DMS Getting Started Guide: Review technical documentation. Features and benefits: Highlights DMS features. Pricing: Prices for replication instances, storage, and data transfer. Support: Post your questions to our Support forum. AWS SDK: Java-based API for creating and managing data migration tasks. AWS Command Line Interface: Start and stop replication tasks with simple commands.
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Resources available to customers - SCT User Guide: Review technical docs at aws.amazon.com/documentation/SchemaConversionTool/ or choose the Download button. Download area: Get installation files for the Schema Conversion Tool. Support forums: Ask questions and review how-to guides. https://forums.aws.amazon.com/forum.jspa?forumID=208.
  49. 49. Submit Session Feedback 1. Tap the Schedule icon. 2. Select the session you attended. 3. Tap Session Evaluation to submit your feedback.
  50. 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×