Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft

275 views

Published on

According to Gartner, “the future of the DBMS is the cloud.” But what does it mean to migrate an on-premises database to the cloud? What database options are available? Do I lift-and-shift, re-platform, or re-factor to a modern database engine? How do I size my database workload in AWS? Join AWS experts for an in-depth discussion based on real-world use cases to explore these topics and more as you plan your journey to the cloud.

  • Be the first to comment

The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft

  1. 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Databases on AWS The Right Tool for the Right Job
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migration & Streaming Services Data Movement Data Warehousing Big Data Processing Interactive Query Operational Analytics Real time Analytics Serverless Data processing Analytics Infrastructure Data Catalog & ETL Security & Management Data Lake Infrastructure & Management Ledger Blockchain Central Trust Dashboards Predictive Analytics Visualization & Machine Learning Databases Relational Key-value Document In-memory Graph Time-Series Broad and deep portfolio, purpose-built for builders
  3. 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Movement Analytics Data Lake Infrastructure & Management Ledger Visualization & Machine Learning Databases + 10 more QuickSight SageMaker Comprehend Lex Polly Rekognition The image part with relationship ID rId5 was not found in the file. Translate Transcribe Deep Learning AMIs Aurora MySQL, PostgreSQL RDS and RDS on VMWARE MySQL, PostgreSQL, Oracle, SQL Server MariaDB DynamoDB Key-value ElastiCache Redis, Memcached Timestream Time Series Neptune Graph DocumentDB Documents Redshift EMR Spark, Hadoop Athena Elasticsearch Service Kinesis Data Analytics Glue Scala, Python S3/Glacier Lake Formation Glue Crawl, ETL Managed Blockchain Quantum Ledger Database (QLDB) Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka Broad and deep portfolio, purpose-built for builders
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Common data categories and use cases Relational Referential integrity, ACID transactions, schema- on-write Lift and shift, ERP, CRM, finance Key-value High throughput, low- latency reads and writes, endless scale Real-time bidding, shopping cart, social, product catalog, customer preferences Document Store documents and quickly access querying on any attribute Content management, personalization, mobile In-memory Query by key with microsecond latency Leaderboards, real-time analytics, caching Graph Quickly and easily create and navigate relationships between data Fraud detection, social networking, recommendation engine Time-series Collect, store, and process data sequenced by time IoT applications, event tracking Ledger Complete, immutable, and verifiable history of all changes to application data Systems of record, supply chain, health care, registrations, financial
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Database Services DynamoDB ElastiCache Neptune Key value | Document Graph Non-relational Database Migration Service Relational Aurora RDS
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Two fundamental areas of focus “Lift and shift” existing apps to the cloud Quickly build new apps in the cloud
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Lift and shift” existing apps to the cloud “Lift and shift” existing apps to the cloud Quickly build new apps in the cloud
  8. 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora MySQL and PostgreSQL-compatible relational database built for the cloud Performance and availability of commercial-grade databases at 1/10th the cost Performance and scalability Availability and durability Highly secure Fully managed 5x throughput of standard MySQL and 3x of standard PostgreSQL; scale-out up to 15 read replicas Fault-tolerant, self-healing storage; six copies of data across three Availability Zones; continuous backup to Amazon S3 Network isolation, encryption at rest/transit Managed by RDS: No hardware provisioning, software patching, setup, configuration, or backups
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Relational Database Service (RDS) Managed relational database service with a choice of six popular database engines Easy to administer Available and durable Highly scalable Fast and secure No need for infrastructure provisioning, installing, and maintaining DB software Automatic Multi-AZ data replication; automated backup, snapshots, failover Scale database compute and storage with a few clicks with no app downtime SSD storage and guaranteed provisioned I/O; data encryption at rest and in transit
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Database Migration Service (AWS DMS) M I G R A T I N G D A T A B A S E S T O A W S Migrate between on-premises and AWS Migrate between databases Automated schema conversion Data replication for zero-downtime migration 100,000+ databases migrated
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Quickly build new apps in the cloud “Lift and shift” existing apps to the cloud Quickly build new apps in the cloud
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Modern apps create new requirements Users: 1 million+ Data volume: TB–PB–EB Locality: Global Performance: Milliseconds–microseconds Request rate: Millions Access: Web, mobile, IoT, devices Scale: Up-down, Out-in Economics: Pay for what you use Developer access: No assembly requiredSocial mediaRide hailing Media streaming Dating
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s take a closer look at… Relational Key-value Graph
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Relational data • Divide data among tables • Highly structured • Relationships established via keys enforced by the system • Data accuracy and consistency Patient * Patient ID First Name Last Name Gender DOB * Doctor ID Visit * Visit ID * Patient ID * Hospital ID Date * Treatment ID Medical Treatment * Treatment ID Procedure How Performed Adverse Outcome Contraindication Doctor * Doctor ID First Name Last Name Medical Specialty * Hospital Affiliation Hospital * Hospital ID Name Address Rating
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Relational use case // Doctors affiliated with Mercy hospital Patient * Patient ID First Name Last Name Gender DOB * Doctor ID Visit * Visit ID * Patient ID * Hospital ID Date * Treatment ID Medical Treatment * Treatment ID Procedure How Performed Adverse Outcome Contraindication Doctor * Doctor ID First Name Last Name Medical Specialty * Hospital Affiliation Hospital * Hospital ID Name Address Rating SELECT d.first_name, d.last_name FROM doctor as d, hospital as h WHERE d.hospital = h.hospital_id AND h.name = ‘Mercy'; // Number of patient visits each doctor completed last week SELECT d.first_name, d.last_name, count(*) FROM visit as v, hospital as h, doctor as d WHERE v.hospital_id = h.hospital_id AND h.hospital_id = d.hospital AND v.t_date > date_trunc('week’, CURRENT_TIMESTAMP - interval '1 week') GROUP BY d.first_name, d.last_name;
  16. 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Key-value data • Simple key value pairs • Partitioned by keys • Resilient to failure • High throughput, low-latency reads and writes • Consistent performance at scale Gamers Primary Key Attributes GamerTag Level Points High Score Plays Hammer57 21 4050 483610 1722 FluffyDuffy 5 1123 10863 43 Lol777313 14 3075 380500 1307 Jam22Jam 20 3986 478658 1694 ButterZZ_55 7 1530 12547 66 … … … … … Gamers Hammer57 21 4050, 483610, 1722 GET { TableName:"Gamers", Key: { "GamerTag":"Hammer57“, “ProjectionExpression“:”Points” } }
  17. 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gamers Primary Key Attributes Gamer Tag Type Hammer57 Rank Level Points Tier 87 4050 Elite Status Health Progress 90 30 Weapon Class Damage Range Taser 87% 50 FluffyDuffy Rank Level Points Tier 5 1072 Trainee Status Health Progress 37 8 Key-value use case // Status of Hammer57 GET { TableName:"Gamers", Key: { "GamerTag":"Hammer57", "Type":"Status” } } // Return all Hammer57 Gamers GamerTag = :a :a Hammer57
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Fast and flexible NoSQL database service for any scale Performance at scale Handles millions of requests per second Delivers microsecond latency Automated global replication Enterprise ready ACID transactions Encryption at rest On-demand backup and restore Serverless Maintenance free Auto scaling On-demand capacity mode
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Document or Key-Value Scales to Any WorkloadFully Managed NoSQL Access Control Event Driven ProgrammingFast and Consistent
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Graph data • Relationships are first-class objects • Vertices connected by Edges PURCHASED PURCHASED FOLLOWS PURCHASED KNOWS PRODUCT SPORT FOLLOWS
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Graph use case // Product recommendation to a user gremlin> V().has(‘name’,’sara’).as(‘customer’).out(‘follows’).in(‘follows’).out(‘purchased’) ( (‘customer’)).dedup() (‘name’) ('name') KNO W S PURCHASED PURCHASED FOLLOWS PURCHASED KNOWS PRODUCT SPORT FOLLOWS FOLLOWS // Identify a friend in common and make a recommendation gremlin> g.V().has('name','mary').as(‘start’). both('knows').both('knows’). where(neq(‘start’)). dedup().by('name').properties('name')
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AMAZON Neptune Fully managed graph database FAST RELIABLEOPEN Query billions of relationships with millisecond latency 6 replicas of your data across 3 AZs with full backup and restore Build powerful queries easily with Gremlin and SPARQL Supports Apache TinkerPop & W3C RDF graph models EASY
  23. 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Open Source Apache TinkerPop™ Gremlin Traversal Language W3C Standard SPARQL Query Language RESOURCE DESCRIPTION FRAMEWORK (RDF) PROPERTY GRAPH Graph Models and Frameworks
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Common customer use cases Ledgers with centralized control Healthcare Verify and track hospital equipment inventory Manufacturers Track distribution of a recalled product HR & Payroll Track changes to an individual’s profile Government Track vehicle title history
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Challenges with building ledgers Adds unnecessary complexity BlockchainRDBMS - audit tables Difficult to maintain Hard to use and slow Hard to build Custom audit functionality using triggers or stored procedures Impossible to verify No way to verify changes made to data by sys admins
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Ledger database concepts C | H J Journal C | H Current | History Current | History Journal Ledger comprises J L Ledger databaseL Journal determines Current | History
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. ID Manufacturer Model Year VIN Owner ID Version Start End Manufacturer Model Year VIN Owner How it works ID Manufacturer Model Year VIN Owner 1 Tesla Model S 2012 123456789 Traci Russell INSERT INTO cars << { 'Manufacturer': 'Tesla', 'Model': 'Model S', 'Year': '2012', 'VIN': '123456789', 'Owner': 'Traci Russel' } >> FROM cars WHERE VIN = '123456789' UPDATE owner = 'Ronnie Nash'FROM cars WHERE VIN = '123456789' UPDATE owner = 'Elmer Hubbard' J ID Version Start End Manufacturer Model Year VIN Owner 1 1 07/16/2012 NULL Tesla Model S 2012 123456789 Traci Russell current.cars C history.cars H ID Version Start End Manufacturer Model Year VIN Owner 1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell 1 2 08/03/2013 NULL Tesla Model S 2012 123456789 Ronnie Nash ID Version Start End Manufacturer Model Year VIN Owner 1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell 1 2 08/03/2013 09/02/2016 Tesla Model S 2012 123456789 Ronnie Nash 1 3 09/02/2016 NULL Tesla Model S 2012 123456789 Elmer Hubbard ID Manufacturer Model Year VIN Owner 1 Tesla Model S 2012 123456789 Ronnie Nash ID Manufacturer Model Year VIN Owner 1 Tesla Model S 2012 123456789 Elmer Hubbard INSERT cars ID:1 Manufacturer: Tesla Model: Model S Year: 2012 VIN: 123456789 Owner: Traci Russell Metadata: { Date:07/16/2012 } H (x) UPDATE cars ID:1 Owner: Ronnie Nash Metadata: { Date:08/03/2013 } H (x) UPDATE cars ID:1 Owner: Elmer Hubbard Metadata: { Date: 09/02/2016 } H (x)
  29. 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Quantum Ledger Database (QLDB) Fully managed ledger database Track and verify history of all changes made to your application’s data Immutable Maintains a sequenced record of all changes to your data, which cannot be deleted or modified; you have the ability to query and analyze the full history Cryptographically verifiable Uses cryptography to generate a secure output file of your data’s history Easy to use Easy to use, letting you use familiar database capabilities like SQL APIs for querying the data Highly scalable Executes 2–3X as many transactions than ledgers in common blockchain frameworks
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Time-series data What is time-series data? What is special about a time-series database? A sequence of data points recorded over a time interval Time is the single primary axis of the data model t
  32. 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Time-series use case Application events IoT Sensor Readings DevOps data Humidity % WATER VAPOR 91.094.086.093.0
  33. 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Existing time-series databasesRelational databases Difficult to maintain high availability Difficult to scale Limited data lifecycle management Inefficient time-series data processing Unnatural for time-series data Rigid schema inflexible for fast moving time-series data Building with time-series data is challenging
  34. 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Timestream Fast, scalable, fully managed time-series database 1,000x faster and 1/10th the cost of relational databases Collect data at the rate of millions of inserts per second (10M/second) Trillions of daily events Adaptive query processing engine maintains steady, predictable performance Time-series analytics Built-in functions for interpolation, smoothing, and approximation Serverless Automated setup, configuration, server provisioning, software patching
  35. 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  36. 36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Purpose-built databases Relational Key-value Document In-memory Graph Time-series Ledger DynamoDB NeptuneAmazon RDS Aurora CommercialCommunity Timestream QLDBElastiCache
  37. 37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Migration Introduction
  38. 38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Challenges Migration patterns AWS Database Migration Service AWS Schema Conversion Tool Database Migration Overview Summary Questions
  39. 39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Migration Challenges Cost Complexity Application downtime Engine specific database code
  40. 40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Value Proposition – What determines value? From an Organizational Perspective… • Value enables a positive outcome • Value IS NOT preventing a negative outcome • Value is a function of Capability, Risk, and Cost. Value = f(Capability, Risk, Cost)
  41. 41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Value Proposition – Operations DevOps NoOpsSustainment Value
  42. 42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Value Proposition – Hosting IaaS PaaSOn-Premises Value
  43. 43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Value Proposition – Migration Re-Host Re-Platform Re-Factor Value Linux à Linux Oracle à Oracle Solaris à Linux Oracle à Aurora Linux à Serverless Oracle à DynamoDB
  44. 44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration Patterns Re-Host (Homogenous) • “Lift-and-Leverage” • Same database engine / Same operating system Re-Platform (Heterogeneous) • Different operating system • Different database engine • Convert the database and/or application code Re-Factor • Decompose into smaller, logical components • Right tool for the workload
  45. 45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Database Migration Service Sources* Targets* Oracle Oracle SQL Server SQL Server Azure SQL PostgreSQL PostgreSQL MySQL MySQL Amazon Redshift SAP ASE SAP ASE MongoDB Amazon S3 Amazon S3 Amazon DynamoDB IBM DB2 Amazon Kinesis Amazon ElasticSearch • Start your first migration in 10 minutes or less • Keep your apps running during the migration • Replicate from within, to, or from AWS • Move data to the same or different database engine Consult CHAP_Source.html and CHAP_Target.html pages for latest DMS sources and targets
  46. 46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Keep your application running during migration On-Premise AWS Cloud Generic database DB Instance Start a replication instance Connect to the source and target Select tables, schemas, or databases Let DMS create the target objects Move data and synchronize objects Switch applications when ready
  47. 47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Migration Options Load is table by table • Configurable number of tables in parallel One time load or Change Data Capture (CDC) • Read from database log on the source and apply to the target Filter criteria available for selective loading • Select only a few tables or a subset of data in your tables Multiple sources and targets. Mix and match. • One side of the migration must be in AWS Ongoing replication support • Keep your replication going until your application is ready to cutover
  48. 48. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Schema Conversion Tool Makes heterogeneous database migrations predictable by automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format compatible with the target database Features Database Migration Assessment report for choosing the right target engine Automatic conversion for eligible database objects and code Code browser to highlight places where manual edits are required
  49. 49. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support for the following conversions Source* Database Target* Database on AWS Oracle database Amazon Aurora, MySQL, PostgreSQL, Oracle Oracle data warehouse Amazon Redshift Azure SQL Amazon Aurora, MySQL, PostgreSQL Microsoft SQL Server Amazon Aurora, Amazon Redshift, MySQL PostgreSQL Teradata Amazon Redshift IBM Netezza Amazon Redshift Greenplum Amazon Redshift HPE Vertica Amazon Redshift MySQL and Maria DB PostgreSQL PostgreSQL Amazon Aurora, MySQL Amazon Aurora PostgreSQL IBM DB2 LUW Amazon Aurora, MySQL, PostgreSQL Apache Cassandra Amazon DynamoDB Consult CHAP_Source.html and CHAP_Target.html pages for latest DMS sources and targets
  50. 50. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS SCT converts objects and code
  51. 51. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pricing and platform support Microsoft Windows Apple Mac Fedora Linux (rpm) Ubuntu Linux (deb) $0You can download AWS Schema Conversion Tool for your platform of choice
  52. 52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Migration Overview Data migration Application code Schema objects Cutover Analysis Testing
  53. 53. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Example migration plan Analysis Assessment 2% Planning 7% Schema conversion TESTING Database conversion 14% Procedure, function, and db script conversion 25% Application conversion Application conversion/remediation 5% Integration with 3rd party applications 2% Data migration Data migration 3% Functional end to end testing 28% Performance testing 2% Cutover Integration and deployment 5% Training and knowledge transfer 2% Documentation and version control 2% Post production support 3%
  54. 54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Resources and skills required Database expertise is required • Working knowledge of target database engine Basic networking knowledge • Familiarity with AWS VPC concepts AWS knowledge is required Software architecture knowledge is an advantage
  55. 55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Educate yourself and your team Getting started FAQS Migration best practices Database migration playbook
  56. 56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Plan your time • Migration projects can take from several weeks to several months to complete depending on the complexity of the code conversion • It may take several iterations and may take longer than you anticipate • Good planning will increase the success and reduce the number of iterations required
  57. 57. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Understand your database Size • Uncompressed size • Number schemas and objects • Relationships between schema objects • Number large tables (>500GB) Datatypes • LOB column number and size • Custom or unusual datatypes • Tables without PKs Transaction profile • Transactions per second • Transaction boundaries Authentication • User authentication • Roles and permissions Integrations • ETL jobs, interfaces, applications
  58. 58. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Understand your network Database access • Firewalls, tunnels, VPN, Direct Connect VPC • Identify the target VPC and subnets you will use IAM permissions • Security groups and IAM permissions needed for DMS Do you have enough bandwidth to move all your data?
  59. 59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Clarify your requirements Can you afford downtime? Do you need the source database to stay alive after migration? Do you know why you prefer one target engine over another? What are your high availability requirements? Does all the data need to move? Does it need to move to the same place? Do you understand RDS benefits (backups, HA, etc.) Do you understand RDS limitations? (storage size, admin user, etc.)
  60. 60. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Create your target schema SCT • For heterogeneous migrations • Orchestrate your migration through SCT by configuring your access keys DMS • Creates only tables and primary keys • Uses transformation rules to convert case, filter, or change schema Native tools for homogeneous migrations
  61. 61. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Start your data migration • Check prerequisites on the source • Check the logs • Check CloudWatch metrics • Proper instance sizing • Migration can take time, especially if your database is large or your network connection is small
  62. 62. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Things that affect your migration speed Size of the source Transaction pressure on the source • Volume of transactions • Run length of transactions Size of the target DMS task settings (# tasks, # tables, settings) Bandwidth available Size of the replication instance Schema configuration LOBs in the schema
  63. 63. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. When to choose native migration tools Target supports native replication Moving ALL the data No transformations required Target is a new database
  64. 64. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. When to use backup/restore or import/export Database is small Downtime window is large enough Moving all the data No transformations required
  65. 65. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Larger database migration What is large? • Is a relative term • Moving all the data takes longer than you are willing to wait What can I do? • Use AWS Snowball • Let AWS Professional Services help you
  66. 66. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Lessons Learned ü Plan, Plan, Plan • 80+% of the time spent during the migration should be planning • Each aspect should be well documented, scripted, and automated ü Test, Test, Test • Conduct multiple test runs of the migration process until it is muscle memory • It is the customer’s responsibility to provide a test plan
  67. 67. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Summary Migrations are complex AWS SCT & DMS can automate a lot of the work Know your environment (existing and new) Bring the right skills Allocate enough time Sometimes native tools are the right choice Many successful migrations AWS is here to help
  68. 68. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Questions?
  69. 69. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS AWS Schema Conversion Tool John Franklin Solutions Architect – Database Specialist
  70. 70. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda AWS Schema Conversion Tool • Overview • Getting started • Menu navigation • Settings
  71. 71. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Overview
  72. 72. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Schema Conversion Tool (SCT) The AWS Schema Conversion Tool helps automate many database schema and code conversion tasks when migrating to a new database engine. Features Schema conversion between database engines Database Migration Assessment report for choosing the best target engine Code browser that highlights places where manual edits are required
  73. 73. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pricing and platform support Microsoft Windows Apple Mac Fedora Linux (rpm) Ubuntu Linux (deb) $0You can download AWS Schema Conversion Tool for your platform of choice
  74. 74. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Getting Started
  75. 75. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Connecting to your database 1. Create a new project 2. Connect to your source database 3. Enter the connection details
  76. 76. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration assessment report 4. Run assessment report 5. Review executive summary 6. Review detailed instructions
  77. 77. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Convert schema and code objects • Functions • Indexes • Packages • Tables • Triggers • Schemas • Sequences • Stored Procedures • Synonyms • User Defined Types • Views
  78. 78. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tree view of objects • Tree view of source • Lazy loaded objects
  79. 79. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Actions at a glance • Action items to help with manual conversion
  80. 80. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Target database schema • Tree view of target • Lazy loaded objects
  81. 81. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source object code • SQL code to create chosen source database object • Properties of chosen source database object
  82. 82. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Target object code • Editable SQL code to create chosen object in the target database • Properties of chosen target database object
  83. 83. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apply to target database Select Apply to Database from the Action menu or Right click on the target schema to apply
  84. 84. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Menu navigation
  85. 85. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Work with projects Save your work Open an existing project from a list of recent ones Connect to a source or target
  86. 86. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Context sensitive actions Menu options change when you work with the source versus the target
  87. 87. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Switch between views • Main view shows the working window for code conversion • Assessment report view shows the summary and action item detail • Database migration view for orchestrating your migration from within SCT
  88. 88. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Convert an application Use the AWS Schema Conversion Tool to convert your application code
  89. 89. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Get help Update AWS SCT User Guide
  90. 90. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Closer look at Settings
  91. 91. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Global settings Applies to your installation of SCT Control • Logging • Drivers • Performance and Memory • Assessment Report • AWS Service Profiles • Security • Notifications • Tree View
  92. 92. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Global settings - logging • Application activity is logged and file size is retained to 100 MB up to 20 files by default • Increase the information in the log by setting Debug mode to true
  93. 93. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Drivers • Add or update drivers for your databases • Drivers are used to connect to the source and target • Use the recommended drivers in the user guide
  94. 94. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Service Profiles Add your AWS Credentials to initiate an AWS DMS migration from within AWS SCT Enter your AWS Access Key and AWS Secret Key Test the connection to check permissions
  95. 95. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Control your application and project settings Default Project Settings Indicates where your project will be stored by default
  96. 96. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Control your application and project settings Project Settings Settings specific to the open project Control • Conversion settings • Project environment • SQL Scripting • Tree View
  97. 97. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Modify or add mapping rules Import or export transformation rules as JSON
  98. 98. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Questions?
  99. 99. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Database Migration Service
  100. 100. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Overview DMS components Database migration options Migration use cases Best practices Questions
  101. 101. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Overview
  102. 102. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Overview • Simple to use • Minimal downtime • Supports widely used databases • Low cost • Fast and easy to setup • Reliable
  103. 103. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simple to use No drivers or applications to install No changes to the source database in most cases Just a few clicks to start a migration from the console DMS manages the complexities of migration for you Automatically replicate changes Can be used for continuous replication
  104. 104. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Minimal downtime Source All changes can be replicated to the target Source database stays fully operational during the migration Target Target database stays in sync with the source for as long as you choose Switch over when the target is fully sync’d and without lag
  105. 105. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Supports widely used databases Sources* Targets** Oracle Oracle SQL Server SQL Server Azure SQL PostgreSQL PostgreSQL MySQL MySQL Amazon Redshift SAP ASE SAP ASE MongoDB Amazon S3 Amazon S3 Amazon DynamoDB IBM DB2 Amazon Kinesis Amazon ElasticSearch On-premises database https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html * **
  106. 106. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Low cost Compute resource Additional log storage Instance Type Hourly Rate Duration Activities Total t2.small $0.036 2 weeks Testing $12.096 c4.large $0.154 2 weeks Initial Load & CDC Until Cutover $51.744 Migrate a 1 TB DB for under $65 ($63.84) Pricing Example https://aws.amazon.com/dms/pricing/
  107. 107. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fast and easy to setup Set up a migration task in minutes Connect to the source database Connect to the target database Create a replication instance to run the migration Create a task Run the task You can use different tasks with different settings for different environments
  108. 108. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reliable • Highly resilient and self–healing • Continual monitoring • Source and target databases • Network connectivity • Replication instance • In case of interruption, it automatically restarts the process and continues the migration from where it was halted • Multi-AZ option for high-availability CloudWatch
  109. 109. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DMS Components
  110. 110. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Migration Service components Replication instance Endpoint Task
  111. 111. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replication instance • T2, C4, or R4 instance types Dedicated EC2 instance • Private: Source or Target inside VPC / VPN / DC • Public: Source or Target outside VPC Public or private IP address • Moves the data from the source to the target • Support for multiple tasks Task execution Replication Instance
  112. 112. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Endpoint Connection information to source or target database • Endpoint type, identifier, engine (source or target) • Server name, port, SSL mode • Username, password Advanced attributes • Extra connection information • KMS key Test with a replication instance
  113. 113. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Endpoint sources Oracle SQL Server Azure SQL PostgreSQL MySQL SAP ASE MongoDB Amazon S3 IBM DB2 https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html
  114. 114. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Endpoint targets Targets Oracle SQL Server PostgreSQL MySQL Amazon Redshift SAP ASE Amazon S3 Amazon DynamoDB Amazon Kinesis Amazon ElasticSearch https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html
  115. 115. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Task Name your task No spaces or underscores Connect to endpoints Source Target Choose migration type Existing data Existing data and replicate changes Replicate changes only Target preparation Do nothing Drop tables on target Truncate Include LOBs Don't include LOBs Full LOB mode Limited LOB mode*✓ Enable validation ✓ Enable logging
  116. 116. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Task – migration type Choose migration type Existing data Existing data and replicate changes Replicate changes only Creates files or tables in the target database Populates the tables with data from the source. Migrate existing data option in the AWS console and Full Load in the API. Captures changes on the source during migration Once initial migration completes, changes are applied to the target as units of completed transactions Migrate existing data and replicate ongoing changes option in the AWS console and full-load-and-cdc in the API. Reads the recovery file on the source database Groups together transactions and applies them to the target. Buffering as needed Replicate data changes only option in the AWS console
  117. 117. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. In Do nothing mode, AWS DMS assumes target tables are pre-created. In full load or full load plus CDC, ensure that the target tables are empty before starting the migration. In Drop tables on target mode, AWS DMS drops the target tables and recreates them before starting the migration. This ensures that the target tables are empty when the migration starts. In Truncate mode, AWS DMS truncates all target tables before the migration starts. Task – Target preparation Target preparation Do nothing Drop tables on target Truncate
  118. 118. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. LOB columns are excluded from the migration. Migrate complete LOBs regardless of size. AWS DMS migrates LOBs piecewise in chunks controlled by the Max LOB size parameter. This mode is slower than using Limited LOB mode. Truncate LOBs to the value of the Max LOB size parameter. This mode is faster than using Full LOB mode. Task – Include LOBs Include LOBs Don't include LOBs Full LOB mode Limited LOB mode*
  119. 119. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. More about replication tasks • Tasks are the workhorse of the migration • Tasks run on a replication instance • Multiple tasks can be run in parallel • Task settings JSON gives you control over performance and debugging
  120. 120. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration scenarios
  121. 121. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Possible migration preparation scenarios Using DMS alone • DMS Creates tables and primary keys using Drop tables on target • Manually add secondary indexes and foreign key objects after initial load has completed. Using SCT and DMS together • Use Schema Conversion Tool to create all objects • Disable secondary indexes and foreign key objects • Load using DMS using Do nothing • Re-enable secondary indexes and foreign key objects Using Native Tools and DMS • Use database scripts to create tables, primary keys, views, and sequences • Load tables with DMS using Truncate • Manually add secondary indexes and foreign key objects after initial load has completed.
  122. 122. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Steps to migrate - overview Start the full load While loading data, also capture changes Load complete - apply captured changes Changes reach steady state Shutdown apps and apply remaining changes Change your application endpoint
  123. 123. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replication Instance Source Target Start full load
  124. 124. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source TargetReplication Instance App While loading data also capture changes
  125. 125. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source TargetReplication Instance App Load complete - apply captured changes
  126. 126. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Changes reach steady state Source TargetReplication Instance App
  127. 127. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source TargetReplication Instance App Shutdown apps and apply remaining changes
  128. 128. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Change your application endpoint Source TargetReplication Instance App
  129. 129. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Load is table by table Replication Instance Source Target
  130. 130. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replication Server Source Target App t 1 t 2 t 1 t 2 Changes are transactional - from db logs
  131. 131. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DMS – change data capture (CDC) “No Touch” design • Reads recovery log of source database • Using the engine’s native change data capture API • No agent required on the source • Changes captured as transactions and applied in order • Activated when load starts • Changes are applied after initial load is complete
  132. 132. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DMS – change data capture (CDC) Some requirements • Oracle: Supplemental logging required • MySQL: Full image row level bin logging required • SQL Server: Recovery model bulk logged or full • Postgres: wal_level = logical; max_replication_slots >= 1; max_wal_Senders >=1; wal_sender_timeout = 0
  133. 133. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Replication Instance Source Target Multiple targets Target Target
  134. 134. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration Server Source Target Multiple Sources Source Source
  135. 135. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration Server Source Target Multiple sources and targets Source Source Target
  136. 136. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. You don’t have to take everything Source L Target Replication Instance
  137. 137. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Homogenous or heterogeneous Replication Instance SQL Server MySQL Replication Instance Oracle Oracle Replication Instance Oracle PostgreSQL
  138. 138. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Size in GB Transaction volume Size Transaction Load DMS limits Too big Toomanychanges
  139. 139. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Setup and configuration tips • Enable CloudWatch logs (not enabled by default) • Choose LOB mode carefully • Replication instance security group is default for VPC – change after creation • Extra connection attributes can alter how the migration task operates • Provide transformation rules for changing case
  140. 140. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source configuration tips • Oracle specific settings for CDC • Enable supplemental logging on tables with no primary key • If you add a table mid-migration add supplemental logging • Even if the table has a primary key • To have DMS automatically configure supplemental logging in Oracle add an extra connection attribute • addSupplementalLogging=Y • Check for unsupported datatypes • Can use a read replica or standby as a migration source
  141. 141. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance tips • Use larger DMS instance for maximum throughput • CPU for type conversions • Check network throughput • Split load across multiple tasks and/or DMS instances • Remember transaction boundaries when capturing changes • Reduce contention on your target • Turn off logging • Run in single AZ Further reading on the AWS Database Blog
  142. 142. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Questions

×