AWS
Data Migration Service
ARUN SIRIMALLA
Senior BigData/AWS DevOps Engineer
Organizer
• Hyderabad Hadoop/AWS user Group
• Portland Bigdata/AWS User Group
• Sydney Bigdata/AWS User Group
• Melbourne Bigdata/AWS User Group
Cloud Computing
• Cloud computing is the on-demand delivery of
compute power, database storage, applications and
other IT resources
• Its a cloud services platform via the internet with pay-
as-you-go pricing
Physical
Servers
X-category	Sports	Marketing
Why Cloud?
• No upfront investment in data centers and
servers
• Stop guessing capacity
• Stop spending money on running and
maintaining data centers
• Go global in minutes
• Disaster recovery
Overview
of Amazon
Web
Services
AWS Global Infrastructure
Regions and Availability Zones
• Amazon EC2 is hosted in multiple locations world-wide
• Each region is a separate geographic area
• Each region has multiple, isolated locations know as Availability zones
VPC
• Virtual datacenter in the cloud
• You can create your own public-facing subnet for your webservers and place your backend
systems such as databases or application servers in private subnet
• You can create a hardware virtual private network connection b/w your corporate datacenter
and AWS
• Assign custom IP address range in each subnet
• Create internet gateways
• Leverage multiple layers of security
S3
• S3 object based allows you to upload files
• Files can be 1 Byte to 5 TB
• Buckets have unique namespace for each region
• Amazon guarantees 99.99% availability
• Guarantees durability of 99.999999999%
RDS
• Allow you to create and scale Relational Databases
• You cannot SSH or RDP to RDS instance
• AWS does not provide you public or private IP address, instead gives you endpoint to connect
• Available and Durable, Secure, Inexpensive
What are AWS DMS and AWS SCT?
AWS Database Migration Service (AWS DMS) easily and securely migrates and/or
replicates your databases and data warehouses to AWS.
AWS Schema Conversion Tool (AWS SCT) converts your commercial database and data
warehouse schemas to open-source engines, Amazon Aurora and Amazon Redshift.
Converts and loads data warehouse data into Amazon Redshift.
When to use AWS DMS?
• Migrate business-critical applications
• Migrate from Classic to VPC
• Migrate data warehouse to Amazon Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Migrate from NoSQL to SQL, SQL to NoSQL or NoSQL to NoSQL
Homogeneous Database Migrations
The source and target database engines are the same
Oracle to Amazon RDS for Oracle, MySQL to Amazon Aurora, MySQL to Amazon RDS for
MySQL, or Microsoft SQL Server to Amazon RDS for SQL Server
Heterogeneous Database Migrations
The source and target databases engines are different
Oracle to Amazon Aurora, Oracle to PostgreSQL, or Microsoft SQL Server to MySQL migrations
Continuous Data Replication
Continuous data replication use cases can be Disaster Recovery instance
synchronization, geographic database distribution and Dev/Test
environment synchronization
You can use DMS for both homogeneous and heterogeneous data
replications for all supported database engines
Data Migration with AWS Database Migration Service
1. Create a Replication Instance
1. Initiates the connection between the source and target databases
• Processing power to migrate data from source database to target
2. Specify Source and Target Endpoints
• The source and target data stores can be on an Amazon EC2 instance, an
Amazon RDS DB instance, or an on-premises database
3. Create a Task
• Create new tables, tables to migrate
Replication Instance
• Initiates the connection between source and target, transfers the data,
caches any changes that occur on the source database during the
initial data load
• Parameters required for creating new replication instance
Ø Instance class
Ø VPC
Ø Replication Engine Version
Ø Allocated Storage
Ø Replication Subnet group
Ø VPC security Group
Ø Availability Zone
Endpoints
• AWS DMS accesses your data sources and targets
using endpoints
• Source endpoint allows DMS to read data from a
Database or from a non-database such as Amazon S3
• Target endpoint allows DMS to write data to a
database, or to a non-database target
Endpoint Options Source Engine
Ø Server name
Ø Port
Ø username
Ø password
Task
• Allows you to migrate existing data, replicate ongoing changes, or
replicate data changes only
• A task can contain one or more table mappings which define
what data is moved from the source to the target
• If a table does not exist on the target, it can be created
automatically
Options to create a Task
Ø Replication Instance
Ø Source endpoint
Ø Target endpoint
Ø Migration type
Ø Target table preparation Mode
DEMO
Thank you!

AWS Data migration services

  • 1.
  • 2.
    Senior BigData/AWS DevOpsEngineer Organizer • Hyderabad Hadoop/AWS user Group • Portland Bigdata/AWS User Group • Sydney Bigdata/AWS User Group • Melbourne Bigdata/AWS User Group
  • 3.
    Cloud Computing • Cloudcomputing is the on-demand delivery of compute power, database storage, applications and other IT resources • Its a cloud services platform via the internet with pay- as-you-go pricing
  • 4.
  • 5.
    Why Cloud? • Noupfront investment in data centers and servers • Stop guessing capacity • Stop spending money on running and maintaining data centers • Go global in minutes • Disaster recovery
  • 6.
  • 7.
  • 8.
    Regions and AvailabilityZones • Amazon EC2 is hosted in multiple locations world-wide • Each region is a separate geographic area • Each region has multiple, isolated locations know as Availability zones VPC • Virtual datacenter in the cloud • You can create your own public-facing subnet for your webservers and place your backend systems such as databases or application servers in private subnet • You can create a hardware virtual private network connection b/w your corporate datacenter and AWS • Assign custom IP address range in each subnet • Create internet gateways • Leverage multiple layers of security
  • 9.
    S3 • S3 objectbased allows you to upload files • Files can be 1 Byte to 5 TB • Buckets have unique namespace for each region • Amazon guarantees 99.99% availability • Guarantees durability of 99.999999999% RDS • Allow you to create and scale Relational Databases • You cannot SSH or RDP to RDS instance • AWS does not provide you public or private IP address, instead gives you endpoint to connect • Available and Durable, Secure, Inexpensive
  • 10.
    What are AWSDMS and AWS SCT? AWS Database Migration Service (AWS DMS) easily and securely migrates and/or replicates your databases and data warehouses to AWS. AWS Schema Conversion Tool (AWS SCT) converts your commercial database and data warehouse schemas to open-source engines, Amazon Aurora and Amazon Redshift. Converts and loads data warehouse data into Amazon Redshift.
  • 11.
    When to useAWS DMS? • Migrate business-critical applications • Migrate from Classic to VPC • Migrate data warehouse to Amazon Redshift • Upgrade to a minor version • Consolidate shards into Aurora • Migrate from NoSQL to SQL, SQL to NoSQL or NoSQL to NoSQL
  • 12.
    Homogeneous Database Migrations Thesource and target database engines are the same Oracle to Amazon RDS for Oracle, MySQL to Amazon Aurora, MySQL to Amazon RDS for MySQL, or Microsoft SQL Server to Amazon RDS for SQL Server
  • 13.
    Heterogeneous Database Migrations Thesource and target databases engines are different Oracle to Amazon Aurora, Oracle to PostgreSQL, or Microsoft SQL Server to MySQL migrations
  • 14.
    Continuous Data Replication Continuousdata replication use cases can be Disaster Recovery instance synchronization, geographic database distribution and Dev/Test environment synchronization You can use DMS for both homogeneous and heterogeneous data replications for all supported database engines
  • 16.
    Data Migration withAWS Database Migration Service 1. Create a Replication Instance 1. Initiates the connection between the source and target databases • Processing power to migrate data from source database to target 2. Specify Source and Target Endpoints • The source and target data stores can be on an Amazon EC2 instance, an Amazon RDS DB instance, or an on-premises database 3. Create a Task • Create new tables, tables to migrate
  • 17.
    Replication Instance • Initiatesthe connection between source and target, transfers the data, caches any changes that occur on the source database during the initial data load • Parameters required for creating new replication instance Ø Instance class Ø VPC Ø Replication Engine Version Ø Allocated Storage Ø Replication Subnet group Ø VPC security Group Ø Availability Zone
  • 18.
    Endpoints • AWS DMSaccesses your data sources and targets using endpoints • Source endpoint allows DMS to read data from a Database or from a non-database such as Amazon S3 • Target endpoint allows DMS to write data to a database, or to a non-database target Endpoint Options Source Engine Ø Server name Ø Port Ø username Ø password
  • 19.
    Task • Allows youto migrate existing data, replicate ongoing changes, or replicate data changes only • A task can contain one or more table mappings which define what data is moved from the source to the target • If a table does not exist on the target, it can be created automatically Options to create a Task Ø Replication Instance Ø Source endpoint Ø Target endpoint Ø Migration type Ø Target table preparation Mode
  • 20.
  • 21.