[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in AWS migrations. by The Pythian Group Inc. - Gleb Otochkin

DATE: 09/06/2017
How to use DMS in AWS migrations.
Taking off to the
clouds.

Gleb Otochkin
Started to work with data in 1992
At Pythian since 2008
Area of expertise:
● Data Integration
● Oracle RAC
● Oracle engineered systems
● Virtualization
● Performance tuning
● Big Data
otochkin@pythian.com
@sky_vst
Principal Consultant
© The Pythian Group Inc., 2017 2

ABOUT PYTHIAN
Pythian’s 400+ IT professionals
help companies adopt and
manage disruptive technologies
to better compete
© 2016 Pythian. Confidential 3

Systems currently
managed by Pythian
EXPERIENCED
Pythian experts
in 35 countries
GLOBAL
Millennia of experience
gathered and shared
over 19 years
EXPERTS
11,800 2400

AGENDA
Migration to a cloud.
AWS schema conversion tool.
AWS Database Migration Service.
Alternative tools.

Migration to a cloud.
Goals, expectations, requirements and obstacles.

Goals.
What we have in premises.
RDBMS
Transactions
BI &
Analysis
App
App
App
Storage
• Cons:
• DC maintenance or rental fee.
• Hardware cost.
• Electricity and cooling.
• Hardware support.
• Long implementation cycle.
• Proc:
• Full control?
• Flexibility in configuration.

Goals.
What we expect or dream of.

Goals.
What we want to have in the cloud.
DB as a
service
Transactions
BI &
Analytics
Cloud
Solutions
App VM
App VM
App VM
Storage
as a
service
• Proc:
• Stable and predictable monthly fee.
• No explicit hardware cost.
• No electricity and cooling bills.
• No hardware support cost.
• Implementation by one click.
• Tightened security.
• Cons:
• Limited control over the environment.
• Pre-fixed configuration.

Migration.
Things to consider.
• Cloud.
• Application layer.
• Location(s).
• Downtime and security.
• Available versions and platforms for
database.
• Network throughput and latency.
• Available PAAS and IAAS options.
• Application:
• Topology or layout.
• OS and platform certification.
• Location(s) for app layer.
• Database:
• Platform and version.
• Memory, CPU and IO requirements.
• Size.

DOWNTIME
Migration.
Different factors affecting downtime.
SIZE Network StorageStorage Conversion

Migration.
How we can reduce downtime
• Network:
• Dedicated network link.
• AWS Direct Connect.
• Conversion.
• Use the same DB engine.
• Same platform and version.
• Conversion tools.
• Replication tools.
▪ Physical replication (Oracle DataGuard).
▪ Logical replication (DMS).
• Size:
• Data purge.
• Data segregation to operational and
archived.
• Compression.
• Storage:
• Optimization on source side.
• Optimization on destination.

Why and when we need it.

Internal network AWS VPC
How it works:
Source DB
Schemas
Tables
Procedures
Other obj
Schemas
Tables
Procedures
Other obj
Dest DB
AWS Schema Conversion Tool (STC)
AWS DMS
JDBC JDBC

Installation and components:
JDBC
AWS Schema
Conversion Tool
(STC)
• Installed to a local machine.
• Supports different OS.
• Written on java.
• JDBC connection to source
and target.
• Stores gathered data in
XML.
• Supports different sources
and targets.

Working with STC:
JDBC
AWS Schema
Conversion Tool
(STC)
source.xml
target.xml
Mapping rules
Conversion Create Report
Apply changes
Add DMS task

Some notes.
• DMS tasks:
• Require IAM settings.
• Require connection between source and
target.
• Conversion.
• Can be stored as SQL.
• Local repository:
• Depends from number of objects.
• Stored in XML.
• Compression.
• Connection:
• JDBC drivers for each target.
• Memory:
• JVMUserOptions in cfg file.
• Global settings in menu.

Interface.

Assessment report.

AWS Database Migration
Service.
What it can do for us. What we need to pay attention to.

AWS VPC
Internal network
AWS VPC
Basic topology
Source DB
Schemas
Tables
Schemas
Tables Dest DB
AWS Schema Conversion Tool (STC)
Replication instance
Source endpoint Target endpoint
AWS DMS tasks

Basic topology. Concepts.
• Logical replication.
• Heterogeneous by the nature.
• Data transformation on the way.
• Stored in a platform independent format on the DMS instance.
• Requires network connection from the DMS to both sides.
• DMS instance preferable in the same VPC as the target.
• Performance depends from data format, network, memory and
cpu.

Components. Source database.
• Transaction logging.
• Archivelog mode and supplemental logging for Oracle.
• DB user with correct privileges.
• DB and ASM user (optional) for Oracle.
• Sufficient retention policy for transaction logs.
• Archivelog deletion policy for Oracle.
• RDS backups and archivelog retention policy.
• Network connection to the database from DMS instance.
• TNS connection for Oracle.

Components. Destination database.
• DB user with correct privileges.
• DB user for Oracle.
• Network connection to the database from DMS instance.
• TNS connection for Oracle.

Components. Replication instance.
• Instance class. From t2.micro to c4.4xlarge.
• Allocated storage.
• VPC and subnet.
• Availability zone settings.
• KMS master key for encryption.
• Maintenance Window. (Only through AWS CLI)
Replication instance

Components. Replication instance.
Glebs-MacBook:~$ aws dms describe-replication-instances --query
'ReplicationInstances[*].{ARN:ReplicationInstanceArn,MaintenanceWindow:PreferredMaintenanceWindow}'
[
{
"MaintenanceWindow": "fri:10:46-fri:11:16",
"ARN": "arn:aws:dms:us-east-1:2221234262:rep:ZXUE23B45HRVGF6ZNHN723GP6E"
}
]
Glebs-MacBook:~$ aws dms modify-replication-instance --replication-instance-arn
arn:aws:dms:us-east-1:2221234262:rep:ZXUE23B45HRVGF6ZNHN723GP6E --preferred-maintenance-window
"sun:10:46-sun:11:16"
…….

Components. Network.
• DMS replication instance is in an AWS VPC.
• At least two subnets and one Availability Zone.
• Security group to allow for all port egress for the VPC.
• Network for target and source.
• Allow access to db port ingress from DMS replication instance.
• Different configurations.
• The source, target and DMS instance are in the same VPC.
• The source in one VPC and target and DMS instance are in another.
• AWS direct connect or VPN for the source.
• Internet connection to the source.

Components. Network configurations.
VPC
VPCVPC
DC VPC
One VPC for all
Two VPC with VPC peering
AWS direct connect or VPN
DC VPC
Connection over Internet

Components. Endpoints.
• Source and Target. Different available engines.
• Allocated storage.
• VPC and subnet.
• Availability zone settings.
• KMS master key for encryption.
• Maintenance Window. (Only through AWS CLI)
Endpoints

Components. Replication tasks. Main settings.
• Task settings:
• Task name. Unique for DMS.
• The instance and endpoints has to
be created already.
• Choose migration or replication.
• Start task on create.

Components. Replication tasks. Optional settings.
• Optional settings:
• Table preparation. Truncate is not
always truncate. DDL changes.
• Lobs are supported with some
limitations.
• Lob chunk size may impact
performance and space
consumption on the replication
instance..
• You need CLI to enable advanced
logging.

Components. Replication tasks. Advanced settings.
• Advanced settings:
• Control of internal DMS tables on
the target.
• Number of concurrent load sessions.
• Waiting timeout.
• Bulk size for commit.

Components. Replication tasks. Mapping. Selection rules.
• Selection rules:
• Schemas can be refreshed by
refreshing endpoint data.
• Table names support wildcard.
• Exclude is applied after include
condition.
• Filter by conditions. May impact
performance.
• Can be written directly in json.

Components. Replication tasks. Mapping. Selection rules.
• Selection rules:
• Schemas can be refreshed by
refreshing endpoint data.
• Table names support wildcard.
• Exclude is applied after include
condition.
• Filter by conditions. May impact
performance.

Components. Replication tasks. Mapping. Transformation rules.
• Transformation rules:
• Rename schema if target endpoint
user is not your desirable schema.
• Switching to uppercase for Oracle
targets.
• Rename target tables and columns if
source names are longer 30 char.
• Order of rules matters. (Rename
after uppercase)

Using CLI.
• AWS CLI:
• Update to the latest version.
• Has more options and flexibility
than GUI.
• Some features can be adjusted
only through CLI.
• Use json format to create tasks
with unique features(advanced
logging)
Glebs-MacBook:~$ aws dms help
Glebs-MacBook:~$ aws dms modify-replication-task help
Glebs-MacBook:~$ aws dms create-replication-task --cli-input-json
dms_test_01.json
Glebs-MacBook:~$ aws dms modify-replication-task
--generate-cli-skeleton
{
"ReplicationTaskArn": "",
"ReplicationTaskIdentifier": "",
"MigrationType": "",
"TableMappings": "",
"ReplicationTaskSettings": "",
"CdcStartTime": "1970-01-01T00:00:00"
}
Glebs-MacBook:~ otochkin$
Glebs-MacBook:~$ aws dms modify-replication-task --cli-input-json
dms_test_01.json

Alternative tools.
What are the other tools we can use for a migration?

Alternative tools for migration.
Oracle Data Guard.
• Proc:
• Block to block replication. Exact copy of the database.
• Performance.
• Network throughput can be reduced using Advanced compression.
• Short downtime.
• Cons:
• Only from Oracle to Oracle.
• Cross-platform migration is limited.
• Cannot be used for RDS targets.

Export - Import.
• Proc:
• Can migrate to different platform using universal formats like CSV.
• Can be uploaded to any database engine.
• Cons:
• Downtime.
• Performance.
• Require manual effort.

Oracle GoldenGate.
• Proc:
• Zero downtime.
• Support different database engines and platforms.
• Supports almost all data types for Oracle source and most for non Oracle sources.
• Cons:
• Doesn’t support all targets on AWS.
• Licensing cost.
• Requires addition EC2 instance.

3-d party replication solutions.
• Proc:
• Zero downtime.
• Performance comparable with DMS and GoldenGate.
• Cons:
• Doesn’t support all targets on AWS.
• Licensing cost.
• Usually requires addition EC2 instance.

THANK YOU
email:otochkin@pythian.com
Twitter: @sky_vst
Blog:https://www.pythian.com/blog/

[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in AWS migrations. by The Pythian Group Inc. - Gleb Otochkin

More Related Content

What's hot

Similar to [db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in AWS migrations. by The Pythian Group Inc. - Gleb Otochkin

More from Insight Technology, Inc.

Recently uploaded

[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in AWS migrations. by The Pythian Group Inc. - Gleb Otochkin