Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Advanced data migration techniques for Amazon RDS

13,849 views

Published on

Migrating on premise data from Oracle and MySQL Databases to AWS Oracle and MySQL RDS. These techniques will work for AWS EC2 as well. Scripts included in the slides.

Published in: Technology
  • Can you please share any perl script to downlod dump from RDS DATA_PUMP_DIR to any directory in ec2. I am able to upload dump from EC2 to RDS data_pump_dir. Thanks in advance!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Anyone placing slides and disabling download is an idiot!!! Thanks for nothing
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Advanced data migration techniques for Amazon RDS

  1. 1. DAT308 – Advanced Data Migration Techniques for Amazon RDS Shakil Langha, Abdul Sathar Sait, Bharanendra Nallamotu Amazon Web Services November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. Next 60 minutes … • • • • • What is new in RDS Types of Data Migration General Considerations Advanced migration techniques for Oracle Near zero downtime migration for MySQL
  3. 3. RDS Recent Releases Oracle Transparent Data Encryption MySQL 5.6 Amazon RDS MySQL Replication to RDS CR1.8XLarge for MySQL 5.6 Oracle Statspack Cross-region Snapshot Copy
  4. 4. One-time Migration
  5. 5. Periodic Migration
  6. 6. Ongoing Replication
  7. 7. General Considerations
  8. 8. RDS Pre-migration Steps • • • • • • Stop applications accessing the DB Take a snapshot Disable backups Use Single AZ instances Optimum instance for load performance Configure security for cross-DB traffic
  9. 9. RDS Post-migration Steps • • • • • Turn on Backups Turn on Multi-AZ Create Read Replicas Tighten down security Notifications via CloudWatch, DBEvents
  10. 10. Data Migration Approaches for Oracle
  11. 11. Let’s move some data
  12. 12. 500 GB
  13. 13. Migration Process
  14. 14. Data Pump Export expdp demoreinv/demo full=y dumpfile=data_pump_exp1:reinvexp1%U.dmp, data_pump_exp2:reinvexp2%U.dmp, data_pump_exp3:reinvexp3%U.dmp filesize=20G parallel=8 logfile=data_pump_exp1:reinvexpdp.log compression=all job_name=reInvExp
  15. 15. Data Pump Export start
  16. 16. Data Pump Export done
  17. 17. Data Pump Files
  18. 18. Compression makes 500 GB to 175 GB 57+62+56 = 175 GB
  19. 19. Upload files to EC2 using UDP Install Tsunami on both the source database server and the EC2 instance Open port 46224 for Tsunami communication $ yum -y install make yum -y install automake yum -y install gcc yum -y install autoconf yum -y install cvs wget http://sourceforge.net/projects/tsunami-udp/files/latest/download?_test=goal tar -xzf tsunami*gz cd tsunami-udp* ./recompile.sh make install
  20. 20. Using UDP tool Tsunami On the source database server start Tsunami server $ cd/mnt/expdisk1 $ tsunamid * On the source database server start Tsunami server $ cd /mnt/data_files $ tsunami $ tsunami> connect source.db.server $ tsunami> get *
  21. 21. Export and Upload in parallel • No need to wait till all 18 files are done to start upload • Start upload as soon as the first set of 3 files are done
  22. 22. Total time to upload175 GB
  23. 23. Transfer files to RDS instance RDS instance has an externally accessible Oracle Directory Object DATA_PUMP_DIR Use a script to move data files to RDS DATA_PUMP_DIR
  24. 24. Perl script to transfer files to RDS instance # RDS instance info my $RDS_PORT=4080; my $RDS_HOST="myrdshost.xxx.us-east-1-devo.rds-dev.amazonaws.com"; my $RDS_LOGIN="orauser/orapwd"; my $RDS_SID="myoradb"; my $dirname = "DATA_PUMP_DIR"; my $fname = $ARGV[0]; my $data = “dummy"; my $chunk = 8192; my $sql_open = "BEGIN perl_global.fh := utl_file.fopen(:dirname, :fname, 'wb', :chunk); END;"; my $sql_write = "BEGIN utl_file.put_raw(perl_global.fh, :data, true); END;"; my $sql_close = "BEGIN utl_file.fclose(perl_global.fh); END;"; my $sql_global = "create or replace package perl_global as fh utl_file.file_type; end;"; my $conn = DBI->connect('dbi:Oracle:host='.$RDS_HOST.';sid='.$RDS_SID.';port='.$RDS_PORT,$RDS_LOGIN, '') || die ( $DBI::errstr . "n") ; my $updated=$conn->do($sql_global); my $stmt = $conn->prepare ($sql_open);
  25. 25. Perl script to transfer files to RDS instance $stmt->bind_param_inout(":dirname", $dirname, 12); $stmt->bind_param_inout(":fname", $fname, 12); $stmt->bind_param_inout(":chunk", $chunk, 4); $stmt->execute() || die ( $DBI::errstr . "n"); open (INF, $fname) || die "nCan't open $fname for reading: $!n"; binmode(INF); $stmt = $conn->prepare ($sql_write); my %attrib = ('ora_type,24); my $val=1; while ($val > 0) { $val = read (INF, $data, $chunk); $stmt->bind_param(":data", $data , %attrib); $stmt->execute() || die ( $DBI::errstr . "n") ; }; die "Problem copying: $!n" if $!; close INF || die "Can't close $fname: $!n"; $stmt = $conn->prepare ($sql_close); $stmt->execute() || die ( $DBI::errstr . "n") ;
  26. 26. Transfer files as they are received • No need to wait till all 18 files are received in the EC2 instance • Start transfer to RDS instance as soon as the first file is received.
  27. 27. Total time to transfer files to RDS
  28. 28. Import data into the RDS instance • • Import from within RDS instance using DBMS_DATAPUMP package Submit a job using PL/SQL script
  29. 29. Import data into the RDS instance declare h1 NUMBER; begin h1 := dbms_datapump.open (operation => 'IMPORT', job_mode => 'FULL', job_name => 'REINVIMP', version => 'COMPATIBLE'); dbms_datapump.set_parallel(handle => h1, degree => 8); dbms_datapump.add_file(handle => h1, filename => 'IMPORT.LOG', directory => 'DATA_PUMP_DIR', filetype => 3); dbms_datapump.set_parameter(handle => h1, name => 'KEEP_MASTER', value => 0); dbms_datapump.add_file(handle => h1, filename => 'reinvexp1%U.dmp', directory => 'DATA_PUMP_DIR', filetype => 1); dbms_datapump.add_file(handle => h1, filename => 'reinvexp2%U.dmp', directory => 'DATA_PUMP_DIR', filetype => 1); dbms_datapump.add_file(handle => h1, filename => 'reinvexp3%U.dmp', directory => 'DATA_PUMP_DIR', filetype => 1); dbms_datapump.set_parameter(handle => h1, name => 'INCLUDE_METADATA', value => 1); dbms_datapump.set_parameter(handle => h1, name => 'DATA_ACCESS_METHOD', value => 'AUTOMATIC'); dbms_datapump.set_parameter(handle => h1, name => 'REUSE_DATAFILES', value => 0); dbms_datapump.set_parameter(handle => h1, name => 'SKIP_UNUSABLE_INDEXES', value => 0); dbms_datapump.start_job(handle => h1, skip_current => 0, abort_step => 0); dbms_datapump.detach(handle => h1); end; /
  30. 30. Total import data into RDS database
  31. 31. Time taken to migrate the database
  32. 32. Optimize the Data Pump Export • Reduce the data set to optimal size, avoid indexes • Use compression and Parallel processing • Use multiple disks with independent IO
  33. 33. Optimize Data Upload • • • • Use Tsunami for UDP based file transfer Use large EC2 instance with SSD or PIOPS volume Use multiple disks with independent IO You could use multiple EC2 instances for parallel upload
  34. 34. Optimize Data File upload to RDS • Use the largest RDS instance possible during the import process • Avoid using RDS instance for any other load during this time • Provision enough storage in the RDS instance for the uploaded files and imported data
  35. 35. Periodic Upload • Oracle Data Pump Network Mode • Oracle Materialized Views • Custom triggers
  36. 36. For small data set one time load • Oracle Import/Export • Oracle Data Pump Network Mode • Oracle SQL*Loader • Oracle Materialized Views
  37. 37. Data Migration Approaches for MySQL
  38. 38. Importing From a MySQL DB Instance Application DB Application mysqldump Staging area Load data scp Tsunami UDP Staging server Replication AWS Region
  39. 39. Importing From a MySQL DB Instance
  40. 40. Importing From a MySQL DB Instance
  41. 41. Check the size of the master database
  42. 42. Create the backup file
  43. 43. Create RDS for MySQL and EC2 Create RDS for MySQL using AWS Management Console or CLI PROMPT>rds-create-db-instance mydbinstance -s 1024 -c db.m3.2xlarge -e MySQL - u <masterawsuser> -p <secretpassword> --backup-retention-period 3 Create EC2 (Staging server) using AWS Management Console or CLI aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type m3.2xlarge --key-name MyKeyPair --security-groups MySecurityGroup Create Replication User on the Master mysql> GRANT SELECT,REPLICATION USER,REPLICATION CLIENT ON *.* TO repluser@„<RDS Endpoint>' IDENTIFIED BY „<password>';
  44. 44. Update /etc/my.cnf on the master server Enable MySQL binlog This enables bin logging which creates a file recording the changes that have occurred on the Master which the Slave uses to replicate the data. [mysqld] server-id = 1 binlog-do-db=mytest relay-log = /var/lib/mysql/mysql-relay-bin relay-log-index = /var/lib/mysql/mysql-relay-bin.index log-error = /var/lib/mysql/mysql.err master-info-file = /var/lib/mysql/mysql-master.info relay-log-info-file = /var/lib/mysql/mysql-relay-log.info log-bin = /var/lib/mysql/mysql-bin
  45. 45. Configure the master database Restart the master database after /etc/my.cnf is updated $ sudo /etc/init.d/mysqld start Record the “File” and the “Position” values. $ mysql -h localhost -u root -p mysql> show master statusG *************************** 1. row *************************** File: mysql-bin.000023 Position: 107 Binlog_Do_DB: mytest Binlog_Ignore_DB: 1 row in set (0.00 sec)
  46. 46. Upload files to EC2 using UDP • Tar and compress MySQL dump file preparation to ship to EC2 staging server. • Update the EC2 security group to allow UDP connection from the server where the dump files being created to your new mysql client server. • On the EC2 staging instance untar the tar.tgz file.
  47. 47. Configure the RDS database Create the database mysql> create database bench; Import the database that you previously exported from the master database. Mysql> load data local infile '/reinvent/tables/customer_address.txt' into table customer_address fields terminated by ','; Mysql> load data local infile '/reinvent/tables/customer.txt' into table customer fields terminated by ','; Configure the slave RDS for MySQL server and start the slave server mysql> call mysql.rds_set_external_master(„<master server>',3306,„<replicationuser>',„<password>','mysql-bin.000013',107,0); mysql> call mysql.rds_start_replication;
  48. 48. RDS for MySQL replication status
  49. 49. Switch RDS MySQL to the Master Switch-over the RDS for MySQL – Stop the service/application that is pointing at the Master Database – Once all changes have been applied to New RDS Database. Stop replication with “call mysql.rds_stop_replication” – Point the service/application at the New RDS Database. – Once Migration is complete. “call mysql. rds_reset_external_master”
  50. 50. Please give us your feedback on this presentation DAT308 As a thank you, we will select prize winners daily for completed surveys!
  51. 51. References • • • • • RDS Home Page RDS FAQs RDS Webinars RDS Best Practices Data Import Guides – MySQL – Oracle – SQL Server

×