Advanced data migration techniques for Amazon RDS

9,958
-1

Published on

Migrating on premise data from Oracle and MySQL Databases to AWS Oracle and MySQL RDS. These techniques will work for AWS EC2 as well. Scripts included in the slides.

Published in: Technology
1 Comment
8 Likes
Statistics
Notes
  • Anyone placing slides and disabling download is an idiot!!! Thanks for nothing
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
9,958
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
8
Embeds 0
No embeds

No notes for slide
  • Why did I say Data SET not Database ?
  • migrate a 150 GB database to RDS MySQL from on-premise MySQL database with greatly reduced downtime using replication. The idea is to restore a fresh backup to RDS MySQL database, turn on replication on the new server after configuring the old server as the Master for the data to be in sync, switch the Slave to become the Master, then point the DNS to the new server all without downtime.
  • Replication between servers in MySQL is based on the binary logging mechanism. The MySQL instance operating as the master (the source of the database changes) writes updates and changes as “events” to the binary log. The information in the binary log is stored in different logging formats according to the database changes being recorded. Slaves are configured to read the binary log from the master and to execute the events in the binary log on the slave's local database.
  • MYSQL.RDS_SET_EXTERNAL_MASTER procedure designates the RDS server as the slave of our master server,It provides the server the correct login credentials, it lets the slave server know where to start replicating from; the master log file and log position come from the numbers we wrote down previously.
  • AWS console will show replicating. New RDS replication is caught up with the source instanceAt this point you may want to take a user snapshot.  
  • Advanced data migration techniques for Amazon RDS

    1. 1. DAT308 – Advanced Data Migration Techniques for Amazon RDS Shakil Langha, Abdul Sathar Sait, Bharanendra Nallamotu Amazon Web Services November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
    2. 2. Next 60 minutes … • • • • • What is new in RDS Types of Data Migration General Considerations Advanced migration techniques for Oracle Near zero downtime migration for MySQL
    3. 3. RDS Recent Releases Oracle Transparent Data Encryption MySQL 5.6 Amazon RDS MySQL Replication to RDS CR1.8XLarge for MySQL 5.6 Oracle Statspack Cross-region Snapshot Copy
    4. 4. One-time Migration
    5. 5. Periodic Migration
    6. 6. Ongoing Replication
    7. 7. General Considerations
    8. 8. RDS Pre-migration Steps • • • • • • Stop applications accessing the DB Take a snapshot Disable backups Use Single AZ instances Optimum instance for load performance Configure security for cross-DB traffic
    9. 9. RDS Post-migration Steps • • • • • Turn on Backups Turn on Multi-AZ Create Read Replicas Tighten down security Notifications via CloudWatch, DBEvents
    10. 10. Data Migration Approaches for Oracle
    11. 11. Let’s move some data
    12. 12. 500 GB
    13. 13. Migration Process
    14. 14. Data Pump Export expdp demoreinv/demo full=y dumpfile=data_pump_exp1:reinvexp1%U.dmp, data_pump_exp2:reinvexp2%U.dmp, data_pump_exp3:reinvexp3%U.dmp filesize=20G parallel=8 logfile=data_pump_exp1:reinvexpdp.log compression=all job_name=reInvExp
    15. 15. Data Pump Export start
    16. 16. Data Pump Export done
    17. 17. Data Pump Files
    18. 18. Compression makes 500 GB to 175 GB 57+62+56 = 175 GB
    19. 19. Upload files to EC2 using UDP Install Tsunami on both the source database server and the EC2 instance Open port 46224 for Tsunami communication $ yum -y install make yum -y install automake yum -y install gcc yum -y install autoconf yum -y install cvs wget http://sourceforge.net/projects/tsunami-udp/files/latest/download?_test=goal tar -xzf tsunami*gz cd tsunami-udp* ./recompile.sh make install
    20. 20. Using UDP tool Tsunami On the source database server start Tsunami server $ cd/mnt/expdisk1 $ tsunamid * On the source database server start Tsunami server $ cd /mnt/data_files $ tsunami $ tsunami> connect source.db.server $ tsunami> get *
    21. 21. Export and Upload in parallel • No need to wait till all 18 files are done to start upload • Start upload as soon as the first set of 3 files are done
    22. 22. Total time to upload175 GB
    23. 23. Transfer files to RDS instance RDS instance has an externally accessible Oracle Directory Object DATA_PUMP_DIR Use a script to move data files to RDS DATA_PUMP_DIR
    24. 24. Perl script to transfer files to RDS instance # RDS instance info my $RDS_PORT=4080; my $RDS_HOST="myrdshost.xxx.us-east-1-devo.rds-dev.amazonaws.com"; my $RDS_LOGIN="orauser/orapwd"; my $RDS_SID="myoradb"; my $dirname = "DATA_PUMP_DIR"; my $fname = $ARGV[0]; my $data = “dummy"; my $chunk = 8192; my $sql_open = "BEGIN perl_global.fh := utl_file.fopen(:dirname, :fname, 'wb', :chunk); END;"; my $sql_write = "BEGIN utl_file.put_raw(perl_global.fh, :data, true); END;"; my $sql_close = "BEGIN utl_file.fclose(perl_global.fh); END;"; my $sql_global = "create or replace package perl_global as fh utl_file.file_type; end;"; my $conn = DBI->connect('dbi:Oracle:host='.$RDS_HOST.';sid='.$RDS_SID.';port='.$RDS_PORT,$RDS_LOGIN, '') || die ( $DBI::errstr . "n") ; my $updated=$conn->do($sql_global); my $stmt = $conn->prepare ($sql_open);
    25. 25. Perl script to transfer files to RDS instance $stmt->bind_param_inout(":dirname", $dirname, 12); $stmt->bind_param_inout(":fname", $fname, 12); $stmt->bind_param_inout(":chunk", $chunk, 4); $stmt->execute() || die ( $DBI::errstr . "n"); open (INF, $fname) || die "nCan't open $fname for reading: $!n"; binmode(INF); $stmt = $conn->prepare ($sql_write); my %attrib = ('ora_type,24); my $val=1; while ($val > 0) { $val = read (INF, $data, $chunk); $stmt->bind_param(":data", $data , %attrib); $stmt->execute() || die ( $DBI::errstr . "n") ; }; die "Problem copying: $!n" if $!; close INF || die "Can't close $fname: $!n"; $stmt = $conn->prepare ($sql_close); $stmt->execute() || die ( $DBI::errstr . "n") ;
    26. 26. Transfer files as they are received • No need to wait till all 18 files are received in the EC2 instance • Start transfer to RDS instance as soon as the first file is received.
    27. 27. Total time to transfer files to RDS
    28. 28. Import data into the RDS instance • • Import from within RDS instance using DBMS_DATAPUMP package Submit a job using PL/SQL script
    29. 29. Import data into the RDS instance declare h1 NUMBER; begin h1 := dbms_datapump.open (operation => 'IMPORT', job_mode => 'FULL', job_name => 'REINVIMP', version => 'COMPATIBLE'); dbms_datapump.set_parallel(handle => h1, degree => 8); dbms_datapump.add_file(handle => h1, filename => 'IMPORT.LOG', directory => 'DATA_PUMP_DIR', filetype => 3); dbms_datapump.set_parameter(handle => h1, name => 'KEEP_MASTER', value => 0); dbms_datapump.add_file(handle => h1, filename => 'reinvexp1%U.dmp', directory => 'DATA_PUMP_DIR', filetype => 1); dbms_datapump.add_file(handle => h1, filename => 'reinvexp2%U.dmp', directory => 'DATA_PUMP_DIR', filetype => 1); dbms_datapump.add_file(handle => h1, filename => 'reinvexp3%U.dmp', directory => 'DATA_PUMP_DIR', filetype => 1); dbms_datapump.set_parameter(handle => h1, name => 'INCLUDE_METADATA', value => 1); dbms_datapump.set_parameter(handle => h1, name => 'DATA_ACCESS_METHOD', value => 'AUTOMATIC'); dbms_datapump.set_parameter(handle => h1, name => 'REUSE_DATAFILES', value => 0); dbms_datapump.set_parameter(handle => h1, name => 'SKIP_UNUSABLE_INDEXES', value => 0); dbms_datapump.start_job(handle => h1, skip_current => 0, abort_step => 0); dbms_datapump.detach(handle => h1); end; /
    30. 30. Total import data into RDS database
    31. 31. Time taken to migrate the database
    32. 32. Optimize the Data Pump Export • Reduce the data set to optimal size, avoid indexes • Use compression and Parallel processing • Use multiple disks with independent IO
    33. 33. Optimize Data Upload • • • • Use Tsunami for UDP based file transfer Use large EC2 instance with SSD or PIOPS volume Use multiple disks with independent IO You could use multiple EC2 instances for parallel upload
    34. 34. Optimize Data File upload to RDS • Use the largest RDS instance possible during the import process • Avoid using RDS instance for any other load during this time • Provision enough storage in the RDS instance for the uploaded files and imported data
    35. 35. Periodic Upload • Oracle Data Pump Network Mode • Oracle Materialized Views • Custom triggers
    36. 36. For small data set one time load • Oracle Import/Export • Oracle Data Pump Network Mode • Oracle SQL*Loader • Oracle Materialized Views
    37. 37. Data Migration Approaches for MySQL
    38. 38. Importing From a MySQL DB Instance Application DB Application mysqldump Staging area Load data scp Tsunami UDP Staging server Replication AWS Region
    39. 39. Importing From a MySQL DB Instance
    40. 40. Importing From a MySQL DB Instance
    41. 41. Check the size of the master database
    42. 42. Create the backup file
    43. 43. Create RDS for MySQL and EC2 Create RDS for MySQL using AWS Management Console or CLI PROMPT>rds-create-db-instance mydbinstance -s 1024 -c db.m3.2xlarge -e MySQL - u <masterawsuser> -p <secretpassword> --backup-retention-period 3 Create EC2 (Staging server) using AWS Management Console or CLI aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type m3.2xlarge --key-name MyKeyPair --security-groups MySecurityGroup Create Replication User on the Master mysql> GRANT SELECT,REPLICATION USER,REPLICATION CLIENT ON *.* TO repluser@„<RDS Endpoint>' IDENTIFIED BY „<password>';
    44. 44. Update /etc/my.cnf on the master server Enable MySQL binlog This enables bin logging which creates a file recording the changes that have occurred on the Master which the Slave uses to replicate the data. [mysqld] server-id = 1 binlog-do-db=mytest relay-log = /var/lib/mysql/mysql-relay-bin relay-log-index = /var/lib/mysql/mysql-relay-bin.index log-error = /var/lib/mysql/mysql.err master-info-file = /var/lib/mysql/mysql-master.info relay-log-info-file = /var/lib/mysql/mysql-relay-log.info log-bin = /var/lib/mysql/mysql-bin
    45. 45. Configure the master database Restart the master database after /etc/my.cnf is updated $ sudo /etc/init.d/mysqld start Record the “File” and the “Position” values. $ mysql -h localhost -u root -p mysql> show master statusG *************************** 1. row *************************** File: mysql-bin.000023 Position: 107 Binlog_Do_DB: mytest Binlog_Ignore_DB: 1 row in set (0.00 sec)
    46. 46. Upload files to EC2 using UDP • Tar and compress MySQL dump file preparation to ship to EC2 staging server. • Update the EC2 security group to allow UDP connection from the server where the dump files being created to your new mysql client server. • On the EC2 staging instance untar the tar.tgz file.
    47. 47. Configure the RDS database Create the database mysql> create database bench; Import the database that you previously exported from the master database. Mysql> load data local infile '/reinvent/tables/customer_address.txt' into table customer_address fields terminated by ','; Mysql> load data local infile '/reinvent/tables/customer.txt' into table customer fields terminated by ','; Configure the slave RDS for MySQL server and start the slave server mysql> call mysql.rds_set_external_master(„<master server>',3306,„<replicationuser>',„<password>','mysql-bin.000013',107,0); mysql> call mysql.rds_start_replication;
    48. 48. RDS for MySQL replication status
    49. 49. Switch RDS MySQL to the Master Switch-over the RDS for MySQL – Stop the service/application that is pointing at the Master Database – Once all changes have been applied to New RDS Database. Stop replication with “call mysql.rds_stop_replication” – Point the service/application at the New RDS Database. – Once Migration is complete. “call mysql. rds_reset_external_master”
    50. 50. Please give us your feedback on this presentation DAT308 As a thank you, we will select prize winners daily for completed surveys!
    51. 51. References • • • • • RDS Home Page RDS FAQs RDS Webinars RDS Best Practices Data Import Guides – MySQL – Oracle – SQL Server

    ×