Your SlideShare is downloading. ×
DAT201 Migrating Databases to AWS - AWS re: Invent 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

DAT201 Migrating Databases to AWS - AWS re: Invent 2012

1,004
views

Published on

In this session, learn how to move your existing database applications to the cloud. We cover the best practices for planning your migrations, moving your data over, sizing your AWS deployment …

In this session, learn how to move your existing database applications to the cloud. We cover the best practices for planning your migrations, moving your data over, sizing your AWS deployment appropriately, and minimizing downtime. You also hear from some of our customers who have successfully migrated their applications about the techniques they used and the reasons they moved onto the cloud.


0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,004
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Internet VPC IAM
  • 2. Sync ReplicationAvailability Zone 1 Availability Zone 2
  • 3. On-Premise AWS
  • 4. T2 T1 T2 T1 Apply Delta Changes On-Premise AWS
  • 5. Sync Replication QA Sync ReplicationProduction Development
  • 6. Data Import Guide for MySQLData Import Guide for Oracle Data Import Guide for SQL Server
  • 7. Big Future: Lessons Learned Delivering OurFirst Million+ Page View/Day Hybrid CloudSolutionRodney Grilli, Solution Architect, The College BoardNovember 28, 2012
  • 8. Audience Takeaways• How to incorporate Amazons growing number of value-added services as you deploy new products to AWS• Criteria for choosing between RDS and self-managed MySQL on EC2• How to setup and size RDS• How to migrate existing on-premise databases to RDS• People, Process, and Technology Lessons that you need to know before starting your journey
  • 9. Who We Are….
  • 10. Our GoalEnsure that every student has the opportunity to prepare for, enroll in, and graduate fromcollege. Our work falls broadly into three categories:College Readiness College Connection and Success – Advanced Placement (AP®) • SAT, SAT Subject Tests™ and SAT Readiness Tools – SpringBoard® • Student Search Service – ReadiStep™ • bigfuture (our college planning site) – CollegeEd® • College Search – CLEP® • CSS/Financial Aid PROFILE® – PSAT/NMSQT® • Scholarship Search – EXCELerator™ Advocacy • Advocacy & Policy Center
  • 11. The Big Future Initiative….
  • 12. PurposeA comprehensive and innovative online resource that helps all studentsaspire, find, connect, and enroll in the right college that sets them on thepath for success in life.VisionBe the most used college guidance website, recommended as the topchoice by all students and counselors, and known as the most helpful andusable source of trusted information and tools.
  • 13. The IT Challenge….
  • 14. • Develop a new product that satisfies all new business goals – Replacing core features in over three legacy products – Targeting AWS for the deployment of a number of components• Migrate all user data from legacy products to new product – This included College Lists for ~10 million students• Maintain all user flows from legacy and dependent products to target locations in the new product – Legacy products process around 1 million pageviews per day – Required extensive redirect rules• Execute the migration in the peak business season
  • 15. The Product….
  • 16. 35Number of new or extended assets? 24%Percentage of College Board pageviews? 23%Percentage of College Board unique daily visitors? 450Number of Web Service calls/second executed by bigfuture?
  • 17. The Architecture….
  • 18. Architecture Approach• Use established cloud and SOA principles to create new system patterns that are optimized for AWS• Design the product to support degraded modes of operation• Conduct sourcing analysis on all architectural components to maximize optimization at all levels of the solution
  • 19. 17 Technical Assets 9 Processes Developed Developed to Pilot State or Extended to Pilot State• Web Service • AWS Logging • Capacity Management & Management System • AWS Testing Capability Planning• AWS Composite • Security Posture & Testing • AWS Core Services Application Hosting • Change Management • AWS Storage • Release Management Stack • AWS Content Delivery • Configuration Management• AWS Business Service • AWS Disaster Recovery • Deployment Management Hosting Stack • Search Index (SOLR) • Automated Test Process• AWS Load Balancing • Operations & Support Deployment Pattern• AWS Service Runbooks • SEO Warming for New Management • AWS Training Domains• AWS Database Service • Site Crawler Handling• AWS Communications for AJAX Heavy Sites• AWS Monitoring
  • 20. AWS Components• Amazon Route53 • Elastic Load Balancers• Amazon Machine • Security Groups Images (AMIs) • Amazon S3• Auto Scale Groups • CloudWatch• AWS Identity & Access Management (IAM)• Amazon RDS (MySQL)
  • 21. RDS Use Cases….
  • 22. Migrating the Data….
  • 23. College Planning Database The biggest table has about 45M rows and the rest are below 10M
  • 24. College List Migration StepsStep Migration Activity Estimate Timing Notes Executed during maintenance 1 Run Extract Script < one hour window 2 Transfer extract files to jumpbox in AWS ~30-60 minutes Use SCP to securely transfer files Truncate tables in existing production RDS 3 ~ 5 minutes Remove test data DB 4 Run SQL Load for Account ~1 hour Run 5 file imports in parallel 5 Run SQL Load for College Lists ~4 hours Run 5 file imports in serial 6 Update the date to current timestamp ~ 35 minutes Take Snapshot of RDS DB and store as Performed on 3/16 cold back up run 7 ~ 3 minutes contingency and 3/19 final run
  • 25. Why Choose RDS….
  • 26. Advantages• High Availability through the Multi-AZ DB feature – The biggest advantage is the stability of the MySQL replication within RDS – This has been the biggest killer in the past when maintaining MySQL replication• MySQL is simple to set up in RDS – Other commercial RDBMSs require many custom configurations that must be tweaked (e.g., memory configuration & I/O configuration)• Less work to maintain
  • 27. Constraints• The biggest perceived constraint is not having access to log files – Limited access to performance info that DBAs traditionally derive from log files• Inability to gain the “Super” privilege• Limited set of configuration settings• Time zone is set to UTC and cant be changed – This presents a challenge for applications set to use a different time zone that use MySQL date/time functions
  • 28. RDS Benefits for Big Future• Removed the need to develop a self managed MySQL relational database capability – Allowed us to focus more effort on the 16 other assets that had to be developed• Provides highly available, Multi-AZ, MySQL deployment model – Removed the need to manage replication within MySQL• Streamlined the deployment of the web service gateway and new business service databases
  • 29. How working with RDS is Different….
  • 30. Activity MySQL RDSCreate, Modify, Reboot, OS Command RDS Console, RDS Command Line Interface, APIsDelete DB Instances LineMonitoring Parse Log Files CloudWatch Console, Command Line Interface, APIsCreate, Modify, Delete Mysql client or Mysql client or mysql adminDatabases mysql adminTuning, DB Parameter Edit my.cnf RDS Console, RDS Command Line Interface, APIsChanges (not all parameters can be changed)Identify slow queries Parse Log Files Enable and query the mysql.slow_log tableDetect locking or SHOW STATUS Enable GoSH and query the global_status_history tablememory issues statement RDS Users Guide - Appendix: Common DBA Tasks for MySQL http://docs.amazonwebservices.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.CommonDBATasks.html
  • 31. RDS Roadmap….
  • 32. What we want to do next• Fully automate RDS provisioning via a Continuous Delivery Pipeline currently under development• Strengthen DB security by implementing SSL between components• Extend database pattern to support multi-region deployments for both read-only and read/write databases• Continue to optimize the cost associated with our RDS databasesHow RDS can help• Provide multi-region replication and management features• Optimize for SSL connections and X509-based authentication/authorization
  • 33. Conclusion• Leveraging value-added services such as RDS can significantly reduce your products’ time to market and ongoing operating costs• RDS is managed differently than a traditional MySQL database, so developers/DBAs need to be educated on these differences – Study the many RDS Documents and keep a copy of the Quick Reference Guide• As with any selection process, you must evaluate the people, process, and technology aspects of each option to ensure you select the best solution for your specific needs