Successfully reported this slideshow.

Migrating from PostgreSQL to MySQL Without Downtime

5

Share

Upcoming SlideShare
EG Reports - Delicious Data
EG Reports - Delicious Data
Loading in …3
×
1 of 50
1 of 50

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Migrating from PostgreSQL to MySQL Without Downtime

  1. 1. Migrating from PostgreSQL to MySQL ... without downtime Matthew Graham Percona Live NYC 2011
  2. 2. Global marketplace for buying and selling handmade goods Total April 2011 • 8 million members • $38.7 million goods sold • 800k active shops • 2 million items sold • 1 billion page views / month • 1.8 million new items listed • 390k new members joined
  3. 3. Global marketplace for buying and selling handmade goods Total April 2011 • 8 million members • $38.7 million = $900 / min • 800k active shops • 2 million items sold • 1 billion page views / month • 1.8 million new items listed • 390k new members joined
  4. 4. Downtime = -$
  5. 5. $320m $240m $160m $80m $0m 2006 2007 2008 2009 2010 Gross Merchandise Sales
  6. 6. Reasons To Migrate • Horizontal Scaling • Reduce Types of Databases • Licensing Costs • Functional Partitioning • Schema Refactor
  7. 7. Deciding to Switch expected quality - transition ----------------- net quality > actual quality
  8. 8. Source Migration Target
  9. 9. Why ?
  10. 10. Foundational Principles • Migrate One Table at a Time • Progressive Ramp Up • Data Duplication During Transition
  11. 11. 5 Steps Per Table 1. Create Target Tables 2. Tee Writes 3. Backfill 4. Read from Target Tables 5. Wrap Up
  12. 12. Create the Target
  13. 13. Merge into the Target
  14. 14. Separate the Target
  15. 15. Source Only Source Target Application Writes
  16. 16. Teed Writes Source Target Application Writes
  17. 17. Generating IDs Source Target ID Name ID Name 1 Adam 1 Adam 2 Bill 2 Bill 3 Charlie 3 David 4 David 4 Charlie Ticket Server: http://bit.ly/dbtickets
  18. 18. Backfill
  19. 19. Application Code Backfill function backfill_user(user_id) { old_row = find_old_user(user_id); new_row = create_new_user(); new_row.id = old_row.id; ... store_new_user(new_row); }
  20. 20. ETL Backfill Source Target Load Extract Transform
  21. 21. Don’t Overload • Script bulk inserts 100-2000 row batches INSERT INTO account (id, firstname, lastname) VALUES (1, ‘Alan’, ‘Alda’), (2, ‘Barry’, ‘Bonds’), (3, ‘Charlie’, ‘Chaplin’);
  22. 22. Backfill Speed Application Code ETL • Easier to Write • Faster Run Time
  23. 23. Backfill Extract Application Code ETL • Easier to Write • Faster Run Time • Less Likely to Get Out • Needs an Extract Source Of Sync
  24. 24. Backfill Reruns Application Code ETL • Easier to Write • Faster Run Time • Less Likely to Get Out • Needs an Extract Source Of Sync • Handles Duplicates from • REPLACE and INSERT Multiple Executions ON DUPLICATE KEY UPDATE
  25. 25. Teed Writes then Backfill or Backfill then Teed Writes
  26. 26. Verification by Diff $ diff_user 111222333 target user row is missing $ diff_user 123456789 - source.user.address_state = ‘CA’ + target.user.address_state = ‘NY’
  27. 27. Verification by Diff COMPARING 200 ROWS From: 111222197 User Id: 111222333 target user row is missing User Id: 111222345 - source.user.address_state = ‘CA’ + target.user.address_state = ‘NY’ SUMMARY: total rows with errors: 2/200
  28. 28. Read from Target
  29. 29. Progressive Ramp Up 100% 0% Source Target Application Reads
  30. 30. Progressive Ramp Up 99% 1% Source Target Application Reads
  31. 31. Progressive Ramp Up 95% 5% Source Target Application Reads
  32. 32. Progressive Ramp Up 75% 25% Source Target Application Reads
  33. 33. Progressive Ramp Up 0% 100% Source Target Application Reads
  34. 34. Ramp Up Example # global configuration use_new_tables: employees: true percent: 1 # application code if (enabled(‘use_new_tables’)) { $result = read_from_target(); } else { $result = read_from_source(); }
  35. 35. Enabled or Disabled? • Check cookies if user already assigned • $enabled = configured threshold > random % • Store $enabled in a cookie for future requests
  36. 36. Continuous Deployment • Ramp Up / Ramp Down • Backfill Fixes • Need Code Running on Prod to Proceed • Makes it Easier
  37. 37. No Foreign Keys
  38. 38. Wrapping Up
  39. 39. Analytical Data
  40. 40. Teed Writes Source Target Application Writes
  41. 41. Target Only Source Target Application Writes
  42. 42. Things to Remove • Code to use old tables • Code to switch on configuration • Configuration • Drop the old tables... eventually
  43. 43. 5 Steps Per Table 1. Create Target Tables 2. Tee Writes 3. Backfill 4. Read from Target Tables 5. Wrap Up
  44. 44. Foundational Principles • Migrate One Table at a Time • Progressive Ramp Up • Data Duplication During Transition
  45. 45. Questions? • Yes, we’re hiring! Specifically MySQL Ops http://bit.ly/etsywantsawesome • http://codeascraft.etsy.com • http://twitter.com/lapsu Matthew Graham Percona Live NYC 2011

×