Your SlideShare is downloading. ×
0
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Migrating from PostgreSQL to MySQL Without Downtime
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Migrating from PostgreSQL to MySQL Without Downtime

8,044

Published on

How Etsy migrates live data into MySQL shards without downtime.

How Etsy migrates live data into MySQL shards without downtime.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,044
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
80
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Migrating from PostgreSQL to MySQL ... without downtimeMatthew GrahamPercona Live NYC 2011
  • 2. Global marketplace for buying and selling handmade goods Total April 2011• 8 million members • $38.7 million goods sold• 800k active shops • 2 million items sold• 1 billion page views / month • 1.8 million new items listed • 390k new members joined
  • 3. Global marketplace for buying and selling handmade goods Total April 2011• 8 million members • $38.7 million = $900 / min• 800k active shops • 2 million items sold• 1 billion page views / month • 1.8 million new items listed • 390k new members joined
  • 4. Downtime = -$
  • 5. $320m$240m$160m$80m $0m 2006 2007 2008 2009 2010 Gross Merchandise Sales
  • 6. Reasons To Migrate • Horizontal Scaling • Reduce Types of Databases • Licensing Costs • Functional Partitioning • Schema Refactor
  • 7. Deciding to Switch expected quality - transition ----------------- net quality quality > actual
  • 8. Deciding to Switch expected quality - transition----------------- net quality > actual quality
  • 9. Deciding to Switch expected quality - transition----------------- net quality > actual quality
  • 10. Source Migration Target
  • 11. Why ?
  • 12. Foundational Principles • Migrate One Table at a Time • Progressive Ramp Up • Data Duplication During Transition
  • 13. 5 Steps Per Table 1. Create Target Tables 2. Tee Writes 3. Backfill 4. Read from Target Tables 5. Wrap Up
  • 14. Create the Target
  • 15. Merge into the Target
  • 16. Separate the Target
  • 17. Source OnlySource Target Application Writes
  • 18. Teed WritesSource Target Application Writes
  • 19. Generating IDs Source TargetID Name ID Name1 Adam 1 Adam2 Bill 2 Bill3 Charlie 3 David4 David 4 Charlie Ticket Server: http://bit.ly/dbtickets
  • 20. Backfill
  • 21. Application Code Backfillfunction backfill_user(user_id) { old_row = find_old_user(user_id); new_row = create_new_user(); new_row.id = old_row.id; ... store_new_user(new_row);}
  • 22. ETL BackfillSource Target Load Extract Transform
  • 23. Don’t Overload • Script bulk inserts 100-2000 row batchesINSERT INTO account (id, firstname, lastname)VALUES(1, ‘Alan’, ‘Alda’),(2, ‘Barry’, ‘Bonds’),(3, ‘Charlie’, ‘Chaplin’);
  • 24. Backfill SpeedApplication Code ETL• Easier to Write • Faster Run Time
  • 25. Backfill ExtractApplication Code ETL• Easier to Write • Faster Run Time• Less Likely to Get Out • Needs an Extract Source Of Sync
  • 26. Backfill RerunsApplication Code ETL• Easier to Write • Faster Run Time• Less Likely to Get Out • Needs an Extract Source Of Sync• Handles Duplicates from • REPLACE and INSERT Multiple Executions ON DUPLICATE KEY UPDATE
  • 27. Teed Writes then Backfill orBackfill then Teed Writes
  • 28. Verification by Diff$ diff_user 111222333target user row is missing$ diff_user 123456789- source.user.address_state = ‘CA’+ target.user.address_state = ‘NY’
  • 29. Verification by DiffCOMPARING 200 ROWS From: 111222197User Id: 111222333target user row is missingUser Id: 111222345- source.user.address_state = ‘CA’+ target.user.address_state = ‘NY’SUMMARY: total rows with errors: 2/200
  • 30. Read from Target
  • 31. Progressive Ramp Up 100% 0%Source Target Application Reads
  • 32. Progressive Ramp Up 99% 1%Source Target Application Reads
  • 33. Progressive Ramp Up 95% 5%Source Target Application Reads
  • 34. Progressive Ramp Up 75% 25%Source Target Application Reads
  • 35. Progressive Ramp Up 0% 100%Source Target Application Reads
  • 36. Ramp Up Example# global configurationuse_new_tables: employees: true percent: 1# application codeif (enabled(‘use_new_tables’)) { $result = read_from_target();} else { $result = read_from_source();}
  • 37. Enabled or Disabled?• Check cookies if user already assigned• $enabled = configured threshold > random %• Store $enabled in a cookie for future requests
  • 38. Continuous Deployment • Ramp Up / Ramp Down • Backfill Fixes • Need Code Running on Prod to Proceed • Makes it Easier
  • 39. No Foreign Keys
  • 40. Wrapping Up
  • 41. Analytical Data
  • 42. Teed WritesSource Target Application Writes
  • 43. Target OnlySource Target Application Writes
  • 44. Things to Remove• Code to use old tables• Code to switch on configuration• Configuration• Drop the old tables... eventually
  • 45. 5 Steps Per Table 1. Create Target Tables 2. Tee Writes 3. Backfill 4. Read from Target Tables 5. Wrap Up
  • 46. Foundational Principles • Migrate One Table at a Time • Progressive Ramp Up • Data Duplication During Transition
  • 47. Questions? • Yes, we’re hiring! Specifically MySQL Ops http://bit.ly/etsywantsawesome • http://codeascraft.etsy.com • http://twitter.com/lapsuMatthew GrahamPercona Live NYC 2011

×