Taking Diaspora from MongoDB to MySQL (RubyConf 2011)

1Image copyright Yukihiro Matsumoto

MongoDB to MySQL 2 Image copyright Yukihiro Matsumoto

MongoDB to MySQL • Data 2 Image copyright Yukihiro Matsumoto

MongoDB to MySQL • Data • Ways to store it 2 Image copyright Yukihiro Matsumoto

MongoDB to MySQL • Data • Ways to store it • Youthful  indiscretion 2 Image copyright Yukihiro Matsumoto

MongoDB to MySQL • Data • Ways to store it • Youthful  indiscretion • Data 2 Image copyright Yukihiro Matsumoto

Sarah Mei@sarahmei 3

Sarah Allen (not me) 4 Photo copyright Lee Lundrigan

5Images copyright Pivotal Labs, Inc.

7Photo by Henrik Moltke

Eben  Moglen 8Photo from The Internet Society, original: http://www.ﬂickr.com/photos/internetsociety/5876916246

9Images copyright (clockwise from top) Diaspora Inc., 10gen Inc., David Heinemeier Hansson.

“Social data isn’t  relational” 10

“Social data isn’t  relational” ‐Twitter 10

OOH SHINY 11Photo from ﬂickr user helen61, original: http://www.ﬂickr.com/photos/helen61/2873090945/

“Social data isn’t  relational” ‐Twitter 12

OOH SHINY 13Photo from ﬂickr user helen61, original: http://www.ﬂickr.com/photos/helen61/2873090945/

Mismatches: 14

Mismatches: • At the data layer 14

Mismatches: • At the data layer • At the mapping layer 14

Mismatches: • At the data layer • At the mapping layerTest‐driving a data migration 14

Mismatches: • At the data layer We are here • At the mapping layerTest‐driving a data migration 14

“document‐oriented  database” 15 Image copyright 10gen Inc.

tv_showsmany seasons many reviews many episodes many cast_members 16

{title: Babylon 5, seasons: [ {season_number: 1, episodes: [ {ordinal_within_season: 1, title: Midnight on the Firing reviews: [{...}], cast_members: [{...}] } ] } ]} 17

usersmany friends many comments many posts many commenters many likes many likers 19

{name: Sarah Mei, friends: [ {name: Thing 1, posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 20

{name: Sarah Mei, friends: [ {name: Thing 1, posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 26

{name: Sarah Mei, friends: [ {user_id: 4b866f08234ae01d21d8960 posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 27

28Photo from ﬂickr user befuddledsenses, original: http://www.ﬂickr.com/photos/befuddledsenses/2903185831

1 “Social data isn’t  relational” 29

1 “Social data isn’t  relational” “Eight‐table joins  ZOMG” 29

2  When they say documents, they really mean documents. 30 Photo from ﬂickr user horrgakx, original: http://www.ﬂickr.com/photos/horrgakx/2963449465

Mismatches: • At the data layer • At the mapping layer We are hereTest‐driving a data migration 31

Faking It 32

Faking It• Faux relational interface bad 32

Faking It• Faux relational interface bad• JS map/reduce good 32

Faking It• Faux relational interface bad• JS map/reduce good • But lower‐level 32

Faking It• Faux relational interface bad• JS map/reduce good • But lower‐level • And harder to test 32

Y U SO HARD? 33

Y U SO HARD?• Less documentation 33

Y U SO HARD?• Less documentation• Harder to google for stuﬀ 33

Y U SO HARD?• Less documentation• Harder to google for stuﬀ• Smaller community 33

Y U SO HARD?• Less documentation• Harder to google for stuﬀ• Smaller community• Less gem support 33

Y U SO HARD?• Less documentation• Harder to google for stuﬀ• Smaller community• Less gem support• Developers had no Mongo experience 33

34Illustration from ﬂickr user katy_tresedder, original: http://www.ﬂickr.com/photos/katy_tresedder/4902794900

Y U SO HARD? In General:• Less documentation• Lower googlability• Smaller community• Less gem support• Team of developers with no Mongo experience 35

36Photo from ﬂickr user nswmaritime, original: http://www.ﬂickr.com/photos/nswmaritime/2963649924

Every non‐standard  technology choice reduces our ability to  iterate quickly. 37

38Photo from ﬂickr user nzgabriel, original: http://www.ﬂickr.com/photos/nzgabriel/2607065194

Mismatches: • At the data layer • At the mapping layerTest‐driving a data migration We are ﬁnally here 40

41Photo from ﬂickr user werkunz, original: http://www.ﬂickr.com/photos/werkunz/5160818883

42Photo from the Seattle Municipal Archives, accessed at: http://www.ﬂickr.com/photos/seattlemunicipalarchives/4328053793

Test‐driving the migration 43

“Test‐driving” the migration 44

“Test‐driving” the migrationThe Easy Way 44

“Test‐driving” the migrationThe Easy Way 1. Test‐drive conversion of a single sub‐ document from JSON to CSV 44

“Test‐driving” the migrationThe Easy Way 1. Test‐drive conversion of a single sub‐ document from JSON to CSV 2. Test‐drive import of that CSV into AR 44

“Test‐driving” the migrationThe Easy Way 1. Test‐drive conversion of a single sub‐ document from JSON to CSV 2. Test‐drive import of that CSV into AR 3. Run it on a copy of the production database* 44

“Test‐driving” the migrationThe Easy Way 1. Test‐drive conversion of a single sub‐ document from JSON to CSV 2. Test‐drive import of that CSV into AR 3. Run it on a copy of the production database* 4. Add more test cases 44

“Test‐driving” the migrationThe Hard(er) Way 45

“Test‐driving” the migrationThe Hard(er) Way • Batching 45

“Test‐driving” the migrationThe Hard(er) Way • Batching • activerecord‐import 45

“Test‐driving” the migrationThe Hard(er) Way • Batching • activerecord‐import • LOAD DATA INFILE 45

“Test‐driving” the migration Gotchas 46

“Test‐driving” the migration Gotchas • character encoding 46

“Test‐driving” the migration Gotchas • character encoding • converting IDs 46

“Test‐driving” the migration Gotchas • character encoding • converting IDs • character encoding 46

Codegithub.com/diaspora/diaspora tag last‐data‐conversion 47

Diaspora@joindiaspora Sarah Mei @sarahmei 48Images not otherwise credited are copyright Sarah Mei and licensed cc-attribution

surprise! 50 Photo from ﬂickr user industry_is_virtue, original: http://www.ﬂickr.com/photos/industry_is_virtue/3209194592

cast_members{character_name: “Cmdr Jeffrey Sinclair”,actor_name: “Michael O’Hare”} 52

So far: 53

So far: • Why MongoDB wasn’t a good ﬁt for  Diaspora’s data 53

So far: • Why MongoDB wasn’t a good ﬁt for  Diaspora’s data • The mechanics of the migration 53

So far: • Why MongoDB wasn’t a good ﬁt for  Diaspora’s data • The mechanics of the migration • Why we switched 53

http://lanyrd.com	321
http://paper.li	4
http://a0.twimg.com	2
http://a0.twimg.com	2
http://translate.googleusercontent.com	1
http://127.0.0.1	1

Taking Diaspora from MongoDB to MySQL (RubyConf 2011)

by Sarah Mei

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 331

Statistics

Taking Diaspora from MongoDB to MySQL (RubyConf 2011) Presentation Transcript