Taking Diaspora from MongoDB to MySQL (RubyConf 2011)
Upcoming SlideShare
Loading in...5
×
 

Taking Diaspora from MongoDB to MySQL (RubyConf 2011)

on

  • 5,054 views

Diaspora is the crowd-funded open-source decentralized social network built on Rails. Full buzzword compliance: on by default. We have many thousands of active users and they generate a lot of social ...

Diaspora is the crowd-funded open-source decentralized social network built on Rails. Full buzzword compliance: on by default. We have many thousands of active users and they generate a lot of social data. But after nine months of full-time development with MongoDB as our primary storage engine, a few months ago we converted it all to MySQL. Wait...what? Most people are going the other way, dropping Mongo into a project in place of MySQL or PostgreSQL. Plus, conventional wisdom says that social data is ill-suited to a traditional data store. Come hear a story about a large-scale Rails project that tried it both ways. You'll see crisis and redemption, facts and figures, nerds, kittens, ponycorns, and, of course, the secret sauce. Hecklers will be piped to /dev/null.

Statistics

Views

Total Views
5,054
Views on SlideShare
4,662
Embed Views
392

Actions

Likes
2
Downloads
35
Comments
0

8 Embeds 392

http://lanyrd.com 378
http://paper.li 4
https://twitter.com 3
http://a0.twimg.com 2
http://a0.twimg.com 2
http://translate.googleusercontent.com 1
http://127.0.0.1 1
http://192.168.1.16 1
More...

Accessibility

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Taking Diaspora from MongoDB to MySQL (RubyConf 2011) Taking Diaspora from MongoDB to MySQL (RubyConf 2011) Presentation Transcript

  • 1Image copyright Yukihiro Matsumoto
  • MongoDB
to
MySQL 2 Image copyright Yukihiro Matsumoto
  • MongoDB
to
MySQL • Data 2 Image copyright Yukihiro Matsumoto
  • MongoDB
to
MySQL • Data • Ways
to
store
it 2 Image copyright Yukihiro Matsumoto
  • MongoDB
to
MySQL • Data • Ways
to
store
it • Youthful
 indiscretion 2 Image copyright Yukihiro Matsumoto
  • MongoDB
to
MySQL • Data • Ways
to
store
it • Youthful
 indiscretion • Data 2 Image copyright Yukihiro Matsumoto
  • Sarah
Mei@sarahmei 3
  • Sarah
Allen (not
me) 4 Photo copyright Lee Lundrigan
  • 5Images copyright Pivotal Labs, Inc.
  • 5Images copyright Pivotal Labs, Inc.
  • 6
  • 7Photo by Henrik Moltke
  • Eben
 Moglen 8Photo from The Internet Society, original: http://www.flickr.com/photos/internetsociety/5876916246
  • 9Images copyright (clockwise from top) Diaspora Inc., 10gen Inc., David Heinemeier Hansson.
  • 9Images copyright (clockwise from top) Diaspora Inc., 10gen Inc., David Heinemeier Hansson.
  • 9Images copyright (clockwise from top) Diaspora Inc., 10gen Inc., David Heinemeier Hansson.
  • “Social
data
isn’t
 relational” 10
  • “Social
data
isn’t
 relational” ‐Twitter 10
  • OOH
SHINY 11Photo from flickr user helen61, original: http://www.flickr.com/photos/helen61/2873090945/
  • “Social
data
isn’t
 relational” ‐Twitter 12
  • OOH
SHINY 13Photo from flickr user helen61, original: http://www.flickr.com/photos/helen61/2873090945/
  • Mismatches: 14
  • Mismatches: • At
the
data
layer 14
  • Mismatches: • At
the
data
layer • At
the
mapping
layer 14
  • Mismatches: • At
the
data
layer • At
the
mapping
layerTest‐driving
a
data
migration 14
  • Mismatches: • At
the
data
layer We
are
here • At
the
mapping
layerTest‐driving
a
data
migration 14
  • “document‐oriented
 database” 15 Image copyright 10gen Inc.
  • tv_showsmany seasons many reviews many episodes many cast_members 16
  • {title: Babylon 5, seasons: [ {season_number: 1, episodes: [ {ordinal_within_season: 1, title: Midnight on the Firing reviews: [{...}], cast_members: [{...}] } ] } ]} 17
  • {title: Babylon 5, seasons: [ {season_number: 1, episodes: [ {ordinal_within_season: 1, title: Midnight on the Firing reviews: [{...}], cast_members: [{...}] } ] } ]} 17
  • {title: Babylon 5, seasons: [ {season_number: 1, episodes: [ {ordinal_within_season: 1, title: Midnight on the Firing reviews: [{...}], cast_members: [{...}] } ] } ]} 17
  • {title: Babylon 5, seasons: [ {season_number: 1, episodes: [ {ordinal_within_season: 1, title: Midnight on the Firing reviews: [{...}], cast_members: [{...}] } ] } ]} 17
  • 18
  • 18
  • usersmany friends many comments many posts many commenters many likes many likers 19
  • {name: Sarah Mei, friends: [ {name: Thing 1, posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 20
  • {name: Sarah Mei, friends: [ {name: Thing 1, posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 20
  • {name: Sarah Mei, friends: [ {name: Thing 1, posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 20
  • {name: Sarah Mei, friends: [ {name: Thing 1, posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 20
  • tv_showsmany seasons many reviews many episodes many cast_members 21
  • usersmany friends many comments many posts many commenters many likes many likers 22
  • usersmany friends many comments many posts many commenters many likes many likers 23
  • tv_showsmany seasons many reviews many episodes many cast_members 24
  • 25
  • 25
  • {name: Sarah Mei, friends: [ {name: Thing 1, posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 26
  • {name: Sarah Mei, friends: [ {user_id: 4b866f08234ae01d21d8960 posts: [ {message: go fly a kite, comments: [{...}], likes: [{...}], reshares: [{...}] } ] } ]} 27
  • 28Photo from flickr user befuddledsenses, original: http://www.flickr.com/photos/befuddledsenses/2903185831
  • 1 “Social
data
isn’t
 relational” 29
  • 1 “Social
data
isn’t
 relational” “Eight‐table
joins
 ZOMG” 29
  • 2 
When
they
say
documents,
they
really
mean documents. 30 Photo from flickr user horrgakx, original: http://www.flickr.com/photos/horrgakx/2963449465
  • Mismatches: • At
the
data
layer • At
the
mapping
layer We
are
hereTest‐driving
a
data
migration 31
  • Faking
It 32
  • Faking
It• Faux
relational
interface
bad 32
  • Faking
It• Faux
relational
interface
bad• JS
map/reduce
good 32
  • Faking
It• Faux
relational
interface
bad• JS
map/reduce
good • But
lower‐level 32
  • Faking
It• Faux
relational
interface
bad• JS
map/reduce
good • But
lower‐level • And
harder
to
test 32
  • Y
U
SO
HARD? 33
  • Y
U
SO
HARD?• Less
documentation 33
  • Y
U
SO
HARD?• Less
documentation• Harder
to
google
for
stuff 33
  • Y
U
SO
HARD?• Less
documentation• Harder
to
google
for
stuff• Smaller
community 33
  • Y
U
SO
HARD?• Less
documentation• Harder
to
google
for
stuff• Smaller
community• Less
gem
support 33
  • Y
U
SO
HARD?• Less
documentation• Harder
to
google
for
stuff• Smaller
community• Less
gem
support• Developers
had
no
Mongo
experience 33
  • 34Illustration from flickr user katy_tresedder, original: http://www.flickr.com/photos/katy_tresedder/4902794900
  • Y
U
SO
HARD? In
General:• Less
documentation• Lower
googlability• Smaller
community• Less
gem
support• Team
of
developers
with
no
Mongo
experience 35
  • 36Photo from flickr user nswmaritime, original: http://www.flickr.com/photos/nswmaritime/2963649924
  • 36Photo from flickr user nswmaritime, original: http://www.flickr.com/photos/nswmaritime/2963649924
  • Every
non‐standard
 technology
choice
reduces
our
ability
to
 iterate
quickly. 37
  • 38Photo from flickr user nzgabriel, original: http://www.flickr.com/photos/nzgabriel/2607065194
  • 39
  • Mismatches: • At
the
data
layer • At
the
mapping
layerTest‐driving
a
data
migration We
are
finally
here 40
  • 41Photo from flickr user werkunz, original: http://www.flickr.com/photos/werkunz/5160818883
  • 42Photo from the Seattle Municipal Archives, accessed at: http://www.flickr.com/photos/seattlemunicipalarchives/4328053793
  • Test‐driving
the
migration 43
  • “Test‐driving”
the
migration 44
  • “Test‐driving”
the
migrationThe
Easy
Way 44
  • “Test‐driving”
the
migrationThe
Easy
Way 1. Test‐drive
conversion
of
a
single
sub‐ document
from
JSON
to
CSV 44
  • “Test‐driving”
the
migrationThe
Easy
Way 1. Test‐drive
conversion
of
a
single
sub‐ document
from
JSON
to
CSV 2. Test‐drive
import
of
that
CSV
into
AR 44
  • “Test‐driving”
the
migrationThe
Easy
Way 1. Test‐drive
conversion
of
a
single
sub‐ document
from
JSON
to
CSV 2. Test‐drive
import
of
that
CSV
into
AR 3. Run
it
on
a
copy
of
the
production
database* 44
  • “Test‐driving”
the
migrationThe
Easy
Way 1. Test‐drive
conversion
of
a
single
sub‐ document
from
JSON
to
CSV 2. Test‐drive
import
of
that
CSV
into
AR 3. Run
it
on
a
copy
of
the
production
database* 4. Add
more
test
cases 44
  • “Test‐driving”
the
migrationThe
Easy
Way 1. Test‐drive
conversion
of
a
single
sub‐ document
from
JSON
to
CSV 2. Test‐drive
import
of
that
CSV
into
AR 3. Run
it
on
a
copy
of
the
production
database* 4. Add
more
test
cases 44
  • “Test‐driving”
the
migrationThe
Hard(er)
Way 45
  • “Test‐driving”
the
migrationThe
Hard(er)
Way • Batching 45
  • “Test‐driving”
the
migrationThe
Hard(er)
Way • Batching • activerecord‐import 45
  • “Test‐driving”
the
migrationThe
Hard(er)
Way • Batching • activerecord‐import • LOAD
DATA
INFILE 45
  • “Test‐driving”
the
migration Gotchas 46
  • “Test‐driving”
the
migration Gotchas • character
encoding 46
  • “Test‐driving”
the
migration Gotchas • character
encoding • converting
IDs 46
  • “Test‐driving”
the
migration Gotchas • character
encoding • converting
IDs • character
encoding 46
  • Codegithub.com/diaspora/diaspora tag
last‐data‐conversion 47
  • Diaspora@joindiaspora Sarah
Mei @sarahmei 48Images not otherwise credited are copyright Sarah Mei and licensed cc-attribution
  • tv_showsmany seasons many reviews many episodes many cast_members 49
  • surprise! 50 Photo from flickr user industry_is_virtue, original: http://www.flickr.com/photos/industry_is_virtue/3209194592
  • tv_showsmany seasons many reviews many episodes many cast_members 51
  • cast_members{character_name: “Cmdr Jeffrey Sinclair”,actor_name: “Michael O’Hare”} 52
  • So
far: 53
  • So
far: • Why
MongoDB
wasn’t
a
good
fit
for
 Diaspora’s
data 53
  • So
far: • Why
MongoDB
wasn’t
a
good
fit
for
 Diaspora’s
data • The
mechanics
of
the
migration 53
  • So
far: • Why
MongoDB
wasn’t
a
good
fit
for
 Diaspora’s
data • The
mechanics
of
the
migration • Why
we
switched 53