Beware of your Hype Value Stores
Upcoming SlideShare
Loading in...5
×
 

Beware of your Hype Value Stores

on

  • 5,358 views

Key value stores are popping all around the web, describing themselves as the best fit for your webapp… but take care of the hype ! Think you’ll have 10000+ writes a sec ? There’s a lot of ...

Key value stores are popping all around the web, describing themselves as the best fit for your webapp… but take care of the hype ! Think you’ll have 10000+ writes a sec ? There’s a lot of unsaid “special features” that you have to know. Don’t trust benchmarks, even your own. Here’s how to choose the right key value store for your app !

The video is also available on youtube: http://www.youtube.com/watch?v=YZD8-EzozKQ

Statistics

Views

Total Views
5,358
Views on SlideShare
5,276
Embed Views
82

Actions

Likes
7
Downloads
37
Comments
2

5 Embeds 82

http://www.altaidevalley.com 46
http://www.slideshare.net 18
http://www.linkedin.com 13
https://www.linkedin.com 4
http://altaide.typepad.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Hi Jeff,
    Of course there are some misrepresentations on these slides, but only in terms of generalisation (you can't say much in 5 minutes ;)).

    First, on your last notice on the error we made on our algorithms, we didn't screw them up. We designed them in what seemed a very good way. We just faced the truth when dealing with billion of records, and guess what ? Tokyo cabinet b+trees are designed in the same fashion, leading to dramatic perf decrease (http://bjclark.me/2009/08/04/nosql-if-only-it-was-that-easy/ for example...).

    About the 'up to 9' random IOs, the actual truth it that it may be less of course, but more importantly, it may be a lot more too, depending on the size of your key, the number of node to visit, if they are in RAM or not etc... So it's much more complicated than that, in both ways.
    Are you sure you want to
    Your message goes here
    Processing…
  • There are a couple of serious misrepresentations in these slides. A decent disk is capable of doing ~300 *random* IOPS, but only a complete idiot lets their data store - be it a database, filesystem, or anything else - generate a seek per operation, and anything serious uses multiple spindles. You say *up to* nine random IOPS per lookup (which is a bit of a stretch even when using only one disk), and then you use that as an *average* for further calculation, which is simply dishonest. When you say you 'had similar algorithms' you obviously screwed them up; just because you didn't know how to implement such a system better doesn't mean others don't.

    You obviously have an axe to grind, or maybe you just want to get some publicity by being contrary. That's fine, but don't let your agenda undermine your integrity.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Beware of your Hype Value Stores Beware of your Hype Value Stores Presentation Transcript

  • Beware of your Hype – Value store ! Ignite Velocity 2009 Jérémie BORDIER - @ahfeel / jeremie.bordier@exalead.com
  • Get
the
Hype
!
 •  Rela&onal
DBMS
are
so
1990…
 •  So
simple,
so
fast,
looks
powerful
!
 BerkeleyDB Persevere Cassandra Hypertable Project Voldemort Redis Dynomite SimpleDB MemcacheDB Dynamo Scalaris Tokyo Tyrant CouchDB MongoDB 2
  • 10
000+
writes
/
sec
!
 •  RDBMS
only
performs
±300
w
/
sec…
wC
?
 Let’s do some simple math ! 3
  • State
of
the
art
mathema8cs
 •  Average
SATA
II
disk
seek
&me:
±6ms
 •  Very
good
SCSI
disk:
±3ms
 1 second / 3ms = ±300 REAL writes / sec 4
  • Eventual
Persistency
 •  Doesn’t
sync
the
file
system
 •  Keep
that
in
mind
!
 Your
servers
WILL
crash,
you
WILL
lose
data
 5
  • Don’t
care
about
writes…
 •  If
can
do
30
000+
lookup
/
sec
!
 Not for long ! 6
  • Eventual
memory
 •  Relies
on
B+Trees
 •  Map
all
the
data
structures
in
memory
 What if your data goes too big ? 7
  • Lookup…
 •  get(“elevator”)
 –  Visit
8
nodes
 –  Read
the
data
from
disk
 A R E S O Data: L 42
 V T E A O S M 8
  • Why
elevator
?
 •  American
elevators
are
wayyyy
too
fast
!
 •  Feels
like
a
NASA
training
 Makes
me
want
to
throw
up
:( 
 9
  • Hardware
limit
!
 Up to 9 RANDOM I/Os per lookup ! 1 second / (9 * 3ms) = 37 get / sec •  On
very
good
hardware
!
 10
  • We
did
that
too
:)
 •  We
had
similar
algorithms
 •  Encountered
horrible
perf
decreases…
 REDUCE I/O ! Ensure only 1 I/O per lookup. 11
  • Don’t
trust
benchmarks
 •  A
few
million
entries
isn’t
enough
 •  You’re
benchmarking
your
Disk
/
OS 
cache
:)
 (Common, DON’T BENCHMARK IN RUBY…) 12
  • Compare
what’s
comparable
 •  Distributed
column
stores
 –  BigTable
like
systems
 •  Key
value
stores
 –  Tokyo,
Dynamo
like
systems
 13
  • Distributed
column
stores
 Google Megastore Persistent Hypertable SimpleDB Eventually Replicated Persistent Mature Distributed Cassandra HBase 14
  • Key
–
value
stores
 Persevere BerkeleyDB Mutable MongoDB Persistent Replicated Dynomite Embedded Distributed Tokyo Cabinet Eventually Scalaris Mature Persistent MemcacheDB Immutable Redis CouchDB Voldemort 15
  • How
to
choose
?
 •  Maturity
is
priceless
 •  Most
suitable
stores:
 –  Persistent:
BerkeleyDB,
MySQL
 –  Ev.
Persistent:
Tokyo
Cabinet
 –  Ev.
Persistent
+
Distributed
+
…
:
Voldemort
 –  Distributed
column
store:
Cassandra
 16
  • You
are
not
Google
 (Well, not all of you ) •  Build
something
 •  Make
it
work
 •  Think
about
scaling
 •  Think
about
being
hyped
 Why ? 17
  • Key
value
pain…
 •  You
will
end
up
crossing
data.
 •  Doing
joins
with
KV
stores
?
 Only for ninjas ! 18
  • Query
8me
joins

 •  Coding
what
RDBMS
are
made
for
?
…
 •  Slow
!
 (you shouldn’t do this…) 19
  • Build
8me
schema
flaTening

 •  Not
flexible
!
 •  Needs
Map
Reduce
(Hadoop…)
to
scale
 Think before doing this :) 20
  • And
finally..
 •  HRD
hard
drive:
160
000
random
IO/sec
!!!
 •  Yahoo
announced
open
sourcing
Sherpa
!
 •  Resources
 –  hip://developer.yahoo.net/blog/archives/2009/06/nosql_meetup.html
 –  hip://metabrew.com/ar&cle/an&‐rdbms‐a‐list‐of‐distributed‐key‐value‐stores/
 –  hip://www.ryanpark.org/2008/04/top‐10‐avoid‐the‐simpledb‐hype.html
 –  hip://project‐voldemort.com/blog/2009/06/building‐a‐1‐tb‐data‐cycle‐at ‐linkedin‐with‐hadoop‐and‐project‐voldemort/
 Thanks ! Contact: @ahfeel :) 21