Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Dirty NoSQL
How simple is your database?




                               25.09.2010
About

Contributor                            Co-founder




                 Felix Geisendörfer

node.js driver          ...
Dirty
Dirty



JavaScript views            In-memory


                    Dirty
Design choices
Non-blocking I/O FTW
150 lines of code
                          Lines of code



   Dirty



CouchDB



   Redis


           0      5.000    ...
Append-only JSON log


$ cat dirty.db

{"key": "first", "val": {"foo": "bar"}}
{"key": "second", "val": "A string"}
{"key"...
No Networking


var db = require('dirty')('test.db');

db.set('foo', 'bar');
db.get('foo'); // => bar
No Networking

var db = require('dirty')('languages.db');

db.set('javascript', {age: 15});
db.set('python', {age: 19});
d...
Performance
Benchmarks


Do your own!
dirty.get()


50 MHz
 (50 million / s)




                    v8: 160 MHz
dirty.set()


5 MHz
 (5 million / s)




                   v8: 12 MHz
dirty.set()
               With flushing to disk




     Numbers                          256 byte string

200 kHz 70 kHz
...
dirty.forEach()


 4.5 MHz
   (4.5 million / s)




                       v8: 33 MHz
The Wall

• Dirty is a wonderful database as long as
  you have < 1 million records


• After that, you hit “The Wall”
Scaling beyond
Possibilities for future node.js based databases
22.000                                                                                         2010




                  ...
22 exabyte / month
  (that is 22 billion gigabytes)
Flexible guarantees

db.set('my-key', 'my-value', function(err) {
  if (err) throw err;

  console.log('Record written to ...
Memory / Disk Hybrids

• “Memcached” built into your database

• Better knowledge about your data than any
  general purpo...
Replication

• Node.js = perfect for streaming between
  instances


• Node could “hold” the connection if a not-
  yet re...
Web Services

• Node.js could act as a proxy for different
  database backends (written in node or not)


• One could quer...
Questions?




✎   @felixge / felix@debuggable.com
Get it

        $ npm install dirty
                 or

http://github.com/felixge/node-dirty
Don’t hit the wall
Questions?




✎   @felixge / felix@debuggable.com
Sources
•   Internet traffic numbers: http://en.wikipedia.org/wiki/Internet_traffic

•   Wonderful dino-rider vs. volcano im...
Memory overhead
        (for setting numeric key / values)




• 20mb overhead / 1 million records

• 20 bytes / key
Dirty - How simple is your database?
Dirty - How simple is your database?
Upcoming SlideShare
Loading in …5
×

Dirty - How simple is your database?

14,975 views

Published on

A tiny & fast key value store with append-only disk log. Ideal for apps with < 1 million records.

Published in: Technology
  • Be the first to comment

Dirty - How simple is your database?

  1. Dirty NoSQL How simple is your database? 25.09.2010
  2. About Contributor Co-founder Felix Geisendörfer node.js driver node.js driver
  3. Dirty
  4. Dirty JavaScript views In-memory Dirty
  5. Design choices
  6. Non-blocking I/O FTW
  7. 150 lines of code Lines of code Dirty CouchDB Redis 0 5.000 10.000 15.000 20.000
  8. Append-only JSON log $ cat dirty.db {"key": "first", "val": {"foo": "bar"}} {"key": "second", "val": "A string"} {"key": "a-number", "val": 23}
  9. No Networking var db = require('dirty')('test.db'); db.set('foo', 'bar'); db.get('foo'); // => bar
  10. No Networking var db = require('dirty')('languages.db'); db.set('javascript', {age: 15}); db.set('python', {age: 19}); db.set('perl', {age: 23}); db.forEach(function(key, doc) { console.log('%s is %d years old.', key, doc.age); });
  11. Performance
  12. Benchmarks Do your own!
  13. dirty.get() 50 MHz (50 million / s) v8: 160 MHz
  14. dirty.set() 5 MHz (5 million / s) v8: 12 MHz
  15. dirty.set() With flushing to disk Numbers 256 byte string 200 kHz 70 kHz (200-thousand / s) (70-thousand / s)
  16. dirty.forEach() 4.5 MHz (4.5 million / s) v8: 33 MHz
  17. The Wall • Dirty is a wonderful database as long as you have < 1 million records • After that, you hit “The Wall”
  18. Scaling beyond Possibilities for future node.js based databases
  19. 22.000 2010 The Internet 20.000 18.000 16.000 2009 14.000 12.000 PB / month 2008 10.000 8000 2007 6000 2006 4000 2005 2000 2004 2003 2002 2000 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Year
  20. 22 exabyte / month (that is 22 billion gigabytes)
  21. Flexible guarantees db.set('my-key', 'my-value', function(err) { if (err) throw err; console.log('Record written to disk.'); }); console.log('Record written to memory.');
  22. Memory / Disk Hybrids • “Memcached” built into your database • Better knowledge about your data than any general purpose algorithm
  23. Replication • Node.js = perfect for streaming between instances • Node could “hold” the connection if a not- yet replicated key is being requested
  24. Web Services • Node.js could act as a proxy for different database backends (written in node or not) • One could query 3rd party services for information
  25. Questions? ✎ @felixge / felix@debuggable.com
  26. Get it $ npm install dirty or http://github.com/felixge/node-dirty
  27. Don’t hit the wall
  28. Questions? ✎ @felixge / felix@debuggable.com
  29. Sources • Internet traffic numbers: http://en.wikipedia.org/wiki/Internet_traffic • Wonderful dino-rider vs. volcano image: Jesse Star / http://www.geekologie.com/ 2009/10/dinorider_geekologie_writer_vs.php
  30. Memory overhead (for setting numeric key / values) • 20mb overhead / 1 million records • 20 bytes / key

×