Dirty NoSQL
How simple is your database?




                               25.09.2010
About

Contributor                            Co-founder




                 Felix Geisendörfer

node.js driver          ...
Dirty
Dirty



JavaScript views            In-memory


                    Dirty
Design choices
Non-blocking I/O FTW
150 lines of code
                          Lines of code



   Dirty



CouchDB



   Redis


           0      5.000    ...
Append-only JSON log


$ cat dirty.db

{"key": "first", "val": {"foo": "bar"}}
{"key": "second", "val": "A string"}
{"key"...
No Networking


var db = require('dirty')('test.db');

db.set('foo', 'bar');
db.get('foo'); // => bar
No Networking

var db = require('dirty')('languages.db');

db.set('javascript', {age: 15});
db.set('python', {age: 19});
d...
Performance
Benchmarks


Do your own!
dirty.get()


50 MHz
 (50 million / s)




                    v8: 160 MHz
dirty.set()


5 MHz
 (5 million / s)




                   v8: 12 MHz
dirty.set()
               With flushing to disk




     Numbers                          256 byte string

200 kHz 70 kHz
...
dirty.forEach()


 4.5 MHz
   (4.5 million / s)




                       v8: 33 MHz
The Wall

• Dirty is a wonderful database as long as
  you have < 1 million records


• After that, you hit “The Wall”
Scaling beyond
Possibilities for future node.js based databases
22.000                                                                                         2010




                  ...
22 exabyte / month
  (that is 22 billion gigabytes)
Flexible guarantees

db.set('my-key', 'my-value', function(err) {
  if (err) throw err;

  console.log('Record written to ...
Memory / Disk Hybrids

• “Memcached” built into your database

• Better knowledge about your data than any
  general purpo...
Replication

• Node.js = perfect for streaming between
  instances


• Node could “hold” the connection if a not-
  yet re...
Web Services

• Node.js could act as a proxy for different
  database backends (written in node or not)


• One could quer...
Questions?




✎   @felixge / felix@debuggable.com
Get it

        $ npm install dirty
                 or

http://github.com/felixge/node-dirty
Don’t hit the wall
Questions?




✎   @felixge / felix@debuggable.com
Sources
•   Internet traffic numbers: http://en.wikipedia.org/wiki/Internet_traffic

•   Wonderful dino-rider vs. volcano im...
Memory overhead
        (for setting numeric key / values)




• 20mb overhead / 1 million records

• 20 bytes / key
Dirty - How simple is your database?
Dirty - How simple is your database?
Upcoming SlideShare
Loading in …5
×

Dirty - How simple is your database?

9,716
-1

Published on

A tiny & fast key value store with append-only disk log. Ideal for apps with < 1 million records.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,716
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
46
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Dirty - How simple is your database?

  1. Dirty NoSQL How simple is your database? 25.09.2010
  2. About Contributor Co-founder Felix Geisendörfer node.js driver node.js driver
  3. Dirty
  4. Dirty JavaScript views In-memory Dirty
  5. Design choices
  6. Non-blocking I/O FTW
  7. 150 lines of code Lines of code Dirty CouchDB Redis 0 5.000 10.000 15.000 20.000
  8. Append-only JSON log $ cat dirty.db {"key": "first", "val": {"foo": "bar"}} {"key": "second", "val": "A string"} {"key": "a-number", "val": 23}
  9. No Networking var db = require('dirty')('test.db'); db.set('foo', 'bar'); db.get('foo'); // => bar
  10. No Networking var db = require('dirty')('languages.db'); db.set('javascript', {age: 15}); db.set('python', {age: 19}); db.set('perl', {age: 23}); db.forEach(function(key, doc) { console.log('%s is %d years old.', key, doc.age); });
  11. Performance
  12. Benchmarks Do your own!
  13. dirty.get() 50 MHz (50 million / s) v8: 160 MHz
  14. dirty.set() 5 MHz (5 million / s) v8: 12 MHz
  15. dirty.set() With flushing to disk Numbers 256 byte string 200 kHz 70 kHz (200-thousand / s) (70-thousand / s)
  16. dirty.forEach() 4.5 MHz (4.5 million / s) v8: 33 MHz
  17. The Wall • Dirty is a wonderful database as long as you have < 1 million records • After that, you hit “The Wall”
  18. Scaling beyond Possibilities for future node.js based databases
  19. 22.000 2010 The Internet 20.000 18.000 16.000 2009 14.000 12.000 PB / month 2008 10.000 8000 2007 6000 2006 4000 2005 2000 2004 2003 2002 2000 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Year
  20. 22 exabyte / month (that is 22 billion gigabytes)
  21. Flexible guarantees db.set('my-key', 'my-value', function(err) { if (err) throw err; console.log('Record written to disk.'); }); console.log('Record written to memory.');
  22. Memory / Disk Hybrids • “Memcached” built into your database • Better knowledge about your data than any general purpose algorithm
  23. Replication • Node.js = perfect for streaming between instances • Node could “hold” the connection if a not- yet replicated key is being requested
  24. Web Services • Node.js could act as a proxy for different database backends (written in node or not) • One could query 3rd party services for information
  25. Questions? ✎ @felixge / felix@debuggable.com
  26. Get it $ npm install dirty or http://github.com/felixge/node-dirty
  27. Don’t hit the wall
  28. Questions? ✎ @felixge / felix@debuggable.com
  29. Sources • Internet traffic numbers: http://en.wikipedia.org/wiki/Internet_traffic • Wonderful dino-rider vs. volcano image: Jesse Star / http://www.geekologie.com/ 2009/10/dinorider_geekologie_writer_vs.php
  30. Memory overhead (for setting numeric key / values) • 20mb overhead / 1 million records • 20 bytes / key
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×