A tiny & fast key value store with append-only disk log. Ideal for apps with < 1 million records.

1. Dirty NoSQL
How simple is your database?
25.09.2010

2. About
Contributor Co-founder
Felix Geisendörfer
node.js driver node.js driver

4. Dirty

5. Dirty
JavaScript views In-memory
Dirty

6. Design choices

7. Non-blocking I/O FTW

8. 150 lines of code
Lines of code
Dirty
CouchDB
Redis
0 5.000 10.000 15.000 20.000

9. Append-only JSON log
$ cat dirty.db
{"key": "first", "val": {"foo": "bar"}}
{"key": "second", "val": "A string"}
{"key": "a-number", "val": 23}

10. No Networking
var db = require('dirty')('test.db');
db.set('foo', 'bar');
db.get('foo'); // => bar

11. No Networking
var db = require('dirty')('languages.db');
db.set('javascript', {age: 15});
db.set('python', {age: 19});
db.set('perl', {age: 23});
db.forEach(function(key, doc) {
console.log('%s is %d years old.', key, doc.age);
});

12. Performance

13. Benchmarks
Do your own!

14. dirty.get()
50 MHz
(50 million / s)
v8: 160 MHz

15. dirty.set()
5 MHz
(5 million / s)
v8: 12 MHz

16. dirty.set()
With flushing to disk
Numbers 256 byte string
200 kHz 70 kHz
(200-thousand / s) (70-thousand / s)

17. dirty.forEach()
4.5 MHz
(4.5 million / s)
v8: 33 MHz

18. The Wall
• Dirty is a wonderful database as long as
you have < 1 million records
• After that, you hit “The Wall”

19. Scaling beyond
Possibilities for future node.js based databases

20. 22.000 2010
The Internet
20.000
18.000
16.000
2009
14.000
12.000
PB / month
2008
10.000
8000
2007
6000
2006
4000
2005
2000 2004
2003
2002
2000
0
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year

21. 22 exabyte / month
(that is 22 billion gigabytes)

22. Flexible guarantees
db.set('my-key', 'my-value', function(err) {
if (err) throw err;
console.log('Record written to disk.');
});
console.log('Record written to memory.');

23. Memory / Disk Hybrids
• “Memcached” built into your database
• Better knowledge about your data than any
general purpose algorithm

24. Replication
• Node.js = perfect for streaming between
instances
• Node could “hold” the connection if a not-
yet replicated key is being requested

25. Web Services
• Node.js could act as a proxy for different
database backends (written in node or not)
• One could query 3rd party services for
information

26. Questions?
✎ @felixge / felix@debuggable.com

27. Get it
$ npm install dirty
or
http://github.com/felixge/node-dirty

29. Don’t hit the wall

30. Questions?
✎ @felixge / felix@debuggable.com

31. Sources
• Internet traffic numbers: http://en.wikipedia.org/wiki/Internet_traffic
• Wonderful dino-rider vs. volcano image: Jesse Star / http://www.geekologie.com/
2009/10/dinorider_geekologie_writer_vs.php

32. Memory overhead
(for setting numeric key / values)
• 20mb overhead / 1 million records
• 20 bytes / key