SlideShare a Scribd company logo
1 of 32
Download to read offline
Dirty NoSQL
How simple is your database?




                               25.09.2010
About

Contributor                            Co-founder




                 Felix Geisendörfer

node.js driver                        node.js driver
Dirty
Dirty



JavaScript views            In-memory


                    Dirty
Design choices
Non-blocking I/O FTW
150 lines of code
                          Lines of code



   Dirty



CouchDB



   Redis


           0      5.000      10.000       15.000   20.000
Append-only JSON log


$ cat dirty.db

{"key": "first", "val": {"foo": "bar"}}
{"key": "second", "val": "A string"}
{"key": "a-number", "val": 23}
No Networking


var db = require('dirty')('test.db');

db.set('foo', 'bar');
db.get('foo'); // => bar
No Networking

var db = require('dirty')('languages.db');

db.set('javascript', {age: 15});
db.set('python', {age: 19});
db.set('perl', {age: 23});

db.forEach(function(key, doc) {
  console.log('%s is %d years old.', key, doc.age);
});
Performance
Benchmarks


Do your own!
dirty.get()


50 MHz
 (50 million / s)




                    v8: 160 MHz
dirty.set()


5 MHz
 (5 million / s)




                   v8: 12 MHz
dirty.set()
               With flushing to disk




     Numbers                          256 byte string

200 kHz 70 kHz
(200-thousand / s)              (70-thousand / s)
dirty.forEach()


 4.5 MHz
   (4.5 million / s)




                       v8: 33 MHz
The Wall

• Dirty is a wonderful database as long as
  you have < 1 million records


• After that, you hit “The Wall”
Scaling beyond
Possibilities for future node.js based databases
22.000                                                                                         2010




                                           The Internet
             20.000




             18.000




             16.000


                                                                                                  2009

             14.000




             12.000
PB / month




                                                                                        2008
             10.000




              8000


                                                                              2007
              6000


                                                                     2006
              4000

                                                            2005

              2000                                 2004
                                            2003
                             2002
                      2000
                 0
                 2000        2001   2002    2003     2004     2005     2006      2007      2008      2009   2010

                                                              Year
22 exabyte / month
  (that is 22 billion gigabytes)
Flexible guarantees

db.set('my-key', 'my-value', function(err) {
  if (err) throw err;

  console.log('Record written to disk.');
});

console.log('Record written to memory.');
Memory / Disk Hybrids

• “Memcached” built into your database

• Better knowledge about your data than any
  general purpose algorithm
Replication

• Node.js = perfect for streaming between
  instances


• Node could “hold” the connection if a not-
  yet replicated key is being requested
Web Services

• Node.js could act as a proxy for different
  database backends (written in node or not)


• One could query 3rd party services for
  information
Questions?




✎   @felixge / felix@debuggable.com
Get it

        $ npm install dirty
                 or

http://github.com/felixge/node-dirty
Don’t hit the wall
Questions?




✎   @felixge / felix@debuggable.com
Sources
•   Internet traffic numbers: http://en.wikipedia.org/wiki/Internet_traffic

•   Wonderful dino-rider vs. volcano image: Jesse Star / http://www.geekologie.com/
    2009/10/dinorider_geekologie_writer_vs.php
Memory overhead
        (for setting numeric key / values)




• 20mb overhead / 1 million records

• 20 bytes / key

More Related Content

What's hot

MongoDB Shell Tips & Tricks
MongoDB Shell Tips & TricksMongoDB Shell Tips & Tricks
MongoDB Shell Tips & Tricks
MongoDB
 
Shell Tips & Tricks
Shell Tips & TricksShell Tips & Tricks
Shell Tips & Tricks
MongoDB
 
Kicking ass with redis
Kicking ass with redisKicking ass with redis
Kicking ass with redis
Dvir Volk
 

What's hot (20)

Node.js in production
Node.js in productionNode.js in production
Node.js in production
 
Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)
 
Node.js - A Quick Tour
Node.js - A Quick TourNode.js - A Quick Tour
Node.js - A Quick Tour
 
MongoDB Shell Tips & Tricks
MongoDB Shell Tips & TricksMongoDB Shell Tips & Tricks
MongoDB Shell Tips & Tricks
 
Shell Tips & Tricks
Shell Tips & TricksShell Tips & Tricks
Shell Tips & Tricks
 
Getting Started with MongoDB
Getting Started with MongoDBGetting Started with MongoDB
Getting Started with MongoDB
 
Kicking ass with redis
Kicking ass with redisKicking ass with redis
Kicking ass with redis
 
MongoDB's New Aggregation framework
MongoDB's New Aggregation frameworkMongoDB's New Aggregation framework
MongoDB's New Aggregation framework
 
Nginx-lua
Nginx-luaNginx-lua
Nginx-lua
 
A language for the Internet: Why JavaScript and Node.js is right for Internet...
A language for the Internet: Why JavaScript and Node.js is right for Internet...A language for the Internet: Why JavaScript and Node.js is right for Internet...
A language for the Internet: Why JavaScript and Node.js is right for Internet...
 
Openstack 簡介
Openstack 簡介Openstack 簡介
Openstack 簡介
 
Introduction to Nodejs
Introduction to NodejsIntroduction to Nodejs
Introduction to Nodejs
 
Building the Enterprise infrastructure with PostgreSQL as the basis for stori...
Building the Enterprise infrastructure with PostgreSQL as the basis for stori...Building the Enterprise infrastructure with PostgreSQL as the basis for stori...
Building the Enterprise infrastructure with PostgreSQL as the basis for stori...
 
Redis - Usability and Use Cases
Redis - Usability and Use CasesRedis - Usability and Use Cases
Redis - Usability and Use Cases
 
Node.js - Best practices
Node.js  - Best practicesNode.js  - Best practices
Node.js - Best practices
 
Redis in Practice: Scenarios, Performance and Practice with PHP
Redis in Practice: Scenarios, Performance and Practice with PHPRedis in Practice: Scenarios, Performance and Practice with PHP
Redis in Practice: Scenarios, Performance and Practice with PHP
 
Event-driven IO server-side JavaScript environment based on V8 Engine
Event-driven IO server-side JavaScript environment based on V8 EngineEvent-driven IO server-side JavaScript environment based on V8 Engine
Event-driven IO server-side JavaScript environment based on V8 Engine
 
Introduction to redis - version 2
Introduction to redis - version 2Introduction to redis - version 2
Introduction to redis - version 2
 
Server-Side JavaScript Developement - Node.JS Quick Tour
Server-Side JavaScript Developement - Node.JS Quick TourServer-Side JavaScript Developement - Node.JS Quick Tour
Server-Side JavaScript Developement - Node.JS Quick Tour
 
Non-blocking I/O, Event loops and node.js
Non-blocking I/O, Event loops and node.jsNon-blocking I/O, Event loops and node.js
Non-blocking I/O, Event loops and node.js
 

Similar to Dirty - How simple is your database?

Why Node.js
Why Node.jsWhy Node.js
Why Node.js
guileen
 
Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)
Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)
Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)
Ontico
 
Varnish, The Good, The Awesome, and the Downright Crazy
Varnish, The Good, The Awesome, and the Downright CrazyVarnish, The Good, The Awesome, and the Downright Crazy
Varnish, The Good, The Awesome, and the Downright Crazy
Mike Willbanks
 
Varnish, The Good, The Awesome, and the Downright Crazy.
Varnish, The Good, The Awesome, and the Downright Crazy.Varnish, The Good, The Awesome, and the Downright Crazy.
Varnish, The Good, The Awesome, and the Downright Crazy.
Mike Willbanks
 

Similar to Dirty - How simple is your database? (20)

Virtual Box Aquarium May09
Virtual Box Aquarium May09Virtual Box Aquarium May09
Virtual Box Aquarium May09
 
From concept to cloud a look at modern software development
From concept to cloud a look at modern software developmentFrom concept to cloud a look at modern software development
From concept to cloud a look at modern software development
 
Squeak DBX
Squeak DBXSqueak DBX
Squeak DBX
 
Web Development Using Microsoft Visual Studio: Now and in the Future
Web Development Using Microsoft Visual Studio: Now and in the FutureWeb Development Using Microsoft Visual Studio: Now and in the Future
Web Development Using Microsoft Visual Studio: Now and in the Future
 
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalabilityIPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
 
Why Nodejs Guilin Shanghai
Why Nodejs Guilin ShanghaiWhy Nodejs Guilin Shanghai
Why Nodejs Guilin Shanghai
 
Why Node.js
Why Node.jsWhy Node.js
Why Node.js
 
Cloud Computing: Is it really new?
Cloud Computing: Is it really new?Cloud Computing: Is it really new?
Cloud Computing: Is it really new?
 
Modernización del manejo de datos con v fabric
Modernización del manejo de datos con v fabricModernización del manejo de datos con v fabric
Modernización del manejo de datos con v fabric
 
Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)
Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)
Database sharding the right way: еasy, reliable, and open source (Esen Sagynov)
 
NoSQL databases and managing big data
NoSQL databases and managing big dataNoSQL databases and managing big data
NoSQL databases and managing big data
 
Desktop Private Cloud
Desktop Private CloudDesktop Private Cloud
Desktop Private Cloud
 
Above the cloud: Big Data and BI
Above the cloud: Big Data and BIAbove the cloud: Big Data and BI
Above the cloud: Big Data and BI
 
Powering the Next Generation Services with Java Platform - Spark IT 2010
Powering the Next Generation Services with Java Platform - Spark IT 2010Powering the Next Generation Services with Java Platform - Spark IT 2010
Powering the Next Generation Services with Java Platform - Spark IT 2010
 
MongoDB, Hadoop and humongous data - MongoSV 2012
MongoDB, Hadoop and humongous data - MongoSV 2012MongoDB, Hadoop and humongous data - MongoSV 2012
MongoDB, Hadoop and humongous data - MongoSV 2012
 
Varnish, The Good, The Awesome, and the Downright Crazy
Varnish, The Good, The Awesome, and the Downright CrazyVarnish, The Good, The Awesome, and the Downright Crazy
Varnish, The Good, The Awesome, and the Downright Crazy
 
Varnish, The Good, The Awesome, and the Downright Crazy.
Varnish, The Good, The Awesome, and the Downright Crazy.Varnish, The Good, The Awesome, and the Downright Crazy.
Varnish, The Good, The Awesome, and the Downright Crazy.
 
Developing realtime apps with Drupal and NodeJS
Developing realtime apps with Drupal and NodeJS Developing realtime apps with Drupal and NodeJS
Developing realtime apps with Drupal and NodeJS
 
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
 
The Megasite: Infrastructure for Internet Scale
The Megasite: Infrastructure for Internet ScaleThe Megasite: Infrastructure for Internet Scale
The Megasite: Infrastructure for Internet Scale
 

Recently uploaded

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Cyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfCyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdf
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 

Dirty - How simple is your database?