• Save
In-memory Databases
Upcoming SlideShare
Loading in...5

In-memory Databases



This is from a 2 hour talk introducing in-memory databases. First a look at traditional RDBMS architecture and some of it's limitations, then a look at some in-memory products and finally a closer ...

This is from a 2 hour talk introducing in-memory databases. First a look at traditional RDBMS architecture and some of it's limitations, then a look at some in-memory products and finally a closer look at OrigoDB, the open source in-memory database toolkit for NET/Mono.



Total Views
Views on SlideShare
Embed Views



7 Embeds 307

http://robertfriberg.se 298
https://www.linkedin.com 2
http://www.linkedin.com 2
http://translate.googleusercontent.com 2
https://twitter.com 1
http://feedly.com 1
http://www.google.co.uk 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Explain the basic operations like insert, seek and scan
  • Explain the basics quickly.Talk about the boundaries of s.Ask: Is an RDBMS ACID? Answer on next slide.
  • Consistency and isolation are not binary.
  • Reporting.In-memory pushes the boundaries
  • Explain each of the bullets relating to previous topics.Recall slide ”What is a database”?
  • Great performance comes for free but could be optimized.
  • Some other frameworks based on or supporting write-ahead command logging and snapshots with a user defined in-memory model.
  • Defining a custom data model is what makes OrigoDB unique.

In-memory Databases In-memory Databases Presentation Transcript

  • About me ◦ Independent Developer and Trainer ◦ Sql Server DBA since 6.5 ◦ C#, javascript, perl, java, +++ ◦ Machine learning, AI ◦ Squash fanatic
  • Agenda ◦ Revisiting Traditional RDBMS ◦ Defining IMDB ◦ A look at a few in-memory products ◦ OrigoDB in depth ◦ Goals ◦ Learn technical stuff ◦ Thinking different
  • What is a database? ◦ An organized collection of information ◦ Allows reading and writing ◦ Provides authorization and authentication ◦ Provides some level of data safety
  • Demand drives change ◦ Performance ◦ Data volume ◦ Scalability ◦ Availability ◦ Modeling • NoSQL • Big data • Graph • Real time analytics • In-memory computing • Column stores One size no longer fits all
  • B-TREE data structure
  • B-trees and Transactions LOG DATA 64KB blocks w 8x8KB pages Logical BTREE of 8kb data pages In the buffer pool (cache) Buffer Manager Transactions append inserted, deleted, original and modified pages to the LOG • Fill factor • Page splits • Clustered index • Checkpoint
  • Transactions A Atomic C Consistent I Isolated D Durable s0 s1 s2t1 t2 What is s?
  • Isolation Levels ◦ SERIALIZABLE ◦ REPEATABLE_READ ◦ READ_COMMITTED_SNAPSHOT (MVCC) (Row versioning) ◦ READ_COMMITTED ◦ READ_UNCOMMITTED – dont worry, be happy consistency performance
  • “the B-tree is optimized for systems that read and write large blocks of data” - Wikipedia
  • The Traditional RDBMS Architecture ”.. is obsolete” -Michael Stonebraker Reference: OLTP through the looking glass, Stonebraker et al
  • OLTP vs. OLAP mismatch Read load OLAP Read intensive, touches a lot of data, benefits from indexes Write load OLTP Write intensive - Small writes - small reads - hot spots Indexes hurt write performance In-memory Disk-based
  • What is an in-memory database? ◦ PRIMARY representation is in-memory ◦ Memory optimized data structures ◦ ALL the data in memory (possibly distributed) (in-memory is not necessarily in-process)
  • Transaction logging ◦ Write Ahead Logging – write to disk before commit ◦ Effect logging – persist the effected datapages ◦ Command logging – persist the cause
  • IMDB Applications ◦ Real time applications with no durability requirements ◦ Embedded, router, online gaming ◦ Real time applications with durability requirements, low latency, high throughput ◦ Traditional applications during test and development (and production) ◦ Whenever data fits in RAM or can be distributed ◦ General OLTP replacement when DB < 2TB
  • Some In-memory Products memcached In-memory computing
  • SQL Server Hekaton ◦ Memory optimized tree structure ◦ Almost Lock-free Mvcc concurrency control ◦ Command logging ◦ Seamlessly Integrated in the traditional model ◦ Indexing ◦ Joins ◦ Querying
  • redis ◦ Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. ◦ Extremely popular and widespread (twitter, flicker, github, digg, disqus, Instagram, stackoverflow) ◦ Written in C, great performance
  • Comparison Matrix Product License Datamodel Interface ACID Distributed Concurrency Control VoltDB OSS Relational Java/sql yes Yes (2PC) Serialized memsql $$ Relational SQL Almost Yes Mvcc aerospike $$ Key/value many yes Yes(2PC) CAS SQL Server $$ Relational + T-SQL Yes (no) No Locking, mvcc NuoDB $$ Relational SQL Yes Hazelcast OSS Key/value+ java Almost Yes (2PC) Gridgain OSS Key/value Java,sql Yes Yes (2PC) mvcc Origodb OSS + User defined NET/REST Yes No Master/slave Serialized + Redis OSS Key/value + Many/LUA Yes No Master/Slave Serialized
  • OrigoDB ◦ Is it a database? (first name was Livedomain) ◦ Database Toolkit - Define your own datamodel ◦ Write ahead command logging + snapshots ◦ Single writer + multiple reader concurrency (serialized) ◦ Open source embedded engine ◦ 100% ACID ◦ Commercial server with master/slave replication
  • Design goals ◦Simplicity and correctness before performance ◦Flexibility ◦Rapid development
  • Modular Kernels ◦ Optimistic Kernel – write ahead logging assumes command will succeed ◦ Royal Food Taster – 2 identical in-memory models ◦ Immutability Kernel – Lock free, writers don’t block readers ◦ Requires immutable model
  • Evolution of OrigoDB ◦ File formats for document-oriented Desktop applications (java 1996) ◦ Cache invalidation experiments ◦ In-memory search indexes (offline built snapshots -> live updates) ◦ Inception: lambda expressions (NET 2008)
  • Cousins of OrigoDB ◦ Prevayler (java) ◦ Bamboo (net) ◦ Madeleine (ruby) ◦ Perlvayler ◦ Twisted-python ◦ Lmax ◦ Java-chronicle
  • Bring your own data model ◦ Generic models = Extra schema + mapping is complex so why? ◦ Relational ◦ Key/Value (value is a blob) ◦ Document (document is structured and queryable) ◦ Graph, nodes and edges ◦ Domain specific models ◦ OO Domain model (DDD) (typed graph) ◦ Javascript V8 environment (persisted node.js) ◦ Machine learning models (Accord.NET) ◦ Lucene.NET indexes
  • Demo time! ◦ TODO example – Anemic model, transaction script pattern (fat commands) ◦ Twitter clone – rich model with proxy, no commands ◦ Geekstream http://geekstream.devrexlabs.com/ ◦ OrigoDB Server http://origodb.com/
  • Last words ◦ Times are changing! Embrace! ◦ One size does not fit all – go polyglot persistence! ◦ Choose the most appropriate data model ◦ If data fits in RAM go in-memory! Thank you! robert@devrexlabs.com @robertfriberg