Tailored database
solutions with Voron
Oren Eini
ayende@ayende.com ayende.com/blog
Hibernating Rhinos
Let’s implement a database
 Time series database, but what are those?
 Heart rate monitor watch.
 Every 30 second sends:
 Watch Id, Time, Heart Rate
 You sold 100,000 of those.
 Users want to log in and see their stats.
Traffic
 Per watch:
 2,880
events per day
 89,280
events per month
 1,051,200
events per year
 Globally (100K watches):
 288,000,000
events per day
 8,928,000,000
events per month
 105,120,000,000
events per year
Just under 3,500 events per second, sustained
100 billion events / year
 How do we approach this?
Design…
 Each watch has an id (string, up to 40 chars)
 We model each watch as a tree in Voron
 The tree uses:
 Big Endian date for key
 Heart rate number as value
 Let us go and build this…
Let us build this
Great, now about those reads
 What kind of reads we need?
 Always on a specific watch. For a given date range.
Let us build this
What is left?
 Real performance testing…
 Concurrent reads & writes
 Connecting to a server…
 Dropping old data / consolidating it.
Questions?

Tailored database solutions: Building a 100 billion records db in 20 minutes

  • 1.
    Tailored database solutions withVoron Oren Eini ayende@ayende.com ayende.com/blog Hibernating Rhinos
  • 2.
    Let’s implement adatabase  Time series database, but what are those?  Heart rate monitor watch.  Every 30 second sends:  Watch Id, Time, Heart Rate  You sold 100,000 of those.  Users want to log in and see their stats.
  • 3.
    Traffic  Per watch: 2,880 events per day  89,280 events per month  1,051,200 events per year  Globally (100K watches):  288,000,000 events per day  8,928,000,000 events per month  105,120,000,000 events per year Just under 3,500 events per second, sustained
  • 4.
    100 billion events/ year  How do we approach this?
  • 5.
    Design…  Each watchhas an id (string, up to 40 chars)  We model each watch as a tree in Voron  The tree uses:  Big Endian date for key  Heart rate number as value  Let us go and build this…
  • 6.
  • 7.
    Great, now aboutthose reads  What kind of reads we need?  Always on a specific watch. For a given date range.
  • 8.
  • 9.
    What is left? Real performance testing…  Concurrent reads & writes  Connecting to a server…  Dropping old data / consolidating it.
  • 10.