SlideShare a Scribd company logo
RAVENDB EMBEDDED AT
MASSIVE SCALES
ABOUT ME
Rodrigo Rosauro, Software Architect Manager at RDI
• Technology passionate
• Developer
• 14 years of experience in software development
• Working with RavenDB since 2013
ABOUT RDI
• We make restaurant automation software for QSR
• Our software runs on more than 36K restaurants
• Estimated number of individual machines: around half a million
• Processing almost USD $50K per second of customer payments
“ONE IN A MILLION IS NEXT TUESDAY”
- Gordon Letwin, Architect for DOS 4
HOW WE USED TO PERSIST DATA ON POS
... and still do (on legacy modules)
LEGACY PERSISTENCY
• Flat files
• Custom data format
• Event sourcing persistence
• Rebuild in-memory state at
every restart
PROBLEMS WITH THAT APPROACH
•No querying – at all
•High complexity to store new data
•No standard data format
•Lack of management tools
WHY WE DECIDED TO USE A DBMS?
WHY A DBMS?
• We started a full re-architecture of the POS platform
• We are definitely not database/persistence specialists
• We wanted to remove complexity to persist data
THE SEEK FOR A DBMS
GOALS / GUIDELINES
• The new architecture is based on plug-ins, so we wanted each
individual plug-in to have its own, isolated, database
• Zero administration
• Easy schema upgrades
• Transparent replication
• Unit tests
WHY WE CHOSE RAVENDB?
WHY RAVENDB?
• .NET
• Embedded operation mode
• In-memory mode for unit tests
• Multi-tenant
• Good transparent replication options
WHAT RAVENDB ALLOWED US TO DO
RAVENDB AT RDI
• Create unit tests without mocking the database
• Have advanced statistics about the data persistence
• Both system-wide and per plug-in
• Transparent replication
• Hot failover, data distribution & real-time backups
• Transparent encryption of data at rest
THE CHALLENGES
... and how we faced them
CHALLENGE #1 – MEMORY CONSTRAINTS
SPECIALIZED, OLD HARDWARE
Old hardware (sometimes 10 years old) means that we have very little
memory and processing power.
MEMORY CONSTRAINTS
• Many iterations with the RavenDB support team to fine-tune its memory
usage
• We learned that under these constraints, caching can be evil
• In the end, the solution was to completely disable caching on RavenDB and
do some cache at the application side, only for the “hot” data.
CHALLENGE #2 – ESENT
WE ARE STILL USING RAVENDB 2.5
• Our software must support Windows XPe
• .NET 4.0
• 32 bits machines
• … this means that ESENT is our only option for now
FACT: WE DON’T LIKE ESENT
(at least not for our usage scenarios)
WHY?
• Any unclean shutdown may cause a completely undetermined result
• Power outages / Application crashes / Task kill
• The possible results are not easy to find out during regular testing
• Crappy hardware doesn’t help
• We still face sporadic full database losses with ESENT
• Copying data between OS versions is awful
• Recovering from most unclean shutdowns requires using ESENTUTL.EXE
ESENTUTL.EXE
• We also hate don’t like ESENTUTL.EXE
• Many different commands to attempt to recover from different kinds of
unclean shutdowns. Some have different syntax on different OS versions
• We had to automate all that (zero maintenance, remember?)
• Almost 500 lines of code
“SUCCESS IS STUMBLING FROM FAILURE TO
FAILURE WITH NO LOSS OF ENTHUSIASM”
― Winston S. Churchill, Ex-prime minister of UK
THE FUTURE
THE FUTURE
• Working with Hibernating Rhinos to improve Voron to our use case
• 32 bits support
• Better reliability on unclean shutdowns
• We may jump straight to RavenDB 4.0 & CoreCLR
• Long-term target: Around 250K individual POS nodes running RavenDB
embedded
THANK YOU VERY MUCH

More Related Content

What's hot

Moving to the Cloud: AWS, Zend, RightScale
Moving to the Cloud: AWS, Zend, RightScaleMoving to the Cloud: AWS, Zend, RightScale
Moving to the Cloud: AWS, Zend, RightScalemmoline
 
Stack Exchange Infrastructure - LISA 14
Stack Exchange Infrastructure - LISA 14Stack Exchange Infrastructure - LISA 14
Stack Exchange Infrastructure - LISA 14
GABeech
 
Drupal High Availability High Performance 2012
Drupal High Availability High Performance 2012Drupal High Availability High Performance 2012
Drupal High Availability High Performance 2012Amazee Labs
 
Why ruby and rails
Why ruby and railsWhy ruby and rails
Why ruby and rails
Reuven Lerner
 
SPA vs. MPA
SPA vs. MPASPA vs. MPA
SPA vs. MPA
Mehmet Ali Tastan
 
In Memory Cahce Structure
In Memory Cahce StructureIn Memory Cahce Structure
In Memory Cahce Structure
Mehmet Ali Tastan
 
Ruby performance - The low hanging fruit
Ruby performance - The low hanging fruitRuby performance - The low hanging fruit
Ruby performance - The low hanging fruit
Bruce Werdschinski
 
Store
StoreStore
Store
ESUG
 
O365Con19 - Sharing Code Efficiently in your Organisation - Elio Struyf
O365Con19 - Sharing Code Efficiently in your Organisation - Elio StruyfO365Con19 - Sharing Code Efficiently in your Organisation - Elio Struyf
O365Con19 - Sharing Code Efficiently in your Organisation - Elio Struyf
NCCOMMS
 
Aws 12 Month Free Tier for Web Designers and Developers
Aws 12 Month Free Tier for Web Designers and DevelopersAws 12 Month Free Tier for Web Designers and Developers
Aws 12 Month Free Tier for Web Designers and Developers
Dylan Burris
 
How & When to Use NoSQL at Websummit Dublin
How & When to Use NoSQL at Websummit DublinHow & When to Use NoSQL at Websummit Dublin
How & When to Use NoSQL at Websummit Dublin
Amazon Web Services
 
Apcera Case Study: The selection of the Go language
Apcera Case Study: The selection of the Go languageApcera Case Study: The selection of the Go language
Apcera Case Study: The selection of the Go language
Derek Collison
 
Building better SQL Server Databases
Building better SQL Server DatabasesBuilding better SQL Server Databases
Building better SQL Server Databases
ColdFusionConference
 
SQL Azure - the good, the bad and the ugly.
SQL Azure - the good, the bad and the ugly.SQL Azure - the good, the bad and the ugly.
SQL Azure - the good, the bad and the ugly.
Pini Krisher
 
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit ChopraI3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
SPS Paris
 
Introduction to CosmosDB - Azure Bootcamp 2018
Introduction to CosmosDB - Azure Bootcamp 2018Introduction to CosmosDB - Azure Bootcamp 2018
Introduction to CosmosDB - Azure Bootcamp 2018
Josh Carlisle
 
A Gentle Introduction to Functions-as-a-Service
A Gentle Introduction to Functions-as-a-ServiceA Gentle Introduction to Functions-as-a-Service
A Gentle Introduction to Functions-as-a-Service
Valeri Karpov
 
Drop acid
Drop acidDrop acid
Drop acid
Mike Feltman
 

What's hot (20)

Moving to the Cloud: AWS, Zend, RightScale
Moving to the Cloud: AWS, Zend, RightScaleMoving to the Cloud: AWS, Zend, RightScale
Moving to the Cloud: AWS, Zend, RightScale
 
Stack Exchange Infrastructure - LISA 14
Stack Exchange Infrastructure - LISA 14Stack Exchange Infrastructure - LISA 14
Stack Exchange Infrastructure - LISA 14
 
Drupal High Availability High Performance 2012
Drupal High Availability High Performance 2012Drupal High Availability High Performance 2012
Drupal High Availability High Performance 2012
 
Why ruby and rails
Why ruby and railsWhy ruby and rails
Why ruby and rails
 
SPA vs. MPA
SPA vs. MPASPA vs. MPA
SPA vs. MPA
 
In Memory Cahce Structure
In Memory Cahce StructureIn Memory Cahce Structure
In Memory Cahce Structure
 
Zend Framwork configurations
Zend Framwork configurationsZend Framwork configurations
Zend Framwork configurations
 
Ruby performance - The low hanging fruit
Ruby performance - The low hanging fruitRuby performance - The low hanging fruit
Ruby performance - The low hanging fruit
 
Store
StoreStore
Store
 
O365Con19 - Sharing Code Efficiently in your Organisation - Elio Struyf
O365Con19 - Sharing Code Efficiently in your Organisation - Elio StruyfO365Con19 - Sharing Code Efficiently in your Organisation - Elio Struyf
O365Con19 - Sharing Code Efficiently in your Organisation - Elio Struyf
 
Infrastructure as Code
Infrastructure as CodeInfrastructure as Code
Infrastructure as Code
 
Aws 12 Month Free Tier for Web Designers and Developers
Aws 12 Month Free Tier for Web Designers and DevelopersAws 12 Month Free Tier for Web Designers and Developers
Aws 12 Month Free Tier for Web Designers and Developers
 
How & When to Use NoSQL at Websummit Dublin
How & When to Use NoSQL at Websummit DublinHow & When to Use NoSQL at Websummit Dublin
How & When to Use NoSQL at Websummit Dublin
 
Apcera Case Study: The selection of the Go language
Apcera Case Study: The selection of the Go languageApcera Case Study: The selection of the Go language
Apcera Case Study: The selection of the Go language
 
Building better SQL Server Databases
Building better SQL Server DatabasesBuilding better SQL Server Databases
Building better SQL Server Databases
 
SQL Azure - the good, the bad and the ugly.
SQL Azure - the good, the bad and the ugly.SQL Azure - the good, the bad and the ugly.
SQL Azure - the good, the bad and the ugly.
 
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit ChopraI3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
I3 - Running SharePoint 2016 in Azure the do's and dont's - Jasjit Chopra
 
Introduction to CosmosDB - Azure Bootcamp 2018
Introduction to CosmosDB - Azure Bootcamp 2018Introduction to CosmosDB - Azure Bootcamp 2018
Introduction to CosmosDB - Azure Bootcamp 2018
 
A Gentle Introduction to Functions-as-a-Service
A Gentle Introduction to Functions-as-a-ServiceA Gentle Introduction to Functions-as-a-Service
A Gentle Introduction to Functions-as-a-Service
 
Drop acid
Drop acidDrop acid
Drop acid
 

Similar to RavenDB embedded at massive scales

BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...
BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...
BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...
Big Data Montreal
 
Serve like a boss (part one)
Serve like a boss (part one)Serve like a boss (part one)
Serve like a boss (part one)
Hamed Nemati
 
Scaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP
120bi
 
Scaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web Applications
Achievers Tech
 
Scaling and Distributing
Scaling and DistributingScaling and Distributing
Scaling and Distributing
Dima Malenko
 
Polyglot Persistence - Two Great Tastes That Taste Great Together
Polyglot Persistence - Two Great Tastes That Taste Great TogetherPolyglot Persistence - Two Great Tastes That Taste Great Together
Polyglot Persistence - Two Great Tastes That Taste Great Together
John Wood
 
The 5 Minute MySQL DBA
The 5 Minute MySQL DBAThe 5 Minute MySQL DBA
The 5 Minute MySQL DBA
Irawan Soetomo
 
Database Provisioning in EM12c: Provision me a Database Now!
Database Provisioning in EM12c: Provision me a Database Now!Database Provisioning in EM12c: Provision me a Database Now!
Database Provisioning in EM12c: Provision me a Database Now!
Maaz Anjum
 
Scaling apps for the big time
Scaling apps for the big timeScaling apps for the big time
Scaling apps for the big time
proitconsult
 
High performance PHP: Scaling and getting the most out of your infrastructure
High performance PHP: Scaling and getting the most out of your infrastructureHigh performance PHP: Scaling and getting the most out of your infrastructure
High performance PHP: Scaling and getting the most out of your infrastructure
mkherlakian
 
Python performance profiling
Python performance profilingPython performance profiling
Python performance profiling
Jon Haddad
 
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey KharlamovRUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
Big Data Spain
 
DCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing Environment
DCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing EnvironmentDCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing Environment
DCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing Environment
Docker, Inc.
 
OSMC 2019 | How to improve database Observability by Charles Judith
OSMC 2019 | How to improve database Observability by Charles JudithOSMC 2019 | How to improve database Observability by Charles Judith
OSMC 2019 | How to improve database Observability by Charles Judith
NETWAYS
 
Cassandra o que nao te contaram
Cassandra   o que nao te contaramCassandra   o que nao te contaram
Cassandra o que nao te contaram
Rafael Monteiro e Pereira
 
eHarmony in the Cloud
eHarmony in the CloudeHarmony in the Cloud
eHarmony in the Cloud
Craig Dickson
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPrem
Kris Buytaert
 
Developing a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure EnvironmentsDeveloping a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure Environments
Ceph Community
 
Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18
BIWUG
 
Big data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymoreBig data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymore
Stfalcon Meetups
 

Similar to RavenDB embedded at massive scales (20)

BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...
BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...
BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...
 
Serve like a boss (part one)
Serve like a boss (part one)Serve like a boss (part one)
Serve like a boss (part one)
 
Scaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP
 
Scaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web Applications
 
Scaling and Distributing
Scaling and DistributingScaling and Distributing
Scaling and Distributing
 
Polyglot Persistence - Two Great Tastes That Taste Great Together
Polyglot Persistence - Two Great Tastes That Taste Great TogetherPolyglot Persistence - Two Great Tastes That Taste Great Together
Polyglot Persistence - Two Great Tastes That Taste Great Together
 
The 5 Minute MySQL DBA
The 5 Minute MySQL DBAThe 5 Minute MySQL DBA
The 5 Minute MySQL DBA
 
Database Provisioning in EM12c: Provision me a Database Now!
Database Provisioning in EM12c: Provision me a Database Now!Database Provisioning in EM12c: Provision me a Database Now!
Database Provisioning in EM12c: Provision me a Database Now!
 
Scaling apps for the big time
Scaling apps for the big timeScaling apps for the big time
Scaling apps for the big time
 
High performance PHP: Scaling and getting the most out of your infrastructure
High performance PHP: Scaling and getting the most out of your infrastructureHigh performance PHP: Scaling and getting the most out of your infrastructure
High performance PHP: Scaling and getting the most out of your infrastructure
 
Python performance profiling
Python performance profilingPython performance profiling
Python performance profiling
 
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey KharlamovRUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
 
DCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing Environment
DCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing EnvironmentDCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing Environment
DCSF19 Transforming a 15+ Year Old Semiconductor Manufacturing Environment
 
OSMC 2019 | How to improve database Observability by Charles Judith
OSMC 2019 | How to improve database Observability by Charles JudithOSMC 2019 | How to improve database Observability by Charles Judith
OSMC 2019 | How to improve database Observability by Charles Judith
 
Cassandra o que nao te contaram
Cassandra   o que nao te contaramCassandra   o que nao te contaram
Cassandra o que nao te contaram
 
eHarmony in the Cloud
eHarmony in the CloudeHarmony in the Cloud
eHarmony in the Cloud
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPrem
 
Developing a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure EnvironmentsDeveloping a Ceph Appliance for Secure Environments
Developing a Ceph Appliance for Secure Environments
 
Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18Pascal benois performance_troubleshooting-spsbe18
Pascal benois performance_troubleshooting-spsbe18
 
Big data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymoreBig data & frameworks: no book for you anymore
Big data & frameworks: no book for you anymore
 

More from Oren Eini

Staying friendly with the gc
Staying friendly with the gcStaying friendly with the gc
Staying friendly with the gc
Oren Eini
 
Rebooting design in RavenDB
Rebooting design in RavenDBRebooting design in RavenDB
Rebooting design in RavenDB
Oren Eini
 
Implementing CQRS and Event Sourcing with RavenDB
Implementing CQRS and Event Sourcing with RavenDBImplementing CQRS and Event Sourcing with RavenDB
Implementing CQRS and Event Sourcing with RavenDB
Oren Eini
 
Should I use a document database?
Should I use a document database?Should I use a document database?
Should I use a document database?
Oren Eini
 
Delving into Documents with Data Subscriptions
Delving into Documents with Data SubscriptionsDelving into Documents with Data Subscriptions
Delving into Documents with Data Subscriptions
Oren Eini
 
Building Codealike: a journey into the developers analytics world
Building Codealike: a journey into the developers analytics worldBuilding Codealike: a journey into the developers analytics world
Building Codealike: a journey into the developers analytics world
Oren Eini
 
RavenDB for modern web apps
RavenDB for modern web appsRavenDB for modern web apps
RavenDB for modern web apps
Oren Eini
 

More from Oren Eini (7)

Staying friendly with the gc
Staying friendly with the gcStaying friendly with the gc
Staying friendly with the gc
 
Rebooting design in RavenDB
Rebooting design in RavenDBRebooting design in RavenDB
Rebooting design in RavenDB
 
Implementing CQRS and Event Sourcing with RavenDB
Implementing CQRS and Event Sourcing with RavenDBImplementing CQRS and Event Sourcing with RavenDB
Implementing CQRS and Event Sourcing with RavenDB
 
Should I use a document database?
Should I use a document database?Should I use a document database?
Should I use a document database?
 
Delving into Documents with Data Subscriptions
Delving into Documents with Data SubscriptionsDelving into Documents with Data Subscriptions
Delving into Documents with Data Subscriptions
 
Building Codealike: a journey into the developers analytics world
Building Codealike: a journey into the developers analytics worldBuilding Codealike: a journey into the developers analytics world
Building Codealike: a journey into the developers analytics world
 
RavenDB for modern web apps
RavenDB for modern web appsRavenDB for modern web apps
RavenDB for modern web apps
 

Recently uploaded

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Generating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using SmithyGenerating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using Smithy
g2nightmarescribd
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 

Recently uploaded (20)

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Generating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using SmithyGenerating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using Smithy
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 

RavenDB embedded at massive scales

  • 2. ABOUT ME Rodrigo Rosauro, Software Architect Manager at RDI • Technology passionate • Developer • 14 years of experience in software development • Working with RavenDB since 2013
  • 3. ABOUT RDI • We make restaurant automation software for QSR • Our software runs on more than 36K restaurants • Estimated number of individual machines: around half a million • Processing almost USD $50K per second of customer payments
  • 4. “ONE IN A MILLION IS NEXT TUESDAY” - Gordon Letwin, Architect for DOS 4
  • 5. HOW WE USED TO PERSIST DATA ON POS ... and still do (on legacy modules)
  • 6. LEGACY PERSISTENCY • Flat files • Custom data format • Event sourcing persistence • Rebuild in-memory state at every restart
  • 7. PROBLEMS WITH THAT APPROACH •No querying – at all •High complexity to store new data •No standard data format •Lack of management tools
  • 8. WHY WE DECIDED TO USE A DBMS?
  • 9. WHY A DBMS? • We started a full re-architecture of the POS platform • We are definitely not database/persistence specialists • We wanted to remove complexity to persist data
  • 10. THE SEEK FOR A DBMS
  • 11. GOALS / GUIDELINES • The new architecture is based on plug-ins, so we wanted each individual plug-in to have its own, isolated, database • Zero administration • Easy schema upgrades • Transparent replication • Unit tests
  • 12. WHY WE CHOSE RAVENDB?
  • 13. WHY RAVENDB? • .NET • Embedded operation mode • In-memory mode for unit tests • Multi-tenant • Good transparent replication options
  • 15. RAVENDB AT RDI • Create unit tests without mocking the database • Have advanced statistics about the data persistence • Both system-wide and per plug-in • Transparent replication • Hot failover, data distribution & real-time backups • Transparent encryption of data at rest
  • 16. THE CHALLENGES ... and how we faced them
  • 17. CHALLENGE #1 – MEMORY CONSTRAINTS
  • 18. SPECIALIZED, OLD HARDWARE Old hardware (sometimes 10 years old) means that we have very little memory and processing power.
  • 19. MEMORY CONSTRAINTS • Many iterations with the RavenDB support team to fine-tune its memory usage • We learned that under these constraints, caching can be evil • In the end, the solution was to completely disable caching on RavenDB and do some cache at the application side, only for the “hot” data.
  • 21. WE ARE STILL USING RAVENDB 2.5 • Our software must support Windows XPe • .NET 4.0 • 32 bits machines • … this means that ESENT is our only option for now
  • 22. FACT: WE DON’T LIKE ESENT (at least not for our usage scenarios)
  • 23. WHY? • Any unclean shutdown may cause a completely undetermined result • Power outages / Application crashes / Task kill • The possible results are not easy to find out during regular testing • Crappy hardware doesn’t help • We still face sporadic full database losses with ESENT • Copying data between OS versions is awful • Recovering from most unclean shutdowns requires using ESENTUTL.EXE
  • 24. ESENTUTL.EXE • We also hate don’t like ESENTUTL.EXE • Many different commands to attempt to recover from different kinds of unclean shutdowns. Some have different syntax on different OS versions • We had to automate all that (zero maintenance, remember?) • Almost 500 lines of code
  • 25. “SUCCESS IS STUMBLING FROM FAILURE TO FAILURE WITH NO LOSS OF ENTHUSIASM” ― Winston S. Churchill, Ex-prime minister of UK
  • 27. THE FUTURE • Working with Hibernating Rhinos to improve Voron to our use case • 32 bits support • Better reliability on unclean shutdowns • We may jump straight to RavenDB 4.0 & CoreCLR • Long-term target: Around 250K individual POS nodes running RavenDB embedded

Editor's Notes

  1. Almost 150 million during this session
  2. https://blogs.msdn.microsoft.com/larryosterman/2004/03/30/one-in-a-million-is-next-tuesday/