1. OLTP-Bench Framework: An Extensible Testbed for
3 September 2014
VLDB14, Hangzhou, CHINA
Benchmarking Relational Databases
Djellel Eddine Difallah, Andy Pavlo, Carlo Curino, Philippe Cudré-Mauroux

2. How many researchers wasted
time writing benchmarking
infrastructure to run their
experiments?
Measuring Reproducibility in Computer Systems Research – Collberg et al. (20% success)

3. Experimental
Research Process
We continuously re-invent the
wheel.
Development
Create the prototype
Search for Benchmarks (ad-hoc workloads)
Search for Datasets (synthetic data)
Building a test toolkit
Measure
Monitor different resources
Analysis
Plot and make sense of the results

4. OLTP-Bench is an open-source
“batteries-included” DBMS
benchmarking testbed tailored
for OLTP/Web workloads

5. OLTP-Bench
16 Workloads
Tested on 8 DBMSs
Rich metrics
Fine-grained Rate control
Workload Mixture
Extensible

6. OLTP-Bench Architecture
● Tight and dynamic control on parallel load generation
● Statistics gathering
● SQL dialect handling
Parallel Connections vs Performance

7. Current Workloads and Benchmarks
Class Benchmark Application Domain
Transactional
AuctionMark
CH-benCHmark
SEATS
TATP
TPC-C
Voter
SmallBank
On-line Auctions
Mixture of OLTP and OLAP
On-line Airline Ticketing
Caller Location App
Order Processing
Talent Show Voting
Banking System
Web-Oriented
Epinions
LinkBench
Twitter
Wikipedia
Social Networking
Social Networking
Social Networking
On-line Encyclopedia
Feature Testing
ResourceStresser
YCSB
JPAB
SIBench
Isolated Resource Stresser
Scalable Key-value Store
Object-Relational Mapping
Transactional Isolation
JDBC Compliant DBMSs
MS SQL Server
Oracle
MySQL
Postgres
DB2
NuoDB
SQLite
Apache Derby
MonetDB

8. Features
showcase

9. Fine-Grained Rate
Control
MySQL running Wikipedia
workload at increasing
throughput. Demonstrating tight
control of transactional
throughput imposed on the
system (and saturation)

10. Simulating Evolving
Skew
MySQL running Twitter with an
evolving skew (alternating zipfian
and uniform). Different skew
imposes stress on different
resources.

11. Multi-tenancy
Comparing 3 relational databases under multi-tenancy resource contention. (No DBMS
is perfect)

12. Changing Mixture over time
MySQL running YCSB with a varying transaction mixture
(highlighting various resource bottlenecks)
Parallel Connections vs Performance

13. Future Work
“We explicitly avoid to define benchmark rules, as they are (often) restrictive,
arbitrary, biased, and not-future-proof”
- Djellel, Andy, Carlo, Philippe
The project received an NSF Grant!!
Possible directions:
● Better tooling (e.g. real-time visualization)
● Synchronized distributed clients
● Support for NoSQL
● A public repository for experiments and results

14. Q&A
https://github.com/oltpbenchmark/
External contributors and acknowledgements:
Evan Jones, Barzan Mozfari, Dimitri Vorona, Ben Reilly, Yu Su, Adam Seering, Simon Krenger, Tommy Reilly, Mark Callaghan, Keving Grittner, David L.
Day, Nik Lanham, Awajeet Arya, Flavio Sousa, Stein Petter Tronstad, Lance FAng, Rodrigo Felix de Almeida, Woonhak kang, Florian Funke, Ahmad
Hassan, Ivo Jimenez,Mickaël Hoerdt, Dimokritos Stamatakis, Benjamin Reilly, Mark Callaghan, Zheng Da, Xi Chen, Jean de Lavarene, Ning Zhang, Erik
Paulson, Alec Jindal, ADD YOUR NAME HERE