Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

OLTPBenchmark is a multi-threaded load generator. The framework is designed to be able to produce variable rate, variable mixture load against any JDBC-enabled relational database. The framework also provides data collection features, e.g., per-transaction-type latency and throughput logs.

Together with the framework we provide the following OLTP/Web benchmarks:

Synthetic Resource Stresser
JPAB (Hibernate)
Voter (Japanese "American Idol")
SIBench (Snapshot Isolation)

Published in: Software
  • Be the first to comment

  • Be the first to like this


  1. 1. OLTP-Bench Framework: An Extensible Testbed for 3 September 2014 VLDB14, Hangzhou, CHINA Benchmarking Relational Databases Djellel Eddine Difallah, Andy Pavlo, Carlo Curino, Philippe Cudré-Mauroux
  2. 2. How many researchers wasted time writing benchmarking infrastructure to run their experiments? Measuring Reproducibility in Computer Systems Research – Collberg et al. (20% success)
  3. 3. Experimental Research Process We continuously re-invent the wheel. Development Create the prototype Search for Benchmarks (ad-hoc workloads) Search for Datasets (synthetic data) Building a test toolkit Measure Monitor different resources Analysis Plot and make sense of the results
  4. 4. OLTP-Bench is an open-source “batteries-included” DBMS benchmarking testbed tailored for OLTP/Web workloads
  5. 5. OLTP-Bench 16 Workloads Tested on 8 DBMSs Rich metrics Fine-grained Rate control Workload Mixture Extensible
  6. 6. OLTP-Bench Architecture ● Tight and dynamic control on parallel load generation ● Statistics gathering ● SQL dialect handling Parallel Connections vs Performance
  7. 7. Current Workloads and Benchmarks Class Benchmark Application Domain Transactional AuctionMark CH-benCHmark SEATS TATP TPC-C Voter SmallBank On-line Auctions Mixture of OLTP and OLAP On-line Airline Ticketing Caller Location App Order Processing Talent Show Voting Banking System Web-Oriented Epinions LinkBench Twitter Wikipedia Social Networking Social Networking Social Networking On-line Encyclopedia Feature Testing ResourceStresser YCSB JPAB SIBench Isolated Resource Stresser Scalable Key-value Store Object-Relational Mapping Transactional Isolation JDBC Compliant DBMSs MS SQL Server Oracle MySQL Postgres DB2 NuoDB SQLite Apache Derby MonetDB
  8. 8. Features showcase
  9. 9. Fine-Grained Rate Control MySQL running Wikipedia workload at increasing throughput. Demonstrating tight control of transactional throughput imposed on the system (and saturation)
  10. 10. Simulating Evolving Skew MySQL running Twitter with an evolving skew (alternating zipfian and uniform). Different skew imposes stress on different resources.
  11. 11. Multi-tenancy Comparing 3 relational databases under multi-tenancy resource contention. (No DBMS is perfect)
  12. 12. Changing Mixture over time MySQL running YCSB with a varying transaction mixture (highlighting various resource bottlenecks) Parallel Connections vs Performance
  13. 13. Future Work “We explicitly avoid to define benchmark rules, as they are (often) restrictive, arbitrary, biased, and not-future-proof” - Djellel, Andy, Carlo, Philippe The project received an NSF Grant!! Possible directions: ● Better tooling (e.g. real-time visualization) ● Synchronized distributed clients ● Support for NoSQL ● A public repository for experiments and results
  14. 14. Q&A External contributors and acknowledgements: Evan Jones, Barzan Mozfari, Dimitri Vorona, Ben Reilly, Yu Su, Adam Seering, Simon Krenger, Tommy Reilly, Mark Callaghan, Keving Grittner, David L. Day, Nik Lanham, Awajeet Arya, Flavio Sousa, Stein Petter Tronstad, Lance FAng, Rodrigo Felix de Almeida, Woonhak kang, Florian Funke, Ahmad Hassan, Ivo Jimenez,Mickaël Hoerdt, Dimokritos Stamatakis, Benjamin Reilly, Mark Callaghan, Zheng Da, Xi Chen, Jean de Lavarene, Ning Zhang, Erik Paulson, Alec Jindal, ADD YOUR NAME HERE