The document provides an overview of various OLTP performance benchmarks including pgbench, sysbench, dbt2, BenchmarkSQL, and DVDStore. It discusses the benchmarks' workloads and metrics. It also outlines the development of new benchmarks like TPC-V and how ensuring they run well on PostgreSQL would benefit the open source database.

1. OLTP Performance Benchmark Review
Jignesh Shah
Product Manager, vFabric Postgres
VMware, Inc
© 2010 VMware Inc. All rights reserved

2. About Me
§ Currently Product Manager of vFabric Postgres
§ Performance Engineer , vFabric Data Director
§ Previously with Sun Microsystems (2000-2010)
§ Team Member that delivered the first published mainstream
benchmark with PostgreSQL
§ Blog at : http://jkshah.blogspot.com
2 2012 Copyright VMware Inc

3. Agenda
§ Introduction
§ pgbench
§ Sysbench
§ Dbt2
§ BenchmarkSQL
§ DVDStore
§ A New benchmark
3 2012 Copyright VMware Inc

4. Introduction
§ Why do we need benchmarks?
• Reference data points
• Stress Test for “Too Big to Fail” scenarios
§ Uses of Benchmark
• Improve Product Quality
• Understand code path usage
• Performance Characteristics
• Baseline metrics (Reference points)
• Release to release
• Against other technologies to do same business operation
§ Abuses of Benchmark
• Benchmarketing
• Fixated only on ones that are favorable
4 2012 Copyright VMware Inc

5. pgbench
PostgreSQL
5 2012 Copyright VMware Inc

6. Pgbench
§ Based on TPC-B workload (circa 1990)
§ Not an OLTP but stress benchmark for database
§ Ratio is Branches: 10 Tellers: 100,000 Accounts Branches
§ Default TPC-B sort-of Tellers Accounts
• Account transactions also impact teller and branch balances
History
• Branch table becomes the biggest bottleneck
6 2012 Copyright VMware Inc

7. Pgbench
§ Hints
• PGSSLMODE disable (Unless you want to factor SSL communication
overhead. Depending on your distribution )
• -M prepared (unless you want to measure overhead of parsing)
§ Various modes of benchmark
§ Default TPC-B sort-of
• Account transactions also impact teller and branch balances
• Branch table becomes the biggest bottleneck
§ -N Simple Update (with select, insert)
• Account Update, Select balance. History insert
• Account table update becomes the biggest bottleneck
§ -S read only test
• AccessShareLock on Accounts table and primary index becomes the
bottleneck
• Fixed in 9.2 (Thanks Robert Haas)
7 2012 Copyright VMware Inc

8. PGBench Select Test
PGBench - Select Test
80000
70000
60000
Transactions Per Second
50000
40000
PG9.1 PGBench Select
30000
9.2 PGBench Select
20000
10000
0
0 20 40 60 80 100 120
Number of Clients
8 2012 Copyright VMware Inc

9. PGBench TPC-B Like Test
PGBench (TPC-B Like)
8000
7000
6000
Transactions Per Second
5000
4000
PG9.1 TPC-B Like
3000
2000 9.2 TPC-B Like
1000
0
0 20 40 60 80 100 120
Number of Clients
9 2012 Copyright VMware Inc

10. Sysbench
MySQL
10 2012 Copyright VMware Inc

11. sysbench
§ Originally developed to test systems
§ Has an OLTP component which was based on MySQL
§ Creates a table sbtest with a pimary key.
§ Various Modes of OLTP operation
§ Simple Read Only (web site primary key lookup)
§ Complex Read Only
§ Complex Read Write test
11 2012 Copyright VMware Inc

12. Sysbench – OLTP Simple Read
Sysbench Simple Read
100000
90000
80000
Transactions Per Second
70000
60000
50000
40000
OLTP Simple Read
30000
20000 9.2 OLTP Simple Read
10000
0
0 20 40 60 80 100 120
Number of Clients
12 2012 Copyright VMware Inc

13. Sysbench – OLTP Complex Read
Sysbench Complex Read
6000
Transactions Per Second
5000
4000
3000
OLTP Complex Read
2000
9.2 OLTP Complex Read
1000
0
0 20 40 60 80 100 120
Number of Clients
13 2012 Copyright VMware Inc

14. Sysbench – OLTP Complex Read/Write
Sysbench OLTP Complex R/W
3000
2500
Transactions Per Second
2000
1500
OLTP Complex Write
1000
500
9.2 OLTP Complex Write
0
0 20 40 60 80 100 120
Number of Clients
14 2012 Copyright VMware Inc

15. Sysbench – Complex R/W Note
§ In 9.0 it was impossible to run sysbench complex r/w without
hitting error - ERROR: duplicate key value violates unique constraint "sbtest_pkey"
§ In 9.1 SSI was introduced and occurrence went down drastically
§ In 9.2 havent encountered the occurence
15 2012 Copyright VMware Inc

16. dbt2
16 2012 Copyright VMware Inc

17. Dbt2 -
§ Fair Use implementation of TPC-C
Warehouses Stock
§ Implemented using C stored procedures using
driver->client->database server architecture District Item
§ Nine Tables Customer Orders
§ Five Transactions – History Order Lines
• New-Order (NOTPM) 45%
New Orders
• Payment 43%
• Delivery 4%
• Order Status 4%
• Stock Level 4%
17 2012 Copyright VMware Inc

18. Dbt2 -
§ Why is it not TPC-C compliant?
• Not audited by TPC
• No Terminal emulator
• Official kit requires commercial Transaction Manager
• Doesn’t’ cover ACID tests
§ Two versions Available
• Libpq
• ODBC
§ One potential problem is 3 network roundtrips per transaction
which causes “Idle in Transaction” at high load
• BEGIN, SELECT StoredProcedure() , END pattern of transactions
18 2012 Copyright VMware Inc

19. Dbt2 – Postgres 9.1
Cached Runs (data in bufferpool)
Short runs (limited checkpoint
and vacuum impacts)
NOTPM = 45% of all DB Trans
DB Trans rate about 3000 TPS
19 2012 Copyright VMware Inc

20. BenchmarkSQL
20 2012 Copyright VMware Inc

21. BenchmarkSQL-
§ Another implementation using TPC-C schema
Warehouses Stock
§ Implemented using JDBC
District Item
§ Nine Tables
§ Five Transactions – Customer Orders
• New-Order (NOTPM) 45% History Order Lines
• Payment 43% New Orders
• Delivery 4%
• Order Status 4%
• Stock Level 4%
§ Surprisingly can do better than dbt2 implementation but still has
“idle in transactions” which means bottlenecked at network/client
level
21 2012 Copyright VMware Inc

22. DVDStore
22 2012 Copyright VMware Inc

23. DVDStore
§ Implementation of Online DVD Store
Customers Cust_hist
§ Postgres support contributed by VMware
Product Inventory
§ Implemented using various stacks
• JSP/Java/JDBC (supports Postgres) Categories Orders
• Linux/Apache/PHP/MySQL (supports Postgres)
Reorder Order Lines
• ASP.NET (not yet implemented for Postgres)
• Stored Procedures (supports Postgres via Npgsql)
§ Eight Tables
§ Main Transactions –
• New-Customers 0-10% (configurable)
• Customer Login
• DVD Browse (By category, by actor, by title)
• Purchase Order (Metric – Orders Per Minute)
• Stock ReOrder (via Triggers)
23 2012 Copyright VMware Inc

24. DVDStore
§ JSP/Java JDBC Implementation
• Tomcat may need tuning
§ PHP-Postgres Implementation
• Suffers from one connection per SQL command
• Needs pg_bouncer (on same server as web server) and configure local
connections to pg_bouncer which does connection caching to actual Postgres
server
§ Stored Procedure Implementation
• Fastest Implementation (> 100,000 orders per minute)
• Idle in transactions can still occur.
§ Metric is Orders Per Minute
• DB Transactions = (6-7 * OPM/60) ~ 10K – 11K TPS
24 2012 Copyright VMware Inc

25. TPC-E/V
25 2012 Copyright VMware Inc

26. Genesis of TPC-V
§ Users are demanding benchmarks to measure performance of
databases in a virtual environment
• Existing virtualization benchmarks model consolidation:
• Many VMs
• Small VMs
• Non-database workloads
§ TPC is developing a benchmark to satisfy that demand: TPC-V
• An OLTP workload typical of TPC benchmarks
• Fewer, larger VMs
• Cloud characteristics:
• Variability: mix of small and large VMs
• Elasticity: load driven to each VM varies by 10X
26 2012 Copyright VMware Inc

27. Benchmark requirements
§ Satisfies the industry need for a benchmark that:
• Has a database-centric workload
• Stresses virtualization layer
• Moderate # of VMs, exercising enterprise applications
• Healthy storage and networking I/O content; emphasizes I/O in a virtualized
environment
• NOT many app environments in an app consolidation scenario
§ Timely development cycle (1-2 years)
• Based on the TPC-E benchmark and borrows a lot from it
27 2012 Copyright VMware Inc

28. What is TPC-E
§ TPC-E is theTPC’s latest OLTP benchmark
• More complex than TPC-C
• Less I/O than TPC-C
• A lot of the code is TPC-supplied
§ Models a brokerage firm
Customers Brokers Market
Invoke the following transactions …
READ-WRITE READ-ONLY
•Market-Feed •Broker-Volume •Security-Detail
•Trade-Order •Customer-Position •Trade-Lookup
•Trade-Result •Market-Watch •Trade-Status
•Trade-Update
… against the following data
Customer Data Brokerage Data Market Data
28 2012 Copyright VMware Inc

29. Abstraction of the Functional Components in an OLTP
Environment
Modeled Business
Presentation
Network Application
User Services Network
Interfaces And Database
Business Logic Services
Services
Market
Exchange Legend
Customer
Sponsor Provided
Stock Market
29 2012 Copyright VMware Inc

30. Functional Components of TPC-E Test Configuration
Driving and Reporting
Sponsor
Provided
CE… MEE… DM…
EGenDriver EGenDriverCE EGenDriverMEE EGenDriverDM TPC Defined
Interfaces
…CE …MEE …DM
EGenDriver Connector
Sponsor
Provided
Network
EGenTxnHarness Connector
EGenTxnHarness TPC-E Logic and Frame Calls TPC Defined
Interface
Sponsor
Provided Frame Implementation
Legend
Database Interface Sponsor Provided
Commercial TPC Provided
Product
DBMS Commercial Product
Sponsor TPC Defined
Provided Database Logic Interface
30 2012 Copyright VMware Inc

31. How does this all matter to the PostgreSQL community?
§ TPC is developing a benchmarking kit for TPC-V
• First time TPC has gone beyond publishing a functional specification
• Full, end-to-end functionality
• Publicly available kit
• Produces the variability and load elasticity properties of the benchmark
• Users need not worry about complexities of simulating cloud characteristics
• Runs against an open source database
• A “reference” kit; companies are allowed to develop their own kit
§ Anyone can install the kit and pound on the server with a cloud
database workload
• Removes the high cost of entry typical to TPC benchmarks
§ The reference kit will run on PostgreSQL
• ODBC interface allows running the workload against other databases
§ Tentative plans to also release a TPC-E kit
• We started out with a kit to run TPC-E; now adding the TPC-V properties
31 2012 Copyright VMware Inc

32. Our dependence on PostgreSQL
§ This reference kit will be a very successful new benchmark
• But only if its performance on the open source database is at least decent
compared to commercial databases
§ PostgreSQL can benefit a lot from being the reference database for a
major new benchmark
• But only its performance is decent!
§ Running the TPC-E prototype on PGSQL 8.4 on RHEL 6.1, we are at
~20% of published TPC-E results
• Very early results
• Current testbed is memory challenged
• Good news: Query plans for the 10 queries implemented look good
• Long, mostly-read queries => issue is the basic execution path, not redo log,
latch contention, etc.
32 2012 Copyright VMware Inc

33. Benchmark Development Status
§ TPC-V Development Subcommittee
• 9 member companies
• 3-4 engineers working actively on the reference kit
• On version 0.12 of the draft spec
• Worked through a lot of thorny issues
• Betting the farm on the reference kit
• But if we produce a good kit, TPC-V will be an immediate success
§ We expect to make a kit available to member companies in Q3 or
Q4
Bottom line: Cooperating to make the TPC-E/TPC-V reference kits
run well on PostgreSQL will greatly benefit all of us
33 2012 Copyright VMware Inc

34. Acknowledgements!
§ VMware Performance Team
• Dong, Reza
§ VMware vFabric Postgres Team
• David Fetter, Dan, Alex, Nikhil, Scott, Yolanda
34 2012 Copyright VMware Inc

35. Thank You
35 2012 Copyright VMware Inc