Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DATABASE MANAGEMENT
SYSTEMS BENCHMARKING
overview
and
trends
ГОНКИ TPC-C
296 results
Simfoware, Oracle DB 7–8, DB2/400 и UDB, Informix, MS SQL Server, Sybase ASE …
TPC-C, APRIL, 2017
SQL
Anywhere
AGENDA
What happened with TPC-C?
•TPC: early history
•TPC-A/B, -С, -H, -E
•TPC: obsolete and new
•Publication issues
The N...
TP1
Mid 1970s,
IBM benchmark
Bank
transactions
processing
Idefix: 100 tps
(1973, “bank
with 1.000
branches and
10.000 tell...
DAVIT DEVITT:
WISCONSIN BENCHMARK
Alternative, more
strictly determined
benchmark
Aimed to terminate
“benchmarking
wars”, ...
JIM GREY: DEBITCREDIT
Counterpoise for Wisconsin benchmark,
based on TP1 subject area
Requires total systems cost, includi...
PRINCIPAL: RESPONSE TIME
BOUND
Изображение: ©Dell, 2013
TPC.ORG
Benchmarking wars continued: how to validated result?
Independent, non-profit organization need…
1988: Transaction...
TPC-A AND TPC-B: PROCESS
READ 100 bytes from TTY (AID, TID, BID, DELTA)
BEGIN TRANSACTION
UPDATE ACCOUNT WHERE ACCOUNT_ID ...
TPC-A AND TPC-B: MODEL
BRANCH
B
ACCOUNT
B*100K
100K
HISTORY
B*2.6M
TELLER
B*1010
10s cycle for each
terminal
1 transaction...
TPC-A AND TPC-B:
DIFFERENCES
TPC-A
Terminals
User response
delay
TPC-B
Server-only
benchmark
Reduced history
(30 days)
TPC-A AND TPC-B: CRITICISM
1995: considered unreliable
Last result (DEC, 1994)
3700 tpsA 4800 $/tpsA
First result (HP, 199...
TPC-C: COMPLICATION
Response thresholds for 90% of transactions
Less than 5s for interactive operation Less than 20s for b...
TPC-C DATA MODEL
WAREHOUSE
W
DISTRICT
W*10
10
CUSTOMER
W*30K
3K
HISTORY
W*30K+
1+
ITEM
100K (fix)
STOCK
W*100K100K W
ORDER...
TPC-C: SCALING
Each new warehouse →
+10 districts, + 100 items, +300k customers
Maximum 1.2 tpmC per terminal
10 terminals...
TPC-C:
CLUSTERED AND NON-CLUSTERED
Clustered >1 nodes
Shared
nothing (fully
federated)
Sharded
Shared storage
(RAC)
Non-cl...
TPC-C: METRICS
tpmC
Transactions per
minute
tpmC / $
Transaction cost
Hardware
acquisition cost
Software and
services cost...
TPC-C INTERPRETATION
By Alan Parker (Alan Parker. Tuning databases on Solaris platform. Prentice-Hall, 2002)
•…not only or...
COULD WE CHEAT TPC-C?
mount -t tmpfs -o size=2048g tmpfs /u01/tablespaces
CREATE UNLOGGED TABLE …
_ALLOW_RESETLOGS_CORRUPT...
RESPONSE FROM COUNCIL
A l l A C I D a s p e c t s c h e c k s s t i f f l y i n t e r p r e t e d i n s p e c i f i c a t ...
TPC-C: CRITICISM OF 1990S
Even wholesale providers works somewhat differently!
Order: some
unsuccessful wildcard
searches
...
TPC: OBSOLETED
TPC-D 1995–1999
First attempt
for OLAP
benchmark
TPC-R 2001–2005 Reporting
TPC-W 2001–2005
Online web-
comm...
TPC-H
1999 :
OLAP vs OLTP
“Ad-hoc
decision
support”
Instead of
TPC-D
(considered
irrelevant)
“Weight
categories”
100 GB
30...
TPC-H: MODEL
1 SF ~ 1 GB
Declarative
constraints
Not star schema
TPC-H Q1
SELECT
l_returnflag, l_linestatus,
sum(l_quantity) as sum_qty,
sum(l_extendedprice) as sum_base_price,
sum(l_exte...
TPC-H: RESULTS, 2017
Tricky to apply
for non-relational DBMS
Variants to
apply for
MOLAP
(MDX)
Reports
with
application
fo...
TPC-E: “PRAGMATIC OLTP”
Still OLTP, but more “hybrid”
Declarative constraints
No transaction monitors
More reads
More kind...
TPC-E:
MORE COMPLEX WORKFLOW
TPC-E: MODEL
Image: ©Transaction Processing Council, 2009
78 results
All non-clustered, all on MS SQL Server for Windows Server x64
11 KTPSE, $144/TPSE (1,6 MIO)
Изображение: ©Lenovo, © TPC, 2015
TPC-ABCEH
BRAND NEW: TPC-X И TPCX-X
TPC-DI
ETL workloads
TPC_DI_RPS
0 results
TPC-DS
Decision support
systems “on Big
Data solutions...
PUBLISHING?
Publishing on TPC.org
Academic or research paper, could not be applicable for
marketing purposes
With definite...
DEWITT CLAUSE IN 2017?
MSFT
EULA
•“You may not disclose the results of any benchmark test
… without Microsoft’s prior writ...
NEW BENCHMARKS FOR
NEW WORKLOADS
2010s
2000s
1990s
1980s
OLTP
ROLAP ROLAP
In-memory
OLAP
TCP-DS
ROLAP TCP-H
Graph
queries
...
PRINCIPAL BENCHMARKS
OF “NEW WAVE”
Terrasort Benchmark
BigBench
Intel HiBench
Yahoo! Cloud Services
Benchmark
Linkbench
• ...
INTEL HIBENCH ДЛЯ
HADOOP
Image:Intel,2013
YAHOO! CLOUD SERVICES
BENCHMARK
New tool for “benchmark marketing”
Used by researchers for wide comparisons
(V. Abramova e...
YCSB: WORKLOADS
LINKBENCH
Facebook workload
•Early graph benchmarks were
graph-analysis oriented
•Real workload of the Internet
Giant
•Tra...
POCKET TOOLS
TO MEASURE
Possible?
Correct?
Representative?
Comparable?
Repeatable?
PGBENCH
TPC-B
Standard
part of
PostgreSQL
distributions
One simple
command to
run
Standard de facto for PostgreSQL
interna...
SYSBENCH
Widely used for internal comparisons in MySQL, MariaDB,
and its forks communities
Sui generis tests, not similar ...
TPCE-MYSQL
TPC-E loader for MySQL by Percona
HAMMERDB
Free Java GUI program for
TPC-C (?) and TPC-H (?) with support a range of DBMSs
Oracle
Database
Microsoft
SQL Ser...
HAMMERDB:
ONE-CLICK TPC-C?
HAMMERDB:
TPC, BUT NOT QUITE…
Modern benchmarks were not been implemented
TPC-E TPC-DS
TPC-H
Single load station
TPC-C
Not...
HAMMERDB: POPULARITY
Used by
hardware and
software
vendors
Indirectly,
in form of a
“blog of our
tech guy”
Hub with
benchm...
SWINGBENCH
TPC-C-like workload for Oracle Database and TimesTen
+ Idiosyncratic OLTP benchmark with a lot of PL/SQL
Oracle...
DELL BENCHMARK FACTORY
FOR DATABASES
Oracle
Database
MySQL
MS SQL
Server
SQLite
SQL
Anywhere
Commercial tool (Quest Softwa...
OSDLDBT.SOURCEFORGE.NET
•TPC-WDBT-1
•TPC-CDBT-2
•TPC-HDBT-3
•TPC-AppDBT-4
•TPC-EDBT-5
While the inspiration for these work...
OLTPBENCH
github.com/oltpbenchmark/oltpbench
Java tool for command line
Supports any JDBC-enabled RDBMS
Special version fo...
TPC TOOLS
Tools by Transaction
Processing
Council
C sources files
Not exists for TPC-C:
just sample in spec
“Do it
yoursel...
TPC-* BY EXAMPLE
По «отчётам о полном раскрытии информации» на TPC.org
RPE2
SAP SD
2-Tier
TPC-C
TPC-HSPEC
jbb2005
SPEC
CPU2006
Supercomposite indicator by Gartner (Ideas)
RPE2-ERP
RPE2-Java
RPE...
BENCHWARE
Swiss kinfe by Manfred Drozd
Peakmarks Benchware
OraCPU
PL/SQL op
•[ops]
PL/SQL alg
•[ops]
OraSRV
In-memory
SQL
...
DATABASE MACHINES?
Pre-configured
appliances for databases
Should be precisely
measured and
benchmarked?
TERADATA
Latest Teradata publications with TPC-H:
Licensed by
«internal Qph» –
tPerf [Traditional Performance]
EXADATA
tps, Qph – not published
“Passport metrics” (X6-8)
෍ 𝑉 × IOPS ≈ Const
IBM PURE DATA FOR
OPERATIONAL ANALYTICS
Qph not published…
“Passport metrics” about input/output
“SQL IOPS”
Measuring SQL IOPS for another DBMS?
Statistical views (…IO_STATS…)
IOPS from Oracle Database side
Orion
(Oracl...
ATOMIZATION OF
AGGREGATE METRICS
tpmC
QpmH
SQL IOPS
SQL
bandwidth
METRICS ATOMIZATION:
PRO
Independent from predefined
models and schemes
Accepted by database
appliance vendors
Pocket tool...
METRICS ATOMIZATION:
…ET CONTRA
Methodology
Different results on
the same
environment from
CALIBRATE_IO
and Benchware
Not ...
RUNNING REAL
WORKLOAD AND
APPLICATIONS
BENCHMARKS
Most
practical
approach?
WORKLOAD REPLAY AS
TRUSTED EXPERIENCE
For systems with full API access
(usually JSON via HTTP)
Logging and
autocapture
Rep...
BOOTSTRAPPING TRICKS
How to
rollout
data?
Clonning
Repeated
with random
shift?
Mixing with
real data
(open data)
Impact to...
PACKAGED APPLICATIONS
1С benchmark series
Metrics: count of concurrent users with acceptable response time
Microsoft Dynam...
QUESTIONS FOR
DISCUSSION
•Could them reach popularity like TPC-C и TPC-H?
•Megaconfigurations problem
TPC-E и TPC-DS
•Runn...
THANK YOU! mailto:anikolaenko@ibs.ru
mailto:anikolaenko@acm.org
Cover image: Paul Hudson, CC-SA
Upcoming SlideShare
Loading in …5
×

0

Share

Download to read offline

DBMS benchmarking overview and trends for Moscow ACM SIGMOD Chapter

Download to read offline

Slides for CXCIV Moscow ACM SIGMOD Chapter workshop

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

DBMS benchmarking overview and trends for Moscow ACM SIGMOD Chapter

  1. 1. DATABASE MANAGEMENT SYSTEMS BENCHMARKING overview and trends
  2. 2. ГОНКИ TPC-C 296 results Simfoware, Oracle DB 7–8, DB2/400 и UDB, Informix, MS SQL Server, Sybase ASE …
  3. 3. TPC-C, APRIL, 2017 SQL Anywhere
  4. 4. AGENDA What happened with TPC-C? •TPC: early history •TPC-A/B, -С, -H, -E •TPC: obsolete and new •Publication issues The New Wave of benchmarks •MapReduce (Hadoop) •Graph benchmarks •Meta-universal, atomic «Pocket tools» to run Input/Output Outcomes
  5. 5. TP1 Mid 1970s, IBM benchmark Bank transactions processing Idefix: 100 tps (1973, “bank with 1.000 branches and 10.000 tellers”) Batch mode without networking no delay for teller response Early 1980s: fantastic victorious reports with 10 ktps $$bln market… … focus on fast- growing OLTP …however, customers could not achieved yet 1 ktps …
  6. 6. DAVIT DEVITT: WISCONSIN BENCHMARK Alternative, more strictly determined benchmark Aimed to terminate “benchmarking wars”, but lit the war even stronger! Weak results of Oracle DB caused ‘DeWitt clause’ DBMS license prohibits publishing benchmarks
  7. 7. JIM GREY: DEBITCREDIT Counterpoise for Wisconsin benchmark, based on TP1 subject area Requires total systems cost, including equipment, licenses, 5 years of support Specified as textual functional requirements, without code requirements or code examples Introduced scaling rules by users and table sizes Postulated a response time limit : 95 % of transactions should complete in 1s
  8. 8. PRINCIPAL: RESPONSE TIME BOUND Изображение: ©Dell, 2013
  9. 9. TPC.ORG Benchmarking wars continued: how to validated result? Independent, non-profit organization need… 1988: Transaction Processing Performance Council Omri Serlin and 8 [consonant] vendors Actian Cisco Cloudera Dell DataCore Fujitsu HPE Hitachi Huawei IBM Inspur Intel Lenovo Microsoft Oracle Pivotal Red Hat SAP Teradata VMWare Members (2017):
  10. 10. TPC-A AND TPC-B: PROCESS READ 100 bytes from TTY (AID, TID, BID, DELTA) BEGIN TRANSACTION UPDATE ACCOUNT WHERE ACCOUNT_ID = AID: READ ACCOUNT_BALANCE FROM ACCOUNT SET ACCOUNT_BALANCE = ACCOUNT_BALANCE + DELTA WRITE ACCOUNT_BALANCE TO ACCOUNT WRITE TO HISTORY: AID, TID, BID, DELTA, TIME_STAMP UPDATE TELLER WHERE TELLER_ID = TID: SET TELLER_BALANCE = TELLER_BALANCE + DELTA WRITE TELLER_BALANCE TO TELLER UPDATE BRANCH WHERE BRANCH_ID = BID: SET BRANCH_BALANCE = BRANCH_BALANCE + DELTA WRITE BRANCH_BALANCE TO BRANCH COMMIT TRANSACTION WRITE 200 bytes to TTY (AID, TID, BID, DELTA) Based on TP1 – retail banking transactions
  11. 11. TPC-A AND TPC-B: MODEL BRANCH B ACCOUNT B*100K 100K HISTORY B*2.6M TELLER B*1010 10s cycle for each terminal 1 transaction per second for each branch Response for 90% of transactions not less than 2s Average transactions per second in 15 min
  12. 12. TPC-A AND TPC-B: DIFFERENCES TPC-A Terminals User response delay TPC-B Server-only benchmark Reduced history (30 days)
  13. 13. TPC-A AND TPC-B: CRITICISM 1995: considered unreliable Last result (DEC, 1994) 3700 tpsA 4800 $/tpsA First result (HP, 1990) 38.2 tpsA 29200 $/tpsA TP1 legacy Too simple to avoid “adjustments” Implausible dispersion of results
  14. 14. TPC-C: COMPLICATION Response thresholds for 90% of transactions Less than 5s for interactive operation Less than 20s for batch More variances… 9 tables Inserts, updates, deletes, operation cancellations Access by primary and secondary keys 5 типов транзакций NEW-ORDER •new customer order PAYMENT •payment event •customer balance refresh DELIVERY •delivery orer •(batch) ORDER-STATUS •checking the status of the last customer order STOCK-LEVEL •checking the stock level at the warehouse 45% 43% 4% 4% 4%
  15. 15. TPC-C DATA MODEL WAREHOUSE W DISTRICT W*10 10 CUSTOMER W*30K 3K HISTORY W*30K+ 1+ ITEM 100K (fix) STOCK W*100K100K W ORDER W*30K+1+ ORDER-LINE W*300K+ 10–15 NEW-ORDER W*5K0–1
  16. 16. TPC-C: SCALING Each new warehouse → +10 districts, + 100 items, +300k customers Maximum 1.2 tpmC per terminal 10 terminals per warehouse Scaling factor: warehouse (W)
  17. 17. TPC-C: CLUSTERED AND NON-CLUSTERED Clustered >1 nodes Shared nothing (fully federated) Sharded Shared storage (RAC) Non-clustered 1 node
  18. 18. TPC-C: METRICS tpmC Transactions per minute tpmC / $ Transaction cost Hardware acquisition cost Software and services cost for 3 years W / ktpmC Energy consumption per 1000 transactions per minute
  19. 19. TPC-C INTERPRETATION By Alan Parker (Alan Parker. Tuning databases on Solaris platform. Prentice-Hall, 2002) •…not only order entrytpmC × 2 •…if transaction monitor does not usedtpmC / 2 •…if Oracle Forms-like client usedtpmC / 3 •…if lightweight client forms used (like curses-based)tpmC × 2 / 3 •…if SQL has not been tunedtpmC / 2 • …if there are heavy reports and batchestpmC / 2
  20. 20. COULD WE CHEAT TPC-C? mount -t tmpfs -o size=2048g tmpfs /u01/tablespaces CREATE UNLOGGED TABLE … _ALLOW_RESETLOGS_CORRUPTION = TRUE _IN_MEMORY_UNDO = TRUE _DB_BLOCK_HASH_LATCHES = 32768 …
  21. 21. RESPONSE FROM COUNCIL A l l A C I D a s p e c t s c h e c k s s t i f f l y i n t e r p r e t e d i n s p e c i f i c a t i o n Single node reboot failure Synchronous commit on two nodes Redo logs and recovering Reboot and recovering Single media failure protection Synchronous commit on two separately powered media Write-ahead logs on separately powered media Commit = written on durable media
  22. 22. TPC-C: CRITICISM OF 1990S Even wholesale providers works somewhat differently! Order: some unsuccessful wildcard searches Printing report after each entry (probably, 3 times) Balance refresh for each operation is impossible in a heavy load environment → insert + batch calc Trivial logic No declarative constraints No trigger logic Non-typical loads What about reporting? Decision support systems?
  23. 23. TPC: OBSOLETED TPC-D 1995–1999 First attempt for OLAP benchmark TPC-R 2001–2005 Reporting TPC-W 2001–2005 Online web- commerce
  24. 24. TPC-H 1999 : OLAP vs OLTP “Ad-hoc decision support” Instead of TPC-D (considered irrelevant) “Weight categories” 100 GB 300 GB 1 TB 3 TB 10 TB 30 TB 100 TB Parallel load 22 kinds of complex queries 2 kinds of DWH refresh
  25. 25. TPC-H: MODEL 1 SF ~ 1 GB Declarative constraints Not star schema
  26. 26. TPC-H Q1 SELECT l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice*(1-l_discount)) as sum_disc_price, sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order FROM lineitem WHERE l_shipdate <= date '1998-12-01' - interval Δ day (3) GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus Δ = random [60…120] “ F u n c t i o n a l Q u e r y D e f i n i t i o n ”
  27. 27. TPC-H: RESULTS, 2017 Tricky to apply for non-relational DBMS Variants to apply for MOLAP (MDX) Reports with application for Apache Hive Exasol MS SQL Server Oracle Database Actian Vector 1 – 10 results for each “weight category”
  28. 28. TPC-E: “PRAGMATIC OLTP” Still OLTP, but more “hybrid” Declarative constraints No transaction monitors More reads More kinds of load
  29. 29. TPC-E: MORE COMPLEX WORKFLOW
  30. 30. TPC-E: MODEL Image: ©Transaction Processing Council, 2009
  31. 31. 78 results All non-clustered, all on MS SQL Server for Windows Server x64
  32. 32. 11 KTPSE, $144/TPSE (1,6 MIO) Изображение: ©Lenovo, © TPC, 2015
  33. 33. TPC-ABCEH
  34. 34. BRAND NEW: TPC-X И TPCX-X TPC-DI ETL workloads TPC_DI_RPS 0 results TPC-DS Decision support systems “on Big Data solutions, such as RDBMS as well as Hadoop/Spark based systems” QphDS@Size 0 results TPCx-BB Express benchmark for “Hadoop- based Big Data Systems” adopted from BigBench BBpm@Size 0 results TPCx-HS Express benchmark for HDFS- compatible systems, adopted from TeraSort HSph@Size From 1 to 4 results in different “weight categories”
  35. 35. PUBLISHING? Publishing on TPC.org Academic or research paper, could not be applicable for marketing purposes With definite clause that outcome is not comparable with TPC.org results With permission of TPC.org Publishing TPC benchmarks is prohibited, except the following cases: Required audit from sizing.com
  36. 36. DEWITT CLAUSE IN 2017? MSFT EULA •“You may not disclose the results of any benchmark test … without Microsoft’s prior written approval” OTN Lic. •“You may not disclose results of any Program benchmark tests without Oracle’s prior consent” IBM IPLA •“Licensee may disclose the results of any benchmark test of the Program or its subcomponents to any third party provided that Licensee, if … A) … B) … C)…”
  37. 37. NEW BENCHMARKS FOR NEW WORKLOADS 2010s 2000s 1990s 1980s OLTP ROLAP ROLAP In-memory OLAP TCP-DS ROLAP TCP-H Graph queries LinkBench MapReduce HiBench OLTP OLTP[E] OLTP[E] TPC-E OLTP Atomic access YCSB TPCx-HS TPCx-BB
  38. 38. PRINCIPAL BENCHMARKS OF “NEW WAVE” Terrasort Benchmark BigBench Intel HiBench Yahoo! Cloud Services Benchmark Linkbench • Adopted in TPCx-HS • Adopted in TPCx-BB • Map-Reduce series • Group of NoSQL etalon runs • Graph-like workload on RDBMS and HBase
  39. 39. INTEL HIBENCH ДЛЯ HADOOP Image:Intel,2013
  40. 40. YAHOO! CLOUD SERVICES BENCHMARK New tool for “benchmark marketing” Used by researchers for wide comparisons (V. Abramova et al. Experimental Evaluation of NoSQL Databases // IJDMS Vol.6, №3, 2014) Cassandra HBase Elastic search MongoDB Oracle NoSQL OrientDB Redis Scalaris Tarantool Voldemort Bombardment from one load station Atomic operation instead of transaction (probably, read a few records)
  41. 41. YCSB: WORKLOADS
  42. 42. LINKBENCH Facebook workload •Early graph benchmarks were graph-analysis oriented •Real workload of the Internet Giant •Transactions (MVCC) Методологично •Statistic laws for data generageion •Measures: average response time for each type of workload •Avg from 99-percentile •Documents in seconds for write •Queries per second for read MySQL (InnoDB vs TokuDB) HBase MongoDB / TokuMX OrientDB T h e Q u e s t i o n o f Fa c e b o o k
  43. 43. POCKET TOOLS TO MEASURE Possible? Correct? Representative? Comparable? Repeatable?
  44. 44. PGBENCH TPC-B Standard part of PostgreSQL distributions One simple command to run Standard de facto for PostgreSQL internal investigations, such as: XFS or ext4? Tables on SSD, indexes on HDD? Block size: 4K or 8K?
  45. 45. SYSBENCH Widely used for internal comparisons in MySQL, MariaDB, and its forks communities Sui generis tests, not similar to standard benchmarks MySQL utility (by Alexei Kopytov) fileio cpu memory threads mutex oltp
  46. 46. TPCE-MYSQL TPC-E loader for MySQL by Percona
  47. 47. HAMMERDB Free Java GUI program for TPC-C (?) and TPC-H (?) with support a range of DBMSs Oracle Database Microsoft SQL Server IBM DB2 TimesTen MySQL MariaDB Postgre SQL Postgres Plus AS Greenplum DB Redis Amazon Aurora (MySQL) Amazon Redshift (ParAccel) Trafodion SQL on Hadoop
  48. 48. HAMMERDB: ONE-CLICK TPC-C?
  49. 49. HAMMERDB: TPC, BUT NOT QUITE… Modern benchmarks were not been implemented TPC-E TPC-DS TPC-H Single load station TPC-C Not emulated full environment, no transaction monitor software Load station in role of single transaction monitor
  50. 50. HAMMERDB: POPULARITY Used by hardware and software vendors Indirectly, in form of a “blog of our tech guy” Hub with benchmark results Section «Performance Data»
  51. 51. SWINGBENCH TPC-C-like workload for Oracle Database and TimesTen + Idiosyncratic OLTP benchmark with a lot of PL/SQL Oracle DB-specific tools for monitoring and analysis (AWR, etc.) Coordinator support GUI and command line Only internal comparability Non-representative in RAC environments
  52. 52. DELL BENCHMARK FACTORY FOR DATABASES Oracle Database MySQL MS SQL Server SQLite SQL Anywhere Commercial tool (Quest Software legacy) TPC-C TPC-D TPC-E TPC-H ASP3AP Supports a number of load stations (Windows)
  53. 53. OSDLDBT.SOURCEFORGE.NET •TPC-WDBT-1 •TPC-CDBT-2 •TPC-HDBT-3 •TPC-AppDBT-4 •TPC-EDBT-5 While the inspiration for these workloads are the TPC-<x…>, workloads are entirely different and results obtained from them should not and can not be compared to TPC results. The use of any supplied results of these tests for commercial purposes is expressly prohibited. MySQL PostgreSQL …расширяемо
  54. 54. OLTPBENCH github.com/oltpbenchmark/oltpbench Java tool for command line Supports any JDBC-enabled RDBMS Special version for Hstore (VoltDB ) TPC-C Wikipedia Synthetic Resource Stresser Twitter Epinions.com TATP AuctionMark SEATS YCSB JPAB (Hibernate) CH- benCHmark Voter SIBench (Snapshot Isolation) SmallBank LinkBench
  55. 55. TPC TOOLS Tools by Transaction Processing Council C sources files Not exists for TPC-C: just sample in spec “Do it yourself”: all connectivity and other stuff
  56. 56. TPC-* BY EXAMPLE По «отчётам о полном раскрытии информации» на TPC.org
  57. 57. RPE2 SAP SD 2-Tier TPC-C TPC-HSPEC jbb2005 SPEC CPU2006 Supercomposite indicator by Gartner (Ideas) RPE2-ERP RPE2-Java RPE2-OLTP RPE2-Compute Intensive
  58. 58. BENCHWARE Swiss kinfe by Manfred Drozd Peakmarks Benchware OraCPU PL/SQL op •[ops] PL/SQL alg •[ops] OraSRV In-memory SQL •[ms] •[dbps] •[tps] •[rps] OraSTO SeqIO •[GBps] •[iops] RandIO •[GBps] •[iops] OraOLTP OLTP Select •[rps] •[tps] OLTP Update •[rps] •[tps] OraLoad TransLoad •[rps] •[tps] BulkLoad •[rps] •[tps] OraAgg OraAgg & Rep •[rps] •[tps] Only for Oracle Database, only PL/SQL and SYS.V_$%
  59. 59. DATABASE MACHINES? Pre-configured appliances for databases Should be precisely measured and benchmarked?
  60. 60. TERADATA Latest Teradata publications with TPC-H: Licensed by «internal Qph» – tPerf [Traditional Performance]
  61. 61. EXADATA tps, Qph – not published “Passport metrics” (X6-8) ෍ 𝑉 × IOPS ≈ Const
  62. 62. IBM PURE DATA FOR OPERATIONAL ANALYTICS Qph not published… “Passport metrics” about input/output
  63. 63. “SQL IOPS” Measuring SQL IOPS for another DBMS? Statistical views (…IO_STATS…) IOPS from Oracle Database side Orion (Oracle IO Numbers) SLOB (recommended by EMC, Flashgrid) Benchware (?) DBMS_RESOURCE_MANAGER .CALIBRATE_IO
  64. 64. ATOMIZATION OF AGGREGATE METRICS tpmC QpmH SQL IOPS SQL bandwidth
  65. 65. METRICS ATOMIZATION: PRO Independent from predefined models and schemes Accepted by database appliance vendors Pocket tools availability Representative for wide class of DBMS and DBMS-like systems • Reprehensive not only for 3NF, snowflakes, stars • Included in passport metrics • Running from DB side • Could be interested for NoSQL
  66. 66. METRICS ATOMIZATION: …ET CONTRA Methodology Different results on the same environment from CALIBRATE_IO and Benchware Not clear application sense The same IO operations in different DBMSs produces different transaction counts Tool not exists for most of DBMS Found just for Oracle DB, IBM DB2, MS SQL
  67. 67. RUNNING REAL WORKLOAD AND APPLICATIONS BENCHMARKS Most practical approach?
  68. 68. WORKLOAD REPLAY AS TRUSTED EXPERIENCE For systems with full API access (usually JSON via HTTP) Logging and autocapture Replay with “sleeps” Splitting workload patterns (user types) Data scaling? Wokrload emulators JMeter LoadRunner … DB-side tools Oracle Real Application Testing (Database Replay) MS SQL Server Distributed Replay
  69. 69. BOOTSTRAPPING TRICKS How to rollout data? Clonning Repeated with random shift? Mixing with real data (open data) Impact to analytics Predictable query results Low selectivity Impact to OLTP Keys, indexes…
  70. 70. PACKAGED APPLICATIONS 1С benchmark series Metrics: count of concurrent users with acceptable response time Microsoft Dynamics AX Application Benchmark Toolkit Oracle E-Business Suite Standard Benchamrks Order-to-Cash OLTP Payroll … SAPS [SAP Application Performance Standard] with SD module
  71. 71. QUESTIONS FOR DISCUSSION •Could them reach popularity like TPC-C и TPC-H? •Megaconfigurations problem TPC-E и TPC-DS •Running TPC-B is simple •Running others is tricky or not fully compliant Pocket tool •Acceptance, adoption, standardization, interpretation •Other metrics of such kind? SQL IOPS & Bandwidth •Atomic (YCSB-type) •Graph (LinkBench-type) Standardization and generalization of benchmarks for new workloads •New types of applications: portals, groupware, … •New database architectures: sharding, in-memory databases, in-memory data grids Future of benchmarks on real up-to-date workload
  72. 72. THANK YOU! mailto:anikolaenko@ibs.ru mailto:anikolaenko@acm.org Cover image: Paul Hudson, CC-SA

Slides for CXCIV Moscow ACM SIGMOD Chapter workshop

Views

Total views

1,108

On Slideshare

0

From embeds

0

Number of embeds

6

Actions

Downloads

22

Shares

0

Comments

0

Likes

0

×