Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DBMS benchmarking overview and trends for Moscow ACM SIGMOD Chapter

370 views

Published on

Slides for CXCIV Moscow ACM SIGMOD Chapter workshop

Published in: Software
  • Be the first to comment

  • Be the first to like this

DBMS benchmarking overview and trends for Moscow ACM SIGMOD Chapter

  1. 1. DATABASE MANAGEMENT SYSTEMS BENCHMARKING overview and trends
  2. 2. ГОНКИ TPC-C 296 results Simfoware, Oracle DB 7–8, DB2/400 и UDB, Informix, MS SQL Server, Sybase ASE …
  3. 3. TPC-C, APRIL, 2017 SQL Anywhere
  4. 4. AGENDA What happened with TPC-C? •TPC: early history •TPC-A/B, -С, -H, -E •TPC: obsolete and new •Publication issues The New Wave of benchmarks •MapReduce (Hadoop) •Graph benchmarks •Meta-universal, atomic «Pocket tools» to run Input/Output Outcomes
  5. 5. TP1 Mid 1970s, IBM benchmark Bank transactions processing Idefix: 100 tps (1973, “bank with 1.000 branches and 10.000 tellers”) Batch mode without networking no delay for teller response Early 1980s: fantastic victorious reports with 10 ktps $$bln market… … focus on fast- growing OLTP …however, customers could not achieved yet 1 ktps …
  6. 6. DAVIT DEVITT: WISCONSIN BENCHMARK Alternative, more strictly determined benchmark Aimed to terminate “benchmarking wars”, but lit the war even stronger! Weak results of Oracle DB caused ‘DeWitt clause’ DBMS license prohibits publishing benchmarks
  7. 7. JIM GREY: DEBITCREDIT Counterpoise for Wisconsin benchmark, based on TP1 subject area Requires total systems cost, including equipment, licenses, 5 years of support Specified as textual functional requirements, without code requirements or code examples Introduced scaling rules by users and table sizes Postulated a response time limit : 95 % of transactions should complete in 1s
  8. 8. PRINCIPAL: RESPONSE TIME BOUND Изображение: ©Dell, 2013
  9. 9. TPC.ORG Benchmarking wars continued: how to validated result? Independent, non-profit organization need… 1988: Transaction Processing Performance Council Omri Serlin and 8 [consonant] vendors Actian Cisco Cloudera Dell DataCore Fujitsu HPE Hitachi Huawei IBM Inspur Intel Lenovo Microsoft Oracle Pivotal Red Hat SAP Teradata VMWare Members (2017):
  10. 10. TPC-A AND TPC-B: PROCESS READ 100 bytes from TTY (AID, TID, BID, DELTA) BEGIN TRANSACTION UPDATE ACCOUNT WHERE ACCOUNT_ID = AID: READ ACCOUNT_BALANCE FROM ACCOUNT SET ACCOUNT_BALANCE = ACCOUNT_BALANCE + DELTA WRITE ACCOUNT_BALANCE TO ACCOUNT WRITE TO HISTORY: AID, TID, BID, DELTA, TIME_STAMP UPDATE TELLER WHERE TELLER_ID = TID: SET TELLER_BALANCE = TELLER_BALANCE + DELTA WRITE TELLER_BALANCE TO TELLER UPDATE BRANCH WHERE BRANCH_ID = BID: SET BRANCH_BALANCE = BRANCH_BALANCE + DELTA WRITE BRANCH_BALANCE TO BRANCH COMMIT TRANSACTION WRITE 200 bytes to TTY (AID, TID, BID, DELTA) Based on TP1 – retail banking transactions
  11. 11. TPC-A AND TPC-B: MODEL BRANCH B ACCOUNT B*100K 100K HISTORY B*2.6M TELLER B*1010 10s cycle for each terminal 1 transaction per second for each branch Response for 90% of transactions not less than 2s Average transactions per second in 15 min
  12. 12. TPC-A AND TPC-B: DIFFERENCES TPC-A Terminals User response delay TPC-B Server-only benchmark Reduced history (30 days)
  13. 13. TPC-A AND TPC-B: CRITICISM 1995: considered unreliable Last result (DEC, 1994) 3700 tpsA 4800 $/tpsA First result (HP, 1990) 38.2 tpsA 29200 $/tpsA TP1 legacy Too simple to avoid “adjustments” Implausible dispersion of results
  14. 14. TPC-C: COMPLICATION Response thresholds for 90% of transactions Less than 5s for interactive operation Less than 20s for batch More variances… 9 tables Inserts, updates, deletes, operation cancellations Access by primary and secondary keys 5 типов транзакций NEW-ORDER •new customer order PAYMENT •payment event •customer balance refresh DELIVERY •delivery orer •(batch) ORDER-STATUS •checking the status of the last customer order STOCK-LEVEL •checking the stock level at the warehouse 45% 43% 4% 4% 4%
  15. 15. TPC-C DATA MODEL WAREHOUSE W DISTRICT W*10 10 CUSTOMER W*30K 3K HISTORY W*30K+ 1+ ITEM 100K (fix) STOCK W*100K100K W ORDER W*30K+1+ ORDER-LINE W*300K+ 10–15 NEW-ORDER W*5K0–1
  16. 16. TPC-C: SCALING Each new warehouse → +10 districts, + 100 items, +300k customers Maximum 1.2 tpmC per terminal 10 terminals per warehouse Scaling factor: warehouse (W)
  17. 17. TPC-C: CLUSTERED AND NON-CLUSTERED Clustered >1 nodes Shared nothing (fully federated) Sharded Shared storage (RAC) Non-clustered 1 node
  18. 18. TPC-C: METRICS tpmC Transactions per minute tpmC / $ Transaction cost Hardware acquisition cost Software and services cost for 3 years W / ktpmC Energy consumption per 1000 transactions per minute
  19. 19. TPC-C INTERPRETATION By Alan Parker (Alan Parker. Tuning databases on Solaris platform. Prentice-Hall, 2002) •…not only order entrytpmC × 2 •…if transaction monitor does not usedtpmC / 2 •…if Oracle Forms-like client usedtpmC / 3 •…if lightweight client forms used (like curses-based)tpmC × 2 / 3 •…if SQL has not been tunedtpmC / 2 • …if there are heavy reports and batchestpmC / 2
  20. 20. COULD WE CHEAT TPC-C? mount -t tmpfs -o size=2048g tmpfs /u01/tablespaces CREATE UNLOGGED TABLE … _ALLOW_RESETLOGS_CORRUPTION = TRUE _IN_MEMORY_UNDO = TRUE _DB_BLOCK_HASH_LATCHES = 32768 …
  21. 21. RESPONSE FROM COUNCIL A l l A C I D a s p e c t s c h e c k s s t i f f l y i n t e r p r e t e d i n s p e c i f i c a t i o n Single node reboot failure Synchronous commit on two nodes Redo logs and recovering Reboot and recovering Single media failure protection Synchronous commit on two separately powered media Write-ahead logs on separately powered media Commit = written on durable media
  22. 22. TPC-C: CRITICISM OF 1990S Even wholesale providers works somewhat differently! Order: some unsuccessful wildcard searches Printing report after each entry (probably, 3 times) Balance refresh for each operation is impossible in a heavy load environment → insert + batch calc Trivial logic No declarative constraints No trigger logic Non-typical loads What about reporting? Decision support systems?
  23. 23. TPC: OBSOLETED TPC-D 1995–1999 First attempt for OLAP benchmark TPC-R 2001–2005 Reporting TPC-W 2001–2005 Online web- commerce
  24. 24. TPC-H 1999 : OLAP vs OLTP “Ad-hoc decision support” Instead of TPC-D (considered irrelevant) “Weight categories” 100 GB 300 GB 1 TB 3 TB 10 TB 30 TB 100 TB Parallel load 22 kinds of complex queries 2 kinds of DWH refresh
  25. 25. TPC-H: MODEL 1 SF ~ 1 GB Declarative constraints Not star schema
  26. 26. TPC-H Q1 SELECT l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice*(1-l_discount)) as sum_disc_price, sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order FROM lineitem WHERE l_shipdate <= date '1998-12-01' - interval Δ day (3) GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus Δ = random [60…120] “ F u n c t i o n a l Q u e r y D e f i n i t i o n ”
  27. 27. TPC-H: RESULTS, 2017 Tricky to apply for non-relational DBMS Variants to apply for MOLAP (MDX) Reports with application for Apache Hive Exasol MS SQL Server Oracle Database Actian Vector 1 – 10 results for each “weight category”
  28. 28. TPC-E: “PRAGMATIC OLTP” Still OLTP, but more “hybrid” Declarative constraints No transaction monitors More reads More kinds of load
  29. 29. TPC-E: MORE COMPLEX WORKFLOW
  30. 30. TPC-E: MODEL Image: ©Transaction Processing Council, 2009
  31. 31. 78 results All non-clustered, all on MS SQL Server for Windows Server x64
  32. 32. 11 KTPSE, $144/TPSE (1,6 MIO) Изображение: ©Lenovo, © TPC, 2015
  33. 33. TPC-ABCEH
  34. 34. BRAND NEW: TPC-X И TPCX-X TPC-DI ETL workloads TPC_DI_RPS 0 results TPC-DS Decision support systems “on Big Data solutions, such as RDBMS as well as Hadoop/Spark based systems” QphDS@Size 0 results TPCx-BB Express benchmark for “Hadoop- based Big Data Systems” adopted from BigBench BBpm@Size 0 results TPCx-HS Express benchmark for HDFS- compatible systems, adopted from TeraSort HSph@Size From 1 to 4 results in different “weight categories”
  35. 35. PUBLISHING? Publishing on TPC.org Academic or research paper, could not be applicable for marketing purposes With definite clause that outcome is not comparable with TPC.org results With permission of TPC.org Publishing TPC benchmarks is prohibited, except the following cases: Required audit from sizing.com
  36. 36. DEWITT CLAUSE IN 2017? MSFT EULA •“You may not disclose the results of any benchmark test … without Microsoft’s prior written approval” OTN Lic. •“You may not disclose results of any Program benchmark tests without Oracle’s prior consent” IBM IPLA •“Licensee may disclose the results of any benchmark test of the Program or its subcomponents to any third party provided that Licensee, if … A) … B) … C)…”
  37. 37. NEW BENCHMARKS FOR NEW WORKLOADS 2010s 2000s 1990s 1980s OLTP ROLAP ROLAP In-memory OLAP TCP-DS ROLAP TCP-H Graph queries LinkBench MapReduce HiBench OLTP OLTP[E] OLTP[E] TPC-E OLTP Atomic access YCSB TPCx-HS TPCx-BB
  38. 38. PRINCIPAL BENCHMARKS OF “NEW WAVE” Terrasort Benchmark BigBench Intel HiBench Yahoo! Cloud Services Benchmark Linkbench • Adopted in TPCx-HS • Adopted in TPCx-BB • Map-Reduce series • Group of NoSQL etalon runs • Graph-like workload on RDBMS and HBase
  39. 39. INTEL HIBENCH ДЛЯ HADOOP Image:Intel,2013
  40. 40. YAHOO! CLOUD SERVICES BENCHMARK New tool for “benchmark marketing” Used by researchers for wide comparisons (V. Abramova et al. Experimental Evaluation of NoSQL Databases // IJDMS Vol.6, №3, 2014) Cassandra HBase Elastic search MongoDB Oracle NoSQL OrientDB Redis Scalaris Tarantool Voldemort Bombardment from one load station Atomic operation instead of transaction (probably, read a few records)
  41. 41. YCSB: WORKLOADS
  42. 42. LINKBENCH Facebook workload •Early graph benchmarks were graph-analysis oriented •Real workload of the Internet Giant •Transactions (MVCC) Методологично •Statistic laws for data generageion •Measures: average response time for each type of workload •Avg from 99-percentile •Documents in seconds for write •Queries per second for read MySQL (InnoDB vs TokuDB) HBase MongoDB / TokuMX OrientDB T h e Q u e s t i o n o f Fa c e b o o k
  43. 43. POCKET TOOLS TO MEASURE Possible? Correct? Representative? Comparable? Repeatable?
  44. 44. PGBENCH TPC-B Standard part of PostgreSQL distributions One simple command to run Standard de facto for PostgreSQL internal investigations, such as: XFS or ext4? Tables on SSD, indexes on HDD? Block size: 4K or 8K?
  45. 45. SYSBENCH Widely used for internal comparisons in MySQL, MariaDB, and its forks communities Sui generis tests, not similar to standard benchmarks MySQL utility (by Alexei Kopytov) fileio cpu memory threads mutex oltp
  46. 46. TPCE-MYSQL TPC-E loader for MySQL by Percona
  47. 47. HAMMERDB Free Java GUI program for TPC-C (?) and TPC-H (?) with support a range of DBMSs Oracle Database Microsoft SQL Server IBM DB2 TimesTen MySQL MariaDB Postgre SQL Postgres Plus AS Greenplum DB Redis Amazon Aurora (MySQL) Amazon Redshift (ParAccel) Trafodion SQL on Hadoop
  48. 48. HAMMERDB: ONE-CLICK TPC-C?
  49. 49. HAMMERDB: TPC, BUT NOT QUITE… Modern benchmarks were not been implemented TPC-E TPC-DS TPC-H Single load station TPC-C Not emulated full environment, no transaction monitor software Load station in role of single transaction monitor
  50. 50. HAMMERDB: POPULARITY Used by hardware and software vendors Indirectly, in form of a “blog of our tech guy” Hub with benchmark results Section «Performance Data»
  51. 51. SWINGBENCH TPC-C-like workload for Oracle Database and TimesTen + Idiosyncratic OLTP benchmark with a lot of PL/SQL Oracle DB-specific tools for monitoring and analysis (AWR, etc.) Coordinator support GUI and command line Only internal comparability Non-representative in RAC environments
  52. 52. DELL BENCHMARK FACTORY FOR DATABASES Oracle Database MySQL MS SQL Server SQLite SQL Anywhere Commercial tool (Quest Software legacy) TPC-C TPC-D TPC-E TPC-H ASP3AP Supports a number of load stations (Windows)
  53. 53. OSDLDBT.SOURCEFORGE.NET •TPC-WDBT-1 •TPC-CDBT-2 •TPC-HDBT-3 •TPC-AppDBT-4 •TPC-EDBT-5 While the inspiration for these workloads are the TPC-<x…>, workloads are entirely different and results obtained from them should not and can not be compared to TPC results. The use of any supplied results of these tests for commercial purposes is expressly prohibited. MySQL PostgreSQL …расширяемо
  54. 54. OLTPBENCH github.com/oltpbenchmark/oltpbench Java tool for command line Supports any JDBC-enabled RDBMS Special version for Hstore (VoltDB ) TPC-C Wikipedia Synthetic Resource Stresser Twitter Epinions.com TATP AuctionMark SEATS YCSB JPAB (Hibernate) CH- benCHmark Voter SIBench (Snapshot Isolation) SmallBank LinkBench
  55. 55. TPC TOOLS Tools by Transaction Processing Council C sources files Not exists for TPC-C: just sample in spec “Do it yourself”: all connectivity and other stuff
  56. 56. TPC-* BY EXAMPLE По «отчётам о полном раскрытии информации» на TPC.org
  57. 57. RPE2 SAP SD 2-Tier TPC-C TPC-HSPEC jbb2005 SPEC CPU2006 Supercomposite indicator by Gartner (Ideas) RPE2-ERP RPE2-Java RPE2-OLTP RPE2-Compute Intensive
  58. 58. BENCHWARE Swiss kinfe by Manfred Drozd Peakmarks Benchware OraCPU PL/SQL op •[ops] PL/SQL alg •[ops] OraSRV In-memory SQL •[ms] •[dbps] •[tps] •[rps] OraSTO SeqIO •[GBps] •[iops] RandIO •[GBps] •[iops] OraOLTP OLTP Select •[rps] •[tps] OLTP Update •[rps] •[tps] OraLoad TransLoad •[rps] •[tps] BulkLoad •[rps] •[tps] OraAgg OraAgg & Rep •[rps] •[tps] Only for Oracle Database, only PL/SQL and SYS.V_$%
  59. 59. DATABASE MACHINES? Pre-configured appliances for databases Should be precisely measured and benchmarked?
  60. 60. TERADATA Latest Teradata publications with TPC-H: Licensed by «internal Qph» – tPerf [Traditional Performance]
  61. 61. EXADATA tps, Qph – not published “Passport metrics” (X6-8) ෍ 𝑉 × IOPS ≈ Const
  62. 62. IBM PURE DATA FOR OPERATIONAL ANALYTICS Qph not published… “Passport metrics” about input/output
  63. 63. “SQL IOPS” Measuring SQL IOPS for another DBMS? Statistical views (…IO_STATS…) IOPS from Oracle Database side Orion (Oracle IO Numbers) SLOB (recommended by EMC, Flashgrid) Benchware (?) DBMS_RESOURCE_MANAGER .CALIBRATE_IO
  64. 64. ATOMIZATION OF AGGREGATE METRICS tpmC QpmH SQL IOPS SQL bandwidth
  65. 65. METRICS ATOMIZATION: PRO Independent from predefined models and schemes Accepted by database appliance vendors Pocket tools availability Representative for wide class of DBMS and DBMS-like systems • Reprehensive not only for 3NF, snowflakes, stars • Included in passport metrics • Running from DB side • Could be interested for NoSQL
  66. 66. METRICS ATOMIZATION: …ET CONTRA Methodology Different results on the same environment from CALIBRATE_IO and Benchware Not clear application sense The same IO operations in different DBMSs produces different transaction counts Tool not exists for most of DBMS Found just for Oracle DB, IBM DB2, MS SQL
  67. 67. RUNNING REAL WORKLOAD AND APPLICATIONS BENCHMARKS Most practical approach?
  68. 68. WORKLOAD REPLAY AS TRUSTED EXPERIENCE For systems with full API access (usually JSON via HTTP) Logging and autocapture Replay with “sleeps” Splitting workload patterns (user types) Data scaling? Wokrload emulators JMeter LoadRunner … DB-side tools Oracle Real Application Testing (Database Replay) MS SQL Server Distributed Replay
  69. 69. BOOTSTRAPPING TRICKS How to rollout data? Clonning Repeated with random shift? Mixing with real data (open data) Impact to analytics Predictable query results Low selectivity Impact to OLTP Keys, indexes…
  70. 70. PACKAGED APPLICATIONS 1С benchmark series Metrics: count of concurrent users with acceptable response time Microsoft Dynamics AX Application Benchmark Toolkit Oracle E-Business Suite Standard Benchamrks Order-to-Cash OLTP Payroll … SAPS [SAP Application Performance Standard] with SD module
  71. 71. QUESTIONS FOR DISCUSSION •Could them reach popularity like TPC-C и TPC-H? •Megaconfigurations problem TPC-E и TPC-DS •Running TPC-B is simple •Running others is tricky or not fully compliant Pocket tool •Acceptance, adoption, standardization, interpretation •Other metrics of such kind? SQL IOPS & Bandwidth •Atomic (YCSB-type) •Graph (LinkBench-type) Standardization and generalization of benchmarks for new workloads •New types of applications: portals, groupware, … •New database architectures: sharding, in-memory databases, in-memory data grids Future of benchmarks on real up-to-date workload
  72. 72. THANK YOU! mailto:anikolaenko@ibs.ru mailto:anikolaenko@acm.org Cover image: Paul Hudson, CC-SA

×