The document provides a comprehensive overview of database benchmarking using Benchmark Factory, highlighting the importance of expertise in database management, tools, and preparation to avoid common mistakes. It discusses industry-standard benchmarks, metrics, and the architecture of Benchmark Factory, emphasizing the need to optimize for average response time rather than just transactions per second. Additionally, it offers resources for further learning and insights into benchmarking best practices.

1. 1 Global Marketing
Bert Scalzo
PhD, MBA & Oracle ACE
Database Expert & Product Architect
Avoiding the Risks of
Database Change
using Benchmark
Factory

2. 2 Quest Software is now a part of Dell
Books by Bert Scalzo

3. 3 Quest Software is now a part of Dell
• Database Benchmarking Basics
• Benchmark Factory Overview
• Benchmarking Options
• Customizing Benchmarking Projects
• Running a Benchmark
• Examining Runtime Reports
• Error Messages and Log Files
• Benchmark Factory Demonstration
• Questions & Answers
Purpose / Agenda

4. 4 Quest Software is now a part of Dell
Database Benchmarking is complicated
Even with a Benchmarking Tool
There is no
substitute for
your
knowledge and
expertise

5. 5 Quest Software is now a part of Dell
Successful Database Benchmarking
Expert – must know database, O/S,
hardware, software, tuning & have
read the benchmark specsStress test the database
?

6. 6 Quest Software is now a part of Dell
Database
Benchmarking
Most
Common
Mistakes

7. 7 Quest Software is now a part of Dell
Wrong Person
Doing the
Heavy Lifting

8. 8 Quest Software is now a part of Dell
Trying to
make do
without the
right tools

9. 9 Quest Software is now a part of Dell
Wrong number
of virtual users
Don’t cut
corners

10. 10 Quest Software is now a part of Dell
Industry Standard Database Benchmarks
• Industry Standard Database Benchmarks have been around for over 20 years
• TPC.ORG
– TPC-C: Older OLTP benchmark. Basic “order entry” type application.
– TPC-H: Data warehousing queries. 22 queries – not star schema design.
– TPC-E: Newer OLTP benchmark. Simulates workload of brokerage firms.
– TPC-DS: New, evolving complete DW. Spec still evolving – not yet in BMF.
– TPC-VMS: New, evolving virtual DB’s. Spec still evolving – not yet in BMF.
• Be Sure to Read the Specs
– Understand basic database design
– Implementation options up to you
– Sizing options for data load and concurrency
– How to interpret results and/or gauge success
• Majority of failures due to insufficient preparation, unrealistic expectations,
and uncertain metrics for success

11. 11 Quest Software is now a part of Dell
Examples of Info in the Spec: TPC-C
BMF Prompts
for DB size –
you must know
how to answer
100 concurrent
users = scale
factor of 10

12. 12 Quest Software is now a part of Dell
Examples of Info in the Spec: TPC-H
BMF does not
know what
Oracle options
you are licensed
for (partitioning),
nor your
hardware setup
such as spindles
per LUN, RAID
level, CPU
count, RAM size,
and son on.
BMF creates
basic & correct
tables and
indexes – you
must manually
optimize them
within spec.

13. 13 Quest Software is now a part of Dell
Database Design – Reference Full Disclosure Reports for
Similar Setups

14. 14 Quest Software is now a part of Dell
Database Design
Reference Full Disclosure Reports for Similar Setups

15. 15 Quest Software is now a part of Dell
Database Design – Appendix B Will Show DDL for Optimal
Design
Very common technique
to load very large data sets
into ETL or staging tables,
and then to do create
table as select (CTAS) to
populate the benchmark
tables

16. 16 Quest Software is now a part of Dell
Common Database Benchmark Metrics
TPS and Avg. Response Time
• Transactions per Second
– Gets far too much attention
– Meaningless to most users
– Sort of like automobile RPM’s (how fast internal engine is working – not how fast car
is moving or how soon we’ll arrive)
– Misconception that TPS equates to IOPS (IO Operations per Second) – ignores
database memory caching and logging
– BMF does not report IOPS (on roadmap for possibly 7.0)
• Average Response Time
– Gets far too less attention
– Meaningful to most users
– Sort of like MPH (or KPH) (how fast car actually is moving – so infers how soon we’ll
arrive or amount of fuel we’ll use)
– When examined in conjunction with TPS, then a generally observable and clear
pattern often emerges (next slide) …
• Be sure to read my blog: The Myth of Transactions per Second (TPS)

17. 17 Quest Software is now a part of Dell
Common Database Benchmark Pattern – True Point of
Saturation
0
1000
2000
3000
4000
5000
6000
7000
100 200 300 400 500 600 700 800 900 1000
TPS Avg Resp Time
Looking for inflection
point where TPS is still
increasing or just
decreasing and close to
max where average
response time is below
customer defined SLA
Notice the line
characteristics between
roughly 750 and 850
concurrent users – for
current configuration
and optimization
benchmark results
interesting
Common mistake to simply attempt maximize TPS – remember TPS is not IOPS

18. 18 Quest Software is now a part of Dell
Benchmark Factory Architecture – Console (Control
Center) and Agents
You must manually install BMF
agent software on servers –
otherwise BMF will simply spawn
agents on your console machine
(major bottleneck)
Each 32-bit BMF agent can handle
500 concurrent user sessions
Each 64-bit BMF agent can handle
1,000 concurrent user sessions
Each server running one or more
agents requires its own separate
BMF license!
Agent servers can be virtual
machines

19. 19 Quest Software is now a part of Dell
Hardware Sizing is Very Difficult
Requires Lots of Thought
Let’s assume doing TPC-C
benchmark for sale factor 100,
so 1,000 concurrent users, and
roughly 8-10 BMF agents
Note we already have a huge
built-in bottleneck – there’s
only one NIC for 1,000
concurrent database sessions!
Let’s assume using virtual
machines for entire BMF
architecture on a seemingly
large VMware ESX / vSphere
server

20. 20 Quest Software is now a part of Dell
Benchmark Factory Architecture
BMF Offers Two Repository Options
DEFAULT

21. 21 Quest Software is now a part of Dell
BMF Offers Many Types of Tests
Industry Standard Benchmarks

22. 22 Quest Software is now a part of Dell
BMF Offers Many Types of Tests
Scalability & Workload Capture/Replay

23. 23 Quest Software is now a part of Dell
BMF Offers Many Types of Tests
Custom Load Scenario

24. 24 Quest Software is now a part of Dell
BMF Offers Many Types of Tests
Goal Testing to Meet Customer SLA
BMF will test from 100
to 1,000 concurrent
users in increments of
100 additional users,
and will stop when
the average response
time is more than 2
seconds (e.g.
common customer
defined SLA).

25. 25 Quest Software is now a part of Dell
Example of Benchmark Factory Project – Default Creation

26. 26 Quest Software is now a part of Dell
Example of Benchmark Factory Project
Custom (Highly Recommended)

27. 27 Quest Software is now a part of Dell
Example of Benchmark Factory Project
Custom (Highly Recommended)

28. 28 Quest Software is now a part of Dell
Example of
Benchmark
Factory
Running

29. 29 Quest Software is now a part of Dell
Benchmark
Factory
Runtime
Reports –
Results from
Runs

30. 30 Quest Software is now a part of Dell
Benchmark Factory Error Logging – Where to Look

31. 31 Quest Software is now a part of Dell
Benchmark Factory Error Logging – Where to Look

32. 32 Quest Software is now a part of Dell
On-Demand
Webcast -
Intro to
Database
Benchmarking
and product
demo
• Download the webcast now

33. 33 Quest Software is now a part of Dell
• www.toadworld.com
– Expert Blogs
– Articles
– How-To Videos
– User Forum
– Wiki
– Direct access to
product experts and
much more
Additional Resources

34. 34 Quest Software is now a part of Dell
Additional
Resources
• Benchmark Factory Home Page
• Download a free 30 day trial edition of Benchmark Factory
• White Paper – Top 10 Benchmarking Misconceptions
• On-Demand Webcast - Introduction to Database
Benchmarking

35. 35 Quest Software is now a part of Dell