We must understand the difference between testing statistics, metrics, and reports to convey and or extract the desired information:
• Statistics captures the desired current status
• Metrics tell the story of how well the desired is performing
• Reports combine statistics and metrics
2. Convey and or extract the desired
information
Price Charlot
We must understand the difference between testing
statistics, metrics, and reports:
Statistics captures the desired current status
Metrics tell the story of how well the desired is
performing
Reports combines statistics and metrics
3. Test Statistics
Price Charlot
Statistics capture what is happening at the moment or within
a specific time frame.
Example: averages, percentages, totals,…
The desired statistics are targeted to the needs of the
organization.
Example: while one must capture the averages, another may
not need that information.
4. Test Metrics
Price Charlot
Metrics are quantitative measurements that tell the story
of how well the system is performing.
Some metrics employ a color scheme:
Green: system performing as intended
Yellow: warns or concerns - unintended behavior(s) has been observed - these behaviors,
current do not impact system performance.
Orange: a system failure is emminent
Red: a system failure or break has occurred
Metrics are used to manage, drive, improve efficiencies and increase
overall satisfaction. In plain language – improved metrics means
improved system monitoring accuracy.
Measurements can include: error rates, response times, resource utilization, usability,…
Metrics should provide a clear picture of the system that identifies its strengths and
weaknesses and help to establish goals to improve or maintain.
5. Test Reports
Price Charlot
Statistics + Metrics = Reports:
Every system wants to promote efficiency and effectiveness.
Reporting must include statistics and metrics.
When both measurements are taken and reported on against
overall goals than the system can project a better picture of
total performance.
6. Price Charlot
Premises and Conclusions
test-case evaluation
GIVEN: defined test-cases
AND: raw test-cases execution results
WHEN: results a tailed
AND: results are interpreted
THEN: system performance is analyzed.
