Key Principles for Efficient Test ReportingProviding the right information to the right stakeholders
Purpose• This document outlines the key principles of efficient test reporting• This is a high level presentation about test reporting that must be implemented differently for every context and stakeholder• Following these key principles will give a good base for setting up efficient test reporting
Key Principle: Stakeholder Expectations• What info is suitable for the intended stakeholders? – What info do they want? – What info do they actually need? – What decisions are being made based on the information?• How do the stakeholders need the information reported to easily be able to use it?• What granularity do they need? High level overview or technical details?• What data to they need to pass along in their consolidated reports for others to make decisions on?• Answers the question: What, how and why do we report?
Key Principle: Risk / Mitigation• Why did we test at all?• What were the goals of the test activities?• What product or project risks initiated the test activity/activities that the report covers?• Did the testing mitigate the problems/risks?• How can remaining risks be mitigated?• Answers the question: Why did we test?
Key Principle: Confidence in Measurements• It must be clear what the scope for the test activities were on different abstraction levels so that stakeholders can get a good overview but also gradually go into more detail• The stakeholder must be able to understand how confident we are in our test results• If status is green, how good are the chances that it will stay green, and how much risk is there that the status will change over time• Are the test results reliable as decision material• Answers the question: What did we test, and what didn’t we test?
Key Principle: Qualitative / Quantitative• We need both data and an analysis of that data from a professional in the report• The analysis leads to a statement about the product, and this statement needs to be backed by empirical data• The empirical data needs context that only a professional can put it in – the product statement is this context• One cannot be reported without the other• If focus should be on the data or the analysis depends on the stakeholders• Answers the question: How do we report?
Key Principle: Traceability in Results• The results reported in the Test Report must be reproducible by stakeholders• There must be traceability between the results and how it was executed• Some or all of the following variables must be covered to create traceability – Which tools were used – How was the test environment configured – Work process – Links to test cases – Links to defects found – Links to logs and other detailed test artifacts• Answers the question: How did we test?
Summary• Using these key principles will not automatically give good test reporting, but not following these key principles will most likely give insufficient test reporting• Know your stakeholders, and understand the key principles of test reporting