Quality assurance is important in the industrial settings

1. Quality Assurance of NAO Value for Money Studies
Why does the NAO ask external reviewers to review its VFM reports?
The external review process was set up in 1993 to allow the NAO to benefit from detailed
and independent scrutiny of its VFM reports by academics. We were the first audit
institution in the world to establish such a process and our aim was to demonstrate that we
were confident about what we did and were prepared to learn lessons from independent
experts. Initially, the academic reviews are carried out on all published reports but, since
2006, on draft reports. Since 2004, the work has been shared between Oxford University
and the London School of Economics (LSE) under contract following competition. Both
Universities have demonstrated that they can draw on a breadth of knowledge and
experience in work across the whole of government, and that they can provide authoritative
comment on study methods and the technical rigor of our analysis.
What criteria do the academic reviewers use to evaluate NAO reports?
LSE and Oxford award each report a set of scores, for the dimensions of performance listed
below. The assessment criteria were developed in conjunction with our first reviewers, the
LSE. They score reports on a scale of one to five: five is excellent, three is a good quality
professional performance, and anything below three falls below the quality standards that we
set ourselves and aim to meet.
Review Criteria prior to 2006
Administrative, managerial and financial To what extent does the report clearly
context Describe the financial, administrative
and managerial context within which
the area examined is carried out?
Structure, presentation and format To what extent is the report well
structured and well-written, and does it
include an effective executive summary?
Graphics and Statistics To what extent were the graphics and
statistics appropriately used, and how
well presented were any that appeared?
Scope To what extent is the rationale for the
report’s scope clearly set out (within
the NAO’s remit, which excludes
questioning the methods of policy
objectives)?
Methodology How far is the methodology clearly set
out? And does it include an appropriate
range of evaluative criteria and
techniques to answer the question posed
in the specification?

2. Conclusions and recommendations To what extent were the report’s
conclusions and recommendations
balanced, logical, consistent and
supported by the evidence cited?
Objectives To what extent was the study successful
in meeting its objectives and providing
useful information to help improve
public services?
Changes to the review process
Prior to 2006, external reviews of published reports carried out by LSE and Oxford
University enabled us to monitor product quality over time, and played an important part in
promoting improvements in the quality of the NAO’s VFM work. But, by taking place post
publication they could not have any impact on the quality of the specific report reviewed.
Since 2006, the external reviews have taken place prior to report publication. The new
arrangements provide an opportunity to enhance the quality of our work by drawing on
external perspectives before publication. LSE and Oxford University have continued their
roles as reviewers, and they examine the draft report against an agreed set of criteria. They
provide a written report with scores against the
criteria so that we have the means to monitor trends in quality of VFM reports over
time.
Review Criteria from 2006 onwards
Scope Is the rationale for undertaking the
study clearly set out together with its
objectives?
Administrative and managerial context Does the report clearly describe the
administrative and managerial context?
Financial Analysis Does the report include sufficient
financial analysis and quantitative
information on costs, benefits and
performance? If not, what would you
have expected?
Structure and presentation Is the report well structured and well
written, does it include an effective
Summary? Does it convey the messages
clearly?
Graphics and statistics Are graphics and statistics appropriately
used and well presented in their draft
form? Could they be improved?
Methods Is the methodology clearly set out and
sufficient to assess the robustness and

3. quality of the data? Does the report
include an appropriate range of
evaluative criteria and techniques? Is
there any discussion of the limitations
of the data?
Value for Money Does the report reasonably conclude
Whether VFM has been achieved, or is
likely to be, together with an indication
of the action needed to achieve better
VFM? Does the report say enough about
opportunities to improve a)
productivity, b) effectiveness and c)
service performance?
Recommendations Are the recommendations sufficiently
Specific, cost-effective, convincing and
supported by the underlying evidence?
Are there any recommendations which
should be added?
Evidence and analysis As a report to Parliament is the draft
sufficiently convincing in its use of
evidence and analysis?
The reviewers take around three weeks to carry out their review. At each institution, an
editorial board oversees the process. It receives the draft review from specialist academics
from around the university, and carries out a moderation process to ensure consistency of
marking and the quality of input made by the reviewer. Where necessary, the board will go
back to the reviewer to ask them to look again at certain aspects.
The reviews are normally about 10 pages in length, providing written comments on each of
the categories.