Techniques for issue analysis and drawing conclusions

1. T
Summer 2009 FEATURE
ECHNIQUES
FOR ISSUE ANALYSIS & DRA
By Brian Vella, Wilfred Aquilina and Melvyn Darmanin
W ING CONCLUSIONS
Introduction directing attention on defining clearly what the gathered
evidence shows, as well as what the audit report and the
The vision of the National Audit Office (NAO) is to accompanying press release should say.
provide a multidisciplinary professional service to
parliament, to government and the taxpayer and to be The IADC methodology was introduced in the National
an agent of change conducive to achieving excellence Audit Office of the United Kingdom1 in 1993. The
in the public sector. One of the most important ways approach was derived in part from concepts developed
of how NAO seeks to realise its vision is by delivering by Barbara Minto in her publication on the Pyramid
audit reports that are focused, accurate, logically Principle2 and has been increasingly used by other audit
rigorous, clear and useful. institutions, including the European Court of Auditors.
To do so effectively, NAO is applying a new approach Workshop on IADC in collaboration
called ‘Issue Analysis/Drawing Conclusions’ (IADC) with the Wales Audit Office
to its audit work. The IADC methodology is a mixture
of logical and practical rules aimed at delivering A workshop was recently organised by the NAO (Malta)
clear and well-structured reports in a timely and cost- in collaboration with the Wales Audit Office (WAO) to
effective manner. The approach involves the following discuss the principles and techniques of IADC.
two specific processes:
Two trainers from WAO, Ms Sue Morgan and Mr James
The Issue Analysis process: The techniques of Verity, conducted the workshop in Malta, sharing their
Issue Analysis are applied during the initial issue expertise and experience on applying the rigorous and
identification, the development of audit questions and structured approach of IADC to all the stages of a
the design of audit fieldwork. The process helps the performance audit study.
audit team to determine the scope of the audit and to
obtain an understanding of the issues that are relevant
to the project. The framework is also used to test the
relevance and feasibility of the audit objectives, and
to build the audit plan and to define what evidence is
needed to complete the work.
The Drawing Conclusions process: The methods
used for Drawing Conclusions are focused on
generating logical explanations based on the collected
evidence from fieldwork and analysis. The techniques
are particularly useful to develop the structure and
summary of the audit report and can also be effectively Participants in the IADC workshop organised by NAO in March 2009,
used during engagement with the audited entities. with the Auditor General, Mr Anthony C. Mifsud, and the Deputy
The process facilitates the work of the audit team by Auditor General, Mr Charles Deguara
1 National Audit Office (2003) User Guidance: The Issue Analysis and Dinner Party Approach (IADPTM). London: National Audit Office.
2 Minto, B. (2002) The Pyramid Principle: Present your thinking so clearly that the ideas jump off the page and into the reader’s mind. Harlow: Pearson
Education Limited.
32

2. FEATURE Summer 2009
33

3. Summer 2009 FEATURE
Real projects were used as case studies during
this two-day training event. NAO auditors Key audit question
Example: Did the outsourcing of service X by Department Y represent good value for money?
had the opportunity to practice critical skills
related to IADC, including the ranking of Level 2 Questions Was the Department right to persue the outsourcing Did the Department obtain a good price Is the Department achieving optimal delivery
and performance of service X?
different issues, the challenging of ideas, as
of service X? for service X?
well as facilitation of IADC sessions. Level 3 Questions
Application of the IADC Level 4 Questions
etc.
approach
Figure 1: The Issue Analysis Pyramid*
Issue Analysis
* According to Minto, issues at any level of the pyramid must always be summaries of the
Issue Analysis takes place once sufficient issues grouped below them. Furthermore, the issues in each grouping must always be logically
information has been collected during the ordered and of the same kind of idea. For more details refer to Minto, B. (2002) The Pyramid
preliminary study stage to allow a substantive Principle: Present your thinking so clearly that the ideas jump off the page and into the reader’s
discussion of the issues that are relevant to mind. Harlow: Pearson Education Limited pages 13-15.
the audit. The initial set of issues is typically
arrived at through desk research, preliminary logical ordering principle in order to provide In addition, the following two practical rules
interviews with stakeholders, brainstorming overall structure and clarity of the issues are also applied to the IADC approach:
and other creative thinking techniques. below the big question. It is important to
point out that there is no “correct” ordering of The ‘Rule of Seven’: The rule states that
The Issue Analysis process leads to the issues and there can be several ways of how the number of sub-issues for any given issue
identification of the most appropriate overall to structure questions. A particular ordering in the pyramid should not exceed seven.
question that the audit will address and a approach might be more practical or suitable The principle is intended to avoid situations
logical hierarchy of supporting questions. depending on the circumstances. What matters whereby an issue is divided in an excessive
The starting point for Issue Analysis is is that the selected ordering principle is valid number of sub-issues making it difficult for
the “Situation-Complication-Key Audit and consistently applied. most readers or stakeholders to comprehend.
Question”, a structure developed by Barbara If a high number of sub-issues are identified
Minto 3,4 : Barbara Minto suggests three approaches to during Issue Analysis, these can be regrouped
The Situation should include a brief, non- ordering thoughts in a logical manner5: so that the Rule of Seven is not breached.
controversial descriptive statement aimed at By time: this is the simplest to understand
reminding and inspiring readers. It should as it shows the causes of an effect, be
provide factual background information. it chronological, systematic or part of
The Complication represents the “so a process, with issues being placed
what?” factor, that is a statement on what is by order of occurrence (first to occur,
complicating the situation. It should highlight second to occur, etc.).
what makes the study worthwhile. By structure: dividing the whole
The overall Key Audit Question should into parts and covering all the pieces
be the obvious question that emerges in the structure, for example, dividing
logically after developing the Situation and issues by Government entity, by
Complication. Its purpose is to clearly indicate relevant organisational functions, by
the main focus and scope of the study. stakeholder, by geographical location/
distribution, by demographic subsets,
The overall structure of questions should or by using a particular framework
take the shape of a pyramid (Figure 1) and (such as the COSO6 or the EFQM7
the questions should be composed in a way models); Presentation of a case study during the IADC
that they are answerable with a ‘yes’ or ‘no’. By rank: grouping (by order of importance) workshop in March 2009
This is important in order to ensure sufficient issues possessing to a different degree a key
rigour during the collection of evidence and characteristic (such as three problems, four The ‘Rule of One’: It is also important that
to direct the audit team’s analysis towards the reasons, or five variables). These can be the number of sub-issues always exceeds
formulation of clear and specific conclusions presented in the order in which they possess a one. If the audit team finds that an issue is
on the area being audited. Using questions that characteristic: first, second, third, etc. underpinned by only one sub-issue then the
are not answerable by ‘yes’ or ‘no’ (such as audit team may need to review how the issue
“how…”, “how well…”, “how effective…”, Furthermore, issues contained in the pyramid has been formulated or else re-check the
“what…”, “why…”) can lead to the risk must always be mutually exclusive of each analysis to ensure that there are no missing
that insufficient evidence is collected during other (that is, different, distinct and not sub-issues.
fieldwork to answer conclusively the key overlapping with one another) and collectively
audit question. exhaustive (the answers to the sub-issues must The role of the facilitator is critical for the
be sufficient to answer the overall high level success of an Issue Analysis meeting. S/he must
The issues should also be grouped using a question).8 possess skills in general facilitation to ensure
3 Ibid. pages 37-49. 4 http://www.barbaraminto.com 5 Ibid. pages 93-112 6 COSO stands for the Committee of Sponsoring Organisations, an organisation that provides
guidance on critical aspects of organisational governance, business ethics, internal control, enterprise risk management, fraud, and financial reporting. 7 EFQM is the European
Foundation for Quality Management. 8 Ibid. page 96.
34

4. FEATURE Summer 2009
optimal participation during the meeting. The up approach through which the full set of rigorous, reasonably brief but comprehensive
facilitator should also be experienced with the emerging audit findings (based on the initial enough to cover the following:
IADC techniques as s/he will need to ensure issue analysis) are used to determine what the The high level answer to the overall key
that the rules are followed. Furthermore, the high-level conclusions and headings should question the audit set out to address;
facilitator should not take a particular position be and how they will be logically structured. The reasons for that answer presented in a
in the discussion. S/he must keep the meeting clear and logical structure; and
focused on concepts, not letting participants The rules of Issue Analysis discussed above A summary of the actual evidence gathered
get bogged down in low level issues at the (for example, use of a facilitator, the Rule during fieldwork to support the
expense of more important top level issues. of Seven, the Rule of One, issues must conclusions.
be mutually exclusive and collectively
Using the approach to carry out the fieldwork exhaustive, etc.) should also be applied to NAO is making increased
Drawing Conclusions.
Issue analysis is not an end in itself but a tool
use of IADC techniques in its
for identifying issues and designing fieldwork The structure of the Drawing Conclusions performance audit work
based on those issues. pyramid can look different from the initial
Issue Analysis pyramid partly due to the The IADC framework is being increasingly
Once the issue analysis pyramid has been refinement of initial thoughts and also because applied by NAO on a diverse range of
developed to its lowest level, the audit team other issues may have emerged during performance audit studies. The approach
should use the established structure to identify fieldwork and analysis. requires sound preparation, good facilitation
the audit tasks that need to be carried out skills, focus, structured logical thinking and
to complete the project. These tasks, once Obviously, Drawing Conclusions can be clarity. Our experience is showing that the
determined, are then used to define, for carried out only when at least most of the additional work and rigour required by IADC
example, detailed fieldwork methodologies, fieldwork and analysis has been completed invariably pays off during planning, fieldwork
evidence sources, resource requirements and the audit team has answers (‘yes’, ‘no’ and report writing by helping our audit teams
(including expertise), and the individuals who and perhaps a few ‘maybe’) to all the issues to produce well structured audit plans, findings
will be carrying out the tasks. and sub-issues. and reports.
Drawing Conclusions The product of the Drawing Conclusions For more information on the National Audit
process should be a clear and interesting outline Office and on the IADC approach, you can
Drawing Conclusions should take place structure that encapsulates briefly all the key send an email to nao.malta@gov.mt or visit
after audit fieldwork and analysis has messages that will be included in the eventual our website at www.nao.gov.mt.
been completed. The process is a bottom- audit report. The outline structure should be
LOCAL SOLUTIONS
FOR LOCAL BUSINESS
ONE INTEGRATED
MODULAR SOLUTION
Shireburn Shireburn Shireburn Shireburn
Financial Inventory Point-of-Sale Payroll System
Manager Management (SPOS) offers (SPS) offers a
(SFM) is a fully System (SIMS) the highest levels wide range of
fledged financial offers effective of management functionality
accounting control of of POS and capable of
solution allowing inventory retail operations handling
an organisation to functions combined with payroll and HR
implement those across the simplicity of use. functions within
modules suitable manufacturing, organisations.
for their particular distribution and
business. retail spectrum.
Tel: 2131 9977 info@shireburn.com www.shireburn.com
35