Satisfying Auditors: Plans and Evidence in a Regulated Environment

MM
PM Tutorial
10/13/2014 1:00:00 PM
"Satisfying Auditors: Plans and
Evidence in a Regulated Environment"
Presented by:
James Christie
Claro Testing
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

James Christie
Claro Testing
James Christie is a testing consultant with thirty-one years of IT experience. Before moving
into testing, James spent six years as an IT auditor, so he has experience on both sides of the
fence. With experience in information security management, project management, business
analysis, and development, he is particularly interested in links between testing, auditing,
governance, and compliance. James spent fourteen years working for a large UK insurance
company, then nine years with IBM working with large clients in the UK and Finland. A member
of the Information Systems Audit and Control Association, James has been self-employed for
the past eight years.

9/11/2014
1
Satisfying Auditors:
Plans and Evidence in a
Regulated Environment
James Christie
How I ended up in software
testing via auditing.
Why Alice in Wonderland was
relevant to my attempts to
understand what goes on in
big companies.
An introduction – to me and the tutorial 1a

9/11/2014
2
“The chief difficulty Alice
found at first was in
managing her flamingo”
An introduction – to me and the tutorial 1a
“When I use a word,”
Humpty Dumpty said in rather a scornful tone,
“it means just what I choose it to mean —
neither more nor less”.
1b
Nothing seemed to make sense

9/11/2014
3
1c
Something like sanity
Something like sanity 1c

9/11/2014
4
Y2K – a testing time 1d
Image courtesy Stuart Miles & FreeDigitalPhotos.net
Information security management – the IBM way 1d

9/11/2014
5
Some internal audit departments have an image problem
Ambiguous? I’m not sure exactly what this means,
but it’s not good.
1d
Back to testing again 1d

9/11/2014
6
And out… 1d
External & internal auditors 2a

9/11/2014
7
“External auditors are watchdogs not bloodhounds”
2b
Providing an opinion to the shareholders
about whether the accounts are true and fair.
Images courtesy Artur84/FreeDigitalPhotos.net
Providing an opinion to the shareholders
about whether the accounts are true and fair.
External auditor independence
Such a big problem it’s more than just a problem.
2b

9/11/2014
8
“Commercial suicide”, alleged quote from current
chair of UK Financial Conduct Authority.
John Griffith-Jones
External auditor independence
Challenging client management?
2b
Images courtesy Artur84/FreeDigitalPhotos.net
Problem #1 - up or out
Images courtesy Stuart Miles, Renjith Krishnanur84/FreeDigitalPhotos.net
”It would seem that about 50% of newly qualified ACAs”It would seem that about 50% of newly qualified ACAs”It would seem that about 50% of newly qualified ACAs”It would seem that about 50% of newly qualified ACAs
do not have enough practical experience to continuedo not have enough practical experience to continuedo not have enough practical experience to continuedo not have enough practical experience to continue
their careers astheir careers astheir careers astheir careers as auditors”auditors”auditors”auditors”
MichaelMichaelMichaelMichael IzzaIzzaIzzaIzza, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009
2b

9/11/2014
9
””””Most internal auditors would join me in assessing theMost internal auditors would join me in assessing theMost internal auditors would join me in assessing theMost internal auditors would join me in assessing the
external audit partners and senior managers as arrogantexternal audit partners and senior managers as arrogantexternal audit partners and senior managers as arrogantexternal audit partners and senior managers as arrogant
beyond their competencebeyond their competencebeyond their competencebeyond their competence””””
NormanNormanNormanNorman Marks, 2010 (Marks, 2010 (Marks, 2010 (Marks, 2010 (chief audit exec at major global
corporations for 20+ years))))
Problem #2 – quality of people
2b
Problem #3 - sampling
Auditors can’t checkAuditors can’t checkAuditors can’t checkAuditors can’t check
all theall theall theall the figures. Thatfigures. Thatfigures. Thatfigures. That
would make auditswould make auditswould make auditswould make audits
far too expensive.far too expensive.far too expensive.far too expensive.
But they can’t justBut they can’t justBut they can’t justBut they can’t just
take figures on trust.take figures on trust.take figures on trust.take figures on trust.
So theySo theySo theySo they sample.sample.sample.sample.
How much do theyHow much do theyHow much do theyHow much do they
samplesamplesamplesample????
How do they chooseHow do they chooseHow do they chooseHow do they choose
the sample?the sample?the sample?the sample?
Cartoons courtesy Scott Adams
2b

9/11/2014
10
Add up everything that moves through the books;Add up everything that moves through the books;Add up everything that moves through the books;Add up everything that moves through the books;
all revenue plus all costs, to get turnover=t. Let’sall revenue plus all costs, to get turnover=t. Let’sall revenue plus all costs, to get turnover=t. Let’sall revenue plus all costs, to get turnover=t. Let’s
say t=£25 million.say t=£25 million.say t=£25 million.say t=£25 million.
Problem #3 – sampling on the
Internal Controls Basis
Apply the accounts total & control score to theApply the accounts total & control score to theApply the accounts total & control score to theApply the accounts total & control score to the
sampling formula to get the sampling interval,sampling formula to get the sampling interval,sampling formula to get the sampling interval,sampling formula to get the sampling interval,
e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.
Assess the internal controls and assign a score, s,Assess the internal controls and assign a score, s,Assess the internal controls and assign a score, s,Assess the internal controls and assign a score, s,
egegegeg 1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.
2b
Internal Controls Basis
Pull out a bank note. Take the last three digits.Pull out a bank note. Take the last three digits.Pull out a bank note. Take the last three digits.Pull out a bank note. Take the last three digits.
Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.
Let’s say 0.851.Let’s say 0.851.Let’s say 0.851.Let’s say 0.851.
Sample your way through the accounts, examiningSample your way through the accounts, examiningSample your way through the accounts, examiningSample your way through the accounts, examining
every transaction you hit at the sample interval,every transaction you hit at the sample interval,every transaction you hit at the sample interval,every transaction you hit at the sample interval,
hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.
Apply the fraction to the interval to get theApply the fraction to the interval to get theApply the fraction to the interval to get theApply the fraction to the interval to get the
starting point, i.e. £42,550.starting point, i.e. £42,550.starting point, i.e. £42,550.starting point, i.e. £42,550.
2b

9/11/2014
11
Internal Controls Basis; gaming the system
(aka cheating)
Why might you want to manipulate the method,Why might you want to manipulate the method,Why might you want to manipulate the method,Why might you want to manipulate the method,
and how would you do it?and how would you do it?and how would you do it?and how would you do it?
You can rig the internal controls score to get theYou can rig the internal controls score to get theYou can rig the internal controls score to get theYou can rig the internal controls score to get the
result you want. The higher the score, the higherresult you want. The higher the score, the higherresult you want. The higher the score, the higherresult you want. The higher the score, the higher
the sampling interval, and the less work thethe sampling interval, and the less work thethe sampling interval, and the less work thethe sampling interval, and the less work the
auditors have to do.auditors have to do.auditors have to do.auditors have to do.
You can rig the formula, but that’s obvious andYou can rig the formula, but that’s obvious andYou can rig the formula, but that’s obvious andYou can rig the formula, but that’s obvious and
you’d have to justify it.you’d have to justify it.you’d have to justify it.you’d have to justify it.
REMEMBER THE FEE IS SET BEFORE THE INTERNALREMEMBER THE FEE IS SET BEFORE THE INTERNALREMEMBER THE FEE IS SET BEFORE THE INTERNALREMEMBER THE FEE IS SET BEFORE THE INTERNAL
CONTROLCONTROLCONTROLCONTROL SCORE IS CALCULATED.SCORE IS CALCULATED.SCORE IS CALCULATED.SCORE IS CALCULATED.
2b
Internal Audit
a totally different perspective
“Internal auditing is an“Internal auditing is an“Internal auditing is an“Internal auditing is an independentindependentindependentindependent, objective, objective, objective, objective
assurance and consulting activity designed toassurance and consulting activity designed toassurance and consulting activity designed toassurance and consulting activity designed to
add value and improveadd value and improveadd value and improveadd value and improve an organization'san organization'san organization'san organization's
operations.operations.operations.operations.
It helps an organization accomplish itsIt helps an organization accomplish itsIt helps an organization accomplish itsIt helps an organization accomplish its
objectives by bringing a systematic, disciplinedobjectives by bringing a systematic, disciplinedobjectives by bringing a systematic, disciplinedobjectives by bringing a systematic, disciplined
approachapproachapproachapproach to evaluate and improve theto evaluate and improve theto evaluate and improve theto evaluate and improve the
effectiveness of risk management, control, andeffectiveness of risk management, control, andeffectiveness of risk management, control, andeffectiveness of risk management, control, and
governance processesgovernance processesgovernance processesgovernance processes.”.”.”.”
Global InstituteGlobal InstituteGlobal InstituteGlobal Institute of Internalof Internalof Internalof Internal AuditorsAuditorsAuditorsAuditors
2c

9/11/2014
12
Internal Audit
The people are different
Image courtesy cooldesign & FreeDigitalPhotos.net
2c
Top six qualities internal auditors need
IIA’s 2013 Global Pulse of the Profession survey
Critical thinking
Communication
skills
Risk-management
IT knowledge
Data mining/analytics
Accounting
2c

9/11/2014
13
1 - Critical thinking
2 - Communication skills
3 - Risk-management
4 - IT knowledge
5 - Data mining/analytics (frauds! ☺)
6 – Accountancy knowledge
Communications
Data mining/analytics
Risk-management assurance
IT knowledge
Accountancy knowledge
Top six qualities internal auditors need
IIA’s 2013 Global Pulse of the Profession survey
2c
Internal auditors know more
Deeper business knowledge
Greater tacit knowledge
Greater nous (streetwise)
More mature & stronger
characters?
Image courtesy Krormrathog & FreeDigitalPhotos.net
2c

9/11/2014
14
Are internal auditors stronger?
You can’t bully good internal
auditors.
(If you can bully them then
they don’t last long).
2c
The internal audit hothouse
Internal audit is used as a
training ground for high
quality staff.
There is a potential
downside to staff rotation.
Where do they go next?
Image courtesy Chanpipat & FreeDigitalPhotos.net
2c

9/11/2014
15
Risk and the financial crash
Risk is a tricky
concept and auditors
didn’t handle it well.
Image courtesy cooldesign & FreeDigitalPhotos.net
3
“...the chance, high or low, of somebody
being harmed by the hazard, and how
serious the harm could be”
(UK Health & Safety Executive)
Image courtesy jscreationzs & FreeDigitalPhotos.net
What is risk anyway?
“the effect of uncertainty on objectives”
(ISO 31000)
“a set of circumstances that hinder the
achievement of objectives”
(David Griffiths)
3a

9/11/2014
16
UK HSE risk matrix
3a
Enrico Fermi – the brilliant nuclear physicist
who worked on the project to develop the
atomic bomb.
1939. The probability that nuclear fission
could be controlled for power or weapons?
10%
1945. The probability that the atomic bomb
would set the atmosphere on fire and wipe
out life on earth?
10%
1950. The probability that humans would
develop the technology to travel faster
than the speed of light by 1960? 10%
3a

9/11/2014
17
Tim O’Riordan & Patrick Cox, 2001.
Science, Risk, Uncertainty & Precaution.
University of Cambridge.
3a
Simple, understandable
and totally misleading?
Complex, accurate(?) and
totally uninformative?
Risk – the big dilemma?
or
Images courtesy Luigi Diamanti, Mr Brightman & FreeDigitalPhotos.net
3a

9/11/2014
18
Rick Buy – Chief Risk Officer.
His stated aim was to ”condense”condense”condense”condense all the risks ofall the risks ofall the risks ofall the risks of
the corporation into a single metricthe corporation into a single metricthe corporation into a single metricthe corporation into a single metric”.”.”.”.
Risk – the big dilemma?
3b
Risk – and how we lost sight of it
Image courtesy of Just2shutter / FreeDigitalPhotos.net
“With half a decade’s
hindsight, it is clear the
crisis had multiple causes.
The most obvious is the
financiers themselves –
especially the irrationally
exuberant Anglo-Saxon sort,
who claimed to have found a
way to banish risk when in
fact they had simply lost
track of it.”
The Economist
Image courtesy pakorn / FreeDigitalPhotos.net
3c

9/11/2014
19
Risk – and how we lost sight of it
Image courtesy of Just2shutter / FreeDigitalPhotos.net
“The weaknesses of
group risk in HBOS
were a matter of
design, not accident.”
Parliamentary
Commission on Banking
Standards;
“An Accident Waiting To
Happen: The Failure of
HBOS”
3c
* Fixed probability
* Time period
* Amount at risk
Eg, 95% probability that the
maximum loss in a week
will not exceed £1m.
Definitely not 5% probability
of losing just £1m in a week.
Value at Risk - or losing sight of risk
3c

9/11/2014
20
Value at Risk – ignoring Black Swans
Decision makers and
auditors lost sight of
what VaR actually means.
Above the “VaR break”
all bets are off – we’re
into Black Swan
territory.
And that’s pretty much
what happened.
3c
Big 4 audit fees for 2007
““““…fees are now coming before independence, objectivity
(and sometimes, even competence) in important parts of
the accounting profession.””””
Paul Moore (ex partner KPMG, ex Head of Group Regulatory
Risk, HBOS - 2013)
3d

9/11/2014
21
Big 6 foul ups in US
US PCAOB Audit
Failures 2012 (2011)
Grant Thornton 65% (43%)
BDO 55% (39%)
Ernst & Young 48% (36%)
PWC 39% (41%)
KPMG 34% (23%)
Deloitte 25% (42%)
(% of audits inspected deemed to
be “audit failures” by regulator)
3d
Has external audit had its day?
“External audit is now largely out-
dated. The binary nature of the
opinion renders it useless.”
Richard Anderson chairman of the
Institute of Risk Management,
2011
“With or without new rules, the
main worry for auditors may be
that people wonder whether
their reports are worth a bean.”
The Economist, April 2014
3e

9/11/2014
22
Has external audit had its day?
“The fact that the audit process
failed to highlight developing
problems in the banking sector
does cause us to question exactly
how useful audit currently is.”
House of Commons Treasury
Committee “Banking Crisis”, 2009
“The problem is that there's not
a lot of evidence that (external)
auditors are very good at
assessing risk.”
Charles Cullinan, Bryant College,
USA
3e
Is internal audit better placed?
PeoplePeoplePeoplePeople
TimeTimeTimeTime
Business knowledgeBusiness knowledgeBusiness knowledgeBusiness knowledge
IndependenceIndependenceIndependenceIndependence
Business modelBusiness modelBusiness modelBusiness model
None of these are guaranteed advantagesNone of these are guaranteed advantagesNone of these are guaranteed advantagesNone of these are guaranteed advantages
Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?
3f

9/11/2014
23
Evidence and Opinion
How do we know anything?
What matters? Who cares?
“To know that we know what
we know, and to know that
we do not know what we do
not know, that is true
knowledge.” Copernicus
4a
4a
Ontology
What is the nature of
reality?
What isisisis a windmill?
What is real?
4a

9/11/2014
24
How can Don Quijote know
anything?
Epistemology
What can he know about
windmills?
From Sancho Panza?
From his senses?
From books!
How can he know about
windmills?
Trigger’s Broom
4a

9/11/2014
25
A positivist worldview?
Have we treated testing, and
auditing, as if they are like
scientific experiments where
we know and control all the
variables?
Have we been too keen to
assume the world we are
investigating is a neater and
more ordered place than it
really is?
4a
Is an interpretivist
worldview more helpful?
A dangerous extreme for
testers?
Certainly for auditors!
There is no single, fixed
reality. Everything is a social
construct so we have to
understand what we are
looking at rather than
criticising or condemning.
4a

9/11/2014
26
A balanced approach?
(just doing the best we can)
We might not know things with
certainty, but we can make
statements based on evidence
& keep refining our opinion.
Positivists might think that
certainty is out there and we
can know it.
Interpretivists might not say
anything useful; they’re all
features, not bugs!
4a
They areThey areThey areThey are overoveroverover----simplifiers. Theysimplifiers. Theysimplifiers. Theysimplifiers. They
take a complicated issue andtake a complicated issue andtake a complicated issue andtake a complicated issue and
deliver a simplistic anddeliver a simplistic anddeliver a simplistic anddeliver a simplistic and
superficially plausible answer.superficially plausible answer.superficially plausible answer.superficially plausible answer.
They offer clear, actionable adviceThey offer clear, actionable adviceThey offer clear, actionable adviceThey offer clear, actionable advice
but…but…but…but…
Thanks toThanks toThanks toThanks to StianStianStianStian Westlake for this.Westlake for this.Westlake for this.Westlake for this.
Berks
4b
Don’t be a berk or a wanker

9/11/2014
27
A wanker (that’s me)
They want to be robustThey want to be robustThey want to be robustThey want to be robust
and comprehensive, andand comprehensive, andand comprehensive, andand comprehensive, and
forget about clarity &forget about clarity &forget about clarity &forget about clarity &
brevity.brevity.brevity.brevity.
4b
Don’t be a berk or a wanker
Rikard Edgren
“Reality isn’t binary… we“Reality isn’t binary… we“Reality isn’t binary… we“Reality isn’t binary… we
don’t know everything indon’t know everything indon’t know everything indon’t know everything in
advance.advance.advance.advance.
We should observe theWe should observe theWe should observe theWe should observe the
software without asoftware without asoftware without asoftware without a
hypothesis to nullify.”hypothesis to nullify.”hypothesis to nullify.”hypothesis to nullify.”
Image courtesy digitalart binary / FreeDigitalPhotos.net
4c
The binary trap

9/11/2014
28
The questions we can answer yes/no with most
certainty are probably those that don't matter.
The danger is that we focus on them because the
light is better there.
The binary trap
4c
It’s not meant to be easy, it’s meant to be valuable.
Test scripts are not testing.
Checklists are not auditing.
4c
The binary trap

9/11/2014
29
Relying on scripts and checklists
assumes that the information we
want is under the streetlight.
It assumes that we can know in
advance what matters, what we
need to look for.
It assumes that the important
questions can be answered with
a “yes” or “no”.
The relevance to testers
4c
If we focus only on what was
specified we will not see what
was needed but neither
specified nor built (5).
And we won’t see what was
not specified or needed, but
was built (6).
Thanks to James Lyndsay, Iain McCowatt,
James Bach & Michael Bolton.
and auditors want to know too
Either could be damaging.
The relevance to testers
4c

9/11/2014
30
Good auditors learn by listening.
Bad auditors don’t listen. Their
checklist tells them the “right
answers”.
UK & US regulators are pushing
auditors away from binary
opinions. EU???
An auditor – “one who hears, a listener
4c
Risk Based Auditing
What is it? How do they do it?
Image courtesy David Castillo Dominici/FreeDigitalPhotos.net
We don’t understand risk well.
We don’t understand auditing.
So do we really know what Risk
Based Auditing means?
5

9/11/2014
31
Risk Based Auditing – what is it?
2- RBA provides
assurance that risks
are being managed
effectively.
3- RBA focuses
effort on the areas
most likely to suffer
problems.
1- RBA identifies
risks so that
management can
eliminate them.
4- RBA focuses on
the risks that pose
the greatest threat to
company objectives.
5a
1- RBA identifies
risks so that
management can
eliminate them.
5a
3- RBA focuses
effort on the areas
most likely to suffer
problems.
2- RBA provides
assurance that risks
are being managed
effectively.
4- RBA focuses on
the risks that pose
the greatest threat to
company objectives.

9/11/2014
32
Controls based auditor; “how can I be
sure no-one will steal bricks while the
house is being built?”
Script driven tester; “what tests should I
write for using these bricks to build a nice
house?”
Risk based auditor; “could someone hit the
cashier over the head with a brick and
steal the payroll? Is that significant?”
The exploratory tester?
Image courtesy Piyachok Thawornmat/FreeDigitalPhotos.net
5a
There’s compliance,
and then there’s compliance
Big difference between the
cops and mere processes!
5a

9/11/2014
33
Reasonable assurance about risks
not absolute
5a
Appropriate…
sufficient…
reasonable…
material
Auditors are looking for
reasonable assurance,
not absolute assurance.
Risks that matter
“Audit priorities (should) align with
those of the board and executive
management. Risks that keep our
stakeholders up at night also should be
of concern to us.”
Richard Chambers, CEO & President of
Institute of Internal Auditors
“The problem is that there's not a lot of
evidence that (external) auditors are
very good at assessing risk.”
(reminder!)
Charles Cullinan, Bryant College
Image courtesy digitalart /FreeDigitalPhotos.net
5a

9/11/2014
34
Attitude of the Institute
of Internal Auditors
Compliance auditing; “tipping
out the pieces of a jigsaw puzzle
on to the Audit Committee table
rather than turning those pieces
into a picture.”
Sarah Blackburn, ex President of
IIA UK
Image courtesy Stuart Miles/FreeDigitalPhotos.net
5a
“In a risk-based approach to security,
compliance is provided by security –
security is not necessarily provided
by compliance.”
John Wheeler, Gartner Inc
Moving this way?
5a
Attitude of the Institute
of Internal Auditors
““““Many organizations look at
compliance as a set of check boxes…
but compliance is not the goal, it’s a
result.”
Mike Rothman, Security Incite

9/11/2014
35
Risk Based Auditing - doing it
There are no right answers
(probably).
The checklist is not the audit. It’s
just a tool.
Auditors who rely on checklists are
unprofessional compliance monkeys. It
demeans and deskills the profession.
5b
Risk Based Auditing - planning it
A development that is under way
RequirementsRequirementsRequirementsRequirements
Test strategyTest strategyTest strategyTest strategy Test resultsTest resultsTest resultsTest resultsDetailed testDetailed testDetailed testDetailed test
plan/scriptsplan/scriptsplan/scriptsplan/scripts
DesignDesignDesignDesign
documentsdocumentsdocumentsdocuments
DevelopmentDevelopmentDevelopmentDevelopment
standardsstandardsstandardsstandards
Problem recordsProblem recordsProblem recordsProblem records
Other?Other?Other?Other?
Handover packHandover packHandover packHandover packProject planProject planProject planProject plan
Change recordsChange recordsChange recordsChange records
A development that went live two months ago
A live system that's been running for four years
5b

9/11/2014
36
A development that is under way
5b
A development that went live two months ago
5b

9/11/2014
37
A live system that's been running for four years
5b
Conway’s Law – a personal
hobby horse.
“Organizations which
design systems ... are
constrained to produce
designs which are copies
of the communication
structures of these
organizations”
Melvin Conway
Image courtesy jscreationzs/FreeDigitalPhotos.net
5b

9/11/2014
38
The communications and
organisational structure
are a useful guide to
where the worst flaws
will be in the project and
system.
My auditor’s corollary (or
heuristic) to Conway’s
Law.
5b
Conway’s Law – a personal
hobby horse.
Image courtesy jscreationzs/FreeDigitalPhotos.net
Risk Based Auditing
IDEF0 & decomposing an application
5b

9/11/2014
39
Risk Based Auditing
Exploratory testing?
Breaking the application
5b
Don’t tell me, show me (auditor’s mantra)
“Don’t tell me the moon
is shining, show me the
glint of light on broken
glass”
Anton Chekhov
5b

9/11/2014
40
And why does it matter?
Different parts of the world
have different models –
with different outcomes.
6
What is Governance?
κυβερνάω [kubernáo] – to steer?
6a
What is Governance?

9/11/2014
41
Corporate governance is the board’s job
Should not involve day to day operational
management by full-time executives
Supervising management & reporting to
shareholders
Setting the strategic aims & values
Leadership to put them into effect
Values based on principles of transparency,
accountability, probity and long term
sustainability
Paraphrased from the UK Financial Reporting
Council’s “UK Corporate Governance Code”
6a
IT governance is the responsibility of corporate
management
Evaluates stakeholders’ needs and sets
objectives to satisfy them
Directs and sets priorities
Monitors performance
Paraphrased from ISACA’s definition
6a

9/11/2014
42
IT management
Plans
Builds
Runs
Monitors
All in alignment with the strategic
direction set by the governance body
Paraphrased from ISACA’s definition
6a
Why governance is a good thing
If we get
governance
wrong then
we suffer
Images courtesy Scott Adams & Stuart Miles/FreeDigitalPhotos.net
6b

9/11/2014
43
Governance - Risk Management
Three Lines of Defence
Functions that own and manage risks;
operational management (the front line)
Functions that oversee risks; risk
management and compliance function
Functions that provide independent assurance;
internal audit
IIA strongly recommended guidance
6c
Governance – comply or explain
“Comply or explain” is the UK approach.
Also Germany and Netherlands.
UK Corporate Governance Code, Deutscher Corporate
Governance Codex & Code Tabaksblat
US style
Comply or else!
(my experience)
Images courtesy Stuart Miles & FreeDigitalPhotos.net
6d

9/11/2014
44
Governance – different countries,
different models, different outcomes
etc
6e
ISACA
Information Systems Audit & Control
Association
ISACA and COBIT 5
Why they matter
7
7

9/11/2014
45
ISO/IEC 38500:2008 Model for
Corporate Governance of IT
7a
COBIT 5 interpretation of IT governance
Control Objectives for Information
and Related Technology
7b

9/11/2014
46
7c
A Quality Management System with quality
standards.
AP011 Manage QualityAP011 Manage QualityAP011 Manage QualityAP011 Manage Quality
“Best practices” to be used as a “reference when
improving and tailoring”.
Based on industry “goodgoodgoodgood practices”.
No mention of specific standards (or even the
need to go looking for standards to adapt).
ISACA expect
the following
7c

9/11/2014
47
“Validate all requirements through approaches
such as peer review, model validation or
operational prototyping”.
BA102 ManageBA102 ManageBA102 ManageBA102 Manage
Requirements DefinitionRequirements DefinitionRequirements DefinitionRequirements Definition
“If appropriate, implement the selected option
as a pilot to determine possible improvements”.
“Review the alternative solutions… and select the
most appropriate one based on feasibility… risk
and cost.”
ISACA expect
the following
7c
“… using agreed-on and appropriate phased or
rapid agile development techniques”.
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
ISACA expect
the following
“Proactively evaluate for design weaknesses (e.g.,
inconsistencies, lack of clarity, potential flaws)
throughout the life cycle”.
BA103.02 Design detailedBA103.02 Design detailedBA103.02 Design detailedBA103.02 Design detailed
solution componentssolution componentssolution componentssolution components
7c

9/11/2014
48
“Develop, resource and execute a QA plan
aligned with the QMS to obtain the quality
specified in the requirements definition and the
enterprise’s quality policies and procedures.”
BA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform quality
assuranceassuranceassuranceassurance
ISACA expect
the following
7c
“1. Define a QA plan & practices including, e.g.,
specification of quality criteria, validation and
verification processes, definition of how quality
will be reviewed, necessary qualifications of
quality reviewers, and roles and responsibilities
for the achievement of quality.”
ISACA expect
the following
7c

9/11/2014
49
“2. Frequently monitor the solution quality
based on project requirements, enterprise
policies, adherence to development
methodologies, quality management
procedures and acceptance criteria.”
ISACA expect
the following
7c
“3. Employ code inspection, test-driven
development practices, automated testing,
continuous integration, walk-throughs and
testing of applications as appropriate.”
ISACA expect
the following
7c

9/11/2014
50
“Establish a test plan and required environments
to test the individual and integrated solution
components, including the business processes
and supporting services, applications and
infrastructure.”
ISACA expect
the following
BAI03.07 Prepare forBAI03.07 Prepare forBAI03.07 Prepare forBAI03.07 Prepare for
solution testingsolution testingsolution testingsolution testing
7c
“Execute testing continually during development.”
ISACA expect
the following
BAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 Execute
Not “keep busy writing scripts till the testing
phase”.
7c

9/11/2014
51
“1. Undertake testing of solutions and their
components in accordance with the testing plan.
Include testers independent from the solution
team…”
ISACA expect
the following
7c
“2. Use clearly defined test instructions, as
defined in the test plan, and consider the
appropriate balance between automated scripted
tests and interactive user testing.”
ISACA expect
the following
7c

9/11/2014
52
“3. Undertake all tests in accordance with the
test plan and practices including the integration
of business processes & IT solution components
and of non-functional requirements (e.g.,
security, interoperability, usability).”
ISACA expect
the following
7c
“4. Identify, log and classify (e.g., minor,
significant and mission-critical) errors during
testing... Ensure that an audit trail of test results
is maintained.”
ISACA expect
the following
7c

9/11/2014
53
“5. Record testing outcomes and communicate
results of testing to stakeholders in accordance
with the test plan.”
ISACA expect
the following
7c
“2. Ensure that the test plan reflects an
assessment of risk from the project.”
BA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptance
teststeststeststests
BA107 Manage ChangeBA107 Manage ChangeBA107 Manage ChangeBA107 Manage Change
Acceptance and TransitioningAcceptance and TransitioningAcceptance and TransitioningAcceptance and Transitioning
Not in BA103 surprisingly.
ISACA expect
the following
7c

9/11/2014
54
“3. Ensure that the test plan addresses the
potential need for internal or external
accreditation of outcomes of the test process
(e.g., financial regulatory requirements).”
ISACA expect
the following
7c
“5. Ensure that the test plan identifies testing
phases appropriate to the operational
requirements and environment.”
ISACA expect
the following
7c

9/11/2014
55
“6. Confirm that the test plan considers test
preparation … training requirements, … test
environment, planning/performing/documenting/
retaining test cases, error and problem handling,
correction and escalation, and formal approval.”
ISACA expect
the following
7c
“6. Consider using clearly defined test
instructions (scripts) to implement the tests.”
BA107.05 PerformBA107.05 PerformBA107.05 PerformBA107.05 Perform
acceptance testsacceptance testsacceptance testsacceptance tests
That’s the end of testing in COBIT 5
ISACA expect
the following
7c

9/11/2014
56
COBIT 5 – big lessons for testers
No insistence on “best practice”
Countless references to ISO standards for;
- Risk management
- Security
- Release management
- Configuration management
- Service level management
- Incident management
- Problem management
- Business continuity
- etc No mention of testing standards
No insistence on detailed scripts or test cases.
None at all!
7d
IIA standards - good news
(seriously!)
8

9/11/2014
57
The Snowflake Theory of IT Audit
“Every IT environment is unique
and represents a unique set of
risks. The differences make it
increasingly difficult to take a
generic or checklist approach
to auditing.”
Global Technology Audit Guide,
Management of IT Audit, 1st edition,
2006
8a
IIA IT Audit Management Standard
“Frameworks and StandardsFrameworks and StandardsFrameworks and StandardsFrameworks and Standards
One challenge auditors face when executing
IT audit work is knowing what to audit
against. Most organizations have not fully
developed IT control baselines for all
applications and technologies. The rapid
evolution of technology could likely render
any baselines useless after a short period of
time.”
Management of IT Audit, 2nd edition,
2013
Image courtesy Salvatore Vuono & FreeDigitalPhotos.net
8b

9/11/2014
58
ISO standards are not mentioned except in an
appendix “… for consideration”.
COBIT 5 is a recommended source of “control
objectives” against which auditors can work. It
offers “robust and generally accepted IT-
specific control objectives… that helps
management to conceptualize an approach for
measuring and managing IT risk”.
Management of IT Audit, 2nd edition, 2013
IIA IT Audit Management Standard
8b
IIA Auditing IT Projects Standard
A basic primer in software development (not a
criticism – humility is not a bad thing).
Every organisation uses a different mix of
methods, standards & tools. Auditors must
understand these. They’re the ones that matter.
Auditing IT Projects, 2009
8c

9/11/2014
59
Mentions ISO project management standards,
but not testing standards.
Favourably disposed towards Agile (one of the
top ten factors for project success).
Importance of COBIT 5 is stressed – though the
IIA does think it’s mainly about project
management.
8c
“Internal auditors should not expect
organizations to fully implement PMBOK,
PRINCE2, COBIT, or any other large set of best
practices. Rather, they should expect to see
that these practices have been customized and
integrated into the organization’s project
management methodology.”
8c

9/11/2014
60
Sarbanes Oxley
Does Sarbox deserve its
scary reputation?
Yes, but…
No, but…
9
Is Sarbanes Oxley scary?
Yes, especially section 404.
That’s the requirement that
management and the external
auditors must report on internal
control over financial reporting.
It’s a lot of work and it scares
people who can make life difficult.
Image courtesy Simon Howden & FreeDigitalPhotos.net
But, it’s only for US
companies, but… but…
9a

9/11/2014
61
No, so long as you don’t
have Wally in charge of
compliance.
Comply with COBIT 5
and Sarbox need not
be a problem for
testers.
That’s one of the
reasons COBIT 5 is so
important.
Cartoon courtesy Scott Adams
9b
“Documentation is never required
‘for the auditors’.
If it is required it is because
it is needed to manage the project,
or it is a requirement of the project
that has to be justified like any
other requirement.”
James Christie
“Do standards keep testers in
the kindergarten?”
Testing Experience, Dec 2009
http://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htm
Image courtesy Simon Howden & FreeDigitalPhotos.net
9b

9/11/2014
62
US Food & Drugs Administration
What does the FDA expect?
10
Strong Credible
Images courtesy Simon Howden, Stuart Miles & FreeDigitalPhotos.net
10a

9/11/2014
63
“Test procedures, test
data, and test results
should be documented in a
manner permitting
objective pass/fail
decisions to be reached.”
General Principles of
Software Validation,
FDA 2002
General Principles of
Software Validation,
FDA 2002
10b
10c
“The FDA is open to agile
processes and realizes that
the current approach to
software validation is not
working”
Griffin Jones
CAST 2011

9/11/2014
64
AAMI TIR45:2012
“Guidance on the use
of AGILE practices in
the development of
medical device
software”
10c
“Agile can be adapted to the
unique needs of medical
device software… … and (can
satisfy) regulatory requirements.”
AAMI TIR45:2012
Shows how Agile maps
onto IEC 62304 (the
standard specifying
lifecycle requirements
for developing medical
software).
10c

9/11/2014
65
“The exploratory stage of
clinical device development is intended to
allow for any iterative improvement of
the design of the device, advance the
understanding of how the device works
and its safety, and to set the stage for the
pivotal study.”
Image courtesy digitalart & FreeDigitalPhotos.net
FDA draft guidance
2011
http://www.fda.gov/MedicalDevices/
DeviceRegulationandGuidance/Guida
nceDocuments/ucm265553.htm
10d
Image courtesy Master & FreeDigitalPhotos.net
10e

9/11/2014
66
Clear
Objective (not requiring
interpretation)
Authentic
Demonstrable integrity
Readable & available
Evidence that will stand up in court
10e
Check out Griffin Jones’ work.
See his talk on YouTube.
http://www.youtube.com/watch?v=i8he7Rejn5s
Attributable and not repudiable
Full record & audit trail for changes
Contemporary
10e
Seriously consider filming testing.

9/11/2014
67
10e
The evidence has to be sufficient
(quality and quantity) so that 3rd
parties will have to come to the
same conclusion if they review it,
without interpretation by the
testers.
10f
“the more energy put in
to preparation, the less
likely direct observations
are captured”
Griffin Jones
on Twitter

9/11/2014
68
10f
Evidence of planning is
emphatically not evidence of
what was done.
Detailed test script
documentation is not
evidence of test execution.
Is a beautifully constructed
project plan evidence that the
project finished on time?
10f
Image courtesy digitalart & FreeDigitalPhotos.net
Get help

9/11/2014
69
Test Strategy & Planning
What does a good auditor expect?
Images courtesy Stuart Miles & Master/FreeDigitalPhotos.net
StrategyStrategyStrategyStrategy not form filling
Relevance, not boiler-plate
11a
Images courtesy Stuart Miles & Master/FreeDigitalPhotos.net
11a
Thoughtfulness, not massive documentation
Honesty, not spurious confidence
What does a good auditor expect?

9/11/2014
70
The strategy is not
the process.
The strategy isn’t part
of the plan, it shapes
the plan.
We’re hopeless at strategy
Cartoon courtesy Scott Adams
Test Strategy & Planning 11a
My experience - we randomly mix up
processes, strategy & planning.
James Bach talking like an auditor sensation!

9/11/2014
71
Brainless optimism.
Problems are not removed
with a stroke of the pen.
Problems do not disappear
if they are ignored.
Budding auditor
or tester?

9/11/2014
72
Kopimism - “the act of
copying is sacred”.
Copy/pasting is not
cool. It’s evidence of a
lack of thought.
Writing a strategy is not
a matter of fleshing out
a template, or recycling
an old strategy.
The Kopimism Heresy
“Strategies” running to 50+ pages.
“Assumptions” & “risks” that are
just wishes that bad things won’t
happen (ifififif they’re even stated).
Failure to learn from experience.
Go live dates announced before
work is sized or staff secured.
Successive draft versions of project
plans that get more optimistic
without obvious plausible reasons.
Images courtesy digitalart/FreeDigitalPhotos.net
11b
More warning signs (a personal list)

9/11/2014
73
Requirements can’t be traced
through to testing.
Images courtesy digitalart/FreeDigitalPhotos.net
11b
“Testing must be traceable to
requirements”.
Vague defect management process.
Environments?
Conflicting demands on resources.
Conway’s Law.
More warning signs (a personal list)
A better
way?
RST Heuristic Test Strategy Model
11c

9/11/2014
74
Really good, but…
it’s not a template,
it won’t think for you,
it won’t stop you
making blunders I’ve
seen with traditional
approaches,
and you have to follow
the spirit, not the letter,
and THINK.
RST Heuristic Test Strategy (Plan? Model
11c
The strategy has to show
how you’ve thought your way
through from the problem to
a plausible answer.
Image courtesy David Castillo Dominici & FreeDigitalPhotos.net
The plan should show
how you’ll implement the
strategy.
“Plan =strategy + logistics”
11d

9/11/2014
75
Images courtesy Simon Howden, Stuart Miles, Artur84 & stockimages/FreeDigitalPhotos.net
What does a
good auditor
expect?
12
Test Execution
Test Execution
COBIT 5
Remember COBIT 5. That says it all.
Record & communicate everything
you said you’d do.
Exploratory testing?
Rapid Software
Testing?
What does a
good auditor
expect?
Images courtesy Simon Howden, Stuart Miles, Artur84 & stockimages/FreeDigitalPhotos.net
12a

9/11/2014
76
Test execution deviating from the
plan. Hmm!
Changes to defect management &
reporting and test priorities
during the test execution.
Lack of an audit trail for defects/fixes
& a lack of reliable, contemporary
evidence.
Image courtesy digitalart/FreeDigitalPhotos.net
Test Execution
Warning signs (an official list from COBIT 5)
12b
Test Execution
Warning signs (an official list from COBIT 5)
12b
In summary, auditors expect the
plan to be relevant.
There are good reasons to change
plans and schedules during
testing, but auditors will be very
suspicious of anything that looks
like winging the testing because
the plan was rubbish, or rigging
the testing schedule to hit the
implementation date.

9/11/2014
77
Test Execution
Warning signs (a personal list)
12c
Reporting that implies a link between
test case passes & progress.
Confusion between defect fix
priority & defect severity.
Massaging defect severity down
and up.
Treating usability issues as cosmetic.
Test Execution
Warning signs (a personal list)
12c
Large numbers of defects being
rejected.
Defects rejected because there’s no
matching test case or requirement.
Defects rejected because the
requirements are assumed to be
correct.
Failure to write reusable automated
tests.

9/11/2014
78
Test Reports
Images courtesy Stuart Miles & David Castillo Dominici/FreeDigitalPhotos.net
13a
What does a good
auditor expect?
The same as a good
test manager.
What does a good
auditor expect?
13b
Test Reports
(with thanks to Rapid Software Testing)
Learning about the
product
Learning about
how the product
was tested.
Learning about
how good the
testing was.

9/11/2014
79
Images courtesy Stuart MilesFreeDigitalPhotos.net
Putting the jigsaw together.
Don’t empty the box
onto the table.
Put the pieces together to
assemble a clear picture, to
tell a compelling story.
Test Reports
13c
Auditors live and die by evidence.
Opinions are not casual observations.
They must be backed by evidence.
Test Reports
13d

9/11/2014
80
Finally, say what you mean and
mean what you say.
Auditors will take your statements at
face value.
Test Reports
13d
Testing Standards
14

9/11/2014
81
Wrap Up
Never follow the letter of the
law and ignore the spirit.
Never do something just
because “that’s what the
auditors will expect”.
Do the right thing and be ready
to justify it.
Go and speak to the auditors.
Say what you mean and mean
what you say. And never lie!
15a
Email: james@clarotesting.com
Twitter: @james_christie
www.clarotesting.wordpress.com
www.clarotesting.com
15b
Wrap Up

Satisfying Auditors: Plans and Evidence in a Regulated Environment

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (8)

Similar to Satisfying Auditors: Plans and Evidence in a Regulated Environment

Similar to Satisfying Auditors: Plans and Evidence in a Regulated Environment (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Satisfying Auditors: Plans and Evidence in a Regulated Environment