Exploratory Testing Audit Survival

The audit survival heuristics of
an FDA regulated exploratory testing team
CAST August 8th, 2011 1Griffin Jones – Congruent Compliance LLC © 2011

Preliminaries
 Who is in the room?
 My goal:
 Stimulate your interest to study the subject more
 Leave with a heuristic to help you organize and present
with confidence your ET results to regulatory auditors
 Have a conversation and try to meet your needs
 Quick Preview
 The context
 The heuristic and how to apply it
 Some of the traps about ET in a regulated industry

Assumptions and Terms
 This is a living presentation
 Based on my experiences of auditing and being audited
 More reference information here than I will present
 Follow the for the key points
 Much of this can be adapted to other contexts
 i.e., not “FDA regulated, Exploratory Testing”
 “Schools of Testing” by Bret Pettichord
 Analytic , Standard, Quality, Context-Driven, Agile
 Exploratory Testing:
 Simultaneous learning, test design and test execution

Terms
 Congruence
 Being balanced between inner feelings & outer actions
 Smells
 Symptom that possibly indicates a deeper problem
 5 Whys
 Questions-asking method to investigate root causes
 “Mary had a little lamb” heuristic
 Emphasize each of the individual words in a statement
 Checking: confirming existing beliefs; versus:
 Testing - finding new information (Michael Bolton)

The Problem
 Let’s assume that you are FDA regulated and trying to
do compliant context driven, Exploratory Testing
 You likely have these concerns about passing an audit:
 Evidence is not sufficient
 Documentation is not sufficient
 Process control is not sufficient
 Can’t clearly explain what you do and why
 Auditors value different things than you, and speak a
different language

Fast Takeaway
 The regulator is not your business partner
 The regulator has police powers
 “Let the Wookie win”
 Auditors are likely of the “Quality” (gatekeepers) or
“Routine” (traceability matrix) testing school model
 You are Context Driven testing school. Deal with it.
 Auditors think “testing” is “demonstration and
checking”
 Don’t try and convert them. Deal with it.

Spoiler
 The regulations are not the problem
 How you are coping with the regulations is the problem
 Give the Auditors what they want:
 Clear traceable requirements and description of risks
 Description and demonstration of control
 Clear objective evidence
 The ability to understand their concerns, speak their
language, and explain how you are compliant
 Abundant, quality evidence mitigates your other
problems

Not going to talk about…
 The Fear, Uncertainly, and Doubt swirling in the field
 Vendor/Experts: “You should be scared, but I have…”
 Silver Bullets and Big Magic
 “… so trust me and just buy my wares. By the way, ..”
 Persistent Myths
 “… IMO the regulators “frown on” ET (… I don’t sell it).”
 The “Typical” Regulatory Affairs Presentation

Regulatory Overview
 Regulations
 For the public good - because people died
 Regulators
 FDA regulates >25% of the Gross Domestic Product
 Regulatory Auditors
 Police Powers
 Industry Auditors
 Assessors and valued advisors to management
 Audits
CAST August 8th, 2011 9
Details
Griffin Jones – Congruent Compliance LLC © 2011

Audit Survival Heuristics
 CHCMWCE
“Chocolate Mousse”
 Congruent
 Honest
 Competent
 Model (Appropriate)
 Willing
 Control
 Evidence
Model
Competent
Honest
Evidence
Control
Willing
Congruent

Let’s take a journey …
 Practice
 Congruent
 Theory
 Less Stressful
Audits

The Congruence Triad
 Congruence is when you are balanced between inner
feelings and outer actions
 The Congruence Triad
 Self, Other, Context
 Being congruent is a process
 A way of communicating with yourself and others
 Incongruence is when part of the triad is missing
 Placating, Blaming, Super-rational, or Irrelevant?
 What is missing and fill it in:
 Self, Others, Context
Other
Context
Self
Details

The Theory Mountains …
 Dishonest
 Incompetent
 Inadequate
 Honest
 Competent
 Appropriate Model
 Self-Incriminating
 Experts and Heroes
 Over-Constrained

Honest
 Integrity, Truthful, Trust, Sincerity in:
 You and your organization
 Words, actions, and documents
 Smells
 Dishonest
 Self-incrimination
 Don’t create even the appearance of a problem
 Tests
 How do you and the organization react to criticism?
 Are you a learning organization? (5 Why)

Competent
 Are you and your organization:
 Capable, credible, understands context, speaks the
language; trained in the industry, technology, and
regulatory obligations
 Smells
 Incompetent
 Experts and heroes
 Tests
 Do you believe you are capable of doing good work?
(5 Why)

Appropriate Model
 Is the process model:
 Complete, reasonable, practical, logical, explainable
 Smells
 Inadequate model
 Over-constrained model
 Test:
 What problem is this model solving? How will it Fail?
 What is required in this model? Missing?
 Do you believe this model is sufficient? (5 Why)

The Practice Mountains …
 Unwilling
 Out-of-Control
 No Evidence
 Excessive or Wasteful
 Micro-Management
 Obsessive-Compulsive
 Willing
 Under Control
 Evidence

Willing
 Motivated, focused, prioritized, committed, resourced,
staffed, supported, given attention, nurtured
 Smells
 Unwilling
 Excessive or Wasteful
 Test
 Do people care? (5 Why)
 Is there sufficient resources for the work and
expectations? (5 Why)

Under Control
 Explain what you are doing and why. Are you living it?
 Coherently explain your:
 configuration control and authorization
 traceability and accountable
 organization, preparation, planning, independent review,
prevention, correction, checking and testing
 Smells
 Out-of control
 Micro-managed
 Tests
 Is the type and level of controls appropriate? (5 Why)

Evidence
 Auditable evidence:
 Clear, objective, retrieval, human readable, attributable,
contemporary evidence that a third party can review or
reconstruct (with minimal outside help); and quickly
reach the same results and conclusions.
 Smells
 No-evidence
 Obsessive-compulsive evidence
 Tests
 Explain why the specific evidence meets the criteria.
(5 Why)

How do you apply this?
 Application is as simple as:
Remembering
to ask the questions.
Follow the energy
of the answers.
Fix the base, first.

During an Audit
 Choosing a regulatory posture
 Manageable issues (within reason)
 Evidence
 Controls
 Willingness (resources and priority)
 Unmanageable issues
 Broken process model
 Lack of competence
 Broken trust
 Incongruence

More Fast Takeaways
 The FDA is open to agile processes and realizes that
the current approach to software validation is not
working
 At the same time, companies are more concerned
about:
 the business risk that the FDA would not accept the
agile process,
 than the product or project risk that is associated with
waterfall type development
 Find the middle option for your context

Natural Evidence
 Periodically , take the observer point-of-view and ask:
 Is what I see and hear, about the theory and practice of
what we do:
 acceptable from both a product qualification and
regulatory compliance point of view?
 If yes, what is the most natural, efficient, and strongest
evidence we could collect?
 Why not a video/audio recordings w/ paper summary?
 Is it being collected? If no, why not? (5 Why)
 organizational problem?

Organizational
Smells
Going Tilt
Traps

Smells that lead to …
 Paint the Village
 Visitors are coming. How shall we work today?
 The “Best Practice” Cargo Cult
 We don’t really understand the details of what we do,
why we do it, or how what we do works. But have faith.
 Testing Death Spiral
 Regulator does not care about testing and management
might only care about regulatory compliance. Spiral.
 The Titanic
 The gigantic engineered process is perfect – people are
the source of problems, not solutions

Organizational Disasters
 Pathetic Compliance
 Following a regulatory compliant procedure in a way
that does not solve the testing problem for which it was
designed.
 Utopian Shelf-ware Procedures
 No one reads them. They are not reality.
 Close Enough
 I don’t have to do it exactly. I know better. No one will
notice or care.
 Read My Mind
 Because that is the only place where the evidence is.

Is the Auditor on Tilt?
 Maybe it is something we said or did, or are doing?
 History
 That you are unaware of, and it might be complicated
 Notches on the gun
 May be making a name for themselves
 Making an example of you
 May be constructing an example to deter others

Classic ET Traps
 Implementation details identified as requirements
 Tighten and simplify your requirements
 Documentation lacks detail to support traceability
 Require less mind reading.
 Control is vague or assumed
 Summarize and document what control is for you

The BIG Trap
 Weak Evidence
 “Clear, objective, retrieval, human readable,
attributable, contemporary evidence that a third party
can review or reconstruct (with minimal outside help);
and quickly reach the same results and conclusions.”
 Check it via “Mary had a little lamb”
 Collect it naturally
 Weak evidence is likely a symptom of other deeper
issues
 Abundant, quality evidence mitigates your other
problems

Audits can be Useful
 Candor can result in free consulting and insight
 Should you take the risk?
 Provides motivation – management cares
 Provides actionable data
 The jiggle that is needed by the organization
 A counter-measure to low expectations & poor practices
If you can’t be a good example,
you are going to be a stern warning.

Recap of the Spoiler
 The regulations are not the problem.
 How you are coping with the regulations is the problem.
 Give the Auditors what they want:
 Clear traceable requirements and description of risks
 Description and demonstration of control
 Clear objective evidence
 The ability to understand their concerns, speak their
language, and explain how you are compliant
 Abundant and quality evidence mitigates your other
problems.

The Big Take Away
 Understand your regulatory context
 Work on your congruence
 Work each level of the model, ask the questions
 Document how you are under control
 Improve your evidence, collect it naturally
 Avoid the smells, disasters, and traps
 Summarize your regulatory story, practice explaining it
 Apply what you learn during the audit
1
2
3

Questions?
Model
Competent
Honest
Evidence
Control
Willing
Congruent

Further Study - A
 FDA presentations and resources:
 Webinar with FDA's John Murray on Software Validation
in the Field of Medical Devices
 Presentation: Preparing for an FDA Medical Device
Sponsor Inspection
 Quality System Inspection Technique – Inspection
Guide
 General Principles of Software Validation; Final
Guidance for Industry and FDA Staff

Further Study - B
 Regulatory Compliance
 “The Art of Compliance: Turning Compliance into
Sustainable Business Advantage” by Robert Rhoades of
Quintiles
 FDA inspections:
 “How to Host an FDA Inspection” by SGS – Life Science
Services
 “Preparation for FDA Inspection” by
NEMA/ADVAMED/PHILIPS
 “FDA Sponsor Inspections: How to Prepare and Survive”
by Medtronic, Inc

Further Study - C
 Audits
 “The ASQ Auditing Handbook” by J. P. Russell
 Congruence
 “Beyond Blaming” by Jean McLendon and Gerald M.
Weinberg
 “The Satir Model: Family Therapy and Beyond” by
Virginia M. Satir
 “More Secrets of Consulting: The Consultant's Tool Kit”
by Gerald M. Weinberg

Further Study - D
 Agile and the FDA
 Business Risk (from the FDA) versus Product Risk
 http://blogs.construx.com/forums/t/432.aspx
 “What is Exploratory Testing? And How it Differs from
Scripted Testing” by James Bach
 “Coping With Complexity: Lessons From a Medical
Device Project” by Yaron Kottler
 Testers and Auditors
 “Testers are like auditors” by James Christie
 Evidence
 “21 CFR Part 11 Electronic Records …” by the FDA

Griffin Jones
Congruent Compliance
Griffin.Jones@CongruentCompliance.com
Thank You!

Exploratory Testing Audit Survival

Recommended

Recommended

More Related Content

Similar to Exploratory Testing Audit Survival

Similar to Exploratory Testing Audit Survival (20)

More from Griffin Jones

More from Griffin Jones (6)

Recently uploaded

Recently uploaded (20)

Exploratory Testing Audit Survival