Surviving an FDA Audit: Heuristics for Exploratory Testing - from CAST, STP, STAR, and KWSQA in 2011 and 2012

Griffin Jones – Congruent Compliance LLC 1March 2012
Test Strategy and Design #602
Surviving an FDA Audit:
The Heuristics for Exploratory Testing
Griffin Jones, Consultant, Congruent Compliance

The Heuristics for Exploratory Testing
2Griffin Jones – Congruent Compliance LLCMarch 2012

Preliminaries
 Who is in the room?
 My goal:
 Stimulate your interest to study the subject more
 Leave with a heuristic to help you organize and present
with confidence your ET results to regulatory auditors
 Have a conversation and try to meet your needs
 Quick Preview
 The context
 The heuristic and how to apply it
 Some of the traps about ET in a regulated industry
Griffin Jones – Congruent Compliance LLC 3

Assumptions and Terms
 More reference information here than I will present
 Follow the for the key points
 Much of this can be adapted to other contexts
 i.e., not “FDA regulated, Exploratory Testing”
 “Schools of Testing” by Bret Pettichord
 Analytic , Standard, Quality, Context-Driven, Agile
 Exploratory Testing
 Simultaneous learning, test design and test execution
 Agile Testing
 Story completion, test automation: Test Driven Dev., etc.
4Griffin Jones – Congruent Compliance LLC

Terms
 Congruence
 Being balanced between inner feelings & outer actions
 Smells
 Symptom that possibly indicates a deeper problem
 5 Whys
 Questions-asking method to investigate root causes
 “Mary had a little lamb” heuristic
 Emphasize each of the individual words in a statement
 Checking: confirming existing beliefs; versus:
 Testing - finding new information (Michael Bolton)

The Problem
 Let’s assume that you are FDA regulated and trying to
do compliant context-driven or Agile, Exploratory
Testing
 You likely have these concerns about passing an audit:
 Evidence is not sufficient
 Documentation is not sufficient
 Process control is not sufficient
 Can’t clearly explain what you do and why
 Auditors value different things than you, and speak a
different language

Fast Takeaway
 The regulator is not your business partner
 The regulator has police powers
 Pick your battles – Sometimes, “Let the Wookie win”
 “Render unto Caesar, that which is Caesar’s …”
 Auditors are likely of the “Quality” (gatekeepers) or
“Routine” (traceability matrix) testing school model
 You are a different testing school. Deal with it.
 Auditors think “testing” is “demonstration and
checking”
 Don’t try and convert them. Deal with it.

Spoiler
 The regulations are not the problem
 How you are coping with the regulations is the problem
 Give the Auditors what they want:
 Clear traceable requirements and description of risks
 Description and demonstration of control
 Clear objective evidence
 The ability to understand their concerns, speak their
language, and explain how you are compliant
 Abundant, quality evidence mitigates your other
problems

Not going to talk about…
 The Fear, Uncertainly, and Doubt swirling in the field
 Vendor/Experts: “You should be scared, but I have…”
 Silver Bullets and Big Magic
 “… so trust me and just buy my wares. By the way, ..”
 Persistent Myths
 “… IMO the regulators “frown on” ET (… I don’t sell it).”
 The “Typical” Regulatory Affairs Presentation

Regulatory Overview
 Regulations
 For the public good - because people died
 Regulators
 FDA regulates >25% of the Gross Domestic Product
 Regulatory Auditors
 Police Powers
 Industry Auditors
 Assessors and valued advisors to management
 Audits
10
Details
Griffin Jones – Congruent Compliance LLC

Audit Survival Heuristics
 CHCMWCE
“Chocolate Mousse”
 Congruent
 Honest
 Competent
 Model (Appropriate)
 Willing
 Control
 Evidence
16
Model
Competent
Honest
Evidence
Control
Willing
Congruent

Let’s take a journey …
17
 Practice
 Congruent
 Theory
 Less Stressful
Audits

The Congruence Triad
 Congruence is when you are balanced between inner
feelings and outer actions
 The Congruence Triad
 Self, Other, Context
 Being congruent is a process
 A way of communicating with yourself and others
 Incongruence is when part of the triad is missing
 Placating, Blaming, Super-rational, or Irrelevant?
 What is missing and fill it in:
 Self, Others, Context
18
Other
Context
Self
Details

Congruence is like a Sailboat
 Because:
 It is a vessel or container, like a basket
 It requires preparation and maintenance
 You don’t “drive” it, and requires skills of crew members
 Subject to weather
 Is vulnerable to sinking
Tools

The Theory Mountains …
 Dishonest
 Incompetent
 Inadequate
22
 Honest
 Competent
 Appropriate Model
 Self-Incriminating
 Experts and Heroes
 Over-Constrained

Honest
 Integrity, Truthful, Trust, Sincerity in:
 You and your organization
 Words, actions, and documents
 Smells
 Dishonest
 Self-incrimination
 Don’t create even the appearance of a problem
 Tests
 How do you and the organization react to criticism?
 Are you a learning organization? (5 Why)

Competent
 Are you and your organization:
 Capable, credible, understands context, speaks the
language; trained in the industry, technology, and
regulatory obligations
 Smells
 Incompetent
 Experts and heroes
 Tests
 Do you believe you are capable of doing good work?
(5 Why)

Appropriate Model
 Is the process model:
 Complete, reasonable, practical, logical, explainable
 Smells
 Inadequate model
 Over-constrained model
 Test:
 What problem is this model solving? How will it Fail?
 What is required in this model? Missing?
 Do you believe this model is sufficient? (5 Why)

The Practice Mountains …
 Unwilling
 Out-of-Control
 No Evidence
26
 Excessive or Wasteful
 Micro-Management
 Obsessive-Compulsive
 Willing
 Under Control
 Evidence

Willing
 Motivated, focused, prioritized, committed, resourced,
staffed, supported, given attention, nurtured
 Smells
 Unwilling
 Excessive or Wasteful
 Test
 Do people care? (5 Why)
 Is there sufficient resources for the work and
expectations? (5 Why)

Under Control
 Explain what you are doing and why. Are you living it?
 Coherently explain your:
 configuration control and authorization
 traceability and accountable
 organization, preparation, planning, independent review,
prevention, correction, checking and testing
 Smells
 Out-of control
 Micro-managed
 Tests
 Is the type and level of controls appropriate? (5 Why)

Evidence
 Auditable evidence:
 Clear, objective, retrieval, human readable, attributable,
contemporary evidence that a third party can review or
reconstruct (with minimal outside help); and quickly
reach the same results and conclusions.
 Smells
 No-evidence
 Obsessive-compulsive evidence
 Tests
 Explain why the specific evidence meets the criteria.
(5 Why)

How do you apply this?
 Application is as simple as:
30
Remembering
to ask the questions.
Follow the energy
of the answers.
Fix the base, first.

During an Audit
 Choosing a regulatory posture
 Manageable issues (within reason)
 Evidence
 Controls
 Willingness (resources and priority)
 Unmanageable issues
 Broken process model
 Lack of competence
 Broken trust
 Incongruence

More Fast Takeaways
 The FDA is open to agile processes and realizes that
the current approach to software validation is not
working
 At the same time, companies are more concerned
about:
 the business risk that the FDA would not accept the
agile process,
 than the product or project risk that is associated with
waterfall type development
 Find the middle option for your context

Natural Evidence
 Periodically , take the observer point-of-view and ask:
 Is what I see and hear, about the theory and practice of
what we do:
 acceptable from both a product qualification and
regulatory compliance point of view?
 If yes, what is the most natural, efficient, and strongest
evidence we could collect?
 Why not a video/audio recordings w/ paper summary?
 Is it being collected? If no, why not? (5 Why)
 organizational problem?

Organizational
Smells
Going Tilt
Traps

Smells that lead to …
 Stop Shaking the Snow Globe
 Hyper-change alongside brittle/heavy formal processes
 The “Best Practice” Cargo Cult
 We don’t really understand the details of what we do,
why we do it, or how what we do works. But have faith.
 Testing Death Spiral
 Regulator does not care about testing and management
might only care about regulatory compliance. Spiral.
 The Titanic
 The gigantic engineered process is perfect – people are
the source of problems, not solutions

Organizational Disasters
 Pathetic Compliance
 Following a regulatory compliant procedure in a way
that does not solve the testing problem for which it was
designed.
 Utopian Shelf-ware Procedures
 No one reads them. They are not reality.
 Close Enough
 I don’t have to do it exactly. I know better. No one will
notice or care.
 Read My Mind
 Because that is the only place where the evidence is.

Is the Auditor on Tilt?
37
 Maybe it is something we said or did, or are doing?
 History
 That you are unaware of, and it might be complicated
 Notches on the gun
 May be making a name for themselves
 Making an example of you
 May be constructing an example to deter others

Classic Agile Traps
 Mixing informal and formal processes
 Start informal - clearly switch to formal when ready
 Emphasizing change; light documents = poke the bear
 Stokes anxiety: control, process model, and competence
 Mistaking team conversation and understanding
 For objective documented evidence
 Speaking “Crazy Agile Moon Language”
 Give the auditor what they want, in their language
 Shows empathy and industry competence
Pass
Fail

Classic ET Traps
 Implementation details identified as requirements
 Tighten and simplify your requirements
 Documentation lacks detail to support traceability
 Require less mind reading
 Control is vague or assumed
 Summarize and document what control is for you

The BIG Trap
 Weak Evidence
 “Clear, objective, retrieval, human readable,
attributable, contemporary evidence that a third party
can review or reconstruct (with minimal outside help);
and quickly reach the same results and conclusions.”
 Check it via “Mary had a little lamb”
 Collect it naturally
 Weak evidence is likely a symptom of other deeper
issues
 Abundant, quality evidence mitigates your other
problems

Audits can be Useful
 Candor can result in free consulting and insight
 Should you take the risk?
 Provides motivation – management cares
 Provides actionable data
 The jiggle that is needed by the organization
 A counter-measure to low expectations & poor practices
41
If you can’t be a good example,
you are going to be a stern warning.

Recap of the Spoiler
 The regulations are not the problem.
 How you are coping with the regulations is the problem.
 Give the Auditors what they want:
 Clear traceable requirements and description of risks
 Description and demonstration of control
 Clear objective evidence
 The ability to understand their concerns, speak their
language, and explain how you are compliant
 Abundant and quality evidence mitigates your other
problems.

The Big Take Away
 Understand your regulatory context
 Work on your congruence
 Work each level of the model, ask the questions
 Document how you are under control
 Improve your evidence, collect it naturally
 Avoid the smells, disasters, and traps
 Summarize your regulatory story, practice explaining it
 Apply what you learn during the audit
43
1
2
3

Questions?
44
Model
Competent
Honest
Evidence
Control
Willing
Congruent

Further Study - A
 FDA presentations and resources:
 Webinar with FDA's John Murray on Software Validation
in the Field of Medical Devices
 Presentation: Preparing for an FDA Medical Device
Sponsor Inspection
 Quality System Inspection Technique – Inspection
Guide
 General Principles of Software Validation; Final
Guidance for Industry and FDA Staff

Further Study - B
 Regulatory Compliance
 “The Art of Compliance: Turning Compliance into
Sustainable Business Advantage” by Robert Rhoades of
Quintiles
 FDA inspections:
 “How to Host an FDA Inspection” by SGS – Life Science
Services
 “Preparation for FDA Inspection” by
NEMA/ADVAMED/PHILIPS
 “FDA Sponsor Inspections: How to Prepare and Survive”
by Medtronic, Inc

Further Study - C
 Audits
 “The ASQ Auditing Handbook” by J. P. Russell
 Congruence
 “Beyond Blaming” by Jean McLendon and Gerald M.
Weinberg
 “The Satir Model: Family Therapy and Beyond” by Virginia M.
Satir
 “More Secrets of Consulting: The Consultant's Tool Kit” by
Gerald M. Weinberg
 Testers and Auditors
 “Testers are like auditors” by James Christie
 Evidence
 “21 CFR Part 11 Electronic Records …” by the FDA

Further Study - D
 Agile and the FDA
 Business Risk (from the FDA) versus Product Risk
 http://blogs.construx.com/forums/t/432.aspx
 “What is Exploratory Testing? And How it Differs from
Scripted Testing” by James Bach
 “Coping With Complexity: Lessons From a Medical Device
Project” by Yaron Kottler
 “Introduction into IEC 62304 Software life cycle for medical
devices” by Christoph Gerber
 http://www.spiq.com/abs/JF200809IEC62304%20SPIQ%20
Rev004.pdf
 “Who says ET is good for Medical Devices? The FDA!” by
James Bach
 http://www.satisfice.com/blog/archives/602

Further Study - E
 Agile and the FDA
 http://rdn-consulting.com/blog/2007/07/25/update-
agile-development-in-a-fda-regulated-setting/
 http://www.agilejournal.com/articles/columns/column-
articles/3463-four-reasons-medical-device-companies-
need-agile-development
 http://rdn-consulting.com/blog/wp-
content/uploads/2007/07/060703ResMed.pdf
 http://scalingsoftwareagility.wordpress.com/2010/11/23/
an-iterative-and-incremental-process-model-for-agile-
development-in-regulated-environments/
 http://scalingsoftwareagility.wordpress.com/category/hi
gh-assurance-and-regulated-environments/

Griffin Jones
Congruent Compliance
Griffin.Jones@CongruentCompliance.com
Thank you for attending this session.
Please fill out the evaluation form.

Surviving an FDA Audit: Heuristics for Exploratory Testing - from CAST, STP, STAR, and KWSQA in 2011 and 2012

More Related Content

Similar to Surviving an FDA Audit: Heuristics for Exploratory Testing - from CAST, STP, STAR, and KWSQA in 2011 and 2012

Recently uploaded

Surviving an FDA Audit: Heuristics for Exploratory Testing - from CAST, STP, STAR, and KWSQA in 2011 and 2012