Successfully reported this slideshow.
Griffin Jones – Congruent Compliance LLC 1March 2012
Test Strategy and Design #602
Surviving an FDA Audit:
The Heuristics ...
The Heuristics for Exploratory Testing
2Griffin Jones – Congruent Compliance LLCMarch 2012
Preliminaries
 Who is in the room?
 My goal:
 Stimulate your interest to study the subject more
 Leave with a heuristi...
Assumptions and Terms
 More reference information here than I will present
 Follow the for the key points
 Much of this...
Terms
 Congruence
 Being balanced between inner feelings & outer actions
 Smells
 Symptom that possibly indicates a de...
The Problem
 Let’s assume that you are FDA regulated and trying to
do compliant context-driven or Agile, Exploratory
Test...
Fast Takeaway
 The regulator is not your business partner
 The regulator has police powers
 Pick your battles – Sometim...
Spoiler
 The regulations are not the problem
 How you are coping with the regulations is the problem
 Give the Auditors...
Not going to talk about…
 The Fear, Uncertainly, and Doubt swirling in the field
 Vendor/Experts: “You should be scared,...
Regulatory Overview
 Regulations
 For the public good - because people died
 Regulators
 FDA regulates >25% of the Gro...
Audit Survival Heuristics
 CHCMWCE
“Chocolate Mousse”
 Congruent
 Honest
 Competent
 Model (Appropriate)
 Willing
 ...
Let’s take a journey …
17
 Practice
 Congruent
 Theory
 Less Stressful
Audits
Griffin Jones – Congruent Compliance LLC
The Congruence Triad
 Congruence is when you are balanced between inner
feelings and outer actions
 The Congruence Triad...
Congruence is like a Sailboat
 Because:
 It is a vessel or container, like a basket
 It requires preparation and mainte...
The Theory Mountains …
 Dishonest
 Incompetent
 Inadequate
22
 Honest
 Competent
 Appropriate Model
 Self-Incrimina...
Honest
 Integrity, Truthful, Trust, Sincerity in:
 You and your organization
 Words, actions, and documents
 Smells
 ...
Competent
 Are you and your organization:
 Capable, credible, understands context, speaks the
language; trained in the i...
Appropriate Model
 Is the process model:
 Complete, reasonable, practical, logical, explainable
 Smells
 Inadequate mo...
The Practice Mountains …
 Unwilling
 Out-of-Control
 No Evidence
26
 Excessive or Wasteful
 Micro-Management
 Obsess...
Willing
 Motivated, focused, prioritized, committed, resourced,
staffed, supported, given attention, nurtured
 Smells
 ...
Under Control
 Explain what you are doing and why. Are you living it?
 Coherently explain your:
 configuration control ...
Evidence
 Auditable evidence:
 Clear, objective, retrieval, human readable, attributable,
contemporary evidence that a t...
How do you apply this?
 Application is as simple as:
30
Remembering
to ask the questions.
Follow the energy
of the answer...
During an Audit
 Choosing a regulatory posture
 Manageable issues (within reason)
 Evidence
 Controls
 Willingness (r...
More Fast Takeaways
 The FDA is open to agile processes and realizes that
the current approach to software validation is ...
Natural Evidence
 Periodically , take the observer point-of-view and ask:
 Is what I see and hear, about the theory and ...
Organizational
Smells
Going Tilt
Traps
34Griffin Jones – Congruent Compliance LLC
Smells that lead to …
 Stop Shaking the Snow Globe
 Hyper-change alongside brittle/heavy formal processes
 The “Best Pr...
Organizational Disasters
 Pathetic Compliance
 Following a regulatory compliant procedure in a way
that does not solve t...
Is the Auditor on Tilt?
37
 Maybe it is something we said or did, or are doing?
 History
 That you are unaware of, and ...
Classic Agile Traps
 Mixing informal and formal processes
 Start informal - clearly switch to formal when ready
 Emphas...
Classic ET Traps
 Implementation details identified as requirements
 Tighten and simplify your requirements
 Documentat...
The BIG Trap
 Weak Evidence
 “Clear, objective, retrieval, human readable,
attributable, contemporary evidence that a th...
Audits can be Useful
 Candor can result in free consulting and insight
 Should you take the risk?
 Provides motivation ...
Recap of the Spoiler
 The regulations are not the problem.
 How you are coping with the regulations is the problem.
 Gi...
The Big Take Away
 Understand your regulatory context
 Work on your congruence
 Work each level of the model, ask the q...
Questions?
44
Model
Competent
Honest
Evidence
Control
Willing
Congruent
Griffin Jones – Congruent Compliance LLC
Further Study - A
 FDA presentations and resources:
 Webinar with FDA's John Murray on Software Validation
in the Field ...
Further Study - B
 Regulatory Compliance
 “The Art of Compliance: Turning Compliance into
Sustainable Business Advantage...
Further Study - C
 Audits
 “The ASQ Auditing Handbook” by J. P. Russell
 Congruence
 “Beyond Blaming” by Jean McLendon...
Further Study - D
 Agile and the FDA
 Business Risk (from the FDA) versus Product Risk
 http://blogs.construx.com/forum...
Further Study - E
 Agile and the FDA
 http://rdn-consulting.com/blog/2007/07/25/update-
agile-development-in-a-fda-regul...
Griffin Jones
Congruent Compliance
Griffin.Jones@CongruentCompliance.com
Thank you for attending this session.
Please fill...
Upcoming SlideShare
Loading in …5
×

Surviving an FDA Audit: Heuristics for Exploratory Testing - from CAST, STP, STAR, and KWSQA in 2011 and 2012

579 views

Published on

In FDA regulated industries, audits are high-stakes, fact-finding exercises required to verify compliance to regulations and an organization’s internal procedures. Although exploratory testing has emerged as a powerful test approach within regulated industries, an audit is the impact point where exploratory testing and regulatory worlds collide. Griffin Jones describes a heuristic model—Congruence, Honesty, Competence, Appropriate Process Model, Willingness, Control, and Evidence—his team used to survive an audit. You can use this model to prepare for an audit or to baseline your current practices for an improvement program.

Griffin highlights the common misconceptions and traps to avoid with exploratory testing in your regulated industry. Avoid mutual misunderstandings that can trigger episodes of incongruous behavior and an unsuccessful audit. Learn how to maintain your composure during a stressful audit and leave with valuable heuristics to help you organize and present your exploratory testing results with confidence.

Published in: Software, Technology, Business
  • Be the first to comment

  • Be the first to like this

Surviving an FDA Audit: Heuristics for Exploratory Testing - from CAST, STP, STAR, and KWSQA in 2011 and 2012

  1. 1. Griffin Jones – Congruent Compliance LLC 1March 2012 Test Strategy and Design #602 Surviving an FDA Audit: The Heuristics for Exploratory Testing Griffin Jones, Consultant, Congruent Compliance
  2. 2. The Heuristics for Exploratory Testing 2Griffin Jones – Congruent Compliance LLCMarch 2012
  3. 3. Preliminaries  Who is in the room?  My goal:  Stimulate your interest to study the subject more  Leave with a heuristic to help you organize and present with confidence your ET results to regulatory auditors  Have a conversation and try to meet your needs  Quick Preview  The context  The heuristic and how to apply it  Some of the traps about ET in a regulated industry Griffin Jones – Congruent Compliance LLC 3
  4. 4. Assumptions and Terms  More reference information here than I will present  Follow the for the key points  Much of this can be adapted to other contexts  i.e., not “FDA regulated, Exploratory Testing”  “Schools of Testing” by Bret Pettichord  Analytic , Standard, Quality, Context-Driven, Agile  Exploratory Testing  Simultaneous learning, test design and test execution  Agile Testing  Story completion, test automation: Test Driven Dev., etc. 4Griffin Jones – Congruent Compliance LLC
  5. 5. Terms  Congruence  Being balanced between inner feelings & outer actions  Smells  Symptom that possibly indicates a deeper problem  5 Whys  Questions-asking method to investigate root causes  “Mary had a little lamb” heuristic  Emphasize each of the individual words in a statement  Checking: confirming existing beliefs; versus:  Testing - finding new information (Michael Bolton) Griffin Jones – Congruent Compliance LLC 5
  6. 6. The Problem  Let’s assume that you are FDA regulated and trying to do compliant context-driven or Agile, Exploratory Testing  You likely have these concerns about passing an audit:  Evidence is not sufficient  Documentation is not sufficient  Process control is not sufficient  Can’t clearly explain what you do and why  Auditors value different things than you, and speak a different language Griffin Jones – Congruent Compliance LLC 6
  7. 7. Fast Takeaway  The regulator is not your business partner  The regulator has police powers  Pick your battles – Sometimes, “Let the Wookie win”  “Render unto Caesar, that which is Caesar’s …”  Auditors are likely of the “Quality” (gatekeepers) or “Routine” (traceability matrix) testing school model  You are a different testing school. Deal with it.  Auditors think “testing” is “demonstration and checking”  Don’t try and convert them. Deal with it. 7Griffin Jones – Congruent Compliance LLC
  8. 8. Spoiler  The regulations are not the problem  How you are coping with the regulations is the problem  Give the Auditors what they want:  Clear traceable requirements and description of risks  Description and demonstration of control  Clear objective evidence  The ability to understand their concerns, speak their language, and explain how you are compliant  Abundant, quality evidence mitigates your other problems 8Griffin Jones – Congruent Compliance LLC
  9. 9. Not going to talk about…  The Fear, Uncertainly, and Doubt swirling in the field  Vendor/Experts: “You should be scared, but I have…”  Silver Bullets and Big Magic  “… so trust me and just buy my wares. By the way, ..”  Persistent Myths  “… IMO the regulators “frown on” ET (… I don’t sell it).”  The “Typical” Regulatory Affairs Presentation 9Griffin Jones – Congruent Compliance LLC
  10. 10. Regulatory Overview  Regulations  For the public good - because people died  Regulators  FDA regulates >25% of the Gross Domestic Product  Regulatory Auditors  Police Powers  Industry Auditors  Assessors and valued advisors to management  Audits 10 Details Griffin Jones – Congruent Compliance LLC
  11. 11. Audit Survival Heuristics  CHCMWCE “Chocolate Mousse”  Congruent  Honest  Competent  Model (Appropriate)  Willing  Control  Evidence 16 Model Competent Honest Evidence Control Willing Congruent Griffin Jones – Congruent Compliance LLC
  12. 12. Let’s take a journey … 17  Practice  Congruent  Theory  Less Stressful Audits Griffin Jones – Congruent Compliance LLC
  13. 13. The Congruence Triad  Congruence is when you are balanced between inner feelings and outer actions  The Congruence Triad  Self, Other, Context  Being congruent is a process  A way of communicating with yourself and others  Incongruence is when part of the triad is missing  Placating, Blaming, Super-rational, or Irrelevant?  What is missing and fill it in:  Self, Others, Context 18 Other Context Self Details Griffin Jones – Congruent Compliance LLC
  14. 14. Congruence is like a Sailboat  Because:  It is a vessel or container, like a basket  It requires preparation and maintenance  You don’t “drive” it, and requires skills of crew members  Subject to weather  Is vulnerable to sinking 20Griffin Jones – Congruent Compliance LLC Tools
  15. 15. The Theory Mountains …  Dishonest  Incompetent  Inadequate 22  Honest  Competent  Appropriate Model  Self-Incriminating  Experts and Heroes  Over-Constrained Griffin Jones – Congruent Compliance LLC
  16. 16. Honest  Integrity, Truthful, Trust, Sincerity in:  You and your organization  Words, actions, and documents  Smells  Dishonest  Self-incrimination  Don’t create even the appearance of a problem  Tests  How do you and the organization react to criticism?  Are you a learning organization? (5 Why) 23Griffin Jones – Congruent Compliance LLC
  17. 17. Competent  Are you and your organization:  Capable, credible, understands context, speaks the language; trained in the industry, technology, and regulatory obligations  Smells  Incompetent  Experts and heroes  Tests  Do you believe you are capable of doing good work? (5 Why) 24Griffin Jones – Congruent Compliance LLC
  18. 18. Appropriate Model  Is the process model:  Complete, reasonable, practical, logical, explainable  Smells  Inadequate model  Over-constrained model  Test:  What problem is this model solving? How will it Fail?  What is required in this model? Missing?  Do you believe this model is sufficient? (5 Why) 25Griffin Jones – Congruent Compliance LLC
  19. 19. The Practice Mountains …  Unwilling  Out-of-Control  No Evidence 26  Excessive or Wasteful  Micro-Management  Obsessive-Compulsive  Willing  Under Control  Evidence Griffin Jones – Congruent Compliance LLC
  20. 20. Willing  Motivated, focused, prioritized, committed, resourced, staffed, supported, given attention, nurtured  Smells  Unwilling  Excessive or Wasteful  Test  Do people care? (5 Why)  Is there sufficient resources for the work and expectations? (5 Why) 27Griffin Jones – Congruent Compliance LLC
  21. 21. Under Control  Explain what you are doing and why. Are you living it?  Coherently explain your:  configuration control and authorization  traceability and accountable  organization, preparation, planning, independent review, prevention, correction, checking and testing  Smells  Out-of control  Micro-managed  Tests  Is the type and level of controls appropriate? (5 Why) 28Griffin Jones – Congruent Compliance LLC
  22. 22. Evidence  Auditable evidence:  Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions.  Smells  No-evidence  Obsessive-compulsive evidence  Tests  Explain why the specific evidence meets the criteria. (5 Why) 29Griffin Jones – Congruent Compliance LLC
  23. 23. How do you apply this?  Application is as simple as: 30 Remembering to ask the questions. Follow the energy of the answers. Fix the base, first. Griffin Jones – Congruent Compliance LLC
  24. 24. During an Audit  Choosing a regulatory posture  Manageable issues (within reason)  Evidence  Controls  Willingness (resources and priority)  Unmanageable issues  Broken process model  Lack of competence  Broken trust  Incongruence 31Griffin Jones – Congruent Compliance LLC
  25. 25. More Fast Takeaways  The FDA is open to agile processes and realizes that the current approach to software validation is not working  At the same time, companies are more concerned about:  the business risk that the FDA would not accept the agile process,  than the product or project risk that is associated with waterfall type development  Find the middle option for your context 32Griffin Jones – Congruent Compliance LLC
  26. 26. Natural Evidence  Periodically , take the observer point-of-view and ask:  Is what I see and hear, about the theory and practice of what we do:  acceptable from both a product qualification and regulatory compliance point of view?  If yes, what is the most natural, efficient, and strongest evidence we could collect?  Why not a video/audio recordings w/ paper summary?  Is it being collected? If no, why not? (5 Why)  organizational problem? 33Griffin Jones – Congruent Compliance LLC
  27. 27. Organizational Smells Going Tilt Traps 34Griffin Jones – Congruent Compliance LLC
  28. 28. Smells that lead to …  Stop Shaking the Snow Globe  Hyper-change alongside brittle/heavy formal processes  The “Best Practice” Cargo Cult  We don’t really understand the details of what we do, why we do it, or how what we do works. But have faith.  Testing Death Spiral  Regulator does not care about testing and management might only care about regulatory compliance. Spiral.  The Titanic  The gigantic engineered process is perfect – people are the source of problems, not solutions 35Griffin Jones – Congruent Compliance LLC
  29. 29. Organizational Disasters  Pathetic Compliance  Following a regulatory compliant procedure in a way that does not solve the testing problem for which it was designed.  Utopian Shelf-ware Procedures  No one reads them. They are not reality.  Close Enough  I don’t have to do it exactly. I know better. No one will notice or care.  Read My Mind  Because that is the only place where the evidence is. 36Griffin Jones – Congruent Compliance LLC
  30. 30. Is the Auditor on Tilt? 37  Maybe it is something we said or did, or are doing?  History  That you are unaware of, and it might be complicated  Notches on the gun  May be making a name for themselves  Making an example of you  May be constructing an example to deter others Griffin Jones – Congruent Compliance LLC
  31. 31. Classic Agile Traps  Mixing informal and formal processes  Start informal - clearly switch to formal when ready  Emphasizing change; light documents = poke the bear  Stokes anxiety: control, process model, and competence  Mistaking team conversation and understanding  For objective documented evidence  Speaking “Crazy Agile Moon Language”  Give the auditor what they want, in their language  Shows empathy and industry competence 38Griffin Jones – Congruent Compliance LLC Pass Fail
  32. 32. Classic ET Traps  Implementation details identified as requirements  Tighten and simplify your requirements  Documentation lacks detail to support traceability  Require less mind reading  Control is vague or assumed  Summarize and document what control is for you 39Griffin Jones – Congruent Compliance LLC
  33. 33. The BIG Trap  Weak Evidence  “Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions.”  Check it via “Mary had a little lamb”  Collect it naturally  Weak evidence is likely a symptom of other deeper issues  Abundant, quality evidence mitigates your other problems 40Griffin Jones – Congruent Compliance LLC
  34. 34. Audits can be Useful  Candor can result in free consulting and insight  Should you take the risk?  Provides motivation – management cares  Provides actionable data  The jiggle that is needed by the organization  A counter-measure to low expectations & poor practices 41 If you can’t be a good example, you are going to be a stern warning. Griffin Jones – Congruent Compliance LLC
  35. 35. Recap of the Spoiler  The regulations are not the problem.  How you are coping with the regulations is the problem.  Give the Auditors what they want:  Clear traceable requirements and description of risks  Description and demonstration of control  Clear objective evidence  The ability to understand their concerns, speak their language, and explain how you are compliant  Abundant and quality evidence mitigates your other problems. 42Griffin Jones – Congruent Compliance LLC
  36. 36. The Big Take Away  Understand your regulatory context  Work on your congruence  Work each level of the model, ask the questions  Document how you are under control  Improve your evidence, collect it naturally  Avoid the smells, disasters, and traps  Summarize your regulatory story, practice explaining it  Apply what you learn during the audit 43 1 2 3 Griffin Jones – Congruent Compliance LLC
  37. 37. Questions? 44 Model Competent Honest Evidence Control Willing Congruent Griffin Jones – Congruent Compliance LLC
  38. 38. Further Study - A  FDA presentations and resources:  Webinar with FDA's John Murray on Software Validation in the Field of Medical Devices  Presentation: Preparing for an FDA Medical Device Sponsor Inspection  Quality System Inspection Technique – Inspection Guide  General Principles of Software Validation; Final Guidance for Industry and FDA Staff 45Griffin Jones – Congruent Compliance LLC
  39. 39. Further Study - B  Regulatory Compliance  “The Art of Compliance: Turning Compliance into Sustainable Business Advantage” by Robert Rhoades of Quintiles  FDA inspections:  “How to Host an FDA Inspection” by SGS – Life Science Services  “Preparation for FDA Inspection” by NEMA/ADVAMED/PHILIPS  “FDA Sponsor Inspections: How to Prepare and Survive” by Medtronic, Inc 46Griffin Jones – Congruent Compliance LLC
  40. 40. Further Study - C  Audits  “The ASQ Auditing Handbook” by J. P. Russell  Congruence  “Beyond Blaming” by Jean McLendon and Gerald M. Weinberg  “The Satir Model: Family Therapy and Beyond” by Virginia M. Satir  “More Secrets of Consulting: The Consultant's Tool Kit” by Gerald M. Weinberg  Testers and Auditors  “Testers are like auditors” by James Christie  Evidence  “21 CFR Part 11 Electronic Records …” by the FDA 47Griffin Jones – Congruent Compliance LLC
  41. 41. Further Study - D  Agile and the FDA  Business Risk (from the FDA) versus Product Risk  http://blogs.construx.com/forums/t/432.aspx  “What is Exploratory Testing? And How it Differs from Scripted Testing” by James Bach  “Coping With Complexity: Lessons From a Medical Device Project” by Yaron Kottler  “Introduction into IEC 62304 Software life cycle for medical devices” by Christoph Gerber  http://www.spiq.com/abs/JF200809IEC62304%20SPIQ%20 Rev004.pdf  “Who says ET is good for Medical Devices? The FDA!” by James Bach  http://www.satisfice.com/blog/archives/602 48Griffin Jones – Congruent Compliance LLC
  42. 42. Further Study - E  Agile and the FDA  http://rdn-consulting.com/blog/2007/07/25/update- agile-development-in-a-fda-regulated-setting/  http://www.agilejournal.com/articles/columns/column- articles/3463-four-reasons-medical-device-companies- need-agile-development  http://rdn-consulting.com/blog/wp- content/uploads/2007/07/060703ResMed.pdf  http://scalingsoftwareagility.wordpress.com/2010/11/23/ an-iterative-and-incremental-process-model-for-agile- development-in-regulated-environments/  http://scalingsoftwareagility.wordpress.com/category/hi gh-assurance-and-regulated-environments/ 49Griffin Jones – Congruent Compliance LLC
  43. 43. Griffin Jones Congruent Compliance Griffin.Jones@CongruentCompliance.com Thank you for attending this session. Please fill out the evaluation form. 50Griffin Jones – Congruent Compliance LLC

×