The audit survival heuristics of
an FDA regulated exploratory testing team
CAST August 8th, 2011 1Griffin Jones – Congruen...
Preliminaries
 Who is in the room?
 My goal:
 Stimulate your interest to study the subject more
 Leave with a heuristi...
Assumptions and Terms
 This is a living presentation
 Based on my experiences of auditing and being audited
 More refer...
Terms
 Congruence
 Being balanced between inner feelings & outer actions
 Smells
 Symptom that possibly indicates a de...
The Problem
 Let’s assume that you are FDA regulated and trying to
do compliant context driven, Exploratory Testing
 You...
Fast Takeaway
 The regulator is not your business partner
 The regulator has police powers
 “Let the Wookie win”
 Audi...
Spoiler
 The regulations are not the problem
 How you are coping with the regulations is the problem
 Give the Auditors...
Not going to talk about…
 The Fear, Uncertainly, and Doubt swirling in the field
 Vendor/Experts: “You should be scared,...
Regulatory Overview
 Regulations
 For the public good - because people died
 Regulators
 FDA regulates >25% of the Gro...
Audit Survival Heuristics
 CHCMWCE
“Chocolate Mousse”
 Congruent
 Honest
 Competent
 Model (Appropriate)
 Willing
 ...
Let’s take a journey …
CAST August 8th, 2011 16
 Practice
 Congruent
 Theory
 Less Stressful
Audits
Griffin Jones – Co...
The Congruence Triad
 Congruence is when you are balanced between inner
feelings and outer actions
 The Congruence Triad...
The Theory Mountains …
 Dishonest
 Incompetent
 Inadequate
CAST August 8th, 2011 21
 Honest
 Competent
 Appropriate ...
Honest
 Integrity, Truthful, Trust, Sincerity in:
 You and your organization
 Words, actions, and documents
 Smells
 ...
Competent
 Are you and your organization:
 Capable, credible, understands context, speaks the
language; trained in the i...
Appropriate Model
 Is the process model:
 Complete, reasonable, practical, logical, explainable
 Smells
 Inadequate mo...
The Practice Mountains …
 Unwilling
 Out-of-Control
 No Evidence
CAST August 8th, 2011 25
 Excessive or Wasteful
 Mic...
Willing
 Motivated, focused, prioritized, committed, resourced,
staffed, supported, given attention, nurtured
 Smells
 ...
Under Control
 Explain what you are doing and why. Are you living it?
 Coherently explain your:
 configuration control ...
Evidence
 Auditable evidence:
 Clear, objective, retrieval, human readable, attributable,
contemporary evidence that a t...
How do you apply this?
 Application is as simple as:
CAST August 8th, 2011 29
Remembering
to ask the questions.
Follow th...
During an Audit
 Choosing a regulatory posture
 Manageable issues (within reason)
 Evidence
 Controls
 Willingness (r...
More Fast Takeaways
 The FDA is open to agile processes and realizes that
the current approach to software validation is ...
Natural Evidence
 Periodically , take the observer point-of-view and ask:
 Is what I see and hear, about the theory and ...
Organizational
Smells
Going Tilt
Traps
CAST August 8th, 2011 33Griffin Jones – Congruent Compliance LLC © 2011
Smells that lead to …
 Paint the Village
 Visitors are coming. How shall we work today?
 The “Best Practice” Cargo Cult...
Organizational Disasters
 Pathetic Compliance
 Following a regulatory compliant procedure in a way
that does not solve t...
Is the Auditor on Tilt?
CAST August 8th, 2011 36
 Maybe it is something we said or did, or are doing?
 History
 That yo...
Classic ET Traps
 Implementation details identified as requirements
 Tighten and simplify your requirements
 Documentat...
The BIG Trap
 Weak Evidence
 “Clear, objective, retrieval, human readable,
attributable, contemporary evidence that a th...
Audits can be Useful
 Candor can result in free consulting and insight
 Should you take the risk?
 Provides motivation ...
Recap of the Spoiler
 The regulations are not the problem.
 How you are coping with the regulations is the problem.
 Gi...
The Big Take Away
 Understand your regulatory context
 Work on your congruence
 Work each level of the model, ask the q...
Questions?
CAST August 8th, 2011 42
Model
Competent
Honest
Evidence
Control
Willing
Congruent
Griffin Jones – Congruent Co...
Further Study - A
 FDA presentations and resources:
 Webinar with FDA's John Murray on Software Validation
in the Field ...
Further Study - B
 Regulatory Compliance
 “The Art of Compliance: Turning Compliance into
Sustainable Business Advantage...
Further Study - C
 Audits
 “The ASQ Auditing Handbook” by J. P. Russell
 Congruence
 “Beyond Blaming” by Jean McLendon...
Further Study - D
 Agile and the FDA
 Business Risk (from the FDA) versus Product Risk
 http://blogs.construx.com/forum...
Griffin Jones
Congruent Compliance
Griffin.Jones@CongruentCompliance.com
Thank You!
CAST August 8th, 2011 47Griffin Jones ...
Upcoming SlideShare
Loading in …5
×

Cast 2011 what do auditors expect from testers - griffin jones

339 views

Published on

Published in: Software, Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
339
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Cast 2011 what do auditors expect from testers - griffin jones

  1. 1. The audit survival heuristics of an FDA regulated exploratory testing team CAST August 8th, 2011 1Griffin Jones – Congruent Compliance LLC © 2011
  2. 2. Preliminaries  Who is in the room?  My goal:  Stimulate your interest to study the subject more  Leave with a heuristic to help you organize and present with confidence your ET results to regulatory auditors  Have a conversation and try to meet your needs  Quick Preview  The context  The heuristic and how to apply it  Some of the traps about ET in a regulated industry CAST August 8th, 2011 2Griffin Jones – Congruent Compliance LLC © 2011
  3. 3. Assumptions and Terms  This is a living presentation  Based on my experiences of auditing and being audited  More reference information here than I will present  Follow the for the key points  Much of this can be adapted to other contexts  i.e., not “FDA regulated, Exploratory Testing”  “Schools of Testing” by Bret Pettichord  Analytic , Standard, Quality, Context-Driven, Agile  Exploratory Testing:  Simultaneous learning, test design and test execution CAST August 8th, 2011 3Griffin Jones – Congruent Compliance LLC © 2011
  4. 4. Terms  Congruence  Being balanced between inner feelings & outer actions  Smells  Symptom that possibly indicates a deeper problem  5 Whys  Questions-asking method to investigate root causes  “Mary had a little lamb” heuristic  Emphasize each of the individual words in a statement  Checking: confirming existing beliefs; versus:  Testing - finding new information (Michael Bolton) CAST August 8th, 2011 4Griffin Jones – Congruent Compliance LLC © 2011
  5. 5. The Problem  Let’s assume that you are FDA regulated and trying to do compliant context driven, Exploratory Testing  You likely have these concerns about passing an audit:  Evidence is not sufficient  Documentation is not sufficient  Process control is not sufficient  Can’t clearly explain what you do and why  Auditors value different things than you, and speak a different language CAST August 8th, 2011 5Griffin Jones – Congruent Compliance LLC © 2011
  6. 6. Fast Takeaway  The regulator is not your business partner  The regulator has police powers  “Let the Wookie win”  Auditors are likely of the “Quality” (gatekeepers) or “Routine” (traceability matrix) testing school model  You are Context Driven testing school. Deal with it.  Auditors think “testing” is “demonstration and checking”  Don’t try and convert them. Deal with it. CAST August 8th, 2011 6Griffin Jones – Congruent Compliance LLC © 2011
  7. 7. Spoiler  The regulations are not the problem  How you are coping with the regulations is the problem  Give the Auditors what they want:  Clear traceable requirements and description of risks  Description and demonstration of control  Clear objective evidence  The ability to understand their concerns, speak their language, and explain how you are compliant  Abundant, quality evidence mitigates your other problems CAST August 8th, 2011 7Griffin Jones – Congruent Compliance LLC © 2011
  8. 8. Not going to talk about…  The Fear, Uncertainly, and Doubt swirling in the field  Vendor/Experts: “You should be scared, but I have…”  Silver Bullets and Big Magic  “… so trust me and just buy my wares. By the way, ..”  Persistent Myths  “… IMO the regulators “frown on” ET (… I don’t sell it).”  The “Typical” Regulatory Affairs Presentation CAST August 8th, 2011 8Griffin Jones – Congruent Compliance LLC © 2011
  9. 9. Regulatory Overview  Regulations  For the public good - because people died  Regulators  FDA regulates >25% of the Gross Domestic Product  Regulatory Auditors  Police Powers  Industry Auditors  Assessors and valued advisors to management  Audits CAST August 8th, 2011 9 Details Griffin Jones – Congruent Compliance LLC © 2011
  10. 10. Audit Survival Heuristics  CHCMWCE “Chocolate Mousse”  Congruent  Honest  Competent  Model (Appropriate)  Willing  Control  Evidence CAST August 8th, 2011 15 Model Competent Honest Evidence Control Willing Congruent Griffin Jones – Congruent Compliance LLC © 2011
  11. 11. Let’s take a journey … CAST August 8th, 2011 16  Practice  Congruent  Theory  Less Stressful Audits Griffin Jones – Congruent Compliance LLC © 2011
  12. 12. The Congruence Triad  Congruence is when you are balanced between inner feelings and outer actions  The Congruence Triad  Self, Other, Context  Being congruent is a process  A way of communicating with yourself and others  Incongruence is when part of the triad is missing  Placating, Blaming, Super-rational, or Irrelevant?  What is missing and fill it in:  Self, Others, Context CAST August 8th, 2011 17 Other Context Self Details Griffin Jones – Congruent Compliance LLC © 2011
  13. 13. The Theory Mountains …  Dishonest  Incompetent  Inadequate CAST August 8th, 2011 21  Honest  Competent  Appropriate Model  Self-Incriminating  Experts and Heroes  Over-Constrained Griffin Jones – Congruent Compliance LLC © 2011
  14. 14. Honest  Integrity, Truthful, Trust, Sincerity in:  You and your organization  Words, actions, and documents  Smells  Dishonest  Self-incrimination  Don’t create even the appearance of a problem  Tests  How do you and the organization react to criticism?  Are you a learning organization? (5 Why) CAST August 8th, 2011 22Griffin Jones – Congruent Compliance LLC © 2011
  15. 15. Competent  Are you and your organization:  Capable, credible, understands context, speaks the language; trained in the industry, technology, and regulatory obligations  Smells  Incompetent  Experts and heroes  Tests  Do you believe you are capable of doing good work? (5 Why) CAST August 8th, 2011 23Griffin Jones – Congruent Compliance LLC © 2011
  16. 16. Appropriate Model  Is the process model:  Complete, reasonable, practical, logical, explainable  Smells  Inadequate model  Over-constrained model  Test:  What problem is this model solving? How will it Fail?  What is required in this model? Missing?  Do you believe this model is sufficient? (5 Why) CAST August 8th, 2011 24Griffin Jones – Congruent Compliance LLC © 2011
  17. 17. The Practice Mountains …  Unwilling  Out-of-Control  No Evidence CAST August 8th, 2011 25  Excessive or Wasteful  Micro-Management  Obsessive-Compulsive  Willing  Under Control  Evidence Griffin Jones – Congruent Compliance LLC © 2011
  18. 18. Willing  Motivated, focused, prioritized, committed, resourced, staffed, supported, given attention, nurtured  Smells  Unwilling  Excessive or Wasteful  Test  Do people care? (5 Why)  Is there sufficient resources for the work and expectations? (5 Why) CAST August 8th, 2011 26Griffin Jones – Congruent Compliance LLC © 2011
  19. 19. Under Control  Explain what you are doing and why. Are you living it?  Coherently explain your:  configuration control and authorization  traceability and accountable  organization, preparation, planning, independent review, prevention, correction, checking and testing  Smells  Out-of control  Micro-managed  Tests  Is the type and level of controls appropriate? (5 Why) CAST August 8th, 2011 27Griffin Jones – Congruent Compliance LLC © 2011
  20. 20. Evidence  Auditable evidence:  Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions.  Smells  No-evidence  Obsessive-compulsive evidence  Tests  Explain why the specific evidence meets the criteria. (5 Why) CAST August 8th, 2011 28Griffin Jones – Congruent Compliance LLC © 2011
  21. 21. How do you apply this?  Application is as simple as: CAST August 8th, 2011 29 Remembering to ask the questions. Follow the energy of the answers. Fix the base, first. Griffin Jones – Congruent Compliance LLC © 2011
  22. 22. During an Audit  Choosing a regulatory posture  Manageable issues (within reason)  Evidence  Controls  Willingness (resources and priority)  Unmanageable issues  Broken process model  Lack of competence  Broken trust  Incongruence CAST August 8th, 2011 30Griffin Jones – Congruent Compliance LLC © 2011
  23. 23. More Fast Takeaways  The FDA is open to agile processes and realizes that the current approach to software validation is not working  At the same time, companies are more concerned about:  the business risk that the FDA would not accept the agile process,  than the product or project risk that is associated with waterfall type development  Find the middle option for your context CAST August 8th, 2011 31Griffin Jones – Congruent Compliance LLC © 2011
  24. 24. Natural Evidence  Periodically , take the observer point-of-view and ask:  Is what I see and hear, about the theory and practice of what we do:  acceptable from both a product qualification and regulatory compliance point of view?  If yes, what is the most natural, efficient, and strongest evidence we could collect?  Why not a video/audio recordings w/ paper summary?  Is it being collected? If no, why not? (5 Why)  organizational problem? CAST August 8th, 2011 32Griffin Jones – Congruent Compliance LLC © 2011
  25. 25. Organizational Smells Going Tilt Traps CAST August 8th, 2011 33Griffin Jones – Congruent Compliance LLC © 2011
  26. 26. Smells that lead to …  Paint the Village  Visitors are coming. How shall we work today?  The “Best Practice” Cargo Cult  We don’t really understand the details of what we do, why we do it, or how what we do works. But have faith.  Testing Death Spiral  Regulator does not care about testing and management might only care about regulatory compliance. Spiral.  The Titanic  The gigantic engineered process is perfect – people are the source of problems, not solutions CAST August 8th, 2011 34Griffin Jones – Congruent Compliance LLC © 2011
  27. 27. Organizational Disasters  Pathetic Compliance  Following a regulatory compliant procedure in a way that does not solve the testing problem for which it was designed.  Utopian Shelf-ware Procedures  No one reads them. They are not reality.  Close Enough  I don’t have to do it exactly. I know better. No one will notice or care.  Read My Mind  Because that is the only place where the evidence is. CAST August 8th, 2011 35Griffin Jones – Congruent Compliance LLC © 2011
  28. 28. Is the Auditor on Tilt? CAST August 8th, 2011 36  Maybe it is something we said or did, or are doing?  History  That you are unaware of, and it might be complicated  Notches on the gun  May be making a name for themselves  Making an example of you  May be constructing an example to deter others Griffin Jones – Congruent Compliance LLC © 2011
  29. 29. Classic ET Traps  Implementation details identified as requirements  Tighten and simplify your requirements  Documentation lacks detail to support traceability  Require less mind reading.  Control is vague or assumed  Summarize and document what control is for you CAST August 8th, 2011 37Griffin Jones – Congruent Compliance LLC © 2011
  30. 30. The BIG Trap  Weak Evidence  “Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions.”  Check it via “Mary had a little lamb”  Collect it naturally  Weak evidence is likely a symptom of other deeper issues  Abundant, quality evidence mitigates your other problems CAST August 8th, 2011 38Griffin Jones – Congruent Compliance LLC © 2011
  31. 31. Audits can be Useful  Candor can result in free consulting and insight  Should you take the risk?  Provides motivation – management cares  Provides actionable data  The jiggle that is needed by the organization  A counter-measure to low expectations & poor practices CAST August 8th, 2011 39 If you can’t be a good example, you are going to be a stern warning. Griffin Jones – Congruent Compliance LLC © 2011
  32. 32. Recap of the Spoiler  The regulations are not the problem.  How you are coping with the regulations is the problem.  Give the Auditors what they want:  Clear traceable requirements and description of risks  Description and demonstration of control  Clear objective evidence  The ability to understand their concerns, speak their language, and explain how you are compliant  Abundant and quality evidence mitigates your other problems. CAST August 8th, 2011 40Griffin Jones – Congruent Compliance LLC © 2011
  33. 33. The Big Take Away  Understand your regulatory context  Work on your congruence  Work each level of the model, ask the questions  Document how you are under control  Improve your evidence, collect it naturally  Avoid the smells, disasters, and traps  Summarize your regulatory story, practice explaining it  Apply what you learn during the audit CAST August 8th, 2011 41 1 2 3 Griffin Jones – Congruent Compliance LLC © 2011
  34. 34. Questions? CAST August 8th, 2011 42 Model Competent Honest Evidence Control Willing Congruent Griffin Jones – Congruent Compliance LLC © 2011
  35. 35. Further Study - A  FDA presentations and resources:  Webinar with FDA's John Murray on Software Validation in the Field of Medical Devices  Presentation: Preparing for an FDA Medical Device Sponsor Inspection  Quality System Inspection Technique – Inspection Guide  General Principles of Software Validation; Final Guidance for Industry and FDA Staff CAST August 8th, 2011 43Griffin Jones – Congruent Compliance LLC © 2011
  36. 36. Further Study - B  Regulatory Compliance  “The Art of Compliance: Turning Compliance into Sustainable Business Advantage” by Robert Rhoades of Quintiles  FDA inspections:  “How to Host an FDA Inspection” by SGS – Life Science Services  “Preparation for FDA Inspection” by NEMA/ADVAMED/PHILIPS  “FDA Sponsor Inspections: How to Prepare and Survive” by Medtronic, Inc CAST August 8th, 2011 44Griffin Jones – Congruent Compliance LLC © 2011
  37. 37. Further Study - C  Audits  “The ASQ Auditing Handbook” by J. P. Russell  Congruence  “Beyond Blaming” by Jean McLendon and Gerald M. Weinberg  “The Satir Model: Family Therapy and Beyond” by Virginia M. Satir  “More Secrets of Consulting: The Consultant's Tool Kit” by Gerald M. Weinberg CAST August 8th, 2011 45Griffin Jones – Congruent Compliance LLC © 2011
  38. 38. Further Study - D  Agile and the FDA  Business Risk (from the FDA) versus Product Risk  http://blogs.construx.com/forums/t/432.aspx  “What is Exploratory Testing? And How it Differs from Scripted Testing” by James Bach  “Coping With Complexity: Lessons From a Medical Device Project” by Yaron Kottler  Testers and Auditors  “Testers are like auditors” by James Christie  Evidence  “21 CFR Part 11 Electronic Records …” by the FDA CAST August 8th, 2011 46Griffin Jones – Congruent Compliance LLC © 2011
  39. 39. Griffin Jones Congruent Compliance Griffin.Jones@CongruentCompliance.com Thank You! CAST August 8th, 2011 47Griffin Jones – Congruent Compliance LLC © 2011

×