Your SlideShare is downloading. ×
Cast 2011   what do auditors expect from testers - griffin jones
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Cast 2011 what do auditors expect from testers - griffin jones

121
views

Published on

Published in: Software, Business, Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
121
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. The audit survival heuristics of an FDA regulated exploratory testing team CAST August 8th, 2011 1Griffin Jones – Congruent Compliance LLC © 2011
  • 2. Preliminaries  Who is in the room?  My goal:  Stimulate your interest to study the subject more  Leave with a heuristic to help you organize and present with confidence your ET results to regulatory auditors  Have a conversation and try to meet your needs  Quick Preview  The context  The heuristic and how to apply it  Some of the traps about ET in a regulated industry CAST August 8th, 2011 2Griffin Jones – Congruent Compliance LLC © 2011
  • 3. Assumptions and Terms  This is a living presentation  Based on my experiences of auditing and being audited  More reference information here than I will present  Follow the for the key points  Much of this can be adapted to other contexts  i.e., not “FDA regulated, Exploratory Testing”  “Schools of Testing” by Bret Pettichord  Analytic , Standard, Quality, Context-Driven, Agile  Exploratory Testing:  Simultaneous learning, test design and test execution CAST August 8th, 2011 3Griffin Jones – Congruent Compliance LLC © 2011
  • 4. Terms  Congruence  Being balanced between inner feelings & outer actions  Smells  Symptom that possibly indicates a deeper problem  5 Whys  Questions-asking method to investigate root causes  “Mary had a little lamb” heuristic  Emphasize each of the individual words in a statement  Checking: confirming existing beliefs; versus:  Testing - finding new information (Michael Bolton) CAST August 8th, 2011 4Griffin Jones – Congruent Compliance LLC © 2011
  • 5. The Problem  Let’s assume that you are FDA regulated and trying to do compliant context driven, Exploratory Testing  You likely have these concerns about passing an audit:  Evidence is not sufficient  Documentation is not sufficient  Process control is not sufficient  Can’t clearly explain what you do and why  Auditors value different things than you, and speak a different language CAST August 8th, 2011 5Griffin Jones – Congruent Compliance LLC © 2011
  • 6. Fast Takeaway  The regulator is not your business partner  The regulator has police powers  “Let the Wookie win”  Auditors are likely of the “Quality” (gatekeepers) or “Routine” (traceability matrix) testing school model  You are Context Driven testing school. Deal with it.  Auditors think “testing” is “demonstration and checking”  Don’t try and convert them. Deal with it. CAST August 8th, 2011 6Griffin Jones – Congruent Compliance LLC © 2011
  • 7. Spoiler  The regulations are not the problem  How you are coping with the regulations is the problem  Give the Auditors what they want:  Clear traceable requirements and description of risks  Description and demonstration of control  Clear objective evidence  The ability to understand their concerns, speak their language, and explain how you are compliant  Abundant, quality evidence mitigates your other problems CAST August 8th, 2011 7Griffin Jones – Congruent Compliance LLC © 2011
  • 8. Not going to talk about…  The Fear, Uncertainly, and Doubt swirling in the field  Vendor/Experts: “You should be scared, but I have…”  Silver Bullets and Big Magic  “… so trust me and just buy my wares. By the way, ..”  Persistent Myths  “… IMO the regulators “frown on” ET (… I don’t sell it).”  The “Typical” Regulatory Affairs Presentation CAST August 8th, 2011 8Griffin Jones – Congruent Compliance LLC © 2011
  • 9. Regulatory Overview  Regulations  For the public good - because people died  Regulators  FDA regulates >25% of the Gross Domestic Product  Regulatory Auditors  Police Powers  Industry Auditors  Assessors and valued advisors to management  Audits CAST August 8th, 2011 9 Details Griffin Jones – Congruent Compliance LLC © 2011
  • 10. Audit Survival Heuristics  CHCMWCE “Chocolate Mousse”  Congruent  Honest  Competent  Model (Appropriate)  Willing  Control  Evidence CAST August 8th, 2011 15 Model Competent Honest Evidence Control Willing Congruent Griffin Jones – Congruent Compliance LLC © 2011
  • 11. Let’s take a journey … CAST August 8th, 2011 16  Practice  Congruent  Theory  Less Stressful Audits Griffin Jones – Congruent Compliance LLC © 2011
  • 12. The Congruence Triad  Congruence is when you are balanced between inner feelings and outer actions  The Congruence Triad  Self, Other, Context  Being congruent is a process  A way of communicating with yourself and others  Incongruence is when part of the triad is missing  Placating, Blaming, Super-rational, or Irrelevant?  What is missing and fill it in:  Self, Others, Context CAST August 8th, 2011 17 Other Context Self Details Griffin Jones – Congruent Compliance LLC © 2011
  • 13. The Theory Mountains …  Dishonest  Incompetent  Inadequate CAST August 8th, 2011 21  Honest  Competent  Appropriate Model  Self-Incriminating  Experts and Heroes  Over-Constrained Griffin Jones – Congruent Compliance LLC © 2011
  • 14. Honest  Integrity, Truthful, Trust, Sincerity in:  You and your organization  Words, actions, and documents  Smells  Dishonest  Self-incrimination  Don’t create even the appearance of a problem  Tests  How do you and the organization react to criticism?  Are you a learning organization? (5 Why) CAST August 8th, 2011 22Griffin Jones – Congruent Compliance LLC © 2011
  • 15. Competent  Are you and your organization:  Capable, credible, understands context, speaks the language; trained in the industry, technology, and regulatory obligations  Smells  Incompetent  Experts and heroes  Tests  Do you believe you are capable of doing good work? (5 Why) CAST August 8th, 2011 23Griffin Jones – Congruent Compliance LLC © 2011
  • 16. Appropriate Model  Is the process model:  Complete, reasonable, practical, logical, explainable  Smells  Inadequate model  Over-constrained model  Test:  What problem is this model solving? How will it Fail?  What is required in this model? Missing?  Do you believe this model is sufficient? (5 Why) CAST August 8th, 2011 24Griffin Jones – Congruent Compliance LLC © 2011
  • 17. The Practice Mountains …  Unwilling  Out-of-Control  No Evidence CAST August 8th, 2011 25  Excessive or Wasteful  Micro-Management  Obsessive-Compulsive  Willing  Under Control  Evidence Griffin Jones – Congruent Compliance LLC © 2011
  • 18. Willing  Motivated, focused, prioritized, committed, resourced, staffed, supported, given attention, nurtured  Smells  Unwilling  Excessive or Wasteful  Test  Do people care? (5 Why)  Is there sufficient resources for the work and expectations? (5 Why) CAST August 8th, 2011 26Griffin Jones – Congruent Compliance LLC © 2011
  • 19. Under Control  Explain what you are doing and why. Are you living it?  Coherently explain your:  configuration control and authorization  traceability and accountable  organization, preparation, planning, independent review, prevention, correction, checking and testing  Smells  Out-of control  Micro-managed  Tests  Is the type and level of controls appropriate? (5 Why) CAST August 8th, 2011 27Griffin Jones – Congruent Compliance LLC © 2011
  • 20. Evidence  Auditable evidence:  Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions.  Smells  No-evidence  Obsessive-compulsive evidence  Tests  Explain why the specific evidence meets the criteria. (5 Why) CAST August 8th, 2011 28Griffin Jones – Congruent Compliance LLC © 2011
  • 21. How do you apply this?  Application is as simple as: CAST August 8th, 2011 29 Remembering to ask the questions. Follow the energy of the answers. Fix the base, first. Griffin Jones – Congruent Compliance LLC © 2011
  • 22. During an Audit  Choosing a regulatory posture  Manageable issues (within reason)  Evidence  Controls  Willingness (resources and priority)  Unmanageable issues  Broken process model  Lack of competence  Broken trust  Incongruence CAST August 8th, 2011 30Griffin Jones – Congruent Compliance LLC © 2011
  • 23. More Fast Takeaways  The FDA is open to agile processes and realizes that the current approach to software validation is not working  At the same time, companies are more concerned about:  the business risk that the FDA would not accept the agile process,  than the product or project risk that is associated with waterfall type development  Find the middle option for your context CAST August 8th, 2011 31Griffin Jones – Congruent Compliance LLC © 2011
  • 24. Natural Evidence  Periodically , take the observer point-of-view and ask:  Is what I see and hear, about the theory and practice of what we do:  acceptable from both a product qualification and regulatory compliance point of view?  If yes, what is the most natural, efficient, and strongest evidence we could collect?  Why not a video/audio recordings w/ paper summary?  Is it being collected? If no, why not? (5 Why)  organizational problem? CAST August 8th, 2011 32Griffin Jones – Congruent Compliance LLC © 2011
  • 25. Organizational Smells Going Tilt Traps CAST August 8th, 2011 33Griffin Jones – Congruent Compliance LLC © 2011
  • 26. Smells that lead to …  Paint the Village  Visitors are coming. How shall we work today?  The “Best Practice” Cargo Cult  We don’t really understand the details of what we do, why we do it, or how what we do works. But have faith.  Testing Death Spiral  Regulator does not care about testing and management might only care about regulatory compliance. Spiral.  The Titanic  The gigantic engineered process is perfect – people are the source of problems, not solutions CAST August 8th, 2011 34Griffin Jones – Congruent Compliance LLC © 2011
  • 27. Organizational Disasters  Pathetic Compliance  Following a regulatory compliant procedure in a way that does not solve the testing problem for which it was designed.  Utopian Shelf-ware Procedures  No one reads them. They are not reality.  Close Enough  I don’t have to do it exactly. I know better. No one will notice or care.  Read My Mind  Because that is the only place where the evidence is. CAST August 8th, 2011 35Griffin Jones – Congruent Compliance LLC © 2011
  • 28. Is the Auditor on Tilt? CAST August 8th, 2011 36  Maybe it is something we said or did, or are doing?  History  That you are unaware of, and it might be complicated  Notches on the gun  May be making a name for themselves  Making an example of you  May be constructing an example to deter others Griffin Jones – Congruent Compliance LLC © 2011
  • 29. Classic ET Traps  Implementation details identified as requirements  Tighten and simplify your requirements  Documentation lacks detail to support traceability  Require less mind reading.  Control is vague or assumed  Summarize and document what control is for you CAST August 8th, 2011 37Griffin Jones – Congruent Compliance LLC © 2011
  • 30. The BIG Trap  Weak Evidence  “Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions.”  Check it via “Mary had a little lamb”  Collect it naturally  Weak evidence is likely a symptom of other deeper issues  Abundant, quality evidence mitigates your other problems CAST August 8th, 2011 38Griffin Jones – Congruent Compliance LLC © 2011
  • 31. Audits can be Useful  Candor can result in free consulting and insight  Should you take the risk?  Provides motivation – management cares  Provides actionable data  The jiggle that is needed by the organization  A counter-measure to low expectations & poor practices CAST August 8th, 2011 39 If you can’t be a good example, you are going to be a stern warning. Griffin Jones – Congruent Compliance LLC © 2011
  • 32. Recap of the Spoiler  The regulations are not the problem.  How you are coping with the regulations is the problem.  Give the Auditors what they want:  Clear traceable requirements and description of risks  Description and demonstration of control  Clear objective evidence  The ability to understand their concerns, speak their language, and explain how you are compliant  Abundant and quality evidence mitigates your other problems. CAST August 8th, 2011 40Griffin Jones – Congruent Compliance LLC © 2011
  • 33. The Big Take Away  Understand your regulatory context  Work on your congruence  Work each level of the model, ask the questions  Document how you are under control  Improve your evidence, collect it naturally  Avoid the smells, disasters, and traps  Summarize your regulatory story, practice explaining it  Apply what you learn during the audit CAST August 8th, 2011 41 1 2 3 Griffin Jones – Congruent Compliance LLC © 2011
  • 34. Questions? CAST August 8th, 2011 42 Model Competent Honest Evidence Control Willing Congruent Griffin Jones – Congruent Compliance LLC © 2011
  • 35. Further Study - A  FDA presentations and resources:  Webinar with FDA's John Murray on Software Validation in the Field of Medical Devices  Presentation: Preparing for an FDA Medical Device Sponsor Inspection  Quality System Inspection Technique – Inspection Guide  General Principles of Software Validation; Final Guidance for Industry and FDA Staff CAST August 8th, 2011 43Griffin Jones – Congruent Compliance LLC © 2011
  • 36. Further Study - B  Regulatory Compliance  “The Art of Compliance: Turning Compliance into Sustainable Business Advantage” by Robert Rhoades of Quintiles  FDA inspections:  “How to Host an FDA Inspection” by SGS – Life Science Services  “Preparation for FDA Inspection” by NEMA/ADVAMED/PHILIPS  “FDA Sponsor Inspections: How to Prepare and Survive” by Medtronic, Inc CAST August 8th, 2011 44Griffin Jones – Congruent Compliance LLC © 2011
  • 37. Further Study - C  Audits  “The ASQ Auditing Handbook” by J. P. Russell  Congruence  “Beyond Blaming” by Jean McLendon and Gerald M. Weinberg  “The Satir Model: Family Therapy and Beyond” by Virginia M. Satir  “More Secrets of Consulting: The Consultant's Tool Kit” by Gerald M. Weinberg CAST August 8th, 2011 45Griffin Jones – Congruent Compliance LLC © 2011
  • 38. Further Study - D  Agile and the FDA  Business Risk (from the FDA) versus Product Risk  http://blogs.construx.com/forums/t/432.aspx  “What is Exploratory Testing? And How it Differs from Scripted Testing” by James Bach  “Coping With Complexity: Lessons From a Medical Device Project” by Yaron Kottler  Testers and Auditors  “Testers are like auditors” by James Christie  Evidence  “21 CFR Part 11 Electronic Records …” by the FDA CAST August 8th, 2011 46Griffin Jones – Congruent Compliance LLC © 2011
  • 39. Griffin Jones Congruent Compliance Griffin.Jones@CongruentCompliance.com Thank You! CAST August 8th, 2011 47Griffin Jones – Congruent Compliance LLC © 2011