By marshaling credible and persuasive evidence, influential testers answer three basic questions: How good is the product? What testing did you do? Why is the testing any good? Feeble evidence and the behaviors associated with it are common testing maladies and threats to your ability to tell a compelling story. What are the qualities of strong evidence, and the anti-patterns and risks of weak evidence? Griffin Jones presents the different qualities and types of evidence. We review threats to the credibility and persuasiveness of your work, such as: The danger of false-negative results reported as “pass – as expected”, “Lullaby Language” as an anti-pattern of Lean’s Genchi Genbutsu (go-and-see), the danger of obsessing on efficiency and how it biases the observer. Finally, we show how over-scripted procedures can superficially conceal fatal evidentiary flaws. Be intentional and critical about the type and quality of your evidence. Leave with the skills to recognize and evaluate the dangers and risks, strengths and weaknesses of the evidence you use for your testing story.

1. What is Good
Evidence?
May 2013 © 2013 Congruent Compliance LLC 1

2. Griffin Jones
Consultant
Agile / Testing / Regulatory
May 2013 © 2013 Congruent Compliance LLC 2
This Presentation Covers …
• Types of Evidence
• Qualities of Evidence
• Traps and Anti-Patterns
Why I Care, and Why You Should
• Civil and Criminal Courts
• Credible and Persuasive Story-of-Testing

3. May 2013 © 2013 Congruent Compliance LLC 3
TYPES OF EVIDENCE
Sources
• FDA’s 21 CFR Part 11 – Electronic Records
• FDA’s Good Manufacturing Practices
• (Adjust for your context)
Records versus Testimony
• Speaks for Itself
• Cross-Examination

4. May 2013 © 2013 Congruent Compliance LLC 4
QUALITIES …
Clear
• To Whom?
Objective
• To What Degree?

5. May 2013 © 2013 Congruent Compliance LLC 5
… QUALITIES …
Authentic
• Originals or Copies?
• Resistant to Falsification?
Integrity
• Protected from Damage / Deterioration?
• Can not be Readily Repudiated?

6. May 2013 © 2013 Congruent Compliance LLC 6
Readable
• Human and Machine?
Changes
• Do Not Obscure Previous Information?
• Single line-out, initial, date, reason?
… QUALITIES …

7. May 2013 © 2013 Congruent Compliance LLC 7
… QUALITIES …
Attributable
• Do we know the Specific Person?
Contemporary
• Created At-The-Moment?
• Are Not Recollections?

8. May 2013 © 2013 Congruent Compliance LLC 8
… QUALITIES …
Audit Trail
• Is there a Chain-of-Custody?
• Is there a Revision History?
Retrievable
• Is this stored where I can Access it?

9. May 2013 © 2013 Congruent Compliance LLC 9
… QUALITIES
3rd Party Review / Reconstruction
• Minimal Outside Assistance?
• Reach the Same Results & Conclusions?

10. May 2013 © 2013 Congruent Compliance LLC 10
TRAPS, THREATS AND
ANTI-PATTERNS
Deficient Qualities
• Leads to Feebleness
Feebleness
• Leads to Unpersuasive Stories
• Maybe a Systemic Risk smell?

11. May 2013 © 2013 Congruent Compliance LLC 11
“PASS-AS EXPECTED”
CHECKBOXES
Fear potential False-Positives
• The danger of the Green-Bar
• Falsification versus Confirmation
What you Intended to do
• Not what you Actually Did

12. May 2013 © 2013 Congruent Compliance LLC 12
Shuts-down Conversation
• Discourages feedback
• Promotes Shallow Agreement
• Lulls your mind into a False Sense of
Security
Styled Ceremony and Ritual
• To provoke a behavior, not inform a
decision
LULLABY LANGUAGE

13. May 2013 © 2013 Congruent Compliance LLC 13
… VERSUS LEAN’S
GENCHI GENBUTU
“… It’s Lies! All Lies!”
• Metrics and reports are just maps …
• Get out of your office
• Go observe where the work happens
• Take your own measurements

14. May 2013 © 2013 Congruent Compliance LLC 14
EFFICIENCY AND TIME PRESSURE
Biasing and Priming the Subject
• Over-Scripting over-focuses attention
• The “Moon-Walking Bear” problem
• Target Lock-On
Unringing the Bell
• The button story and pixel-drop out

15. May 2013 © 2013 Congruent Compliance LLC 15
Recognize the Different Qualities
• Clear, Objective, Authentic, Integrity,
Readable, Changes, Attributable,
Contemporary, Audit Trail, Retrievable,
Reconstructable
How is your evidence deficient?
SUMMARY … A

16. May 2013 © 2013 Congruent Compliance LLC 16
Recognize the Traps
• Pass-As-Expected
• Lullaby Language
• Moon-Walking Bear
Go Observe the Work
SUMMARY… B

17. May 2013 © 2013 Congruent Compliance LLC 17
QUESTIONS AND STORIES

18. Thank you for attending this session.
Please fill out the evaluation form.
Griffin Jones
Griffin.Jones@CongruentCompliance.com
May 2013 © 2013 Congruent Compliance LLC 18