Regulated Software Testing - Griffin Jones - TISQA 2014

1,161 views
1,114 views

Published on

Regulated software is consequential software, subject to authoritative outside review. Even experienced software testers can have an Alice-In-Wonderland feeling the first time they test a regulated product. Many aspects of software testing are surprisingly exactly the same as testing unregulated software, while others are surprisingly completely different. Why is that? What parts are surprisingly similar or different? Griffin shares experiences of testing FDA regulated systems, and similar software (e.g., financial systems, aircraft controls, insurance, and online gaming) – drawing examples from the participants of the Workshop on Regulated Software Testing (WREST), and himself. We examine the what, why, and how certain software is regulated - and the duties imposed on companies and individuals. We highlight some of the surprising aspects, such as: test design and execution; tools and automation; accuracy and rigor; traceability and authorization; evidence and record keeping; the halting problem; process standardization and predictability; human variability and adaptability; technology choices; morality and ethics; and business pressures. Leave with more insight into and less surprise about the challenges of testing regulated software.

Published in: Software, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,161
On SlideShare
0
From Embeds
0
Number of Embeds
607
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Regulated Software Testing - Griffin Jones - TISQA 2014

  1. 1. Regulated Software Testing March 2014 © 2014 Congruent Compliance LLC 1
  2. 2. Griffin Jones Consultant Agile / Testing / Regulatory March 2014 © 2014 Congruent Compliance LLC 2 Why Testing Is Not Dead … … In this Context • Because there are “Survival Rules” associated with these systems • these systems need strong “harsh tests” (in a Karl Popper way) • not just simplistic, mindless checking
  3. 3. March 2014 © 2014 Congruent Compliance LLC 3 THIS PRESENTATION … Outline • What/Why/How Software is Regulated • 11 Surprising Aspects of Regulated SW • Same as Unregulated • Different from Unregulated • Cognitive Dissonance • Questions
  4. 4. March 2014 © 2014 Congruent Compliance LLC 4 WHAT IS REGULATED SOFTWARE? Working Definition • Software subject to review by an internal or external regulatory body WREST (Workshop on Regulated Software Testing) • Share ideas and provide a forum for people who are interested in improving the testing of regulated systems
  5. 5. March 2014 © 2014 Congruent Compliance LLC 5 WHY I CARE AND WHY YOU SHOULD Authorization • Criminal, Civil, and Administrative Law The Regulators are Police • Inspect, Search, Question, Confiscate, Fine, Debarment • Deputies • Self-Policing
  6. 6. March 2014 © 2014 Congruent Compliance LLC 6 WELCOME TO WONDERLAND “Skill, Pill, and Will” • Consequences • Scrutiny of Your Work • Moral Hazard • “Just World” Hypothesis • Good decisions can still have bad outcomes
  7. 7. March 2014 © 2014 Congruent Compliance LLC 7 BUT … SURPRISE! cts of Software Testing • Some Aspects are Surprisingly Similar • Some Aspects are Surprisingly Different Unregulated versus Regulated, Aspects of Software Testing
  8. 8. March 2014 © 2014 Congruent Compliance LLC 8 11 ASPECTS OF SOFTWARE TESTING Surprisingly Similar Surprisingly Different
  9. 9. March 2014 © 2014 Congruent Compliance LLC 9 ASPECT 1 A … Test Design and Execution • Schools of Testing: Quality and Standards • Test Design Patterns [http://kaner.com/?p=100] • Checking / Demonstration of Requirements
  10. 10. March 2014 © 2014 Congruent Compliance LLC 10 ASPECT 1 B … Test Design and Execution • Failure Mode and Effects Analysis • Problem Investigation
  11. 11. March 2014 © 2014 Congruent Compliance LLC 11 ASPECT 2 … Tools and Automation • Commercial Tools • Open-Source • Customized Jigs • Testability Built into the Product • Lack of Tools
  12. 12. March 2014 © 2014 Congruent Compliance LLC 12 Accuracy and Rigor • “If it is not documented, it didn’t happen” • Attention to Detail is a Tell • Seriousness, Under Control, Honest, Professional • A Proactive, Self-Healing Culture ASPECT 3 …
  13. 13. March 2014 © 2014 Congruent Compliance LLC 13 ASPECT 4 … Traceability and Authorization • Under Control • Accountability
  14. 14. March 2014 © 2014 Congruent Compliance LLC 14 ASPECT 5 … Evidence and Record Keeping • Reasonably recreate “the project” [Design History File] • Retained for expected life of the device (after last sale), plus two years
  15. 15. March 2014 © 2014 Congruent Compliance LLC 15 ASPECT 6 … The Halting Problem • How and on what basis does someone decide they have enough information to stop testing?
  16. 16. March 2014 © 2014 Congruent Compliance LLC 16 ASPECT 7 … Process Standardization and Predictability • Process Police • Measurement Obsession • Mechanization • Reductionism versus Holistic System Thinking
  17. 17. March 2014 © 2014 Congruent Compliance LLC 17 ASPECT 8 … Human Variability and Adaptability • Ordinary, flawed people • Their water also boils at 100 oC • Dependent on their skill, experience and judgment
  18. 18. March 2014 © 2014 Congruent Compliance LLC 18 ASPECT 9 … Technology Choices • Old but well understood technology; or • Just invented technology
  19. 19. March 2014 © 2014 Congruent Compliance LLC 19 ASPECT 10 … Morality and Ethics • Harms Innocent and Vulnerable • Destroys the Business • Becomes Public • Not obvious: Online Games • “Can you handle the Truth?”
  20. 20. March 2014 © 2014 Congruent Compliance LLC 20 ASPECT 11 Business Pressures • Competitive Markets • Reimbursement Codes • Regulated Marketing • Regulatory Uncertainty
  21. 21. March 2014 © 2014 Congruent Compliance LLC 21 THE BIG TAKE AWAY … USE COGNITIVE DISSONANCE When working on Unregulated SW … • I constantly ask myself: “Would we be doing this for regulated SW?” • Reconsider the purpose and form of activities where you answer “NO!”
  22. 22. March 2014 © 2014 Congruent Compliance LLC 22 … THE BIG TAKE AWAY USE COGNITIVE DISSONANCE When working on Regulated SW … • I constantly ask myself: “What basic unregulated industry practices are we are not doing?” • Reconsider adopting those practices • How will you justify not doing them?
  23. 23. March 2014 © 2014 Congruent Compliance LLC 23 Regulated Software Testing • What/Why/How Software is Regulated • 11 Surprising Aspects of Regulated SW • Same and Different from Unregulated • Test Design and Execution • Tools and Automation • Accuracy and Rigor SUMMARY … A
  24. 24. March 2014 © 2014 Congruent Compliance LLC 24 Regulated Software Testing • 11 Surprising Aspects of Regulated SW • Traceability and Authorization • Evidence and Record Keeping • The Halting Problem • Process Standardization and Predictability SUMMARY … B
  25. 25. March 2014 © 2014 Congruent Compliance LLC 25 Regulated Software Testing • 11 Surprising Aspects of Regulated SW • Human Variability and Adaptability • Technology Choices • Morality and Ethics • Business Pressures • Cognitive Dissonance SUMMARY … C
  26. 26. March 2014 © 2014 Congruent Compliance LLC 26 QUESTIONS AND STORIES
  27. 27. March 2014 © 2014 Congruent Compliance LLC 27 IMAGE CREDITS http://www.morguefile.com/archive/#/?q=target&sort=pop&photo_lib=morgueFile http://www.morguefile.com/archive/#/?q=old%20technology&sort=pop&photo_lib=morgueFile http://www.morguefile.com/archive/#/?q=key&sort=pop&photo_lib=morgueFile http://www.flickr.com/photos/bexross/2636921208/in/photostream/ http://en.wikipedia.org/wiki/File:HAL9000.svg http://upload.wikimedia.org/wikipedia/commons/a/af/All_Gizah_Pyramids.jpg http://upload.wikimedia.org/wikipedia/commons/9/96/Waymarker_at_Southern_Upland_Way.JPG http://en.wikipedia.org/wiki/File:Painted_blaze.JPG http://www.morguefile.com/archive/#/?q=rubber%20duck&sort=pop&photo_lib=morgueFile http://www.flickr.com/photos/minnesotahistoricalsociety/5494632378/sizes/o/in/photostream/ http://www.ebay.com/itm/Star-Trek-Original-Series-Science-Tricorder-Replica-/190807969198?_trksid=p2054897.l4276 http://upload.wikimedia.org/wikipedia/en/7/72/Alicesadventuresinwonderland1898.jpg http://upload.wikimedia.org/wikipedia/commons/7/78/Paris_2010_-_Le_Penseur.jpg http://en.wikipedia.org/wiki/File:Keyboard_typing.png http://upload.wikimedia.org/wikipedia/commons/b/bc/Library_of_Ashurbanipal_The_Flood_Tablet.jpg http://en.wikipedia.org/wiki/File:Storage_containers_in_Svalbard_Global_Seed_Vault_01.jpg http://upload.wikimedia.org/wikipedia/commons/1/19/Sevens_scrum.jpg http://en.wikipedia.org/wiki/File:July_4_crowd_at_Vienna_Metro_station.jpg http://blogs.msdn.com/b/geektester/archive/2010/12/30/life-saver-or-life-taker-therac-25-impact-of-poor-testing-testing-tragedies-1- learning-from-past.aspx http://en.wikipedia.org/wiki/File:Jack-in-the-box.jpg http://www.wrestworkshop.com/
  28. 28. Thank you for attending this session. Please fill out the evaluation form. Griffin Jones Griffin.Jones@CongruentCompliance.com March 2014 © 2014 Congruent Compliance LLC 28

×