Join us for the second installment of our webinar series, during which we explore the interesting and controversial aspects of quality and test solutions used in engineering for medical devices. In this session, we'll weigh the pros, cons, motivations and alternatives for the canonical forms of software tests. We'll also differentiate Medical Device Verification from other forms of testing to ensure you don't pay twice for the same result. And, we'll discuss how the concept of "reliability" in medical devices has evolved for software, and how "durability" might have more value. If you’re developing medical devices and are trying to improve the value and efficacy of your quality budget, this session is a can't-miss!

1. Sub-Header Text
Welcome
1

2. About ICS
Established in 1987, Integrated
Computer Solutions, Inc. (ICS)
delivers innovative software
solutions with a full suite of
services to accelerate development
of successful next-gen products.
ICS is headquartered outside
Boston in Waltham, Mass. with
offices in California, Canada and
Europe. Currently 160+ people.

3. www.ics.com
Delivering a Full Suite of
Medtech Services
● IEC 62366-UX/UI Design
● Custom Frontend & Backend Software Development
● Development with IEC 62304-Compliant Platform
● Low-code Tools that Convert UX Prototype to
Product
● Medical Device Cybersecurity
● AWS and Azure Cloud Services and Analytics
● ISO 14971-Compliant Hazard Analysis
● Software Verification Testing
● Digital Twins for Training and Simulation
● Complimentary Software Technology Assessments

4. Quality and Test in Medical Devices
1. Part 1 - Recap
2. Part 2 - Test Methods, Pros, Cons
3. The complete picture Quality and Test
Page #

5. Part 1 Recap
1. Usability to systematically reduce Use Errors through product design - impacts quality
2. Architecture for quality - maintainable, testable, extensible, reliable, available
a. Tiered layers - visibility / access
b. Error injection - difficult to test sub-systems
c. Dependency inversion - simulators and visibility
3. Code analyzers
a. A quality tool, with tradeoffs
b. Necessary but not sufficient (for complex projects)
Usability and Architecture for Quality
5

6. Part 2
➢ Test Canon
● Automated GUI testing
● System testing
● Integration testing
● Unit tests
➢ High-Value testing
○ Key goals
○ Example
➢ Verification vs. test canon
➢ Software “Reliability"
Pros and Cons of Testing
6

7. Automated UI
Testing
Page #

8. Using a 3rd party application to automate mouse clicks on a GUI (Graphical User Interface)
Test tools manage scripts and collections and support automation
Good for:
- Stable User Interfaces (not undergoing changes)
- Enduring value - runs ‘for free’
- Functional/system/black-box testing
Automated Graphical
User Interface Testing
8
Lambda Test
Selenium
AutoHotKey
Squish
Test Complete
Cypress

9. Challenges
- Brittle - Superficial changes to UI can cause tests to fail
- Test strategy - create modular/reusable tests that can be daisy-chained
- No variability - tests the same thing over and over
- Mostly tells you something changed
- Changes system - additional process running on system with threading considerations
- i.e. a graphic bug example (triangles)
Automated User Interface Testing
9
Lambda Test
Selenium
AutoHotKey
Squish

10. System Testing
Page #

11. UI vs. System testing → same?
● Depends on architecture and functionality of the UI
● Are all system behaviors available from the UI?
System Testing
● May require additional test software/framework (i.e. LIS, Cloud services)
● Focuses on positive and negative test cases
● Can be harder to automate
Both
Are tested vs. product requirements
System Testing
11

12. Integration Testing
Page #

13. Why integration testing is important
- Behavior obscured by system
complexity
- FDA language indicating that functional
testing (Black box), is not sufficient as it
does not exercise hidden dependencies
- Rigorously test behavior for each block
and each interface
- Essential for large or complex products
Integration Testing
13
Pro
- Essential, irreplaceable form of test for
a complex system
Con
- Labor intensive
- Time-consuming

14. Unit Tests
Page #

15. Unit Tests
Background
- The smallest unit of test
- Software functions that call product functions
- They usually are numerous, multiple test functions for
a single product function. Executed by a framework in
an automated build/test sequence.
- Necessary, but far from adequate
As valuable as we thought?
15

16. Unit Tests
Pros
- Finds bugs early, at check-in
- Easily automated - Part of Continuous Integration
- Automated coverage metrics
- Enduring value
- Mature tool set (Google Test, xUnit, Mocha)
- Gold standard
Cons
- Increases initial development effort
- Finds few bugs
- Static
- Obvious/trivial
- Low value/effort ratio
- Small code changes can take much longer
- False confidence in code quality
Pros / Cons?
16
Commonly seen:
- Low value unit tests that rarely find
a bug
- Unit test code > product source
code - making changes difficult
- High effort to get the last 10%
coverage - with little apparent
return
- But,... still essential

17. Unit Tests
Keep it light
Don’t try for 100% everywhere
Differentiate highly critical vs. non-critical
Pin down the corners vs. exhaustive testing
Don’t:
Add high levels of test if product in flux
Add trivial testing to improve code coverage metrics
Best applied to
Stable products
When losing experienced staff
Sophisticated/complex designs
Automation: Continuous Integration, automated emails
Best practices
Page #
Best practice
Non-trivial tests written for stable
products with low staffing levels in an
automated environment
Worst practice
Simple tests written for 100% code
coverage on products undergoing change
TL;DR

18. High Value Testing
Page #

19. ➢ Analyze software sub-components for criticality
○ Comes from IEC 62304: risk based
analysis for each software
sub-component
○ Adjust your process according to risk
➢ Allows you to discriminate between critical
sub-systems and non-critical sub-systems
➢ Overall safety class = MAX(each functional
aspect)
High Value Testing
19
C B C C B C C A B A C A

20. Key goals
1) Balance test resources around criticality of
component to Intended Use, Safety, Efficacy
2) Minimize development overhead
3) Minimize regulatory burden
Example
- “Offline Data Access” omits unit, integration,
system testing
Pro
- Less budget overall
- More budget on critical areas
Con
- Expertise: More complicated to plan and apply
- Hard to keep track of →Test Plan
High Value Testing
20
A
C B C C B C C A B C A

21. ➢ Test vs. Verification
➢ Reliability
Page #

22. Quality
team
V&V
team
Engineering
Quality Assurance
- Confirm process
- Checking that SDP steps were followed
- Issuing/processing CAPAs
Engineering Test Engineer
- Test automation (scripts, CI integration)
- Ad-hoc testing
- Smoke testing
- Build management
Verification Test Engineer
- Formal test protocol writing
- Formal test protocol execution
- Test coverage
- Requirement focus
Validation Test
- Formal test protocol writing
- Formal test protocol execution
- Clinical focus
- Statistical test treatment
HF/UE Test
Engineer
- Summative test
HF/UE Test
Engineer
- Formative test
Engineering
Vs
Quality
Developer
- Unit tests
- Ad-hoc testing
Quality driven
Plan → Review → Report
Plan
Accounts for engineering tests
Analyze/Review
Determines outcome
Report
Formally documents
Engineering driven
- Development process
- Includes ad-hoc
Engineering Test vs. Verification & Validation

23. ● Software is logical not physical so traditional concepts of failure
don’t apply
○ Mechanical or electrical systems have Bathtub failure curves
● Software Reliability is a nuanced concept different from Software
Quality
○ Reliability is Software Quality over time
○ i.e. How long has the software been meeting it’s requirements
● Reliability
○ Considers Total Inputs vs Total Outputs
○ Tests a sample of Total Inputs
○ Probability = Number failing cases / Total cases
○ Testing types: Feature/Load/Regression test
○
● Takeaway:
○ Reliability is a sample based, predictive metric different from
QA/QC/Test
○ A complete, quantitative, system for assessing software
Software Reliability
23

24. Tying it together:
Part 1 & 2
Quality vs. Test
Page #

25. Quality → Test
What’s the relationship between quality and test?
Quality as an outcome, Test as a method?
Can you test quality into a product?
Usability
Use Errors
Prevent unintended behaviors
Architecture
Separation
Clear responsibilities
Access / Visibility
Prevent unintended functionality
Test canon (unit, integration, system, UI)
Quality/Test relationship
25
Quality Assurance
Quality Control
Testing
Prevent Defects
Correct Defects
Detect Defects

26. Thank You
Page #