The document discusses Maquet Critical Care AB's introduction and use of session-based testing (SBTM) to better manage exploratory testing in their FDA-regulated medical device development environment. They created an in-house SBT tool to facilitate SBTM and provide metrics on testing activities. Using SBTM and the tool increased visibility into exploratory testing and engaged more employees in the process. Future goals include enhancing the tool and training more employees on SBTM.

1. Exploratory Session-Based Testing … with a Twist Session-Based Testing in an FDA Regulated MED-Tech Environment Presented by Alexander Andelkovic Copyright Maquet Critical Care AB 2009. All Rights Reserved. 1/22

2. Introduction Background Challenges Success factors Pitfalls Tool Summary Questions Contents Copyright Maquet Critical Care AB 2009. All Rights Reserved. 2/22

3. Research, development and manufacturing of MED-Tech equipment World Leader Servo-i ventilator Breath support intensive care Flow-i anesthesia equipment Situated in Solna, Sweden Total ~450 Employees R&D ~130 persons Test group ~25 persons Introduction – Maquet Critical Care AB Servo-i ventilator equipment launched 2001 (> 100 000 units sold) Flow-i anesthesia equipment under development Copyright Maquet Critical Care AB 2009. All Rights Reserved. 3/22

4. U.S. is Maquet’s largest market U.S. FDA regulations need to be followed FDA requires documentation and traceability of performed verification tests Introduction – Regulatory requirements Copyright Maquet Critical Care AB 2009. All Rights Reserved. 4/22

5. Main test method Software / Hardware tested Requirements needed in FDA regulated environment Lots of documentation required Supports traceability Inefficient for finding bugs Background – Requirement based testing at Maquet Software tested Hardware tested Copyright Maquet Critical Care AB 2009. All Rights Reserved. 5/22

6. Complimentary test method ~60% of all bugs are found with exploratory testing (ET) 1-3 testers of project resources with 100% ET focus in larger projects 10 - 20 persons with various competencies doing scheduled ET + Cover test areas outside requirement testing + Cost effective (more bugs found / test time used) + Good test method for clinical and other expert competencies + Reduced animal testing / user acceptance testing - Hard to know what has been tested - Poor knowledge transfer - Difficult to manage Background – Exploratory testing at Maquet Copyright Maquet Critical Care AB 2009. All Rights Reserved. 6/22

7. Problem: Method needed to manage and measure ET Poor Knowledge sharing Solution: Test method: ”Session-Based Test Management (SBTM)” provides management control and metrics reporting for exploratory testing Tool: User friendly SBTM tool needed for broader participation of exploratory testing and knowledge capture in ET session reports Background – How to improve ET? Exploratory test session board Copyright Maquet Critical Care AB 2009. All Rights Reserved. 7/22

8. Elements of SBTM Charter: goal or agenda for a test session Session: time spent testing (45 – 120 minutes) Session report: records of the test session Debrief: short talk about the session between test manager and tester Metrics: number of testers, number of sessions, testing hours, number of defects found, defect rate: defects / hour Background – Session-Based Test Management Session-based testing was developed in 2000 by Jonathan and James Bach. More info at h ttp://www.satisfice.com/sbtm Copyright Maquet Critical Care AB 2009. All Rights Reserved. 8/22

9. Sell in the concept of SBTM Increased control and visibility for management of exploratory testing External expertise (James Bach) emphasize use of SBTM Implementation Test cost visible on weekly basis through SBTM statistics report Lessons learned (External experiences) Simple inexpensive tool Freedom vs control Challenges James Bach - test expert consultant Copyright Maquet Critical Care AB 2009. All Rights Reserved. 9/22

10. Buy or create own SBT tool? Create own tool, start small and build in more features Who should have ownership of SBT tool in the test group? Dedicated tester(s) When should SBTM be used in the test process? - System testing (functional / non-functional) - System integration testing Success factors – Planning Copyright Maquet Critical Care AB 2009. All Rights Reserved. 10/22

11. Announce at large meeting for awareness of new way of working Teach how to use SBT tool and SBTM concept Dedicate SBTM workstations for resources outside test group Communicate SBTM statistics at project status meeting Start small and tune process Bug hunt competition to introduce SBT tool for larger group Success factors – SBTM Launch Reserved test area for exploratory testing Copyright Maquet Critical Care AB 2009. All Rights Reserved. 11/22

12. What should be the frequency of session follow-ups? Minimum once a week How many resources are ideal for SBTM? 1-2 persons responsible, 10-20 persons testing How much should be reported after a session? 5-10 minutes maximum time spent on reporting Pitfalls Copyright Maquet Critical Care AB 2009. All Rights Reserved. 12/22

13. In-house tool named ‘SBT’ (Session-Based Testing) Web-based for increased availability User friendly interface Reported data stored and backed up on dedicated database server Developed in Perl and uses SQL server Maintained by part-time working student Tool - Introduction All charters done? Report / Debrief Report / Debrief Test session Test session Charters define / assign Test resources Weekly Statistics report Test mission no yes SBT tool workflow Copyright Maquet Critical Care AB 2009. All Rights Reserved. 13/22 … …

14. Tool – Main menu Features: - Add session - Tester / Project administration - Archive - Statistics - Edit / Export / Print / Reassigning / Copy / Delete Copyright Maquet Critical Care AB 2009. All Rights Reserved. 14/22

15. Tool – Add session Purpose: Add sessions to the database Session charter Project Test notes Tester name Duration Example (template) Customized fields such as Machine serial number , Software version and Defects Copyright Maquet Critical Care AB 2009. All Rights Reserved. 15/22

16. Tool – Administration Purpose: Administrate tester and project names in SBT tool Tester name Project name Contact information Add / Remove / Edit Copyright Maquet Critical Care AB 2009. All Rights Reserved. 16/22

17. Tool – Archive Purpose: Archive old session items and projects Avoids cluttered main menu Dearchive Regular backups Copyright Maquet Critical Care AB 2009. All Rights Reserved. 17/22

18. Tool – Statistics Purpose: Show weekly statistics reports Export to word / excel Testers breakdown Copyright Maquet Critical Care AB 2009. All Rights Reserved. 18/22

19. Managed exploratory testing achieved Increased visibility of exploratory testing through weekly statistics report More people interested and involved in exploratory testing Other groups within company have started to use SBTM for other test activities Increased / better communication between colleagues Common ET process Summary – SBTM introduction results Copyright Maquet Critical Care AB 2009. All Rights Reserved. 19/22

20. Get more people to use SBTM More features in SBT tool (Electronic signatures, more statistics) SBTM introduction / training program for more employees Improve debrief and test reporting Summary – Future improvements Copyright Maquet Critical Care AB 2009. All Rights Reserved. 20/22

21. E-mail: [email_address] Maquet: http://www.maquet.com Questions? Copyright Maquet Critical Care AB 2009. All Rights Reserved. 21/22

22. Thank you! Copyright Maquet Critical Care AB 2009. All Rights Reserved. 22/22