Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical Device - EuroSTAR 2010

Thomas KaudersAgile Test Design and Automationof a Life-Critical Medical Device
A usable model for Agile Test Automation
1
© PrettyGoodTesting 2010
Agile Test Design and Automation of a Life-Critical Medical Device

Project Summary-about me, the project and our test team
Project Challenge-why we needed to automate test
Our Approach-how we designed the automation process
Automation Implementation-how did it take place in practice
Lessons Learned-our successes and concerns
Agenda
2

Thomas KaudersM.Sc., Test and test manager since 2002
ISEB/ISTQB Certified
Currently:
Assigned at a large medical company
About
3

More than
11.000 pages
of specifications
About the Project
“The Device”
Not the actual device
depicted! Used forillustration only!
700 Man-years
...as from Guthenberg
...to ”The Internet”
4

The Test Team1 Test Manager1 Test Designer2 Manual Testers3 Test Automators...1 Model-Based Test (MBT) Automator...2 Functional Test Automators
5

Large and complex systemLife critical systemVolatile requirements R&DAgile technical requirementsFormal V&V(Verification & Validation)
U.S. Food and
Drug Administration
The Project Challenge
Test
Development
Requirements
6

Importance of formal test
Huge volume of manual test
Short development sprints
Limited ressources
Strict deadline
Possible solutions:
•More ressources
•Time extension
•Test Automation
The Testing Challenge
7

Considering test automationfrom the very beginning of test design
Cooperating Manual-and Automated Teams
Reusing Test Design across teams
Modularising the test
Gradually adding more ”Pay As You Go”
A New Approach
8

SRS
MMI
Consider Test Automation from the Start
Modular mindset, during analysis andTestCase design!
Aut. TestCases
Man. TestCases
TestDesign
Requirements
ActionWords
Traceability to Reqs?
Based on SRS and MMI?
TestCases OK for Test Automation? 9

To keep it simple we developed a concept of:
PRIMITIVES(aka ActionWords, KeyWords etc.)
Primitives is a known term by key projectmembers
Stickingto thisterm eased our ”selling the idea”
Naming the Principle10

Close cooperation btw. Man. & Aut. teams
Re-using Test Design across teams
11

Use of ”mnemonics” resulted in ”readable” names, e.g.: ST-PRM-FTUS : System Test, Primitive, First Time UseST-PRM-LGBK : System Test Primitive, Log Book
Using “Call to test” to use Primitives
Call to test
12

PTC:
Set Year ()
PTC:
Set Month ()
PTC:
Set Day ()
Primitives (PTC)
Call to test PTC:
Set Month ()
Call to test PTC:
Set Day ()
Call to test PTC:
Set Time ()
Call to test PTC:
Confirm Setup
PTC:
Confirm Cancel Setup
PTC:
Set Time ()
PTC:
Cancel Setup
Cancel Cancel Setup
PTC:
Confirm Setup
Test Cases (STC)
Call to test PTC:
Set Year ()
TC:
Navigate to Setup D&T
Call to test PTC:
Set Month ()
Call to test PTC:
Set Day ()
Call to test PTC:
Cancel Setup
Call to test PTC:
Confirm Cancel Setup
Call to test PTC:
Set Year ()
Call to test PTC:
Set Month ()
Call to test PTC:
Set Day ()
Call to test PTC:
Cancel Setup
Call to test PTC:
Cancel Cancel Setup
Call to test PTC:
Set Year ()
TC:
Call to test PTC:
Set Time ()
Call to test PTC:
Confirm Setup
PTC:
Verify Setup Complete
PTC:
Call to test PTC:
Verify Setup Canceled
PTC:
PTC:
TC:
Think of Lego Bricks
13 © PrettyGoodTesting 2010

Manual test not coping with the volume of testsAutomated tests running on unstable test systemThe ”Death by test script maintenance” issueMilestones not reachable if not: -increasing the extent of the automated regression-increasing the quality of the automated regression-creating better manual regression testing
-developing smarter approach to test scriptingNot possiblewithouteffectivetest automation!
The Test Automation Challenge
14

Navigation
Go from A to B (Start to adjust Time)
Action
Input to system (Set Time)
Verification
Expectedresult(VerifyTime set)
Execution
Practicaltest implementation(Note Test Start Time)
Use parameters to keep numbers of mode
Primitive Classification
15

Traditional Sequential
TestCases
A pseudo test case broken down intoPrimitives,
eachPrimitive is unique to the SUT (Software Under Test)
PRIM 6
PRIM 5
PRIM 4
PRIM 3
PRIM 2
PRIM 1
Products of Module-based Test Design
16

PRIM 7
PRIM 6
PRIM 3
PRIM 2.1
PRIM 1
PRIM 7
PRIM 5.1
PRIM 4
PRIM 3
PRIM 2.2
PRIM 1
PRIM 7
PRIM 5.2
PRIM 4
PRIM 3
PRIM 2.3
PRIM 1
1
2.1
2.2
2.3
3
6
4
5.1
7
Saving above 50% of work for just this example.
5.2
An Advantage
17

During test analysistry to break down eachTest Case in unique and identifiable user action sequences
A uniquesequencemust not existelsewhere
(stick to this…)
The sequences are to bere-usedand combined in anyorderrepresentingcomplete user actions
Why does this work?
18

New test delivered once a week
Tests fixedfrom day to day
Change identified (By Developer or Tester)
Impact assesed
AffectedPrimitive(s) identified
Change(s) implemented
Test Scripts tested
Deliveredand verifiedthattests are running
Dealing with Change in Automation
19

Deliver tests
Verify tests
Test TestScripts
Implement
Change
Identifyaffected
Primitive(s)
Assessimpact
Change
identified
Must be ableto respond quickly
to change
Daily Change Management
20

Tracebility from Requirements -> TC´s involved
Change only the involved Primitives
1:1 manual TC´s -> automated TC’s
* SCRUM = Dynamic Requirements!
When Requirement’s are changing*
21

Easy indication of what has to be changed
Full traceability of what has been changed
1:1 traceability eases quicker validation*
Use of Primitives made the test suite scalable
Use of Primitives made the test suite flexible
* FDA requires full validation of:
Designs, Implementation, Tools, TestCases etc.
What was reached?
22

Initial Test Analysis…still to be carried out* (no Quick-fix)
Change Impact Analysis…still required* (no Silver Bullet)
* Exploratory Testing and Model-basedTesting...is still to be carried out (not a stand-alone thing)!
What was NOT reached?
23

Equals 7-14 days of man. regression tests...nightly! -September 2010, 7 days
-December 2010, 14 days (expectedly)
40% WorkLoad Reduction
Every team member plays an important role-Happy Testing!
And…Milestones are now met!
Outcome
24

Too many Primitives (incorrect partitioning)
Improper mindset during review
-This seems possible! But is it?
Lack of review of Manual and Automated tests
Lack of review of Test Scripts
Perfecting / ”goldplating” test breakdown
Pitfalls
25

Cooperation between Test Designer, Manual Test Team and Automation Test Team reduces effort, by: -Thinking Automation First-Reusing of Test Documentation/Primitives
Primitives (aka ActionWords, KeyWordsetc.): -Promotes flexible & easy maintainable Test Suites
Exploratory Testing and Model Based Testing...still to be carried out (not a stand-alone thing)!
Summary
26

Time for Questions and Comments
Thomas Kauders: (+45) 3163 0205THKA@PrettyGoodTesting.com
PrettyGoodTesting ApSLautruphøj 1-3DK-2750 BallerupDenmark (+45) 3163 0200
www.PrettyGoodTesting.com
27

Test Running
Additional Slide

Step 1: Using Primitives by calling with new param. values
Step 2: In making test, needed Primitives are implemented
Step 3: Putting it all together in a test script
Step 4: Making a test suite for Hudson
Capture Replay
Additional Slide

Continuous Integration
Additional Slide

Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical Device - EuroSTAR 2010

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical Device - EuroSTAR 2010

Similar to Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical Device - EuroSTAR 2010 (20)

More from TEST Huddle

More from TEST Huddle (20)

Recently uploaded

Recently uploaded (20)

Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical Device - EuroSTAR 2010