Test automation is difficult to get right. Working under FDA regulation presents its own challenges. Combining the two is a scary proposition because the FDA requires—and will scrutinize—the validation of any test automation used. Despite this, working in a regulated environment only magnifies the value of test automation. Aware that automation is a driver of quality and consistency, the FDA welcomes automated tests as part of an audit submission. The key to success is demonstrating quality in a way that the FDA recognizes. Chris Crapo and David Nelson lay out the road map to validation of a test automation system and highlight the critical thinking, planning, and types of maintenance that form the core of any successful validation strategy. By understanding the focal points of validation, you can set your project up for regulatory success while maintaining a lean, focused execution that drives results, not paperwork.

1. !!
T3
Test!Automation!
10/16/2014!9:45:00!AM!
!
A Path through the Jungle:
Validating a Test Automation
System for the FDA
!
Presented by:
Chris Crapo and David Nelson
Boston Scientific Neuromodulation
!
!
!
Brought(to(you(by:(
(
(
(
340!Corporate!Way,!Suite!300,!Orange!Park,!FL!32073!
888G268G8770!H!904G278G0524!H!sqeinfo@sqe.com!H!www.sqe.com

2. Chris Crapo
Boston Scientific Neuromodulation
A senior software engineer with Boston Scientific, Chris Crapo is currently the
team lead for the test automation team. Chris has guided the in-house
automation system from prototype to a full-featured system that supports unit,
integration, and system testing for a number of Class III medical devices. His
experience includes developing, testing, and validating software and tools in a
regulated environment.
David Nelson
Boston Scientific Neuromodulation
David Nelson is a principle software engineer with Boston Scientific and has
been with them since 1998. David has worked on projects containing medical
device software including implantable cardiac pacemakers and defibrillators,
implantable stimulators, and external computers to program these devices. He
has been involved in both the software development and verification/validation of
these devices. David is also on the AAMI faculty and is an instructor for their
validation class.

3. A"Path"through"the"Jungle:"
Valida3ng"a"Test"Automa3on"System"
for"the"FDA"
Chris&Crapo&–&Boston&Scien1fic&Neuromodula1on&
David&Nelson&–&Boston&Scien1fic&Neuromodula1on&
Course"Objec3ves"
• What"is"SoCware"Valida3on?"
• Intended"Use"/"Requirements"/"Objec3ve"Evidence"
• Development"Process"/"Configura3on"Management"
• Defect"Tracking"/"Change"Control"
• Risk"Based"Tes3ng"
• Maintenance"Planning"/"Ac3vi3es"
• Sample"Valida3on"Report"

4. Our"Test"Automa3on"System"
Remote&
Control Stimulator
Clinician&Programmer
Test&Station&Host&Computer
"FDA"Regula3on"
• FDA"Regula3on:"
“When"computers"or"automated"data"processing"
systems"are"used"as"part"of"produc3on"or"the"quality"
system,"the"manufacturer"shall"validate(computer(
so0ware(for(its(intended(use(according"to"an"
established"protocol"…”"
21&CFR&820.70&(i)&
"

5. What"is"SoCware"Valida3on?"
• “Confirma3on"by"examina3on"and"provision"of"
objec&ve(evidence"that"soCware"specifica3ons"
conform"to"user(needs(and(intended(uses,"and"
that"the"par3cular"requirements"implemented"
through"soCware"can"be"consistently"fulfilled.”"
General&Principles&of&SoGware&Valida1on,&Final&Guidance,&Sec1on&3.1.2,&
January&2002&
• Building"Confidence"
What"is"SoCware"Valida3on?"
!
!
!
!
!
!
!
!
!
!
Software Validation Activities Zone
Planning A
Requirements A
Traceability A,B
Change management A
User site testing D
Defect resolution A,B
Risk management A
Intended use/user needs ID A
Evaluations B
Design reviews B
Software testing C
Regression testing C,D
Reproduced with permission, from Medical Device Software Verification, Validation, and Compliance by David A. Vogel,
Ph.D. © 2011 Artech House, Inc., Norwood, MA

6. Course"Objec3ves"
• What"is"SoCware"Valida3on?"
• Intended(Use(/(Requirements(/(Objec&ve(
Evidence(
• Development"Process"/"Configura3on"Management"
• Defect"Tracking"/"Change"Control"
• Risk"Based"Tes3ng"
• Maintenance"Planning"/"Ac3vi3es"
• Sample"Valida3on"Report"
Intended"Use"
• Defines"the"use"of"the"computer"soCware"and"specifies"what"the"
soCware"is"automa3ng"
– This"should"be"defined"as"narrowly"as"possible"to"cover"all"the"intended"
uses"that"are"an3cipated"
– Not"too"detailed,"should"be"at"a"fairly"highXlevel"
• Defines"the"scope"of"Valida3on"
– A"narrower"intended"use"helps"clarify"what"needs"to"be"validated"and"it"
allows"one"to"focus"on"what"is"important"
• Defines"the"users"and"constraints"of"the"soCware"
– Iden3fying"who"uses"that"soCware"helps"to"understand"where"to"focus"
valida3on"effort"
– Number"of"users,"skill/training"of"users,"language,"loca3on"of"users,"
etc."
– How"many"different"environments"is"the"tool"used"in?"
– When"is"the"tool"used?"
• Provides"the"basis"for"an"ini3al"risk"analysis"

7. Requirements"
• Specify"the"func3onal"need"of"the"soCware"
– NOT"resta3ng"the"Intended"Use"
– Similar"to"a"typical"SoCware"Requirements"
Document"
– API"defini3on"
• Basis"for"verifica3on/valida3on"tes3ng"
– Trace"tests"to"the"requirements"to"ensure"
coverage"
– Specify"risk"of"each"requirement"/"API"call"in"order"
to"op3mize"level"of"tes3ng""
Objec3ve"Evidence"
• “OBJECTIVE(EVIDENCE:(Informa3on"which(can(
be(proven(true,"based"on"facts"obtained"
through"observa3on,"measurement,"test,"or"
other"means.”""
[IEC&60601R1:2005&defini1on&3.72,ISO&14971:2000&
defini1on&2.8, &ISO&9000:2005&defini1on&3.8.1]"
• Signatures"/"dates"
• Electronic"Signatures"(Part"11"Compliance)"

8. Course"Objec3ves"
• What"is"SoCware"Valida3on?"
• Intended"Use"/"Requirements"/"Objec3ve"Evidence"
• Development(Process(/(Configura&on(
Management(
• Defect"Tracking"/"Change"Control"
• Risk"Based"Tes3ng"
• Maintenance"Planning"/"Ac3vi3es"
• Sample"Valida3on"Report"
Configura3on"Management""
• Primary"Focuses"of"Configura3on"
Management"
– Traceability"
– Good"Development"Prac3ce"
– Source"Analysis"

9. Traceability"
– Unambiguous(Iden&fica&on(
• Your"source"control"ID"is"usually"your"ul3mate"iden3fier"for"a"state"
of"source"code."If"it"doesn’t"have"an"ID,"then"it"doesn’t"exist."
– Universal"Iden3fica3on"
• Source"control"ID"is"always"used"when"referencing"the"automa3on"
system"–"for"defect"tracking,"change"control,"release"notes,"and"
especially"when"iden3fying"the"eventual"system"that"is"validated."
– Users"of"the"system"should"be"trained"to"always"pull"
versions"from"source"control"so"that"iden3fier"numbers"
are"good."
– AdXhoc"changes"that"need"to"be"referenced"should"be"
placed"in"a"known"loca3on"of"source"control"if"they"need"
to"be"referenced."
Good"Source"Control"Workflow"
– Allows"you"to"group"and"organize"changes"so"that"
they"are"introduced"in"a"logical/meaningful"group."""
• A"bug"fix"and"corresponding"changes"to"regression"tests"can"
be"found"in"the"same"checkXin"
– A"staged"workflow"allows"for"much"easier"
iden3fica3on"of"when"defects"are"introduced."
– Makes"for"much"more"efficient"overall"planning."
• Feature"addi3ons"and"closeXouts"have"defined"endpoints"
• Ul3mately"allows"you"to"use"source"control"IDs"as"
clean"objec3ve"evidence"

10. Source"Code"Analysis"
– Along"with"good"tools"for"performing"Diffs,"and"for"
robust"full"text"searches,"your"source"control"
system"allows"you"to"disclose"analyses"(plural)."
(e.g."From"version"A"to"version"B"there"were"no"
changes"to"the"source"code"in"module"X)"
– This"capability"forms"the"basis"for"disclosing"ripple"
effect"analyses"and/or"inchXup"valida3ons,"or"for"
releasing"a"“noXchange”"review"report."
– This"is"also"a"legi3mate"way"of"giving"suppor3ng"
evidence"for"disposi3oning"defects"in"the"system."
More"than"Just"Source"
– Anything"that"needs"to"be"referenced"from"your"
valida3on"doc"should"be"in"source"control"somewhere"
–"valida3ons"scripts,"peer"review"reports,"design"
documenta3on,"and"even"test"results."
– Anything"that"is"needed"to"run"your"automa3on"
system"should"be"in"source"control"as"well"–"binaries,"
user"documenta3on,"and"configura3on"files."
– It"may"even"be"worth"checking"in"the"reports"from"
your"change"control"or"defect"management"system"so"
that"they"can"be"referenced."
– Some"source"control"systems"do"not"like"binaries,"so"
plan"ahead"for"poten3al"complica3ons."

11. Warning:"Changing"History"
– Do"not"rely"on"features"that"can"be"changed"
(mutability)"for"traceability"
– A"source"control"iden3fier"should"be"a"hard"
guarantee"of"contents"
– AdminXonly"privileges"that"alter"history"can"be"
permissible,"but"expect""to"answer"how"its"use"is"
controlled"and"reported"
Course"Objec3ves"
• What"is"SoCware"Valida3on?"
• Intended"Use"/"Requirements"/"Objec3ve"Evidence"
• Development"Process"/"Configura3on"Management"
• Defect(Tracking(/(Change(Control(
• Risk"Based"Tes3ng"
• Maintenance"Planning"/"Ac3vi3es"
• Sample"Valida3on"Report"

12. Defect"Management"
– All"about"follow"through!"
• Not"every"defect"needs"to"be"fixed,"but"every"defect"
needs"to"be"understood"
• Fix"/"No"Fix"decisions"should"be"backed"up"by"reasoning"
about"risk"and"intended"use."
"
– Tracing"to"specific"versions"of"the"automa3on"
system"when"bugs"were"fixed"is"a"big"confidence"
booster."
• A"tester’s"signXoff"verifying"a"fix"is"objec3ve"evidence"
Change"Control"
– A"large,"formal,"change"control"process"is"not"
required"
– Knowing"what"the"list"of"features"in"the"system"
will"be"important"for"documen3ng"intended"use"
– Some"level"of"control"as"to"what"goes"into"the"
system"will"save"you"from"a"lot"of"lateXproject,"risk"
mi3ga3on"tes3ng."

13. Course"Objec3ves"
• What"is"SoCware"Valida3on?"
• Intended"Use"/"Requirements"/"Objec3ve"Evidence"
• Development"Process"/"Configura3on"Management"
• Defect"Tracking"/"Change"Control"
• Risk(Based(Tes&ng(
• Maintenance"Planning"/"Ac3vi3es"
• Sample"Valida3on"Report"
Valida3on"is"More"than"Tes3ng"
– Valida3on"is"reasoning"of"sufficient"quality"based"on"
one"or"more"of:"
• intended"use"
• exper3se"of"users"
• development"process"
• review"(including"use)"
• tes3ng"
• development"process"of"scripts"
– Not"all"elements"of"a"system"are"as"high"risk"as"the"
overall"risk"
• With"jus3fica3on,"different"combina3ons"of"the"above"are"
sufficient"for"different"intended"uses"in"the"system"

14. Different"Strategies"for"Different"Risks"
– ReadXonly,"or"informa3onal,"parts"of"the"system"
may"not"need"tes3ng"
– Parts"of"the"automa3on"system"that"make"Quality"
decisions"always"need"thorough"tes3ng."
– Always"use"good"judgment"in"tes3ng"–"super"rare/
difficult"scenarios"may"be"ruled"out"by"user"
training"or"review"of"automa3on"scripts."
• Good"judgment"becomes"objec3ve"evidence"when"
included"in"a"report"or"documenta3on"
Clarifying"Intended"Use"
– User"documenta3on,"including"func3on"headers,"
is"an"appropriate"place"for"these"intended"uses."
• There"must"be"objec&ve"evidence"that"people"do"
regard"these"intended"uses."
• Your"average"user"doesn’t"look"at"the"valida3on"docs."
Thus"just"sta3ng"it"in"the"doc"is"not"a"good"plan."
• Intended"use"+"training"addresses"the"ques3on"“what"if"
someone"does"something"inappropriate"with"it?”"

15. Third"Party"Tools"
– Must"be"disclosed."
• The"intended"uses"of"the"tools"are"essen3al"for"avoiding"
having"to"perform"full"valida3on"for"every"tool.""
– Must"be"under"configura3on"management"
• Many"vendors"do"not"guarantee"that"specific"versions"will"
stay"available."
– Must"be"assessed"for"risk"the"same"way"the"rest"of"the"
system"is."
• Intended"use"
• known"level"of"tes3ng"
• scope"of"adop3on"
• known"bug"lists"
Valida3ng"an"API"
• Divide"and"Conquer"
– Break"func3ons"into"logical"groups"(classes).""
– For"each"func3on"evaluate:"
• Requirements"
• intended"use"
• risk"analysis"
– Trace"each"requirement"to"a"sufficient"level"of"
mi3ga3on"for"its"risk"
• Tests"
• Code"reviews"
• Suppor3ng"tools"(and"their"valida3on)"
• Development"cycle"

16. Course"Objec3ves"
• What"is"SoCware"Valida3on?"
• Intended"Use"/"Requirements"/"Objec3ve"Evidence"
• Development"Process"/"Configura3on"Management"
• Defect"Tracking"/"Change"Control"
• Risk"Based"Tes3ng"
• Maintenance(Planning(/(Ac&vi&es(
• Sample"Valida3on"Report"
Maintenance"Planning"
– Dealing"with"defects"that"are"found"aCer"
deployment"
– Revalida3ng"the"system"aCer"making"changes"
– Expanding"the"intended"use"

17. Example"Simple"Strategy:"
"Sta3c"Valida3on"
– All"defects"that"are"found"against"validated"
soCware"are"kept"in"a"specified"bug"tracking"
system"and"are"disposi3oned"for"risk."
– Any"bugs"of"sufficient"risk"will"be"fixed"
– Any"changes"to"the"system"will"trigger"a"
revalida3on"and"issuing"another"valida3on"report"
– Any"change"to"the"scope"of"intended"use"will"
trigger"a"revalida3on"and"another"valida3on"
report."
Example"Involved"Strategy:"
"Rolling"Valida3on"with"Updates"
– All"defects"that"are"found"are"tracked"in"a"specified"
bug"tracking"system,"keeping"track"of"major"and"
minor"versions"of"the"soCware."
– All"changes"to"the"soCware"are"issued"as"part"of"a"
Change"Request"and"are"traced"to"a"bug"or"feature"
request."
– All"changes"to"the"system"are"traced"to"a"new"
regression"test"or"an"expansion"of"an"old"one.""
– All"releases"to"deployment"are"tested"(and"results"
documented!)"with"the"regression"suite."
– A"new"report"is"only"issued"aCer"major"changes"to"the"
soCware"or"major"changes"to"the"intended"use."

18. Course"Objec3ves"
• What"is"SoCware"Valida3on?"
• Intended"Use"/"Requirements"/"Objec3ve"Evidence"
• Development"Process"/"Configura3on"Management"
• Defect"Tracking"/"Change"Control"
• Risk"Based"Tes3ng"
• Maintenance"Planning"/"Ac3vi3es"
• Sample(Valida&on(Report(
Conclusion"
• SoCware"valida3on"is"only"providing"objec3ve"
evidence"that"you"are"following"industry"
standard"steps"to"produce"quality"soCware."
• Valida3ng"an"automa3on"system"follows"the"
same"principals"of"good"development"
prac3ce.""
• Just"remember"to"focus"on"appropriate"scaling"
of"intended"uses"of"parts"of"the"API."