The document discusses various testing methodologies, focusing on test automation and keyword-driven testing. It highlights the advantages and disadvantages of different approaches including manual testing, capture-replay testing, script-based testing, and model-based testing. Keyword-driven testing is presented as a flexible and maintainable option that abstracts test case design from automation, making it accessible to non-programmers.

1. Test Automation and
Keyword-driven testing
Brian Nielsen,
bnielsen@cs.aau.dk

2. Testing as a process
g p
requirements Planning&
Control
ologi
preparation
specification test plan
tekno
(abstract) abstract/logical
Test test cases
specification
generation
tionst
concrete/
Test executablel
implemen-
tation
test cases
ormat
Test verdict/logs
implementation execution
Info
Completion quality report
CISS

3. Test Automation
requirements Planning&
Control
ologi
preparation
specification test plan
tekno
(abstract) abstract/logical
Test test cases
specification
generation
tionst
concrete/
Test executablel
implemen-
tation
test cases
ormat
Test verdict/logs
implementation execution
Info
Completion quality report
CISS

4. 1 Manual Testing
1.
+ easy & cheap to start
+ flexible testing
- expensive every execution
- no auto regression testing
i i
- ad-hoc coverage
- no coverage measurement
M. Utting ”Practical Model Based Testing”

5. 2 Capture Replay Testing
2. Capture-Replay
+ flexible testing
expensive first execution
+ auto regression testing
- fragile tests break easily
- ad-hoc coverage
- no coverage measurement

6. 3 Script Based Testing
3. Script-Based
+/- test impl. = programming
+ automatic execution
+ auto regression testing
fragile tests break easily?
(depends on abstraction)
- ad-hoc coverage
- no coverage measurement

7. 4 Keyword Driven Testing
4. Keyword-Driven
+ abstract tests
+ automatic execution
+ auto regression testing
- robust tests
- ad-hoc coverage
ad hoc
- no coverage measurement

8. 5 Model-Based
5. Model Based
Testing
g
+ abstract tests
+ automatic execution
+ auto regression testing
+ auto design of tests
+ systematic coverage
+ measure coverage of model
and requirements
- modelling overhead

9. Script-based
Script based Testing
Test inputs and observations (verdict
assignment) programmed in scripts in
g )p g p
dedicated or general-purpose languages.
+ Repeatable (for regression testing)
+ Data driven testing: Fixed scripts, each parameterized
with different data
d e e da a
– Mixes test harness and logical test cases
– Must be updated when specs (or impl) change
– Very lengthy
– Good programmers may write well-structured
well structured
reusable test code but ”it is just test code”

10. Keyword Driven Testing
y g
• Script based automation where test case
design is
d i i separated f
t d from automation
t ti
– Focus on actions user/environment can do on
objects in SUT (at different abstraction levels)
• ”Actions” appear in scripts,
pp p
• ”Action-code” implements the action
• aka “Action-word” testing (or table driven
Action-word (or,
testing), < 1994
• Black-box, subsystem, accept tests
• Automated execution using a “framework”.
framework .

11. Keyword Driven Testing
y g
+ Concise, flexible, maintainable,
+ Read-/writeable by non-programmers
÷ Expressiveness of a scripting language
– C t l structures and complex computations
Control t t d l t ti
– (Branching in test cases, complex data, matching of expected results)

12. Keyword Driven Test
y
Eg. Sequence
Object Keyword Value
LoginDialog Login ”badUserId”, ”GoodPass”
VerifyLogin ”Login failed”
Login failed
LoginDialog Login ”goodUserId”, ”badPass”
VerifyLogin ”Login failed”
LoginDialog Login ”goodUserId”, ”GoodPass”
VerifyLogin ”welcome”
WelcomeDialog
W l Di l changeLogin
h L i ”newUserName”,”newpwd”
” U N ”” d”
confirm ”goodUserId”, ”newpwd”
confirmChange ”Change Failed”

13. KWD Process
Test Expert Automation Expert
http://www.softwaretestinghelp.com/wp-content/qa/uploads/2010/01/keyword-driven-testing.pdf

14. Automation Fixture
Keywod Script
Open(Scripts)
While(!EOF){
Engine/driver Keyword, data=readFile();
Generic Libs Case(keyword){
(fixtures) Login: id t > b it(d t )
L i widget->submit(data);
verifyLogin: …
Application Libs }
(keyword functions) Close()
SUT
}

15. Tools
• SAFS, open source
• EMOS Framework, open source
• FIT/FitNesse,
FIT/FitNesse open source
• Certify, Worksoft
• Unified TestPro, SDT
• TestFrame,
TestFrame LogicaCMG
• TestArchitect, LogiGear
• Tools with built-in keyword support:
– TestQuest Pro
Pro,
– QuickTest Professional

16. Example
http://localhost:8080/FitBookExamples.Chapter4TestingActions.Fig6TestChatServer

17. http://fit.c2.com/wiki.cgi?FitBook
http://my.safaribooksonline.com/0321269349

18. Comparison
p
Criteria Manual Script KWD Model-Based
Cost Development L M M L
Maintenance H? H L
Tools/equip
q p L M M H
Time Time to first test L M M M-H
Re-execution H L L L
Maintenance L H H L
Learn- Skills L H M* H*
ability Maturity L M M-H H
Readability L L M H*
Interest / Ch ll
I t t Challenging
i L M M H
Effective- Bug detection H L L H
ness
Tracability L L M H
Coverage L L M H

19. References
• http://en.wikipedia.org/wiki/Keyword-driven_testing
• http://www.cs.waikato.ac.nz/~marku/mbt/
• http://www.onestoptesting.com/automation-
framework/keyword-driven-testing/
• http://www.softwaretestinghelp.com/wp-
content/qa/uploads/2010/01/keyword-driven-testing.pdf
• http://www.musala.com/press/PR_2008_03_17/pr/Keyw
ord-driven%20Testing.pdf
ddi %20T ti df
• Mark Fewster and Dorothy Graham, Software Test
Automation,
Automation 1999 (Chapter 22)
• M. Utting Practical Model-based testing