Free Test 2009 conference slides

1. Exploring no man's land
with keyword driven testing
Martin Gijsen, M.Sc.
Martin@DeAnalist.nl
Free Test Conference
March 23, 2009

2. Presentation overview
● Automated testing
● When (not) to use Record & Playback
● Automated testing as no man's land
● Keyword driven testing
● The process
● Where to get the automator

3. Automated (functional) testing
● A computer performs the test
● Fast and consistent
● Potential benefits:
● Time (to market) / cost / quality
● More effective use of (test) resources
● More challenged (= happier) testers
● Accurate and up2date test status
● Continuity
● Business goals are drivers

4. The Record & Playback approach
● Record test actions, replay as needed
● Checks are added to scripts
● Fundamental problem is maintenance
● Software evolves until it is decommissioned
● Maintainability requires structure and abstractions
● Recorded test cases are long, unstructured and low
level programs
● Maintenance sensitivity is high
● Maintainability is low

5. So why is it still used?
● Automated testing is often the job of a tester
● Most testers are:
● willing to record, but
● unwilling or unable to create good automation
solutions
● Vendor sales pitch is not always clear on this

6. Result and conclusion
It is often initially unclear what
effective automated testing requires
2 out of every 3 TA projects
fails sooner or later
– Brian LaSuer
Use R&P only in case of
little manual maintenance

7. What about the developers?
● Test automation is software engineering
● Developers have the skills to create a good
automation solution
● Many developers are unwilling or unable to
create good test sets
● … or automate test cases that testers wrote

8. Welcome to no man's land
● Most testers cannot or will not automate well
● Most developers cannot or will not test well
● Automated testing is like a no man's land in
between testing and development
● Is automated testing doomed?

9. Exploring no man's land
Testing
Automated testing
Development

10. Exploring no man's land
Testing
Automated testing
= analysis (what to test)
+ automation (how to test)
Development

11. What is in between?
Testing
Test analysis
?
Test automation
Development

12. No more no man's land?
Testing
Test analysis
Test automation
Development

13. Keywords – The DS(T)L
● Domain Specific (Test) Language
● Used to express test cases
● Defined by (or with) the testers
● Specific to the application under test
● No programming
● High level

14. Bad example
select (“a box”);
type (“some text”);
click (“OK”);
...

15. Better example
create account;1234567890;John;Doe
create account;2345678901;Jane;Doe
deposit;1234567890;12345,67
transfer;1234,56;1234567890;2345678901
check balance;1234567890;11111,11
check balance;2345678901;1234,56

16. Good example
Account First name Last name
create account 1234567890 John Doe
create account 2345678901 Jane Doe
Account Amount
deposit John 12345,67
Amount From To
transfer 1234,56 John Jane
Account Amount
check balance John 11111,11
check balance Jane 1234,56

17. Keywords / Instructions
● Often have arguments
● No (irrelevant) test execution details
● No (irrelevant) interfacing details
● No tooling details
● Only the essence of a test case remains
● Easy to read, write and maintain test cases
● Low maintenance sensitivity

18. Sample instruction documentation
Name and aliases 'begin test case' and 'begin testcase'
Description Indicates the beginning of a (new) test case.
Parameters Two parameters, both optional
1 The identification of the test case, printable characters, preferably
unique within the test. Optional.
2 The test case description, printable characters. Optional.
Validity Anywhere except inside a procedure definition.
Pre-condition None.
Post-condition If the previous input line was part of a test case, that test case is
closed. A new test case is opened and assigned the next sequence
number for the report(s), starting at one.
Error condition If in a procedure definition.

19. Automating (basic)
Test
Test
Automation report
Application under test

20. Automating (more detailed)
Engine Test
Test
report
Instructions
Interfacing
Application under test

21. The ETA Framework 'engine'
● Can read from and report to a spreadsheet
● Supports test cases, variables and procedures
● Implement keywords in Java
● Use any Java library (Abbot, WebDriver, MQ,
generated web service client, ...)
● Documented, actively used and developed
● Freeware

22. The process (waterfall)
Test cases
Keyword defs
Automate Test
Requirements
Product development

23. The process (agile)
Test cases
Keyword defs
Requirements Automate Test
Product development
Test cases
Keyword defs
Requirements Automate Test
Product development
Test cases
Keyword defs
Requirements Automate Test
Product development

24. Where to get the automator
● Main criteria (for continuity):
● Skills
● Availability
● Cost
● Educate someone from test team
● Borrow from development team
● Hire or reserve developer for test team
● Hire external consultant

25. Summary
● Automated testing can help achieve your goals
● Avoid Record & Playback
● Good keywords help:
● The tester can focus on the testing
● The developer can focus on developing
● Easy to write test cases and little maintenance
● A good test engine also helps
● Use skilled automators
● The automators must be available when needed

26. Questions
?

27. Automation design
Engine Test
Test
report
Instructions
Interfacing
Application under test

28. Design for a telephony switch
Engine
Instructions (C++)
Existing library (iTCL)
Telephony switch

29. Design for a voice portal
Engine
Instructions (C++)
Tool library
Tool
Voice portal

30. Design for web services (1)
Engine
Generic web service
instructions (Java)
Generated
web service client (Java)
Web services

31. Design for web services (2)
Engine
Specific web service
instructions (Java)
Generated
web service client (Java)
Web services

32. Design for a payments system
Engine
Iso20022 protocol SWIFT MT protocol Web service Web instructions
instructions Java) instructions (Java) instructions (Java) (Java)
Generated web service Web driver
Messaging library (Java)
client (Java) library (Java)
SEPA banking system