Exploring no man's land
with keyword driven testing
       Martin Gijsen, M.Sc.
       Martin@DeAnalist.nl

       Free Te...
Presentation overview
●   Automated testing
●   When (not) to use Record & Playback
●   Automated testing as no man's land...
Automated (functional) testing
●   A computer performs the test
●   Fast and consistent
●   Potential benefits:
    ●   Ti...
The Record & Playback approach
●   Record test actions, replay as needed
●   Checks are added to scripts
●   Fundamental p...
So why is it still used?
●   Automated testing is often the job of a tester
●   Most testers are:
    ●   willing to recor...
Result and conclusion
   It is often initially unclear what
effective automated testing requires


    2 out of every 3 TA...
What about the developers?
●   Test automation is software engineering
●   Developers have the skills to create a good
   ...
Welcome to no man's land
●   Most testers cannot or will not automate well
●   Most developers cannot or will not test wel...
Exploring no man's land

          Testing



     Automated testing



       Development
Exploring no man's land

           Testing


      Automated testing
   = analysis (what to test)
  + automation (how to ...
What is in between?

        Testing

     Test analysis
          ?
    Test automation

     Development
No more no man's land?

          Testing


       Test analysis

      Test automation


       Development
Keywords – The DS(T)L
●   Domain Specific (Test) Language

●   Used to express test cases
●   Defined by (or with) the tes...
Bad example
select (“a box”);
type (“some text”);
click (“OK”);
...
Better example

create account;1234567890;John;Doe
create account;2345678901;Jane;Doe

deposit;1234567890;12345,67

transf...
Good example
                 Account      First name   Last name
create account   1234567890   John         Doe
create ac...
Keywords / Instructions
●   Often have arguments

●   No (irrelevant) test execution details
●   No (irrelevant) interfaci...
Sample instruction documentation
Name and aliases   'begin test case' and 'begin testcase'

Description        Indicates t...
Automating (basic)

                                     Test
Test
                Automation          report




        ...
Automating (more detailed)

                      Engine             Test
Test
                                        rep...
The ETA Framework 'engine'
●   Can read from and report to a spreadsheet
●   Supports test cases, variables and procedures...
The process (waterfall)

                               Test cases


               Keyword defs


                       ...
The process (agile)


                   Test cases
         Keyword defs
Requirements        Automate        Test

      ...
Where to get the automator
●   Main criteria (for continuity):
    ●   Skills
    ●   Availability
    ●   Cost
●   Educat...
Summary
●   Automated testing can help achieve your goals
●   Avoid Record & Playback
●   Good keywords help:
    ●   The ...
Questions




    ?
Automation design

                  Engine             Test
Test
                                    report


           ...
Design for a telephony switch

                Engine



          Instructions (C++)



         Existing library (iTCL)
...
Design for a voice portal
             Engine



        Instructions (C++)



           Tool library




              T...
Design for web services (1)

                Engine


          Generic web service
           instructions (Java)


     ...
Design for web services (2)

                Engine


          Specific web service
           instructions (Java)


    ...
Design for a payments system

                                      Engine


Iso20022 protocol    SWIFT MT protocol       ...
Upcoming SlideShare
Loading in …5
×

Exploring No Mans Land with Keyword-Driven Testing

696 views
616 views

Published on

Free Test 2009 conference slides

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
696
On SlideShare
0
From Embeds
0
Number of Embeds
36
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Exploring No Mans Land with Keyword-Driven Testing

  1. 1. Exploring no man's land with keyword driven testing Martin Gijsen, M.Sc. Martin@DeAnalist.nl Free Test Conference March 23, 2009
  2. 2. Presentation overview ● Automated testing ● When (not) to use Record & Playback ● Automated testing as no man's land ● Keyword driven testing ● The process ● Where to get the automator
  3. 3. Automated (functional) testing ● A computer performs the test ● Fast and consistent ● Potential benefits: ● Time (to market) / cost / quality ● More effective use of (test) resources ● More challenged (= happier) testers ● Accurate and up2date test status ● Continuity ● Business goals are drivers
  4. 4. The Record & Playback approach ● Record test actions, replay as needed ● Checks are added to scripts ● Fundamental problem is maintenance ● Software evolves until it is decommissioned ● Maintainability requires structure and abstractions ● Recorded test cases are long, unstructured and low level programs ● Maintenance sensitivity is high ● Maintainability is low
  5. 5. So why is it still used? ● Automated testing is often the job of a tester ● Most testers are: ● willing to record, but ● unwilling or unable to create good automation solutions ● Vendor sales pitch is not always clear on this
  6. 6. Result and conclusion It is often initially unclear what effective automated testing requires 2 out of every 3 TA projects fails sooner or later – Brian LaSuer Use R&P only in case of little manual maintenance
  7. 7. What about the developers? ● Test automation is software engineering ● Developers have the skills to create a good automation solution ● Many developers are unwilling or unable to create good test sets ● … or automate test cases that testers wrote
  8. 8. Welcome to no man's land ● Most testers cannot or will not automate well ● Most developers cannot or will not test well ● Automated testing is like a no man's land in between testing and development ● Is automated testing doomed?
  9. 9. Exploring no man's land Testing Automated testing Development
  10. 10. Exploring no man's land Testing Automated testing = analysis (what to test) + automation (how to test) Development
  11. 11. What is in between? Testing Test analysis ? Test automation Development
  12. 12. No more no man's land? Testing Test analysis Test automation Development
  13. 13. Keywords – The DS(T)L ● Domain Specific (Test) Language ● Used to express test cases ● Defined by (or with) the testers ● Specific to the application under test ● No programming ● High level
  14. 14. Bad example select (“a box”); type (“some text”); click (“OK”); ...
  15. 15. Better example create account;1234567890;John;Doe create account;2345678901;Jane;Doe deposit;1234567890;12345,67 transfer;1234,56;1234567890;2345678901 check balance;1234567890;11111,11 check balance;2345678901;1234,56
  16. 16. Good example Account First name Last name create account 1234567890 John Doe create account 2345678901 Jane Doe Account Amount deposit John 12345,67 Amount From To transfer 1234,56 John Jane Account Amount check balance John 11111,11 check balance Jane 1234,56
  17. 17. Keywords / Instructions ● Often have arguments ● No (irrelevant) test execution details ● No (irrelevant) interfacing details ● No tooling details ● Only the essence of a test case remains ● Easy to read, write and maintain test cases ● Low maintenance sensitivity
  18. 18. Sample instruction documentation Name and aliases 'begin test case' and 'begin testcase' Description Indicates the beginning of a (new) test case. Parameters Two parameters, both optional 1 The identification of the test case, printable characters, preferably unique within the test. Optional. 2 The test case description, printable characters. Optional. Validity Anywhere except inside a procedure definition. Pre-condition None. Post-condition If the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one. Error condition If in a procedure definition.
  19. 19. Automating (basic) Test Test Automation report Application under test
  20. 20. Automating (more detailed) Engine Test Test report Instructions Interfacing Application under test
  21. 21. The ETA Framework 'engine' ● Can read from and report to a spreadsheet ● Supports test cases, variables and procedures ● Implement keywords in Java ● Use any Java library (Abbot, WebDriver, MQ, generated web service client, ...) ● Documented, actively used and developed ● Freeware
  22. 22. The process (waterfall) Test cases Keyword defs Automate Test Requirements Product development
  23. 23. The process (agile) Test cases Keyword defs Requirements Automate Test Product development Test cases Keyword defs Requirements Automate Test Product development Test cases Keyword defs Requirements Automate Test Product development
  24. 24. Where to get the automator ● Main criteria (for continuity): ● Skills ● Availability ● Cost ● Educate someone from test team ● Borrow from development team ● Hire or reserve developer for test team ● Hire external consultant
  25. 25. Summary ● Automated testing can help achieve your goals ● Avoid Record & Playback ● Good keywords help: ● The tester can focus on the testing ● The developer can focus on developing ● Easy to write test cases and little maintenance ● A good test engine also helps ● Use skilled automators ● The automators must be available when needed
  26. 26. Questions ?
  27. 27. Automation design Engine Test Test report Instructions Interfacing Application under test
  28. 28. Design for a telephony switch Engine Instructions (C++) Existing library (iTCL) Telephony switch
  29. 29. Design for a voice portal Engine Instructions (C++) Tool library Tool Voice portal
  30. 30. Design for web services (1) Engine Generic web service instructions (Java) Generated web service client (Java) Web services
  31. 31. Design for web services (2) Engine Specific web service instructions (Java) Generated web service client (Java) Web services
  32. 32. Design for a payments system Engine Iso20022 protocol SWIFT MT protocol Web service Web instructions instructions Java) instructions (Java) instructions (Java) (Java) Generated web service Web driver Messaging library (Java) client (Java) library (Java) SEPA banking system

×