Introducing Keyword-driven Test Automation

  • 267 views
Uploaded on

In both agile and traditional projects, keyword-driven testing has proven to be a powerful way to attain a high level of automation—when it is done correctly. Many testing organizations use keyword …

In both agile and traditional projects, keyword-driven testing has proven to be a powerful way to attain a high level of automation—when it is done correctly. Many testing organizations use keyword testing but aren’t realizing the full benefits of scalability and maintainability that are essential to keep up with the demands of testing today’s software. Hans Buwalda outlines how you can meet what he calls the 5 percent challenges—automating 95 percent of your tests with no more than 5 percent of your total testing effort—using the proven, keyword-driven test method he uses. Hans discusses how keywords relate to other automation techniques like scripting and data-driven testing. The information and real-world application Hans presents enables you to attain a very high level of automation with the lowest possible effort.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
267
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
16
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1.       orial      Presented by:  Hans Buwalda  Brought to you by:      340 Corporate Way, Suite   Orange Park, FL 32073  888‐2 MI  PM Tut 4/7/2014  1:00 – 4:30 PM          “Introducing Keyword‐Driven Test Automation”      LogiGear                    300, 68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com   
  • 2.                          Hans Buwalda has been working with inform hnology since his high school r, s   Hans Buwalda LogiGear   ation tec years. In his thirty year career, Hans has gained experience as a developer, manage and principal consultant for companies and organizations worldwide. He was a pioneer of the keyword approach to testing and automation, now widely used throughout the industry. His approaches to testing, like Action Based Testing and Soap Opera Testing, have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges. Hans is a frequent speaker at STAR conference and is lead author of Integrated Test Design and Automation: Using the Testframe Method.
  • 3. 2/4/2014 1 STARCANADA 2014 Toronto, Canada Tutorial MI Monday, April 7, 13.00 – 16.30 Introducing Keyword-driven Test Automation © 2014 LogiGear Corporation. All Rights Reserved Hans Buwalda LogiGear hans @ logigear.com Introduction − industries roles in testing © 2014 LogiGear Corporation. All Rights Reserved − roles in testing
  • 4. 2/4/2014 2 About LogiGear Software testing company, around since 1994 Testing and test automation expertise, services and www.logigear.com www.testarchitect.com g p , tooling − consultancy, training − test development and automation services − "test integrated" development services Aims to be thought leader, in particular for large and complex test projects © 2014 LogiGear Corporation. All Rights Reserved p p j Products: − TestArchitect™, TestArchitect for Visual Studio™ − integrating test development with test management and automation − based on modularized keyword-driven testing About Hans Dutch guy, living and working in California since 2001, as CTO of LogiGear Background in math, computer science, management Original career in management consultancy, since 1994 focusing on testing and test automation − keywords, agile testing, big testing, . . . © 2014 LogiGear Corporation. All Rights Reserved www.happytester.com hans @ logigear.com
  • 5. 2/4/2014 3 Topics for this tutorial Introduction to keyword driven testing − including "Action Based Testing", my own flavor of it... Comparison to other techniques for automationp q Recommendations for a successful application of keyword driven testing − test design − automation − organization Some ideas for specific situations: − data driven testing © 2014 LogiGear Corporation. All Rights Reserved data driven testing − non-ui testing − multi-media − protocols − initial data Not everything will be equally interesting, or accessible, to everybody Testing Under Pressure DEADLINE specification development test © 2014 LogiGear Corporation. All Rights Reserved
  • 6. 2/4/2014 4 Testing Under Pressure DEADLINE specification development test Develop tests in time: © 2014 LogiGear Corporation. All Rights Reserved Develop tests in time: • Test design • Auditing, acceptance • Preparations • Automation The 5% Rules of Test Automation No more than 5% of all test cases should beNo more than 5% of all test cases should be executed manually No more than 5% of all efforts around testing should involve automating the tests © 2014 LogiGear Corporation. All Rights Reserved testing should involve automating the tests
  • 7. 2/4/2014 5 Why a High Automation Degree ?? The best way to prepare for efficiency in the crunch zone − good manual test cases can help too, but marginally Buy time to do more "exploratory testing", and better test development Credible pay-off for the cost of introduction of automation − initial costs are: tooling, learning curve, adaptation of existing tests Automation is better positioned to identify “bonus bugs” − on average 15% of fixes cause new bugs − many of these bugs are hard to find without integral testing • often a result of violating overall architectures • the bugs occur because data is left in inconsistent state © 2014 LogiGear Corporation. All Rights Reserved Automated tests have a better chance of being kept up to date if they form the majority of the testware Automation can be re-run, for example as part of the continuous integration process − either specific based on code changes, or integral, to also catch bonus bugs Why < 5% Automation Efforts ?? Automation should not dominate testing − it is not a goal in itself − may never be a bottleneck automation should deliver, not dominate… may never be a bottleneck Testers should be able to focus on testing − better tests (higher ambition level) − communication with stake holders High automation efforts can aggravate the © 2014 LogiGear Corporation. All Rights Reserved High automation efforts can aggravate the “crunch zone”, instead of relieving it − “invitation to Murphy’s law”
  • 8. 2/4/2014 6 Record and Playback select window "Logon" enter text "username", "administrator" enter text "password", "testonly" push button "Ok"push button "Ok" select window "Main" push button "New Customer" expect window "Customer Information" select field "First Name" type "Paul" select field "Last Name" type "Jones" © 2014 LogiGear Corporation. All Rights Reserved type Jones select field "Address" type "54321 Space Drive" . . . Scripting Test Case Design Test Case Automation TEST DESIGNER AUTOMATION ENGINEER © 2014 LogiGear Corporation. All Rights Reserved Test Case Execution AUTOMATION ENGINEER MR. PLAYBACK
  • 9. 2/4/2014 7 Example scripting /// <summary> /// AddItems ‐ Use 'AddItemsParams' to pass parameters into this method. /// </summary> State of the art, but stuff for coders . . . /// </summary> public void AddItems() { #region Variable Declarations WinControl uICalculatorDialog = this.UICalculatorWindow.UICalculatorDialog; WinEdit uIItemEdit = this.UICalculatorWindow.UIItemWindow.UIItemEdit; #endregion // Type '{NumPad7}' in 'Calculator' Dialog © 2014 LogiGear Corporation. All Rights Reserved Keyboard.SendKeys(uICalculatorDialog, this.AddItemsParams.UICalculatorDialogSendKeys, ModifierKeys.None); // Type '{Add}{NumPad2}{Enter}' in 'Unknown Name' text box Keyboard.SendKeys(uIItemEdit, this.AddItemsParams.UIItemEditSendKeys, ModifierKeys.None); } Keywords, Action Words number name quantity new product P-9009 Sledge Hammer 5 number quantity add quantity P-9009 20 number name quantity new product P-9009 Sledge Hammer 5 number quantity add quantity P-9009 20 actions, each with a keyword and arguments read from top to bottom fragment from a test with actions Common in automation tools nowadays (but with different styles) Identify tasks for both test development and automation q y add quantity P-9009 3 add quantity P-9009 6 number quantity check quantity P-9009 34 q y add quantity P-9009 3 add quantity P-9009 6 number quantity check quantity P-9009 34 "34" is the expected value here to bottom © 2014 LogiGear Corporation. All Rights Reserved y p The test developer creates tests using actions Each action consists of a keyword and arguments The automation task focuses on automating the actions Each action is automated only once
  • 10. 2/4/2014 8 Potential benefits of keywords More tests, better tests − more breadth − more depth Fast, results can be quickly available − the test design directly drives the automation Separates the tests from the technical scripting language − easier to involve business subject matter experts − the action format allows for easy readability Less efforts for automation − even "script free" in many cases © 2014 LogiGear Corporation. All Rights Reserved Automation more stable and maintainable − limited and manageable impact of changes in the system under test Develop tests more early in the life cycle − deal with execution details later Risks of keyword approaches Often seen as silver bullet, complications are underestimated − often treated as a technical "trick" − testers can get squeezed and marginalized • developers and users dictating tests • automation engineers dictating actions − testers can end up with an automation responsibility, thus becoming pseudo programmers The method needs understanding and experience to be successful © 2014 LogiGear Corporation. All Rights Reserved − pitfalls are many, and can have a negative effect on the outcome Lack of method and structure can risk manageability − maintainability not as good as hoped − results can be disappointing, approach will be blamed
  • 11. 2/4/2014 9 Combining Approaches . . . Use keywords for the automation-ready description of test cases Use scripting to set up structured automation for the actions © 2014 LogiGear Corporation. All Rights Reserved Use record and playback to record keywords Enter a user id that is greater than 10 characters, enter proper information for all other fields, and click on the "Continue" button There should be an error message stating that "User Id must be less than 10 characters". Comparing Formats Most values are implicit. The tester has to figure them out during execution…. classic format Execution instructions are repeated in multiple test cases and click on the Continue button characters . Enter a User Id with special character's), enter proper information for all other fields and click on the "Continue" button An error message should be displayed indicating that "User Id cannot contain some special characters". Enter the information, with a password of 4 characters and click on the "Continue" button Check for an error message saying: "Password must contain at least 5 characters". keywords © 2014 LogiGear Corporation. All Rights Reserved user id message check registration dialog aaaaabbbbbc User Id must be less than 10 characters user id message check registration dialog résoudre User Id cannot contain some special characters password message check registration dialog test Password must contain at least 5 characters
  • 12. 2/4/2014 10 Keywords is not just test automation Can also be used for other than testing: − data entry chores − training purposesg p p Can also be used for manual testing − for example with a manual testing dialog − even can show instructions, with placeholders for values Action login <user> <password> Enter "<user>" in the user name field, and "<password>" in the password field. Action login <user> <password> Enter "<user>" in the user name field, and "<password>" in the password field. © 2014 LogiGear Corporation. All Rights Reserved Test line user name password login hansb secret Test line user name password login hansb secret What the manual tester would see Enter "hansb" in the user name field, and "secret" in the password field. What the manual tester would see Enter "hansb" in the user name field, and "secret" in the password field. Keywords need a method By themselves keywords don't provide much scalability − they can even backfire and make automation more cumbersome − a method can help tell you which keywords to use when, and how to i thorganize the process Today we'll look at Action Based Testing (ABT) − addresses test management, test development and automation − large focus on test design as the main driver for automation success Central deliveries in ABT are the "Test Modules" © 2014 LogiGear Corporation. All Rights Reserved Central deliveries in ABT are the Test Modules − developed in spreadsheets − each test module contains "test objectives" and "test cases" − each test module is a separate (mini) project, each test module can involve different stake holders
  • 13. 2/4/2014 11 Don't just automate manual testing © 2014 LogiGear Corporation. All Rights Reserved Don't just automate manual testing © 2014 LogiGear Corporation. All Rights Reserved
  • 14. 2/4/2014 12 Don't just automate manual testing © 2014 LogiGear Corporation. All Rights Reserved Good automated testing is not the same as automating good manual testing. . . Action Based Testing (ABT) Uses the keyword format as a basis for a method − covers test management, test development and automation − with a large focus on test design as the main driver for automation success − method is specific, but concepts are generic The central product in ABT is the "Test Module", not the test case − like chapters in a book − test cases are part of the test modules, they are typically the result (rather than the input) of test development © 2014 LogiGear Corporation. All Rights Reserved (rather than the input) of test development − test development is seen as having both analytical and creative aspects − developed as spread sheets, external from the automation, with a well defined flow − easier to manage: each test module is a separate (mini) project, each test module can involve different stake holders
  • 15. 2/4/2014 13 Test Module Plan Test Module 1 Test Module 2 Test Module N Overview Tests Objectives Tests Tests Actions . . . Objectives Objectives © 2014 LogiGear Corporation. All Rights Reserved AUTOMATION user password log in jdoe StarEast first last brand model rent car John Renter Ford Escape rent car John Renter Chevrolet Volt last total check bill Renter 140.42 interaction test business test window control value enter log in user name jdoe enter log in password StarEast window control property expected check prop log in ok button enabled true car guy Example of business level test module Consists of an (1) initial part, (2) test cases and (3) a final part Focus is on business functionality, with a clear business scope Navigation details are avoidedNavigation details are avoided TEST MODULE Car Rental Payments user start system john TEST CASE TC 01 Rent some cars first name last name car rent car John Doe Ford Escape © 2014 LogiGear Corporation. All Rights Reserved rent car John Doe Ford Escape rent car John Doe Chevvy Volt last name amount check payment Doe 140.4 FINAL close application
  • 16. 2/4/2014 14 Example of an interaction level test module Lay-out the same, with an initial part, test cases and a final part Interaction details that are the target of the test are not hidden Focus is not on business ("is the payment amount correct"), but on( p y ) interaction ("do I see the payment amount") TEST MODULE Screen Flow user start system john TEST CASE TC 01 Order button window button click main create order © 2014 LogiGear Corporation. All Rights Reserved window check window exists new order FINAL close application Variables and expressions with keywords TEST CASE TC 02 Rent some more cars car available get quantity Chevvy Volt >> voltsget quantity Chevvy Volt volts first name last name car rent car John Doe Chevvy Volt rent car John Doe Chevvy Volt car expected check quantity Chevvy Volt # volts - 2 © 2014 LogiGear Corporation. All Rights Reserved This test does not need an absolute number for the available cars, just wants to see if a stock is updated As a convention we denote an assignment with ">>" The "#" indicates an expression
  • 17. 2/4/2014 15 Data driven testing with keywords DATA SET cars car first last value Chevvy Volt John Doe 40000 TEST CASE TC 03 Check stocks data set use data set /cars y Ford Escape Mary Kane 22500 Chrysler 300 Jane Collins 29000 Buick Verano Tom Anderson 23000 BMW 750 Henry Smyth 87000 Toyota Corolla Vivian Major 16000 car available get quantity # car >> quantity first name last name car rent car # first # last # car car expected check quantity # car # quantity - 1 repeat for data set © 2014 LogiGear Corporation. All Rights Reserved The test lines will be repeated for each row in the data set The values represented by "car", "first" and "last" come from the selected row of the data set Automating keyword tests Keywords are useful, but technical not complex. It is not hard to make a simple keyword interpreter. Many test tools also have keyword options in some form or another. Function Interpret While not end of test Read next line Split the line into arguments Look up the keyword in the "action list" Execute the function belonging to the keyword ReportInterpreter © 2014 LogiGear Corporation. All Rights Reserved g g y Report the results of this line Repeat for next line End
  • 18. 2/4/2014 16 Example: script for an action "check sort order" The following action script will verify whether the rows in a table are sorted: def action checkSortOrder(): # get table object, column number and column count windowName = LIBRARY.NamedArgument("window") tableName = LIBRARY.NamedArgument("table") columnName = LIBRARY.NamedArgument("column") table = ABT.OpenElement(windowName, tableName) column = table.GetColumnIndex(columnName) rowCount = table.GetRowCount() # check the sort order, row by row previous = table.GetCellText(0, column) f i i (1 C t) find the table in the UI if a value is smaller than before, fail the test get arguments from the test line _ () © 2014 LogiGear Corporation. All Rights Reserved for i in range(1, rowCount): current = table.GetCellText(i, column) if current < previous : LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0) return previous = current LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1) if all rows are ascending, pass the test Using the new action By keeping an action generic it can be applied for a variety of situations S l f i " h k t d "Some examples of using "check sort order": window table column check sort order view orders orders table ID window table column check sort order annual results regions revenue © 2014 LogiGear Corporation. All Rights Reserved window table column check sort order inventory cars price window table column check sort order registration students last name
  • 19. 2/4/2014 17 Example application © 2014 LogiGear Corporation. All Rights Reserved A Test Module for the application © 2014 LogiGear Corporation. All Rights Reserved We click a tree node, and then do a check The actions here are built-in in the framework
  • 20. 2/4/2014 18 Making a new "action" name of the new action create a node path the arguments of the new action © 2014 LogiGear Corporation. All Rights Reserved This action definition uses existing actions to create a new action called "check bitrate" Argument names can be used in cell expressions, that start with "#", and support the usual string and numeric operators p from the first two argumentsthe expected value is given by the 3rd argument Using the action in a test © 2014 LogiGear Corporation. All Rights Reserved These test lines don't care about the navigation in the UI of the application, the focus is functional: verify data Such functional tests are easier to read with high level actions, and the reduced dependency on navigation makes them (much) easier to maintain in the long term
  • 21. 2/4/2014 19 4000 In a good application of the keywords approach a large increase in test cases (like doubling the amount) should result in a modest increase in actions, and a minor increase, if any, in programmed action functions. Scalability 4000 tests 2000 © 2014 LogiGear Corporation. All Rights Reserved 250 actions tests 22 functions 200 actions 20 functions Identifying controls © 2014 LogiGear Corporation. All Rights Reserved Identify windows and controls, and assign names to them These names encapsulate the properties that the tool can use to identify the windows and controls when executing the tests
  • 22. 2/4/2014 20 Mapping an interface INTERFACE ENTITY library interface entity setting title {.*Music Library} name class label interface element title text Title:interface element title text Title: interface element artist text Artist: interface element file size text File size (Kb): name class position interface element playing time text textbox 4 interface element file type text textbox 5 interface element bitrate text textbox 6 name class position interface element music treeview treeview 1 © 2014 LogiGear Corporation. All Rights Reserved An interface mapping (common in test tools) will map windows and controls to names When the interface of an application changes, you only have to update this in one place The interface mapping is a key step in your automation success, allocate time to design it well, in particular naming and choosing identifying properties Some Tips to Get Stable Automation Press for an automation-friendly system design − consider this a key requirement ("must have") − having testing and automation in agile teams helpshaving testing and automation in agile teams helps Test the automation, in particular complex actions − before running tests with them Keep an eye on the test design © 2014 LogiGear Corporation. All Rights Reserved Use "active" timing, don't use hard coded waits
  • 23. 2/4/2014 21 Automation friendly system design (examples) Timing hooks − to know when a UI or a background process is ready Hidden identifiers for controlsHidden identifiers for controls − like "name" in Java and "automation id" in WPF Access to data input − ability to set up special values for calculations − functions to shut off or steer random generators Behind-the-scenes access to displayed data © 2014 LogiGear Corporation. All Rights Reserved − graphic representation is different from value correctness − in a game: knowing where the monsters are Emulation features, like time-travel and fake locations (note: be aware of security considerations. Non-UI access should be safe) Look for properties a human user can't see, but a test tool can This approach can lead to speedier and more stable automation − interface mapping is often bottleneck, and source of maintenance problems − with predefined identifying property values in interface map can be created without "spy" tools not sensitive to changes in the system under test Automation-friendly design: hidden properties − not sensitive to changes in the system under test − not sensitive to languages and localizations Examples: − "id" attribute for HTML elements − "name" field for Java controls − "AccessibleName" or "Automation ID" properties in .Net controls (see below) © 2014 LogiGear Corporation. All Rights Reserved number
  • 24. 2/4/2014 22 Mapping the interface using hidden identifiers INTERFACE ENTITY library interface entity setting automation id MusicLibraryWindow ta name ta class automation id interface element title text TitleTextBoxinterface element title text TitleTextBox interface element artist text SongArtistTextBox interface element file size text SizeTextBox interface element playing time text TimeTextBox interface element file type text TypeTextBox interface element bitrate text BitrateTextBox ta name ta class automation id interface element music treeview MusicTreeView © 2014 LogiGear Corporation. All Rights Reserved Instead of positions or language dependent labels, an internal property "automation id" has been used The interface definition will be less dependent on modifications in the UI of the application under test If the information can be agreed upon with the developers, for example in an agile team, it can be entered (or pasted) manually and early on Active Timing Passive timing − wait a set amount of time − in large scale testing, try to avoid passive timing altogether: • if wait too short, test will be interrupted if it t l ti i t d• if wait too long, time is wasted Active timing − wait for a measurable event − usually the wait is up to a, generous, maximum time − common example: wait for a window or control to appear (usually the test tool will do this for you) Even if not obvious, find something to wait for... © 2014 LogiGear Corporation. All Rights Reserved Involve developers if needed − relatively easy in an agile team, but also in traditional projects, give this priority If using a waiting loop − make sure to use a "sleep" function in each cycle that frees up the processor (giving the AUT time to respond) − wait for an end time, rather then a set amount of cycles
  • 25. 2/4/2014 23 Active Timing How much passive timing do you have in your scripts? If you're not sure, find out... ... and let me know "First action I took upon my return was to evaluate the percentage of © 2014 LogiGear Corporation. All Rights Reserved First action I took upon my return was to evaluate the percentage of passive time in our code and found passive time 68% versus active time 32%. Needless to say our automation test cases were very expensive time operations and now I know why..." Raed Atawneh, 2012 (extract) Things to wait for... Wait for a last control or elements to load − developers can help knowing which one that is Non UI criteriaNon-UI criteria − API function − existence of a file Criteria added in development specifically for this purpose, like: − "disabling" big slow controls (like lists or trees) until they're done loading − API functions or UI window or control properties Use a "delta" approach: © 2014 LogiGear Corporation. All Rights Reserved Use a delta approach: − every wait cycle, test if there was a change; if no change, assume that the loading time is over: − examples of changes: • the controls on a window • count of items in a list • size a file (like a log file)
  • 26. 2/4/2014 24 Test Design Effective test breakdown (into test modules) − make sure every test module has a clear focus keep different kinds and levels of tests separate− keep different kinds and levels of tests separate Right level of actions − as “high level” if possible, hiding as many details as much as possible − but not if the details are relevant for the test © 2014 LogiGear Corporation. All Rights Reserved I have become to believe that successful automation is not a technical challenge. It is most of all a test design challenge. Why Better Test Design? Quality and manageability of test − many tests are often quite "mechanical" now − one to one related to specifications, user stories or requirements, hi h ft i k b t l k iwhich often is ok, but lacks aggression − no combinations, no unexpected situations, lame and boring − such tests have a hard time finding (interesting) bugs Better automation − when unneeded details are left out of tests, they don't have to be maintained − avoiding "over checking": creating checks that are not in the scope of a test but may fail after system changes © 2014 LogiGear Corporation. All Rights Reserved a test, but may fail after system changes − limit the impact of system changes on tests, making such impact more manageable I have become to believe that successful automation is usually less of a technical challenge as it is a test design challenge.
  • 27. 2/4/2014 25 The Three “Holy Grails” of Test Design Metaphor to depict three main steps in test design Using "grail" to illustrate that there is no one perfect solution, but that it matters to pay attention (to search) About quality of tests, but most of all about scalability and maintainability in big and complex projects Organization of tests into test modules © 2014 LogiGear Corporation. All Rights Reserved Right approach for each test module Proper level of detail in the test specification What's the trick... © 2014 LogiGear Corporation. All Rights Reserved
  • 28. 2/4/2014 26 What's the trick... Have or acquire facilities to store and organize your content Select your stuff Decide where to put what − assign and label the shelves P t it th © 2014 LogiGear Corporation. All Rights Reserved Put it there If the organization is not sufficient anymore, add to it or change it Breakdown Criteria Main Criteria − Business tests versus interaction tests − Data versus presentation ("is data correct" versus "can I view data") Architecture of the system under test (client server protocol sub− Architecture of the system under test (client, server, protocol, sub systems, components, modules, ...) − Functionality (customers, finances, management information, ...) − Kind of test (navigation flow, negative tests, response time, ...) − Ambition level (smoke test, regression, aggressive, …) Additional Criteria © 2014 LogiGear Corporation. All Rights Reserved − Stakeholders (like "Accounting", "Compliance", "HR", ...) − Complexity of the test (put complex tests in separate modules) − Technical aspects of execution (special hardware, multi-station, ...) − Overall project planning (availability of information, timelines, sprints, ...) − Risks involved (extra test modules for high risk areas)
  • 29. 2/4/2014 27 Approach 1: Workshop Gather a meeting with relevant participants − test developers − domain expertsdomain experts − automation engineer (focus on efficiency of automation) − experienced moderator − also consider: developers, managers If necessary, provide training of participants before the discussion © 2014 LogiGear Corporation. All Rights Reserved before the discussion Approach 2: Design and Feedback One or two experienced test designers create a first draft The draft is delivered/discussed to relevant parties Ask the parties to verify: 1. Structure: does it make sense © 2014 LogiGear Corporation. All Rights Reserved 2. Completeness: are all relevant areas covered Based on feedback, further modify the design
  • 30. 2/4/2014 28 Properties of a good Breakdown Reflects the level of tests Well differentiated and clear in scope Balanced in size and amount Modules mutually independent © 2014 LogiGear Corporation. All Rights Reserved Fitting the priorities and planning of the project "Thou Shall Not Debug Tests..." Large and complex test projects can be hard to "get to run" If they are however, start with taking a good look again at your test design... Rule of thumb: don't debug tests. If tests don't run smoothly, make sure: − lower level tests have been successfully executed first -> UI flow in © 2014 LogiGear Corporation. All Rights Reserved o e e e tests a e bee success u y e ecuted st U o the AUT is stable − actions and interface definitions have been tested sufficiently with their own test modules -> automation can be trusted − are you test modules not too long and complex?
  • 31. 2/4/2014 29 Breakdown examples CRUD tests (Create, Read, Update, Delete) for all entity types in the app − like "order", "customer", "well", etc − for all: various types and situations Forms, value entry − does each form work (try to test form by form, not entity by entity) − mandatory and optional fields valid and invalid values etcmandatory and optional fields, valid and invalid values, etc − UI elements and their properties and contents − function keys, tab keys, special keys, etc Screen and transaction flows − like cancel an order, menu navigation, use a browser back and forward buttons, etc − is the data in the database correct after each flow Business transactions, business rules − identify situations that the tests need to try Function tests, do individual functions work − can I count orders, can I calculate a discount, etc End-to-end tests © 2014 LogiGear Corporation. All Rights Reserved d to e d tests − like enter sale order, then check inventory and accounting Tests with specific automation needs − like multi station tests Tests of non-UI functions High ambition tests (aggressive tests) − can I break the system under test Identifying the modules Step 1: top down: establish main structure: analyze what the business is and what the system does? how is it technically organized? do other “primary criteria” apply? use the list in the "breakdown examples" slide as a starting point − see it as "low hanging fruit": items that tend to apply well in many projects also visit the “secondary criteria” − not always applicable, but can help to refine the design further Step 2: bottom up: refine, complete: study individual functionalities and checks (like from exist test cases) © 2014 LogiGear Corporation. All Rights Reserved and identify test modules for them if needed identify and discuss any additional criteria and needed testing situations review and discuss the resulting list(s) of test modules if necessary try to create some early drafts of test modules and adjust the list Repeat steps 1 and 2 if needed.
  • 32. 2/4/2014 30 Example of an application under test Various item types − tests − actions − interface definitions d t t− data sets − folders − ... Various operations − open − cut, copy, paste − check out − ... Various ways to initiate an operation − context menu, with or without accelerator key © 2014 LogiGear Corporation. All Rights Reserved , y − main menu, with or without accelerator key − toolbar − short cut key − function key − drag and drop − double click − .... Defining some modules Test modules for operations − primary and alternate paths − various values for fields like "comment" in check-in − paste in other projects d t i− copy and paste various groups − not necessarily on each item type Test modules for items − address all item types at least once − on each item type perform each operation − not necessarily each variant of each operation UI handling − try for UI command if it starts the intended operation © 2014 LogiGear Corporation. All Rights Reserved y p − not necessarily on each item type or operation variant Concurrency − try concurrency sensitive operations with multiple stations − in varying concurrency scenarios, with and without local "refreshes" − not necessarily each item type or operation variant − certainly not each UI command included
  • 33. 2/4/2014 31 What about existing tests? Compare to moving house: − some effort can't be avoided − be selective, edit your stuff, look at the future, not the past d id h t t h t th t it th− decide where to put what, then put it there − moving is an opportunity, you may not get such chance again soon Adopt a module approach − define the modules and their scope as if from scratch − use the existing test cases in two ways: • verify completeness • harvest and re-use them for tests and for actions avoid porting over "step by step" in particular avoid over checking © 2014 LogiGear Corporation. All Rights Reserved − avoid porting over step by step , in particular avoid over-checking Defining test runs using "test suites" Build Acceptance Test Smoke Test System Test © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. Functional Acceptance Test Delivery Test
  • 34. 2/4/2014 32 Test Module Scope Prio Status Model Life Cycles Create, store, delete Models (=formula + data), as part of SYS  sessions 1 pass Result Life Cycles Create, store outputs. See them in the process store. 1 pass Formula Life Cycles Create, edit, manage, remove formulas 2 pass Formula Editor buttons, operations, undo 3 pass Repository display of the Modeler repository, presence of user formulas,  drag and drop usage. Effect of changing repository folder  (environment variable) 1 failed Model Store in Repository presence, re‐run, delete 1 pass Repository UI example: selecting an item shows its description 2 errorsRepository UI example: selecting an item shows its description 2 errors Formula Evaluation Correctness of results, valid/invalid arguments, boundary  analyses, special arguments 1 pass Built‐in Formulas Presence, correctness, valid/invalid arguments, boundaries,  special arguments, equivalence classes 1 pass Data Table Association Associate tabels view, change and remove associations, data  applicability, for existing and defined formulas 2 pass Quick Access buttons Life cycle of Quick Access buttons, correctnes for the built‐in  ones 3 dev Formula arguments presence, argument types, argument entry, parameters,  defaults 2 pass arguments for Built‐in  l arguments, argument types  and defaults for each pre‐defined  f l 2 failed © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. Formulas formula 2 failed Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass Model Execution Model times, start, stop (cancel), restart ("chunks",  "timeboxes", ... needs more information) 3 pass Graphics graphical representation of various data types and data sets 1 pass Graphics Viewing zoom, select, drag and drop (no 3d now) 1 pass Administration users, projects, authorization 1 pass Model results in central  database storing, removing, using, correctness, … (there are some other  applications, mostly legacy, that can do the same Models to  compare) 1 pass Modeler UI various controls, panels, tabs 2 pass Test Module Scope Prio Status Model Life Cycles Create, store, delete Models (=formula + data), as part of SYS  sessions 1 pass Result Life Cycles Create, store outputs. See them in the process store. 1 pass Formula Life Cycles Create, edit, manage, remove formulas 2 pass Formula Editor buttons, operations, undo 3 pass Repository display of the Modeler repository, presence of user formulas,  drag and drop usage. Effect of changing repository folder  (environment variable) 1 failed Model Store in Repository presence, re‐run, delete 1 pass Repository UI example: selecting an item shows its description 2 errorsRepository UI example: selecting an item shows its description 2 errors Formula Evaluation Correctness of results, valid/invalid arguments, boundary  analyses, special arguments 1 pass Built‐in Formulas Presence, correctness, valid/invalid arguments, boundaries,  special arguments, equivalence classes 1 pass Data Table Association Associate tabels view, change and remove associations, data  applicability, for existing and defined formulas 2 pass Quick Access buttons Life cycle of Quick Access buttons, correctnes for the built‐in  ones 3 dev Formula arguments presence, argument types, argument entry, parameters,  defaults 2 pass arguments for Built‐in  l arguments, argument types  and defaults for each pre‐defined  f l 2 failed © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. Formulas formula 2 failed Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass Model Execution Model times, start, stop (cancel), restart ("chunks",  "timeboxes", ... needs more information) 3 pass Graphics graphical representation of various data types and data sets 1 pass Graphics Viewing zoom, select, drag and drop (no 3d now) 1 pass Administration users, projects, authorization 1 pass Model results in central  database storing, removing, using, correctness, … (there are some other  applications, mostly legacy, that can do the same Models to  compare) 1 pass Modeler UI various controls, panels, tabs 2 pass
  • 35. 2/4/2014 33 Test Module Scope Prio Status Model Life Cycles Create, store, delete Models (=formula + data), as part of SYS  sessions 1 pass Result Life Cycles Create, store outputs. See them in the process store. 1 pass Formula Life Cycles Create, edit, manage, remove formulas 2 pass Formula Editor buttons, operations, undo 3 pass Repository display of the Modeler repository, presence of user formulas,  drag and drop usage. Effect of changing repository folder  (environment variable) 1 failed Model Store in Repository presence, re‐run, delete 1 pass Repository UI example: selecting an item shows its description 2 errorsRepository UI example: selecting an item shows its description 2 errors Formula Evaluation Correctness of results, valid/invalid arguments, boundary  analyses, special arguments 1 pass Built‐in Formulas Presence, correctness, valid/invalid arguments, boundaries,  special arguments, equivalence classes 1 pass Data Table Association Associate tabels view, change and remove associations, data  applicability, for existing and defined formulas 2 pass Quick Access buttons Life cycle of Quick Access buttons, correctnes for the built‐in  ones 3 dev Formula arguments presence, argument types, argument entry, parameters,  defaults 2 pass arguments for Built‐in  l arguments, argument types  and defaults for each pre‐defined  f l 2 failed © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. Formulas formula 2 failed Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass Model Execution Model times, start, stop (cancel), restart ("chunks",  "timeboxes", ... needs more information) 3 pass Graphics graphical representation of various data types and data sets 1 pass Graphics Viewing zoom, select, drag and drop (no 3d now) 1 pass Administration users, projects, authorization 1 pass Model results in central  database storing, removing, using, correctness, … (there are some other  applications, mostly legacy, that can do the same Models to  compare) 1 pass Modeler UI various controls, panels, tabs 2 pass Test Module Scope Prio Status Model Life Cycles Create, store, delete Models (=formula + data), as part of SYS  sessions 1 pass Result Life Cycles Create, store outputs. See them in the process store. 1 pass Formula Life Cycles Create, edit, manage, remove formulas 2 pass Formula Editor buttons, operations, undo 3 pass Repository display of the Modeler repository, presence of user formulas,  drag and drop usage. Effect of changing repository folder  (environment variable) 1 failed Model Store in Repository presence, re‐run, delete 1 pass Repository UI example: selecting an item shows its description 2 errorsRepository UI example: selecting an item shows its description 2 errors Formula Evaluation Correctness of results, valid/invalid arguments, boundary  analyses, special arguments 1 pass Built‐in Formulas Presence, correctness, valid/invalid arguments, boundaries,  special arguments, equivalence classes 1 pass Data Table Association Associate tabels view, change and remove associations, data  applicability, for existing and defined formulas 2 pass Quick Access buttons Life cycle of Quick Access buttons, correctnes for the built‐in  ones 3 dev Formula arguments presence, argument types, argument entry, parameters,  defaults 2 pass arguments for Built‐in  l arguments, argument types  and defaults for each pre‐defined  f l 2 failed © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. Formulas formula 2 failed Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass Model Execution Model times, start, stop (cancel), restart ("chunks",  "timeboxes", ... needs more information) 3 pass Graphics graphical representation of various data types and data sets 1 pass Graphics Viewing zoom, select, drag and drop (no 3d now) 1 pass Administration users, projects, authorization 1 pass Model results in central  database storing, removing, using, correctness, … (there are some other  applications, mostly legacy, that can do the same Models to  compare) 1 pass Modeler UI various controls, panels, tabs 2 pass
  • 36. 2/4/2014 34 Test Module Scope Prio Status Model Life Cycles Create, store, delete Models (=formula + data), as part of SYS  sessions 1 pass Result Life Cycles Create, store outputs. See them in the process store. 1 pass Formula Life Cycles Create, edit, manage, remove formulas 2 pass Formula Editor buttons, operations, undo 3 pass Repository display of the Modeler repository, presence of user formulas,  drag and drop usage. Effect of changing repository folder  (environment variable) 1 failed Model Store in Repository presence, re‐run, delete 1 pass Repository UI example: selecting an item shows its description 2 errorsRepository UI example: selecting an item shows its description 2 errors Formula Evaluation Correctness of results, valid/invalid arguments, boundary  analyses, special arguments 1 pass Built‐in Formulas Presence, correctness, valid/invalid arguments, boundaries,  special arguments, equivalence classes 1 pass Data Table Association Associate tabels view, change and remove associations, data  applicability, for existing and defined formulas 2 pass Quick Access buttons Life cycle of Quick Access buttons, correctnes for the built‐in  ones 3 dev Formula arguments presence, argument types, argument entry, parameters,  defaults 2 pass arguments for Built‐in  l arguments, argument types  and defaults for each pre‐defined  f l 2 failed © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. Formulas formula 2 failed Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass Model Execution Model times, start, stop (cancel), restart ("chunks",  "timeboxes", ... needs more information) 3 pass Graphics graphical representation of various data types and data sets 1 pass Graphics Viewing zoom, select, drag and drop (no 3d now) 1 pass Administration users, projects, authorization 1 pass Model results in central  database storing, removing, using, correctness, … (there are some other  applications, mostly legacy, that can do the same Models to  compare) 1 pass Modeler UI various controls, panels, tabs 2 pass Test Module Scope Prio Build 1 Build 2 Build 3 Model Life Cycles Create, store, delete Models (=formula + data), as part of SYS  sessions 1 pass pass pass Result Life Cycles Create, store outputs. See them in the process store. 1 pass pass pass Formula Life Cycles Create, edit, manage, remove formulas 2 pass pass pass Formula Editor buttons, operations, undo 3 pass pass pass Repository display of the Modeler repository, presence of user formulas,  drag and drop usage. Effect of changing repository folder  (environment variable) 1 failed failed failed Model Store in Repository presence, re‐run, delete 1 pass pass pass Repository UI example: selecting an item shows its description 2 errors errors errorsRepository UI example: selecting an item shows its description 2 errors errors errors Formula Evaluation Correctness of results, valid/invalid arguments, boundary  analyses, special arguments 1 pass pass pass Built‐in Formulas Presence, correctness, valid/invalid arguments, boundaries,  special arguments, equivalence classes 1 pass pass pass Data Table Association Associate tabels view, change and remove associations, data  applicability, for existing and defined formulas 2 pass pass pass Quick Access buttons Life cycle of Quick Access buttons, correctnes for the built‐in  ones 3 dev dev dev Formula arguments presence, argument types, argument entry, parameters,  defaults 2 pass pass pass arguments for Built‐in  l arguments, argument types  and defaults for each pre‐defined  f l 2 failed failed failed © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. Formulas formula 2 failed failed failed Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass pass pass Model Execution Model times, start, stop (cancel), restart ("chunks",  "timeboxes", ... needs more information) 3 pass pass pass Graphics graphical representation of various data types and data sets 1 pass pass pass Graphics Viewing zoom, select, drag and drop (no 3d now) 1 pass pass pass Administration users, projects, authorization 1 pass pass pass Model results in central  database storing, removing, using, correctness, … (there are some other  applications, mostly legacy, that can do the same Models to  compare) 1 pass pass pass Modeler UI various controls, panels, tabs 2 pass pass pass
  • 37. 2/4/2014 35 Grail 2: Approach per Test Module • Name the test modules recommendation: subject area only, leave words like "Verify", "Test", "Can" from titles of test modules (and test objectives and test cases) • Plan the test module:• Plan the test module: when to develop: is enough specification available when to execute: make sure the functionality at action level is well-tested and working already • Process: analysis of requirements formulation of "test objectives" create "test cases" © 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved. • Identify stakeholders and their involvement: users, subject matter experts developers auditors • Choose testing techniques if applicable: boundary analysis, decision tables, transition diagrams, soap opera testing, ... Eye on the ball, Scope Always know the scope of the test module Th h ld b biThe scope should be unambiguous The scope determines many things: − what the test objectives are − which test cases to expect − what level of actions to use © 2014 LogiGear Corporation. All Rights Reserved − what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)
  • 38. 2/4/2014 36 State your Objectives . . . ... TO-3.51 The exit date must be after the entry date ... test objective TO-3.51 name entry date exit date enter employment Bill Goodfellow 2002-10-02 2002-10-01 © 2014 LogiGear Corporation. All Rights Reserved enter employment Bill Goodfellow 2002-10-02 2002-10-01 check error message The exit date must be after the entry date. Grail 3: Specification Level, choosing actions Scope of the test determines the specification level As high level as appropriate, as little arguments as iblpossible − Use default values for non-relevant arguments Clear names (usually verb + noun usually works well) − to standardize action names: standardize both the verbs and the nouns, so "check customer" versus "verify client" (or vice versa) − tests are not C++ code: avoid "technical habits", like mixed case and (worse) underlines © 2014 LogiGear Corporation. All Rights Reserved Manage the Actions Document the Actions By-product of the test design
  • 39. 2/4/2014 37 Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function • the "#" means an expression, in this case a variable • the ">>" assign to a variable for use later on in the test key key navigate F7 key navigate 3y g page tab locate page tab Scan Criteria w indow wait for controls loaded search text check breadcrumb general functions > search w indow control value select search scan direction Backward © 2014 LogiGear Corporation. All Rights Reserved w indow control value enter value search business date match # bus date source control click search go w indow wait for controls loaded search results w indow control variable get search results sequence number >> seq num Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function • the "#" means an expression, in this case a variable • the ">>" assign to a variable for use later on in the test variable get sequence number >> seq num © 2014 LogiGear Corporation. All Rights Reserved
  • 40. 2/4/2014 38 An example test for the Modeler model name arguments formulamodel name arguments formula create model vegas winner x 10*x argument value set argument x some money model name expected run model vegas winner a lot more money © 2014 LogiGear Corporation. All Rights Reserved run model vegas winner a lot more money Too detailed? Step Name Description Expected step 16 Click the new formula button to start a new  calculation. The current formula is cleared. If it had not  been save a message will show step 17 Enter "vegas winner" in the name field The title will show "vegas winner" step 18 Open the formula editor by clicking the '+'  button for the panel "formula editor" The formula editor will show with an empty  formula (only comment lines) step 19 Add some lines and enter "10*x;" The status bard will show "valid formula".  There is a "*" marker in the title step 20 Click the Save formula button The formula is saved, the "*" will disappear  from the title step 21 Open the panel with the arguments by  clicking the '+' button There two lines, for 'x' and 'y' © 2014 LogiGear Corporation. All Rights Reserved g step 22 Click on the value type cell and select  "currency" A button to select a currency appears, with  default USD step 23 Click on the specify argument values link The argument specification dialog is shown
  • 41. 2/4/2014 39 Another example TEST MODULE Order processing start system TEST CASE TC 01 Enter an order for two articles user password login jdoe testdesign window check window exists welcome order id cust id priority create order AB123 W3454X normal © 2014 LogiGear Corporation. All Rights Reserved order id article amount price add to order AB123 tablet 3 219.95 add to order AB123 case 3 3.19 id total check order total AB123 669.42 . . . Environments, configurations Many factors can influence details of automation − language, localization − hardware i f th t d t t− version of the system under test − system components, like OS or browser Test design can reflect these − certain test modules are more general − others are specific, for example for a language © 2014 LogiGear Corporation. All Rights Reserved But for tests that do not care about the differences, the automation just needs to "deal" with them − shield them from the tests
  • 42. 2/4/2014 40 Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there). Can be a feature in a test playback tool, or something you do with a global variable or setting. "Variations" "Master Switch" Actions, Interface Definitions . . . © 2014 LogiGear Corporation. All Rights Reserved Variation Variation Variation Possible set up of variations linked variation keyworded variation © 2014 LogiGear Corporation. All Rights Reserved Specify for example in a dialog when you start an execution:
  • 43. 2/4/2014 41 Non-UI Testing Examples − application programming interfaces (API’s) − embedded software protocols− protocols − files, batches − databases − command line interfaces (CLI’s) − multi-media − mobile devices Impact is mainly on the automation test design should in most cases be transparent towards the specific testing devices © 2014 LogiGear Corporation. All Rights Reserved − test design should in most cases be transparent towards the specific interfaces Often non-UI automation can speed up functional tests that do not address the UI Multiple System Access Test Modules, driving either one or multiple interfaces Automation Scheme protocol access UI access one or multiple interfaces © 2014 LogiGear Corporation. All Rights Reserved System (part) Under TestAPI access access access database access
  • 44. 2/4/2014 42 Device Testing Testing Host Device Software Under Test Agent ABT Automation Interface I f Device Andr oid © 2014 LogiGear Corporation. All Rights Reserved Info Multimedia: The "Play List" Approach Approach applicable for graphics, videos, sound fragments, etc The test includes "questions": − what the tester should see or hear − like "Are the matching areas blue?" − actions like "check picture" The test tool keeps a "play list" − during the run items are captured and stored − after the run, the tester is presented with the items, d h hi i © 2014 LogiGear Corporation. All Rights Reserved and the matching questions − the tester acknowledges/falsifies − the system stores those passed items − if during the next run the items are the same as earlier passed ones, the tester is not asked again
  • 45. 2/4/2014 43 Performance Testing The topic is complex, but to create tests can be quite straightforward − actions like "generate load <how much>" and "check response time <max wait>" − use one tool to generate load (like JMeter), another to run the "normal" functional test Often performance testing isn't testing, but more close to research − analysis bottle necks and hot spots (for example discontinuities in response times, means buffers are full) − application of statistical techniques like queuing theory © 2014 LogiGear Corporation. All Rights Reserved pp q q g y − how to realistically mimic large scale productions situations in smaller test environments The three controls you can/should address: hardware (equipment, infrastructure, data centers, etc) software (programs, database models, settings, etc) demands (1 second may cost 10 times more than 2 seconds) See also: "Load Testing for Dummies", Scott Barber, gomez.com Organization Much of the success is gained or lost in how you organize the process − part of the teams − who does test designwho does test design − who does automation − what to outsource, what to keep in-house Write a plan of approach for the test development and automation − scope, assumptions, risks, planning − methods, best practices − tools, technologies, architecture − stake holders, including roles and processes for input and approvals − team © 2014 LogiGear Corporation. All Rights Reserved team − . . . Assemble the right resources − testers, lead testers − automation engineer(s) − managers, ambassadors, ... Test design is a skill . . . Automation is a skill . . . Management is a skill . . . . . . and those skills are different . . .
  • 46. 2/4/2014 44 Life Cycles Product life cycles, rather than task life cycles The project planning and execution largely determines when the products are created system development test d l t © 2014 LogiGear Corporation. All Rights Reserved development test automation Typical Time Allocation TEST DEVELOPMENT efforts © 2014 LogiGear Corporation. All Rights Reserved AUTOMATION time
  • 47. 2/4/2014 45 Keywords and ABT in Agile Keywords are suitable for agile projects: − tests are easier to create and understand, in particular for non- programmers th ll t t d l t ith t d f d t il th t h 't− they allow test development without a need for details that haven't been defined yet − automated tests can quickly follow changes in the system under test Action Based Testing in itself is quite agile − focused on products and cooperation − flexible in process, in fact each test module can have its own process − test modules are usually very suitable to drive system development H ABT li hi h l l t t d i f b t © 2014 LogiGear Corporation. All Rights Reserved However, ABT relies on high level test design for best results − identifying test modules − in larger scale projects this may require at least some overall test planning activities that are not necessarily easy to do in a single scrum team Test Development and Automation in sprints product owner team prod owner & team Agile life cycle Test Module Definition (optional) Test Module Development Interface Definition Action Automation Sprint Products Product Backlog Test re-use Automation re-use © 2014 LogiGear Corporation. All Rights Reserved Test Execution User stories Documentation Domain understanding Acceptance Criteria PO Questions Situations Relations Test development Main Level Test Modules Interaction Test Modules Cross over Test Modules
  • 48. 2/4/2014 46 Automated testing in sprints Try keep the main test modules at a similar level as the user stories and acceptance criteria − test modules can double as modeling device for the sprint Aim for "sprint + zero", meaning: try to get test development and automation "done" in the same sprint, not the next one − next one means work clutters up, part of team is not working on the same sprint, work is done double (manually and automated), ... Make sure you can do the interface mapping by hand (using developer provided identifications) © 2014 LogiGear Corporation. All Rights Reserved (using developer provided identifications) − can do earlier, before UI is finalized, and − recording of actions will go better Make sure to have sufficient technical access to the AUT − for timing − for data, conditions etc Fitting in sprints Agreement on the approach: − questions like does "done" include tests developed and automated? − do we see testing and automation as distinguishable tasks and kill tskillsets − is testability a requirement for the software Understand subject matter and testing approaches Automation technology should be readily available Have a strategy for dealing with base and input data M k t ti d t ti t f th l ti © 2014 LogiGear Corporation. All Rights Reserved Make testing and automation part of the evaluations Address tests and automation also in hardening sprints Just like for development, use discussions with the team and product owners to deepen understanding: − also to help identify negative, alternate and unexpected situations
  • 49. 2/4/2014 47 Summary Keywords is one of the techniques for automated testing, in addition to record & playback and scriptingscripting In itself keywords are not a silver bullet, it needs a good approach, careful planning and good organization to be successful © 2014 LogiGear Corporation. All Rights Reserved Keywords can work for GUI testing, but equally well for a variety of other purposes 1. Testing Computer Software, Cem Kaner, Hung Nguyen, Jack Falk, Wiley 2. Lessons Learned in Software Testing, Cem Kaner, James Bach, Bret Pettichord, Wiley 3. Experiences of Test Automation, Dorothy Graham, Mark Fewster, Addison Wesley, 2012 4. Automating Software Testing, Dorothy Graham, Mark Fewster, Addison Wesley Some References 5. Action Based Testing (overview article), Hans Buwalda, Better Software, March 2011 6. Action Figures (on model-based testing), Hans Buwalda, Better Software, March 2003 7. Integrated Test Design & Automation, Hans Buwalda, Dennis Janssen and Iris Pinkster, Addison Wesley 8. Soap Opera Testing (article), Hans Buwalda, Better Software Magazine, February 2005 9. Testing with Action Words, Abandoning Record and Playback, Hans Buwalda, Eurostar 1996 10. QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September © 2014 LogiGear Corporation. All Rights Reserved 10. QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September 2006 11. The 5% Solutions, Hans Buwalda, Software Test & Performance Magazine, September 2006 12. Happy About Global Software Test Automation, Hung Nguyen, Michael Hackett, e.a., Happy About 13. Misconceptions About Test Automation, Hans Buwalda, LogiGear Magazine, April 29th, 2013