Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. An AutomAted testing institute Publication - www.automatedtestinginstitute.com AutomAted ....... SoftwAre teSting . MAY 2009 $8.85 MAGAZINE Is Your AutomAtIon AgIle or FrA Building Flexible Automated Tests gile? For a Flexible Environment BUSINESS CASE FOR TEST AUTOMATION MAKING A PROJECT SPECIFIC CASE AND COMMUNICATING ROI LAYING A FOUNDATION FOR TEST AUTOMATION POSITIONING YOUR AUTOMATION TO LAST May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 1 X MARKS THE SPOT: Building An OBject MAp with XpAth
  2. 2. Ad 2 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  3. 3. AutomAted S t oftwAre eSting May 2009, Volume 1, Issue 1 Contents Cover Story IS YOUR AUTOMATION AGILE OR FRAGILE? 22 In order to create an effective test automation regime, you must be able to bend and adjust to the environment with which you are presented. Lack of flexibility will ultimately lead to broken test automation, therefore, it’s important to learn how to be flexible. By Dion Johnson FeatureS THE BUSINESS CASE FOR AUTOMATED SOFTWARE TESTING 12 Articulate the expected Return on Investment (ROI), and employ strategies and techniques for developing the automated software testing business case for your project. By Elfriede Dustin LAYING A FOUNDATION FOR AUTOMATION 28 Overcoming management resistance to test automation, and laying the groundwork for lasting success. By Randall W. Rice ColumnS & DepartmentS EDITORIAL 4 UP-TO-DATE WITH ATI 32 Scripting Calisthenics Find out what’s happening on the Automated Testing Keep your automation skills in shape with Calisthenics. Institute’s Online Reference. AUTHORS AND EVENTS 6 HOT TOPICS IN AUTOMATION 34 Learn about AST authors and upcoming events. IAST Is Hot! Read this review of Elfriede Dustin’s newest book entitled SWEAT THE TECHNIQUE 8 Implementing Automated Software Testing. XPath Marks the Spot Learn how to create an Object Map feature with XPath. Automated Software Testing (AST) Magazine is an Automated Testing Institute (ATI) publication that serves as a companion to the ATI Online Reference (www.automatedtestinginstitute. OPEN SOURCERY 11 com). Tool Enthusiasts Unite! Small groups are meeting all over to country to discuss For more information regarding the magazine visit: test automation and the open source tools they love. http://www.astmagazine.automatedtestinginstitute.com May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 3
  4. 4. Editorial Scripting Calisthenics by Dion Johnson Test automation is like a resistance training workout program; the more you do, the better and stronger you get. The dilemma with test automation projects, however, is that they are very often fleeting. When a test cycle ends, test automation often doesn’t resume until much later after a stable version of the next release comes to the test team. Other times, a project or contract ends, and the next contract does not utilize your test automation skills at all. Either of these situations could have an automator away from the test automation keyboard for an extended, even indeterminate period of time, which presents To support you in your maintenance, this access to the heavy duty equipment that we a serious challenge. Anyone that is used to article defines a program of Automation previously used. This heavy duty equipment weight training knows that if you stop for Calisthenics. Calisthenics in physical training may include the Application Under Test (AUT) an extended period of time, your muscles are exercises consisting of a variety of simple that we were automating, the commercial begin to atrophy or waste away. Then, if you movements, usually performed without tools we used, and/or framework in which later attempt to resume working out with test automation was the same weight performed. Automation or intensity that you previously If you stop using your test automa- Calisthenics don’t require any of this, but enjoyed, risk you injuring tion skills for an extended period of instead can use some light weight equipment yourself. same applies to The time, they begin to atrophy. including scripting languages, open source test automation. If tools, and freeware you stop using your test automation skills equipment or with limited equipment (pull-up applications. for an extended period of time, they begin to bar, small dumbbells, etc.), that are intended atrophy. You forget basic scripting syntax and to increase body strength and flexibility using Scripting caliSthEnic ExErciSES techniques, and if you try to restart a vigorous the weight of one’s own body for resistance. Automation Calisthenics involve the test automation effort, you risk getting This will allow you to remain in shape, and development and implementation of various injured. Injury in the automation arena is maintain the physical gains that you’ve made. scripting components. seen in the form of failed test automation In the world of test automation, calisthenics Class development and implementation is implementation, loss of credibility, loss are simple exercises that allow automators to a powerful exercise for maintaining scripting of confidence, increased stress due to maintain the automation skills that have been fitness, because it requires a certain level of overwhelming expectations, or even job loss. acquired. comfort with objects, properties and methods. When recessing from an extremely Remaining comfortable with these concepts intense automation effort, the best thing to EquipmEnt is useful, because they are at the core of do is go into maintenance mode, to maintain When we step away from an intensive test several other concepts that are important to the knowledge and skill that you’ve attained. automation effort, we often no longer have successful test automation implementation; 4 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  5. 5. concepts such as object maps, AUT object verification points that allow your scripts to on your automation skills. File manipulation models, runtime objects used by automated identify failures in expected outcomes. The normally involves the instantiation and use of test tools, and the properties and methods most basic assertions may be created using operating system objects, which allows you to native to each of these objects. branching constructs that signal a positive or exercise skills dealing with objects, properties Control flow function creation is negative result, depending on the condition and methods. In addition, since these objects an excellent exercise, because much of being evaluated. To get the maximum burn allow you to check and perform actions on the scripting that takes place during test from this exercise, however, it is best to various operating system components, they automation involves behave much like the use of control an AUT. Also, Exercises flow functions. you can use these After you learn the objects to create and objects associated write information with the AUT, and to simple text the statements in the files, which allows automated testing you to exercise Class Development tool that allow you your automation to manipulate those reporting skills. objects, the main thing that is left for concluSion you to do is control Maintaining test the execution of those statements Control Flow Functions automation fitness is paramount to with control a test automator, flow functions. and with Test Therefore, simple development of Function Creation Automation Calisthenics, scripts that utilize downtime doesn’t if-then-else or have to result in case branching constructs, and for- File Manipulation the loss of skills. On the contrary, next or do-while it could actually looping constructs give you a chance will keep you sharp to tone your skills in this area. so that you will be Developing and even healthier and implementing classes will provide you with implement the assertion from an external more prepared for the next big automation a chance to create functions in the form of function library. Don’t just add a branching effort that comes your way. class methods, but this is not the same type of construct within a script. Create the assertion exercise as the creation and implementation within a function with arguments, and return Many of the exercises that of a function library. This typically requires values. This will allow you to exercise the may be conducted as part of that you organize the library of functions in automation skills associated with AUT Automation Calisthenics are some sort of logical grouping, load the library verification, and error handling. detailed in the Techniques in the script in which it is to be used, and then File Manipulation is an exercise that section of the Automated Testing use the functions in the proper way. These may be deemed the “squats” of automation. Institute’s Online Refence. For are all important actions to many real life Anyone that regularly does physical exercise access to this Techniques section automation efforts, which is why developing knows about squats. A squat is a lower body today visit: function libraries is an excellent exercise. exercise that is touted as one of the best The creation of Assertions is excellent resistance exercises for toning and building http://www.techniques.automatedtestinginstitute.com for keeping automation verification muscle all over the body. File manipulation in skills from cooling. Assertions are basic automated test scripts can have the same affect May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 5
  6. 6. Authors and Events Who’s In This Issue? AutomAted Elfriede Dustin provides the blue print for making a business case for test automation. Elfriede has authored SoftwAre teSting various white papers on the topic of automated software Editor testing, teaches testing tutorials and is a frequent speaker Dion Johnson at various software testing conferences. With a Bachelor of Science degree in Computer Science and over 15 years of IT Contributing Editor experience, Elfriede has been responsible for implementing Jeffrey Lyles automated testing for several projects. She is the coauthor of several books such as Effective Software Testing and Automated Software Testing, and Director of Marketing and Events among her many accolades now graces the Automated Software Testing Magazine with Christine Johnson an excerpt from her latest book entitled Implementing Automated Software Testing. A PUBLICATION OF THE AUTOMATED TESTING INSTITUTE In the lead feature, Dion Johnson provides guidance for keeping your test automation agile in an Agile development environment. Dion has several years of experience in providing IT services to both government and private industry in a manner that has demonstrated expertise in multiple areas of the software development lifecycle. With a Bachelor of Science degree in Electrical Engineering, he has spent much of his professional CONTACT US AST Magazine career as a consultant in the areas of quality assurance (QA), quality control (QC), astmagazine@automatedtestinginstitute.com software process improvements, requirements analysis and software test automation. As a regular conference speaker and presenter, Dion has delivered award winning and ATI Online Reference highly acclaimed presentations at many of the most prestigious industry conferences, contact@automatedtestinginstitute.com including the StarEast and StarWest International Conference on Software Testing, Analysis and Review, the Quality Assurance Institute Conference and Better Software Conference. He also teaches college and business level classes relative to testing and test automation, and has several published articles in various IT publications. ATI and Partner Events April 1 - June 30, 2009 Randall W. Rice is a leading author, speaker and ATI Honors Nomination Period consultant in the field of software testing and software http://www.atihonors.automatedtestinginsti- quality. He has over 30 years experience building tute.com and testing mission-critical projects in a variety of environments and has authored over 50 training courses July 25 - 30, 2009 in software testing and software engineering. He is a Black Hat 2009 USA Briefings and popular speaker at international conferences on software testing and is also publisher of The Software Quality Trainings http://www.blackhat.com/ Advisor newsletter. He is coauthor with William E. Perry of the book, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems, published by Dorset House Publishing Co. Randy also serves on the board of directors of the American Software Testing Qualifications Board (ASTQB). You can contact Randy at rrice@riceconsulting.com, or from his website at www.riceconsulting.com. 6 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  7. 7. May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 7
  8. 8. Sweat the Technique XPath Marks the Spot Building An Object Map with XPath As you probably already know, an Object Map is a file used in test automation to maintain a logical representation of each Application Under Test (AUT) object. This file serves the purpose of separating application object property data, known as the Physical Description, from the automated script. Without an Object Map, referencing an object in a script would be done using the physical description of that object, similar to the following statement: GetElement(“Class:=Button”, “Text:=O Button”).Click The Class and the Text properties make up the physical description of the object represented information. In essence, XPath queries for across and porting to various platforms. In in the above statement. With an object map, information in an XML document much like addition, information storage and retrieval the same object would be identified using a SQL queries for data in a database. Why use become standardized. Following are steps for Logical Name as in the following statement: XML? XML is written in a text file, which building an Object Map: simplifies data storage, and simplifies sharing • Identify the basic types of objects the GetElement(“OvalButton”).Click repository will support <Object_Map> • Create the XML Object Map structure This separation helps increase an automated • Create the utility script(s) that use(s) <Object logicalName = “UserName”> test framework’s robustness, because it is XPath to read the repository <propertyClass>textbox</propertyClass> likely that an object is referenced in several • Update automated library functions to <propertyName>id</propertyName> locations among the automated test scripts <propertyValue>userID</propertyValue> reference the XPath utility script(s) within that framework. If the properties of the </Object> object change in the application, an Object <Object logicalName = “Password”> Step 1. Identify the basic types of objects the Map ensures that this change only needs to <propertyClass>textbox</propertyClass> repository will support be reflected in a single location as opposed <propertyName>id</propertyName> The object types will largely depend to every place in the automated test scripts in <propertyValue>pass</propertyValue> on the technology of the application being which the object is referenced. </Object> tested. It’s probably a safe bet, however, Not all tools come with an Object Map when automating graphical user interfaces feature, however, so reaping these benefits <Object Name= “Login”> (GUIs), to begin with the following list of <propertyClass>button</propertyClass> often requires that you build your own. An objects: <propertyName>value</propertyName> excellent way to accomplish this is by using • Textbox <propertyValue>loginButton</propertyValue> XML and XPath. XML – extensible markup • Listbox </Object> language – is an open standard for structuring • Checkbox </Object_Map> information. XPath is a language used to • Radio Button navigate through elements and attributes in an XML document in order to find specific Figure 1: XML Object Map You can build from this list of objects as 8 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  9. 9. necessary. » <PropertyValue> – Defines the value of Essentially, we want to be able to query Step 2. Create XML Object Map structure the property for the object by its Logical Name, and then You have a lot of freedom regarding how you Note: This particular set of rules is most have the property class, name and value of choose to create the structure, since you’ll effective when one property is defined that object return. To accomplish this, we also be responsible for writing the script that per object. There are many approaches to need to understand and use basic XPath will interpret that structure. The XML Object defining an Object Map, but for simplicity syntax as it applies to our specific Object Map discussed in this article will comply this is the definition that we will use in this Map. Following basic XPath statement that with the following set of rules: article. With these rules, the Object Map may will be used. • XML must contain one element – the appear shown in Figure 1, which defines root element – that is the parent of all three objects: //Object[@logicalName=‘Login’]/propertyClass other elements. This element may be • A UserName textbox object that is called <Object_Map>. defined in the application by an id property The //Object portion of the statement will obtain • Each object in the Object Map is defined that is equal to “userID” all Object elements in the XML document. within an <Object> element. This • A Password textbox object that is Adding [@logicalName=‘Login’] will obtain element is a child of the <Object_Map> defined in the application by an id property all elements in the document that have a element, and it has an attribute known as that is equal to “pass” logicalName attribute that is equal to ‘Login’. ‘logicalName’ which describes the name • A Login button object that is defined in Finally, the /propertyClass portion will obtain all that will be used to reference the object the application by a value property that is propertyClass elements that are children of in a script. This name should be unique, equal to “loginButton” elements that have the ‘Login’ logicalName to prevent conflicts with other objects. attribute. Ultimately, this statement would • Each Object will have three child Step 3. Create the utility script(s) that use(s) return “button” from our file. To return the elements that will collectively define a XPath to read the repository propertyName and propertyValue similar single object property: This is where a little scripting know- statements are constructed. » <PropertyClass> – Defines the type of how and knowledge of XPath come in handy. The way you use these statements in object being used (i.e. textbox, listbox, While this article will not be a full-blown your code, will largely depend on your etc.) XPath tutorial, we will discuss some of the platform and scripting language. If using » <PropertyName> – Defines the name of basics. Ruby in a Windows environment, you may the property have a function similar to the one shown in 1 def parseXML(objName, objMap) Figure 2. The method shown in the figure, 2 called parseXML has two arguments: 3 xmlDoc = WIN32OLE.new(“Microsoft.XMLDOM”) • objName – The logical name of the 4 xmlDoc.async=”false” object in the Object Map 5 xmlDoc.load(objMap) • objMap – The full file path of the Object 6 Map XML file. 7 #’’’Write Batch File Lines for 1st Environment 8 Lines 3 through 4 load the XML file so that 9 propertyNameNode = xmlDoc.selectNodes(“//Object[@Name=’” + objName +”’]/propertyName”) it may be queried, while lines 9 through 11 10 propertyValueNode = xmlDoc.selectNodes(“//Object[@Name=’” + objName +”’]/propertyValue”) obtain the propertyName, propertyValue and 11 propertyClassNode = xmlDoc.selectNodes(“//Object[@Name=’” + objName +”’]/propertyClass”) propertyClass nodes of the desired object 12 (based on the logical name assigned to 13 the objName argument). Lines 14 through 14 propName = propertyNameNode.item(0).text 16 obtain the actual text associated with 15 propValue = propertyValueNode.item(0).text the propertyName, propertyValue and 16 propClass = propertyClassNode.item(0).text propertyClass nodes, while lines 18 through 17 19 return this text in the form of an array. 18 propData = [propName, propValue, propClass] Therefore, if we assigned “UserName” 19 return(propData) to the objName argument, and set the objMap 20 end argument equal to the path of our Object Map, this function would return the following Figure 2: XPath Utility Script (in Ruby) array: (Continued on page 31) May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 9
  10. 10. How Do You Network? Whatever your choice, the Automated Testing Institute is there! YouT Twitter Facebook Myspace Stay up-to-date with test automation by following the Automated Testing Institute on your so- cial networking site of choice. Facebook Myspace Twitter YouTube LinkedIn Blogger For more information, visit http://www.networking.automatedtestinginstitute.com 10 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  11. 11. Open Sourcery Tool Enthusiasts Unite! Open Source Meetups For The Love of Tools If you’ve been tuning into the Automated automator, announced on his twitter site, Testing Institute’s video news series entitled http://twitter.com/jasnow, that he’d organize a “This Week in Automation”, you’ll recall similar event on March 7th called the Atlanta that we’ve been reporting on various open Open Source “Test Automation Shareathon, source meetups that have been taking place aka AOSTAS. But while the “Love-In” took all over the country, beginning with an event place in California, the “Shareathon” took known as OSTATLI, conducted by Ms. place in Atlanta, Georgia. Elisabeth Hendrickson on February 5th, in Now, Frank Cohen, a “leading authority Pleasanton, CA. OSTATLI, which stands for for testing and optimizing software developed Open Source Test Automation Tools Love- with Web, SOA, AJAX and REST designs In, was created for the purpose of allowing and implementations” will lead a slew of participants to express their “mutual love of open source meet-ups starting in late April open source test automation tools”. in Texas and concluding in early June in Las could continue with an informal conversation According to Hendrickson’s blog, the Vegas, Nevada. For more information on about “geek stuff with a beer in our hands”. meeting was sort of free form, relying on these meetings, visit http://www.pushtotest. From the accounts of the meeting from sticky notes, and small groups to set and com/services/workshop. participants - including Hendrickson’s carry out a spontaneous agenda. Later in the The open source meeting craze seems account on http://www.testobsessed.com, day the group gathered for a demo conducted contagious! Get your shots, or you might be and Dave Liebreich’s account on his blog by meeting participant Ken Pier, Product and the next one to conduct a meeting. site, http://www.daveliebreich.com/blog - the Quality Assurance manager at Socialtext, meeting seems to have been well received. To see the “This Week In highlighting the test harness that his company has built around the Selenium open source That gathering also seems to have been Automation” video series and successful at touching a nerve and inspiring other automation news, visit: automated test tool. After that demo a second some other folks to conduct similar events. demo was conducted, then the meeting was http://www.news.automatedtestinginstitute.com Al Snow, self described tool developer and W moved to a local pub in which the participants hat are the SkillS NeceSSary for aN automated teSt ProfeSSioNal? http://www.abok.automatedtestinginstitute.com May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 11
  12. 12. The Business Ca Automated Soft Testing By Elfriede Dustin 12 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  13. 13. ase for tware As discussed throughout our book Implementing Automated Software Testing (IAST), Addison Wesley, March, 2009, there are many benefits to implementing Automated Software Testing (AST). Chapter 2 of IAST discusses in detail the why’s of automated testing, i.e. “Why Automate?” If applied correctly, those automated testing benefits can include reducing the time and cost of software testing, improving software quality, and many other factors that will positively impact your software test program. Understanding these potential benefits and being able to explain them are important when introducing the idea of AST to your management and test teams. Additionally, it is important to have buy-in from upper management and the various stakeholders affected in order to secure the funding required for AST. It is important to be able to articulate for your particular project the specific business needs that will be met, the expected Return on Investment (ROI), and any other considerations beyond ROI. This article is an excerpt from the IAST book and is focused on strategies and techniques for developing the AST business case for your project, so that you can gain the necessary buy-in and support required. May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 13
  14. 14. Definition of the Business Case Various definitions of business case ...large-scale AST exist, much has been written about how to write a business case, and various related projects are much more templates are available. A quick search on successful when the team Google for the term brings up thousands of references. The purpose of this discussion is has thought through not to redefine what a business case is, but to use existing definitions to develop a business the specific areas and case for AST. Our definition is a combination reasons for of the existing ones. To summarize, the purposes of a business case are to implementing AST... • Identify the business needs—provide the reasoning for initiating a project or task, in this case AST • Justify the business needs in terms of costs and benefits—calculate the ROI • Include the expected risks tests will soon be running. The team then justify them in terms of cost and benefits, An example of a summarized business identifies the tests that look like the easiest to we should ask ourselves a few questions, to need for AST could be that an AST program automate so they can run some with the new narrow down our specific needs. For example: will support the timeline decrease of the automation tool, and through hard work and • Do we want to reduce the lifecycle cost software testing lifecycle, facilitate defect perseverance they get some tests automated. of a software product? identification, and thus help improve Then the questions come: “Now that we have • Do we want to deliver software products quality. Naturally, improved quality through these tests automated, how much will this with fewer defects, and have extremely minimal defects leads to improved customer save us in the future? How much better testing high confidence that certain functionality satisfaction. did we do?” The answers are disappointing or capability is working as expected; for In our experience, large-scale AST because the tests that were automated were example are some tests nearly impossible projects are much more successful when the selected because they were easy, not because to run manually, requiring AST? team has thought through the specific areas they have a big impact on the test program, • Do we want to remove the mundane tasks and reasons for implementing AST, is able to or they result in minimal savings because for members of the testing team, whose describe the expected benefits and costs, and they are not even run very often because they skills are best applied on more complex has also taken into account what is needed cover low risk areas. The consensus becomes areas of the AUT than rerunning manual to be successful in terms of time, skills, and “We tried AST once but it didn’t really make tests over and over? processes. Additionally, it is important that much of a difference for our project.” The upper management and other stakeholders team blocks existing or future automation Most often the answer to these questions buy in to the business case associated with because of a bad experience based upon the is “All of the above!” We want to deliver the AST effort. wrong ingredients and false starts. faster, at a lower cost, with fewer defects On the other hand, too often we see We have found that spending the time and higher confidence and at the same type projects that start out with the purchase up front to identify how AST can support a increase test team efficiency. In this scenario, of an automation tool without any proper project’s goals and what the expected benefits then, the consideration of how to best apply automation skill acquisitions or real analysis are to be an important step that is often left AST to a project should take into account all of what the tool can reasonably be expected out. The process of developing the business areas that contribute to each of these factors, to accomplish, other than speeding up case is a discipline we highly encourage such as testing. No thought has been given to how when introducing AST to your project. • The need for speeding up the testing this “speeding up” is to be accomplished. The effort and increase the testing efficiency. automation tool is provided to a team that has Identifying the • The need for decreasing the testing cost little or no experience or training with the Business Needs • The need for applying test team member tool, with an expectation that some automated As we identify the business needs and skills most effectively 14 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  15. 15. Need for Speeding Up the Testing Effort efficient testing team can’t be successful if analyzed carefully and AST is applied to the and Efficiency there isn’t enough time to implement and appropriate areas. How much would speeding up the execute the test program’s best laid plans. testing really impact the product delivery Test program activities should be The need for applying test team member timeline? If a particular test activity is not tailored to your business needs and your skills most effectively really in the critical path, automating it will specific project. Then, once you complete Chapter 10 in the IAST book talks about probably not impact delivery time much. On an inventory of the test activities and assess putting the right people on the project and the other hand, if a test or set of test program them against your business needs, you can describes the skills required of members of activities can be identified that are truly on provide an objective recommendation of the an effective test program. Manual testing the critical path to improving a product benefits of automation areas, which phases can be tedious, cumbersome, error prone delivery timeline, automating it, can improve of the Software Testing Lifecycle (STL) to and very mundane. It can be exciting, when software delivery time. For example, a test automate, which specific tests to automate, a new software release of the Application setup activity, i.e. restoring a test baseline and why you selected those areas and tests, and Under Test (AUT) is received the first time setting up the required pre- and testers start testing it to conditions for a test are For calculating the business benefit, find defects. This might still very time-consuming, but be interesting to do when automating these activities receiving the second and would reduce this activity you want to determine how much third release, but running by 60 percent. Using these the same mundane manual types of “potential time testing currently costs. tests over and over on the savings” metrics allows fourth and nth AUT release for an assessment of the can become boring very potential impact on schedule of reducing an expectation of the impact. In most cases, fast. During the fifth or later releases, often the execution time and can be used for the test activities that contribute to multiple testers decide they tested an area previously developing the business case. Generally we business needs are the ones that are most and thought that certainly it will still work the can speed up the testing effort and decrease beneficial to automate. same and they don’t repeat the test again on cost if we increase the testing efficiency. the nth release. This decision to not rerun a When assessing the testing efficiency, various Need for Decreasing the Testing Cost test often can be detrimental to the quality of questions should be asked: What category If testing is sped up and efficiencies are a product and that’s how often times defects of software defects is most important to increased, one of the resulting benefits can be slip into production. AST takes away this minimize? Which set of test activities, tests a decrease in testing cost. For calculating the guesswork as to whether to rerun a test and is and test techniques have proven to be the business benefit, you want to determine how ideal for this type of mundane, cumbersome most important in discovering these types much testing currently costs. If you break and tedious testing. Test team member of software defects? Which are the tests that down the cost of testing into the various skills are best applied in the more complex are critical to be able to consistently repeat phases such as unit testing, component areas, while the automated tests play back and also likely to be the ones you will want testing, system testing, and other testing the mundane tests behind the scenes. AST to run most often, because they cover a phases discussed in Appendix B in the IAST can help improve tester morale, because not complex, high risk or problematic area and book, which phase costs the most? Which only can they learn new AST skills, but also, have the tendency to uncover most defects? tests add most value, i.e. cover the highest they can focus on new or more interesting Also consider the tests that manual testing risk areas and absolutely have to be run? In activities, rather than rerunning a manual test can’t hardly achieve, because they are so other words, are the tests that cost the most for the nth time. complex, such as memory leak detection, to execute adding the most value? Of these concurrency, performance, and so forth and tests, which ones are repeated the most often? Justifying automation is needed. Consider what would How much time and effort are required to Automation in Terms the impact on software quality be if they were run these tests? What would be the impact on of Cost and Benefits run more often as overnight regression tests? confidence in the software delivery if these Estimating ROI Do we have the right skills on board to allow tests could be run more often because they As discussed in the previous section on for testing efficiency? Are we finding defects are now automated and the test coverage assessing business needs, estimating the early enough or too late in the testing cycles? from these tests could be expanded? Testing potential cost savings is a critical part of Do we have adequate schedules? The most cost can be decreased if the test activities are evaluating and articulating the impact of (Continued on page 18) May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 15
  16. 16. The Automated Testing Institute (ATI) Online Reference is the web’s foremost comprehen- sive online resource for obtaining, exchanging and certifying industry standard test automation knowledge, information, and techniques. Visit us today!
  17. 17. (Continued from page 15) implementing AST. This section describes over numerous days to do some of the AST measure the impact of increased test coverage how to calculate the potential cost savings or activities. Additionally, those test personnel if using different data but, for example, using ROI when implementing AST in comparison can focus on additional testing tasks or the same data paths, resulting in different to performing the same tests manually. projects which is an additional ROI, but often scenarios. Here we need to evaluate whether ROI accounts for not only the time savings too difficult to put into an actual metric that running additional test data scenarios really from automation but also the investment it shows the gains. will benefit the testing effort or if we are just will take to implement AST to give you a duplicating the same test path without adding complete picture. Lab Equipment Requirements any value—ignoring the rules of equivalence AST ROI can also be non-tangential, i.e. Also, as part of AST ROI considerations, partitioning, for example. Note: Wikipedia automation pays off for areas of the test you should consider additional equipment has a good definition of Equivalence program that are difficult to be described in required, considering the most efficient Partitioning: Equivalence partitioning is a metrics and numbers, but are simply more way to support AST, such as evaluating the software testing technique in which test cases beneficial to automate. Some of these areas feasibility of implementing the test lab using are designed to execute representatives from are described in Chapter 3 of the IAST book. VMWare or other OS image/virtualization each equivalence partition, i.e. a partition of For the tangential areas, we have developed software. input values undergoing similar treatment. an ROI “calculator” that allows for entering In principle, test cases are designed to cover all the factors needed to determine AST time Test Case Maintenance each partition at least once. This has the goal: and cost savings. As presented in the IAST So far we’ve discussed original test case book, the ROI calculator is broken down into setup, but test case maintenance should also 1. To reduce the number of test cases to a a set of worksheets focusing on each of the be included as part of the ROI considerations. necessary minimum. major activities associated with testing: When evaluating AST ROI, it is also 2. To select the right test cases to cover all important to compare test case maintenance possible scenarios. • test setup, for the manual versus automated testing • test development, efforts. How many data combinations to run • test execution, and through the same data paths is a strategic • test evaluation. Intangible Automation Savings decision, depending on complexity and Some intangible or implementation criteria. An additional Each of these categories are described in difficult-to-measure automated difficult non-tangential benefit is detail in the ROI section of the IAST book. ROI can be achieved. For to measure automated test instance, it is difficult success that manual Other ROI Considerations to measure the testing couldn’t Personnel Requirements impact on an achieve, i.e., When calculating the AST cost savings, automated tests that were keep in mind that different skills are required testing previously to implement an effective automated testing not program from those required for manual testing. The cost of possibly training existing personnel or the cost of hiring qualified AST engineers (aka software developers) needs to be considered and included in the ROI considerations. Chapter 10 in IAST discusses the skill set required for implementing an r u n effective automated testing program. effort when because they During this ROI consideration also compare test coverage would have the number of people required for manual is increased been too cost- versus automated testing efforts. Although via automated prohibitive or almost you might have to hire software developers to test data generation impossible to execute, implement the AST tasks, it might be possible and increased test scenario such as memory leak detection, to retrain the testers who run the various coverage that only automation stress testing, performance testing, labor-intensive manual regression tests can achieve. It is difficult to concurrency testing, and so forth. ROI 18 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  18. 18. needs to be added for a high priority defect More ROI Metrics found using AST that a manual test could Once automated testing has been implemented, another useful ROI metric not have uncovered because one didn’t exist. Additionally, some of our customers tell us to consider is the cost savings calculations for defects detected via automation that currently they don’t run a regression test versus defects detected via manual testing. This, however, is another challenging as often as they would like to, because it’s metric to calculate, because most often defects are uncovered during the labor-intensive and a lot of setup is involved. automated test creation—one could argue that the test uncovered a defect as However, once the regression test has been part of manual testing (tests were still being set up), not as part of an automated automated, they want to run the test more test run. often. How do we measure “peace of mind”? However, as mentioned previously if automated testing adds test cases that manual testing didn’t cover before, and these uncover additional defects, this Risks could be a valuable ROI metric to track. Automated test program assumptions, Generally it is difficult to compare defects found in manual versus prerequisites, and risks need to be understood automated testing, because we have to be sure to compare apples to apples. To and presented as part of the business case. calculate an accurate comparison, we would have to compare the same software These include any events, actions, or component in the same software testing stage running automated and manual circumstances that will prevent the automated tests in parallel to determine the difference automation would make. Only this test program from being implemented or executed successfully, such as lack of skills, parallel effort could truly show the defects the manual effort uncovers versus late budget approvals, the delayed arrival of the ones the automated tests uncover. test equipment, fluid and constantly changing Additionally, it is a good practice to track the savings produced by detecting requirements and software, or late availability defects during automated testing, especially if defects are uncovered early in of the software application. the testing lifecycle, as opposed to the savings over discovering the same defect The risk description needs to incorporate later or even in production. Different weights can be assigned when developing ways that help minimize it and prevent this type of metric; i.e., a high-severity defect uncovered by automation gets a failure. Failure is defined in terms of cost greater weight than a low-severity defect and so forth. overruns, schedule slippage, undetected critical software errors, rework required, and so on, despite the automated testing effort. Most risk is caused by a few factors. • Short time-to-market - Risks include a short time-to-market schedule for not available in-house. The limited tested because of complexity and lack of the software product. As mentioned availability of engineering resources on documentation will also pose a high risk previously, testing budgets and schedules the project would be considered another to an AST effort. are often determined at the onset of a potential risk to AST success. project, during proposal efforts, without • New technology - If new technology is As risks are identified, they need to be any “real” or “actual” estimates or inputs implemented, the risk can be high that assessed for impact and then mitigated with from testing personnel, based on past no automated testing tool exists and the a strategy for overcoming them should they experiences or other effective estimation tool needs to be developed in-house. be materialized as part of the business case. techniques. However, a test manager • Difficult to automate requirements Carefully examine potential risks in order can quickly see when the short time- or features - Some system features to try to prevent them from being realized; to-market schedule does not allow for exist that are not documented, are hiring people with the right skills for the adequate testing. As part of the business complex, and most often are the job, reassessing the deadline or listing all case, this high-risk issue needs to be biggest problem area. Learning curves the requirements needed in support of a pointed out. Either schedules should be of the functionality, subject matter successful AST effort, are ways to mitigate adjusted, or risk mitigation strategies expert support, and documentation those listed risks. However, if risks are should be developed. requirements need to be considered, realized even though all precautions have • Lack of adequate skills - Other risks before claiming a requirement or feature been taken, a mitigation strategy needs to be may exist because new skills are required can successfully be tested in an automated in place. to implement the automated testing fashion. Functional or nonfunctional Other Considerations effort and currently adequate skills are requirements that are difficult to be In earlier sections we described the benefits May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 19
  19. 19. that AST can offer, ideas for how to evaluate So far we’ve discussed the importance business needs and assess AST against those of a business case and addressed how to ATI Newsletter needs, and estimating ROI. This section lists identify the business needs, how to justify other ways to approach the topic and other AST in terms of cost and benefits, and how questions to consider in relation to AST. to identify risks. The strategic approach to AST may be the solution if you don’t have an implementing AST and measuring the key answer to some of the following questions: success indicators also needs to be defined 1. How will your test team keep up with as part of the business case. Strategy is the ever increasing complexity of new discussed in Chapter 6 of “IAST,” and the technologies implemented? keys to success and metrics are discussed 2. How will your test team’s productivity throughout this book. keep up with the rate of productivity improvement expected from the Summary development team? In order to get buy-in from upper The ATI Newsletter keeps 3. What is your testing strategy to support management and all stakeholders involved, you abreast of the latest the increasing level of software and the business case for AST needs to be made. component reuse that is planned? Our experience has shown that if buy-in information relative to our 4. What artifacts can you provide to exists (i.e., all stakeholders have bought into Online Reference. Sent to demonstrate the software was tested to the automated testing effort), all parties feel registered members of the meet specified standards? How quickly responsible for the success of that effort. If site on a bi-weekly basis, can those be produced and reproduced all contribute, chances for success are much this newsletter keeps you for each test run? higher. It is therefore an important goal to 5. How do you capture and reuse the develop the business case and get approval. informed on the follow: knowledge of the subject matter experts The market demands that software products your test program currently depends continue to be delivered faster and cheaper ATI’s Featured Content on? with increasing reliability. Testing is a big • Featured Videos 6. What is your plan for being able to part of the time, cost, and quality equation, • Featured Tutorials document and re-create problems found and without changes in how testing is done, • Featured Articles during testing for a development team software projects will not be able keep up. • Featured Tools that is geographically distributed? In fact, there is much evidence that this is • Featured Publication 7. When the software product can be already the case. AST can play a big part in • Featured Poll installed remotely, what is your addressing these needs. More specifically, • Featured Forum Post approach to providing verification or AST provides an opportunity to reduce the diagnostic tests that can be executed time and cost of software testing, improve Popular Content remotely? software quality, helps identify defects • Popular Tools 8. How will you run tests that manual earlier in the development lifecycle, and • Popular Articles testing can hardly accomplish, such as improve your software test program in memory leak detection, stress testing, measurable and significant ways. Automation OMG! performance testing, and concurrency In order to implement AST successfully, • Interesting News testing? an approved business case can set the right • Current Events 9. Will you need to justify having testers tone. We are now off to a good start. work night shifts, weekends, and overtime when an automated tool can Register on the ATI site today The Automated Testing Institute just be kicked off at day’s end and to receive this email newsletter. (ATI) conducted a podcast interview results viewed the next morning? To register visit: with Elfriede Dustin, where IAST was discussed. To listen in, visit Other such questions can be added, registration.automatedtestinginstitute.com and in considering these questions and the http://www.podcasts.automatedtestinginstitute.com. future of your test program, AST should be evaluated as part of the potential solution. 20 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  20. 20. Real Adventures in Automation Submit Your Stories! Now’s Your Chance to be Heard Hav e so me thin g to say ? o listen ATI i s here t Ad The Automated Testing Institute believes that one of the best ways to improve upon test au- tomation is to listen to the lessons learned by others. So, if you have an interesting story that you’d like to share with the world, contact us at stories@automatedtestinginstitute.com to find out how. May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 21
  21. 21. Is Your AutomAtIon AgIle or FrA gile? Building Flexible Automated Tests For a Flexible Environment By Dion Johnson 22 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  22. 22. Recently, I came across a question posted on every action there is an equal and opposite an online message board that asked whether or reaction”. I bet you never thought you’d have not test automation was possible in an Agile to dust off that physics book to understand environment. I suggested that success would test automation, did you? This law, known as be tied to the ability to effectively customize Newton’s third law of motion is useful, however, the automation approach to the environment in in explaining our current dilemma. The frivolous which it was to be implemented, and therefore nature with which many consistently treat test may result in a framework that is scaled down automation is an extreme action that is naturally in structure. This didn’t really sit well, however, met with the opposite and equally extreme reaction with those that believe a heavily structured of trying to apply an intensive test automation and extremely robust framework is the answer framework to every test automation situation. It’s to every automation question. It’s not hard to like a tug of war in which no one wins, because understand why this belief is held, though; it’s a what many automation loyalist don’t realize is direct result of the fact that test automation is too that both of these actions are prone to failure. often treated as an afterthought. It is relegated to a spare time event that calls for no planning or Don’t get me wrong, I am a strong advocate for skills beyond the ability to record manual structured test automation frameworks, and push test execution; and this treatment has for them whenever possible, but I am also a realist. led to failure in test automation And reality lets me know that there is no one- implementation time and size-fits-all solution to successful test automation time again. Therefore, implementation. There are basic principles that the response by many can be applied in every situation, but you need automation professionals to be flexible in your approach, both in an Agile may be summed up environment and outside of an Agile environment, by the law of because project constraints always require you physics which to bend and adjust what you plan on doing. suggests Lack of flexibility in your automation approach that “to will ultimately lead to that approach breaking. I’ve worked in a couple of Agile projects that have used the Scrum methodology and even took the “Scrum Master” training. Why they call it “Scrum Master”, I don’t know, because I definitely didn’t feel like a master when it was over. I understood it well enough to do my job, however. May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 23
  23. 23. any circumstance, but it can be extremely Sprinting and ScrumMing difficult in Agile environments. This is because many people still don’t Scrum involves the compilation of a prioritized list of features see the value of actually breaking up planned for a product Increment. This compiled list, known as the automated framework development and Product Backlog, is evaluated at the start of each Sprint – a timeboxed implementation into tasks that can be period of software development that is often 15 or 30 days – and the tracked and burned down on a daily basis. And if it can’t be burned highest priority items that are deemed attainable are transferred to down on a daily basis, it’s what is known as the Sprint Backlog. Project members form difficult to justify having time a Scrum Team, often consisting of 5 to 9 people, and in a Scrum environment devoted the team breaks down the prioritized functionality to getting it done. into detailed tasks. A short meeting is conducted • Difficulty Securing at the beginning of each day to obtain progress Time – Even if you are able and set priorities for that day. If work is to get buy-in for a highly not completed at the end of the Sprint, it structured test automation is placed back on the Product Backlog framework, it is difficult until the next Sprint comes along. The to secure the time that it entails, point of all this is to produce constant, particularly frameworks for measureable results, and to facilitate change. system test automation. Test automation frameworks require an up- front investment in time as well as money. The purpose of this article is not to assert or members are constantly The investment is normally paid back in deny the merits of Agile or Scrum. That’s for shifting, activities a positive return on investment, but the others to do. Instead, this article attempts to occur within a small window of investment is necessary, none-the-less. address how automation may survive in a time (timeboxed), and progress is The investment is spent in analysis, rapidly changing environment such as one constantly checked and measured. design, development, documentation that uses Agile. This presents the following and implementation, and there is a good The “Sprinting and Scrumming” challenges for highly structured chance that you will not have much time summary provided above tells us something test automation frameworks: for each of these items. In addition, you very important about Scrum and Agile may also be assigned other project tasks environments, and that is that everything • Difficulty Obtaining Buy-in – that need to be completed in concert is extremely dynamic. Priorities are set Obtaining buy-in for an effective test with the automation tasks, resulting in and changed often, teams are small and its automation program is challenging under the automation tasks getting pushed by the wayside. • Difficulty Sustaining Participation – It is very difficult getting agreement for a dedicated automation team, yet the project team is expected to have multiple people involved in automated test creation and implementation. Each person that will need to implement test automation has a million and one other daily tasks to consider, so it’s highly probable that they are not going to be interested in learning the framework that you’ve developed, Figure 1: Scrum Process (Wikipedia) obtained or customized. 24 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009
  24. 24. While these challenges are not specific all 700 users were successfully created within training for the team on how to use it in a way to Agile, they do seem to get amplified in an 2 hours, so not only did the process take that would yield useful reports. Agile environment. Two common responses less than a full day, the other two resources This new set of priorities provided to these challenges are: didn’t have to suffer through manually me with some freedom and time to focus • Throw up your hands and say, “We just won’t automate until the environment stabilizes” ...project constraints always require or you to bend and adjust what you • Dig in your hills and say, “We need to plan on doing. make the framework even more robust to handle the changes.” implementing the repetitive task of creating attention on some functional test automation. users. There were previous attempts by other The first response never works, because Another opportunity presented itself resources to produce automated tests, but the environment never stabilizes. The second upon learning that the team was regularly they continuously failed and they couldn’t response may appear to work at times, but producing a requirements traceability matrix figure out why. I quickly realized it was it’s probably an illusion. Strong frameworks (RTM) manually. The team owned a test because there was no framework in which require strong support from upper level management solution, but it did not produce the tests were created and implemented, management and your project team; absent the RTM in the desired format, when used making the automated tests extremely fragile. that, the framework will probably not be directly out-of-the box. It’s worth noting that I then set out on a mission to create a robust effective. it wasn’t officially called an RTM, because the framework, complete with libraries and a lot project didn’t like the word ‘requirements’. of reusability components, which ultimately Project Scenario The official name was Acceptance Criteria resulted in the scripts lasting longer than This scenario describes an automation Traceability Matrix, but everyone called it one Sprint. In addition, I produced some experience that I had with a Scrum project an RTM. Anyway, since I was hired for tool detailed documentation for the framework so on which I worked. I was hired primarily support, I was able to customize the tool to others could understand and use it. When I for test tool support, but was immediately produce the RTM in the desired format, a presented this to the team, they loved it! They thrown into several other tasks. Although congratulated me on creating lasting scripts, I was focused on several tasks, I kept my and on producing thorough documentation. eyes open for areas that could be supported The problem arose when it was time for by tools. For example, there was a request others to actually start developing automated for assistance in creating test users through a scripts. They had no interest in learning the GUI separate from the application under test guidelines for developing automated tests (AUT). Someone on the team previously tried within the framework, nor did they have to automate the user creation process with the any interest in using the documentation. The functional automated test tool that was on automated test framework was unsuccessful the project’s proverbial shelf. The attempt at being institutionalized into the project, failed because the tool’s record & playback so the automated tests were ultimately still feature couldn’t record actions in the user fragile. creation application. The test team lead, What ended up being more successful therefore, enlisted about 3 people to manually move that was very well received by all. was when the project team got together and create more than 700 users that needed to be Unfortunately, once I began producing the made the decision to slightly segment the created, a task that would collectively take at RTM directly from the test management testers into two groups. One group focused least a full day. Upon assessing the situation, solution, it became clear, based on the useless on the daily Sprint testing and interfacing I found that although the application couldn’t data coming out of it, that the tool was not heavily with development, while the second be recorded, I could still use a combination of being used properly. Since management group focused on big picture validation and Send keys and the application object model was now aware of the usefulness of the test interfaced more with the user. The Sprint to automate the user creation process. The management tool, they decided to allow me focused testers focused on manual testing automation took no more than 30 minutes and dedicated time to set up the tool, and produce of acceptance criteria, while utilizing May 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 25
  25. 25. small, quick and easy to develop functional beginning, my focus was more on the test automation solutions, as it did for automated tests created by developers for broader view of test automation. This me. front-end “unit” testing. The tests weren’t view defines test automation as tool • Reduction in structure – In the name of very maintainable, but nobody cared (at least initially), because the focus was on quick returns. The big picture testers ...making your automation more focused on gaining a broader understanding of the system beyond what was stated in Agile involves expanding your view acceptance criteria, ensuring the customers expectations were of what test automation is. being met and on developing support for all aspects of a test project, maintainability, robustness and a host of maintainable and not just automation of test execution. other quality attributes, we as automators automated tests and Approaching test automation from can sometimes create frameworks that test components that this point of view worked well, aren’t used enough to justify the effort would cover some of because it provided put into ensuring these quality attributes the acceptance criteria quick returns, are met. Sometimes the conventional in a traceable fashion. These which is what wisdom that is used to determine when automated tests would primarily be used Agile a given quality attribute is important, in later Sprints and during FIT (Factory just doesn’t apply. For example, Integration Testing). Again, I must reiterate an organization that is extremely that this article is not meant to assert or dynamic and has many environments deny the merits of this specific approach, thrives in which testing must occur, typically but it is meant to illustrate the importance on. With its calls for a framework with increased that agility has to successful automated test small teams, maintainability and robustness. That implementation. short cycles, didn’t work in the scenario described and daily Scrum in this article. By focusing more on Lessons Learned meetings, Agile is built usability, and reducing the structure, Many might suggest that the team’s lack on receiving regular, the automation was much more useful of interest in a working framework is their small bursts of information for the Sprint focused team. own personal problem, but I saw it differently. and progress. Anything I wasn’t interested in simply displaying my that automation can do to Making Your Automation More ability to create a framework, but instead facilitate that is a plus. Agile interested in creating something that would And as the case study One of the most powerful be effective. So it was important to assess the revealed, this approaches to making your automation above case study to determine what worked approach more agile involves expanding your view and what didn’t. may also of what test automation is. As previously result stated, the broader view of test automation What didn’t work: in more defines it as tool support for all aspects of a • Too Much Structure – I was no stranger freedom test project. Some examples of implementing to being able to evaluate an environment, being given to test test automation in the broader definition are and design a framework that works best automators to explore and as follows: with that environment. Unfortunately, attempt more robust • Data Creation – The creation of test I’d underestimated the resistance that a functional data often involves highly repetitive, highly structured framework would have yet simple tasks. Therefore, it is a great in an Agile environment (at least the candidate for automation. ones in which I worked), • Report Generation – One of the most costly, time consuming overhead items What worked: in IT is the production of reports, yet this • Broader view of automation – In the also is very repetitive. Automating the 26 Automated Software Testing Magazine www.automatedtestinginstitute.com May 2009