Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

  • Be the first to comment

  • Be the first to like this


  1. 1. An AutomAted testing institute Publication - AutomAted ....... SoftwAre teSting . AuguST 2009 $8.95 MAGAZINE Evolution of AutomAtEd SoftwArE tESting evolutionary 5 StepS to Framework roaD Development inTroduCing your ProjeCT How To CreATe A FrAMework To new TeCHnology For MAinTAinAbiliTy August 2009 Automated Software Testing Magazine 1 Give me the key...worD: Building A Keyword driver Script in 4 StepS
  2. 2. Still Testing SAP Manually? Worksoft Certify® Get ahead of the curve and maximize the value of your SAP investments with Worksoft Certify®, the for SAP® only code-free approach to end-to-end, cross-platform business process validation and Automate automated testing. Accelerate Worksoft Certify® for SAP® is a specialized lifecycle management solution for SAP that Optimize automates and optimizes SAP lifecycle tasks from change impact analysis through functional testing, performance testing and audit documentation. Certify for SAP is unmatched in the industry for accelerating time to value and reducing overall project costs by up 40-80%. Are you ready to test smarter and deploy faster?  Contact us at 1.866.836.1773 or visit to learn more.
  3. 3. AutomAted S t oftwAre eSting August 2009, Volume 1, Issue 2 Contents Cover Story Evolution of AutomAtEd SoftwArE tESting 14 Reviewing how test automation and its tools have evolved is useful only if it helps to understand where we go from here. To do that, you must appreciate the technology and market forces that have shaped them and why, and what that portends for the future. By Linda G. Hayes FeatureS EvolutionAry roAd 22 Software technology changes in existing applications often present roadblocks on the road to successful test automation. Learn what steps will help you navigate your test automation over and around these barriers. By J.L. Perlin 5 StEpS to frAmEwork dEvElopmEnt 28 Developing an automated test framework can greatly increase the ROI of your test automation efforts, but it can also be a daunting task. Read this article for a step-by-step framework development process. By Dion Johnson ColumnS & DepartmentS EditoriAl 4 hot topicS in AutomAtion 38 evolve, adapt and Build microblogging is hot! Keeping automation on pace with three modes of change. Test automation meets microblogging. AuthorS And EvEntS 6 i ‘B’log to u 40 Learn about AST authors and upcoming events. Read featured blog posts from the web. SwEAt thE tEchniquE 8 up-to-dAtE with Ati 42 Find out what’s happening on the Automated Testing hand me the key...word Institute’s Online Reference. Learn how to build a keyword driver script. Automated Software Testing (AST) Magazine is an Automated opEn SourcEry 12 Testing Institute (ATI) publication that serves as a companion Contribute to open Source projects in 4 Steps to the ATI Online Reference. Learn how to contribute to your favorite open source For more information regarding the magazine visit: projects. August 2009 Automated Software Testing Magazine 3
  4. 4. editorial Evolve, Adapt and Build by Dion Johnson For anything to survive, it must be able to change itself by three different ways: evolving, adapting and building. This has been evident in the natural world, and it also holds true in the world of software. This reality should be especially clear for software testers and test automators, because the very mission and reason for their existence is to ensure systems are effectively evolving, adapting and being built. This conceptual trilogy is not confined, however, to only the systems tested using automation, but also applies to test automation itself. Evolution is defined as a gradual process to aid in this understanding, this issue of the frameworks. In addition, Hayes describes in which something changes into a different Automated Software Testing Magazine is where the automation evolutionary approach and usually more complex or better form. dedicated to exploring each of these modes has come up wanting, and proposes what the This change is normally an innate response of change. next test automation evolutionary response to some measured environmental stimulus In the cover story, Linda Hayes delivers should be. that occurs over time. Adaptation is the act of what is possibly one of the greatest articles Next, J.L. Perlin provides a featured adjusting oneself to perspective on a different conditions or a shifting ...possibly one of the greatest modern day test automation adaptation environment. This type of change is articles ever on the topic of the test with a peek into a project’s encounter automation evolution typically marked with contemporary by a deliberate and changes to the direct response software environment. to immediate modification in the environment. Building ever on the topic of the test automation In this article, an existing software application is the act of increasing or strengthening, and evolution. Titled “The Evolution of with a substantial suite of automated tests is may be a response to some outside stimulus, Automated Software Testing”, this article faced with changes to existing application but it could also be based on a self-imposed looks at software test automation from a components. These changes threaten to render decision to improve one’s situation. macroscopic point of view and explores the all existing automated tests useless as the The discipline of test automation has gradual changes in software systems over automated test tool contends with problems experienced and been shaped by each of the years. It takes us through a trip down communicating with the new components. these modes of change in the past, and software computing memory lane, moving us This is a problem that many automators have continue to be defined by them even today. from mainframe to Internet domination, and come across, and if you haven’t, there is Understanding these three modes of change explaining how the test automation discipline a good chance that you will. So J.L. Perlin will help to better understand the current state has responded to each evolutionary shift by offers an approach for effectively navigating of test automation, and will therefore help moving from text capture/playback tools your test automation through such modern us all to chart a course forward. Therefore, to the broad use of commercial automation day technological changes. (Continued on page 37) 4 Automated Software Testing Magazine August 2009
  5. 5. How Do You Network? Whatever your choice, the Automated Testing Institute is there! YouT Twitter Facebook Myspace Stay up-to-date with test automation by following the Automated Testing Institute on your so- cial networking site of choice. Facebook Myspace Twitter YouTube LinkedIn Blogger For more information, visit August 2009 Automated Software Testing Magazine 5
  6. 6. authors and events Who’s In This Issue? linda g. hayes, in the lead feature, offers an AutomAted eyewitness account of test automation evolution. With over 20 years of experience in software development and testing, SoftwAre teSting Linda is the founder of three software companies including AutoTester, the first PC-based test automation tool, and Managing Editor Worksoft, Inc., pioneer of next generation code-free test Dion Johnson solutions. Linda holds degrees in accounting, tax and law and is a frequent industry speaker and award-winning author Contributing Editor on software quality. Named one of Fortune Magazine’s People to Watch and one of the Edward Torrie Top 40 Under 40 by Dallas Business Journal, she is a regular columnist and contributor to numerous publications, including the Automated Software Testing Magazine, Director of Marketing and Events and Better Software magazine. She is the author of the Automated Christine Johnson Testing Handbook, contributing author to Testing SAP R/3:A Manager’s Guide, and co- editor of Dare to be Excellent with Alka Jarvis on best practices in the software industry. A publICATIon of ThE AuTomATED TEsTIng InsTITuTE Her article “Quality is Everyone’s Business” won a Most Significant Contribution award from the Quality Assurance Institute and was published as part of the Auerbach Systems Development Handbook. You can contact Linda at J.l. perlin offers a featured article that provides an approach for handling modern day technology changes. Perlin has over 20 years of manual & automated testing ConTACT us experience and has managed QA teams in both the AST Magazine public and private sector for organizations including Lockheed Martin, Social Security Administration (SSA), General Electric, and the Centers for Medicare & ATI Online Reference Medicaid Services (CMS). Mr. Perlin is currently a lead test automation engineer working on critical care systems for Philips Patient Monitoring Informatics. You can email Mr. Perlin at ATI and Partner Events dion Johnson provides step-by-step instructions for building an automated test framework in which you August 17 may develop and maintain your automated tests. Dion has ATI honors finalists Announced several years of experience in providing IT services to http://www.atihonors.automatedtestinginsti- both government and private industry in a manner that has demonstrated expertise in multiple areas of the software development lifecycle. With a Bachelor of Science August 17 - 31, 2009 degree in Electrical Engineering, he has spent much of ATI honors Voting period his professional career as a consultant in the areas of quality assurance (QA), quality http://www.atihonors.automatedtestinginsti- control (QC), software process improvements, requirements analysis and software test automation. As a regular conference speaker and presenter, Dion has delivered award winning and highly acclaimed presentations at many of the most prestigious industry conferences, including the StarEast and StarWest International Conference on Software Testing, Analysis and Review, the Quality Assurance Institute Conference and Better Software Conference. He also teaches college and business level classes relative to testing and test automation, and has several published articles in various IT publications. 6 Automated Software Testing Magazine August 2009
  7. 7. August 2009 Automated Software Testing Magazine 7
  8. 8. Sweat the technique hand me the key...word Building A Keyword Driver Script In 4 Steps Often called “Table Driven”, a Keyword advantage is that it is easier to read keyword framework interprets automated tests that files than traditional scripts. Keyword files are developed in a tabular format using typically mirror manual test procedures, and a vocabulary of keywords. A keyword is the natural language of the verb-like phrases typically a verb or verb-like phrase that is makes reading a keyword file similar to associated with an external library function reading a collection of sentences. Finally, responsible for executing the application while the Keyword framework itself may commands necessary to satisfy the objective be more technical, the keyword files created of the test. during everyday application automation are Therefore, in a keyword framework, a a lot less technical than files with statements script that may normally appear as illustrated written in code. This allows less technical in Figure 1 will instead appear as illustrated individuals to effectively contribute to in Figure 2. automated test creation. These advantages don’t mean that we’ve escaped scripting altogether, however. We haven’t evaded scripting; we’ve merely changed the nature of the scripting that needs to be performed. There are at least two forms of scripting that remain: function because many of the functions are created development and driver script creation. in such a way to not only be reusable for If you purchase a commercial keyword multiple tests within a single application, but framework or obtain an open source keyword also for tests across multiple applications. framework, you may eliminate the need for Redundancy may therefore be decreased custom development of the driver script and Figure 1: Code-based interface across all applications that an organization may also reduce the function development to may be responsible for automating. Earlier only those functions that are specific to your Figure 2 illustrates a tabular automated script development also results from using application and necessary for more efficient test known as a keyword file, with the this type of framework, due to an increased automation. If you choose to create your own keywords that dictate the actions to be ability for automation to begin before the framework however, the script development performed shown in the Action column. application is delivered. Using information is solely up to you. There are several advantages to using gathered from corresponding manual test This article will focus on the development keyword files for automated test development cases, requirements or other documentation, of the driver script that interprets a keyword including increased reusability. Reusability is keyword files can be created well before the file. Following are steps for successfully increased with a Keyword Driven framework, application is ready for automation. Another creating and implementing a keyword driver Figure 2: keyword Driven interface 8 Automated Software Testing Magazine August 2009
  9. 9. script: about the step to anyone reading the application under test 1. Create the keyword file structure keyword file. 2. Identify potential keywords Step 3. IdentIfy Keyword utIlIty 3. Identify keyword utility functions Step 2. IdentIfy potentIal KeywordS funCtIonS, argumentS and return ValueS 4. Create driver script The keywords created will largely depend Function definitions are a product of the on the functions deemed necessary for framework and of the automator’s personal Step 1. IdentIfy ColumnS In Keyword fIle implementation of the associated framework. preferences for how the function is to be The keyword file columns will vary from Not all of the keywords will be identified implemented. One of the keys in developing framework to framework. The only constant during framework design and development, an effective automation framework utility Keyword fIle drIVer SCrIpt The Driver ScripT callS funcTionS baSeD on The KeyworD in The acTion column Figure 3: Driver Script pseudocode and keyword File interaction is that a column will almost certainly exist but will rather continuously become apparent function, however, is to make sure it is for keywords to be identified. In addition, as tests are automated. If developed properly, independent, reusable, and able to be the keyword file may have one or more of the keyword framework will be seamlessly implemented as intuitively as possible. the columns illustrated in Figure 2. These scalable with respect to the addition of new Function arguments play a big role in columns are defined as follows: keywords over time, but there are some ensuring your functions met these criteria. • Action – Contains the step’s main generic keywords that may be identified For example, a function associated with the keyword, and identifies the action that upfront, including: INPUT keyword might need the following will be executed • INPUT – Inputs a single data item arguments: • Screen – Contains the name of the screen • PRESS – Presses or clicks a button or • Screen – This argument contains the on which the action is to be performed link screen on which the input action is to • Object – Contains the name of the object • VERIFY_SCREEN – Verifies a screen occur on which the action is to be performed has appeared • Object – This argument contains the • Value – Contains values that may be • VERIFY_FIELD_VALUE – Verifies object on which the input action is to entered into the application entered of selected field data • Value – This argument contains the value • Recovery – Contains Exception • VERIFY_TEXT – Verifies dynamic to be entered into an object Handling keywords. screen text These arguments correspond to columns in • Comment – Contains helpful information • INVOKE_APPLICATION – Opens the the keyword file shown in Figure 2, therefore, August 2009 Automated Software Testing Magazine 9
  10. 10. ati newsletter The ATI Newsletter keeps you abreast of the latest information relative to our Online Reference. Sent to registered members of the site on a bi-weekly basis, this newsletter keeps you informed on the follow: ATI’s featured Content • Featured Videos • Featured Tutorials • Featured Articles • Featured Tools • Featured Publication • Featured Poll • Featured Forum Post popular Content • Popular Tools • Popular Articles Automation omg! • Interesting News • Current Events Figure 4: Sample Driver Script the arguments will be fed values from these If the driver script is developed using Ruby, it three keyword file columns via a Driver may appear as shown in Figure 4. Register on the ATI site today to Script. Line 1 loads the UtilityFunctions.rb file. receive this email newsletter. To This is a file you develop that contains your register visit: Step 4. deSIgn and deVelop drIVer SCrIpt user defined utility functions. As illustrated in Figure 3, the driver script Lines 3 through 7 are responsible for is responsible for calling the keyword file, setting up and initializing the keyword file reading all of the steps, and executing utility which is developed in an Excel spreadsheet. functions based on the keywords in each step Line 9 begins the loop that reads all of of the keyword file. the rows in the keyword file. (Continued on page 37) 10 Automated Software Testing Magazine August 2009
  11. 11. Tired of searches that return irrelevant information? google - AutomAted teSting SeArch engine The Automated Testing institute has partnered with google to create a software test automation search engine. if you’re looking for software test automation information, you will get the best results by using this search engine, because it only searches the sites that matter most to automators. you can also restrict searches to blogs and forum sites. August 2009 Automated Software Testing Magazine 11
  12. 12. open Sourcery Ask not what your tool can do for you Learn to Contribute to Open Source Projects in 4 Steps Ask not what your tool can do for you. interested in automated software Ask what you can do for your tool. testing or interested in one of the My fellow automators, how often authors writing about automated have you pointed out the shortcomings software testing. Either way, this is of tools with which you have worked? a good indicator that a project that Now, how often have you sought to maintains an automated test tool may take an active role in fixing these be a good fit for you. shortcomings? Aside from simply picking a project I’m sure many of you are thinking, because you like it, you may also “Fixing tool shortcomings is not my want to pick a project that is modest in job! I use these tools to help me better size; particularly if you are a beginner perform my job. If I spend time fixing to open source development. Smaller tools, then I won’t have any time to do projects typically have smaller code my real job!” bases, provide easier access to other I hear all loud and clear, and you developers, and have simpler bug make good points, but while fixing you on how this may be accomplished. In our submission processes. Therefore, tool shortcomings is not your responsibility, research we came up with 4 steps. smaller projects may make it easier for you to it does have several benefits. Fixing tool quickly become productive. shortcomings is normally not an option with Step 1 – ChooSIng a projeCt Another option is to start your own open commercial tools, but the beauty of open There are numerous open source projects source project. Maybe there is a browser source tools, is that it is very much a viable extension or automated test tool plug-in that ready and willing to receive contributions option. you’d like to see created to support automation from eager scripters, and the project list The May 2009 issue of the Automated of a particular technology; if so, you can lead grows with each passing day. Following are Software Testing Magazine contained an the effort in developing the desired extension links that identify and provide access to some editorial by Dion Johnson called, “Scripting or plug-in. Just search around first and make of these projects: Calisthenics”, and this article presented sure a project doesn’t already exist for what various options for keeping automation you intend on doing. Not sure how to create skills sharp during periods of inactivity. • a project? Some pretty good instructions are The article was referenced in various • laid out at ( automation Internet forums with a question • how_2091738_start-open-source-project- posed to the forum’s community that sourceforge.html) for creating an open source asked, “How do you keep your skills The challenge is now in determining project with SourceForge. sharp during periods of inactivity?” One which project you should contribute to. The of the community’s users suggested that simplest way to pick an open source project Step 2 – pICKIng your ContrIbutIon contributing to open source projects is a good is to look for one that maintains a tool that Now that you’ve picked a project, the next way to keep scripting skills sharp. We here you use and/or are interested in. For example, step is determining what you’re going to at the Automated Testing Institute thought given that you are reading the Automated contribute to that project? You may begin by this was an enterprising answer and decided Software Testing magazine right now, it asking yourself what shortcoming you’d like to research and provide information for all of may be safe to assume that you are either to strengthen, and determining if it’s going 12 Automated Software Testing Magazine August 2009
  13. 13. to be a ‘fix’ or a ‘feature’? In other words, important to do it with a productive tone. or it may be as complicated as modifying do you want to fix an existing bug, or do you One way that you’ll be able to communicate several code modules. However extensive want to add a new feature? At this point in with people on the project is by joining the your scripting task, be sure to use the code the process you may have already decided project’s chat sessions and/or mailing list if conventions defined by the project, including what you want your contribution to be, and they have one. Mailing lists are often used conventions for indenting, variable naming, that decision may have played a big part in for discussing recent code changes or to and commenting. the project you chose to join. If not, however, propose new code changes, so they offer a Upon completion you need to post there are some tips you your changes to the can use for picking a contribution. ...if key players on the project don’t project, a task that is accomplished The first and most obvious thing to do is like the ‘new guy’, there is a good differently different projects. for chance that your proposals will get to actually use open Some projects source tools. There are submit changes via a bunch of them that the mailing list, perform a wide range rejected and sometimes the of functions. If you project uses a version work on a computer control system. at all, you should have no problem finding at great avenue for proposing your changes and Whatever vehicle you use for sharing your least one open source tool that can help you soliciting feedback. Making your intentions update, that update is normally presented in with some of your tasks. Pick one, and then known to the key group members early will the form of a patch. A patch is a small piece identify some of the features you’d like to see improve the chances of your change being of software that contains your fix or new changed. What you come up with, could be accepted, or it may allow you to receive feature for a computer application. One way your contribution to the project. feedback explaining why your proposed to create a patch is to run what’s called a diff In addition, some projects maintain change should be reconsidered. program. The diff program compares original readily available “to-do” lists that identify Using the bug system to study existing files or directories against modified files items that need to be fixed. Also, the project’s bug reports is also a great way to learn the or directories, and produces an output file bug reporting system can be used for culture of the project, because the reports called a patch that contains the differences. gathering ideas for what to fix. indicate who the key developers are and what Upon submitting the patch, be prepared to types of contributions normally get accepted. make changes as suggested by the project Step 3 – learn the projeCt Culture Aside from the bug system, you’ll want to maintainers. For information on creating and Much like countries and commercial appropriately utilize other project tools, such submitting a patch, visit (http:// businesses, each open source project has as the version control system. Developers its own unique culture, and your success in on these projects often have little patience open-source-project.html). a project largely depends on your ability to for individuals that can’t seem to follow Before submitting your patch to the learn and fit into that project’s culture. As rules, standards and directions. To obtain open source project, be sure you understand much as you may like to believe success is information on tools and conventions used the open source licensing used by the project only about technical abilities, it unfortunately by the project, locate and read the developer to ensure the patch is not rejected for legal goes beyond that. No matter how good your docs. reasons, and to ensure any copyright you may coding skills are, if key players on the project wish to impose on your developed code is don’t like the ‘new guy’, there is a good Step 4 – Code and poSt preserved. chance that your proposals will get rejected. At this point, it’s time to get down to the It is, therefore, important for you to be business of developing your fix or feature, ConCluSIon aware of who the core maintainers are, who and it’s imperative to begin by making sure Therefore, my fellow automators, if you the key contributors are, and who the more you have the most up-to-date project code. find yourself complaining about what your vocal members of the project are. You’ll also Next, search the code for the location in test automation tool is not doing for you, want to study how they communicate with which the change is to be applied – a task ask yourself, “What am I doing for my test one another. This will provide insight into that is often simplified by using an Integrated automation tool?” If more people had this who you may want to reach out to, and how Development Environment (IDE). Now you mentality, we’d all be a little more skilled, best to reach out to them. It will be important can begin scripting. Your scripting may be and the discipline of test automation would to make yourself known, but it’s equally as simple as changing a line or function, probably be a lot better off. August 2009 Automated Software Testing Magazine 13
  14. 14. Evolution of AutomAtEd SoftwArE tESt An Eye Witness Account by linda Hayes 14 Automated Software Testing Magazine August 2009
  15. 15. Test automation has a history of failure. Despite the explosion of n software into every aspect of our lives, only a small percentage of tests are actually automated today. Not that there aren’t pockets of success or flashes of promise, but after more than a quarter century of investment by vendors, customers and consultants alike, manual testing predominates. So reviewing how test automation tools have evolved is useful only if it helps to understand where we go from here. To do that, you need to appreciate the technology ting and market forces that have shaped them and why, and what that portends for the future. For that reason, this article sets the context of each evolutionary period within the state of software development and the applications market at the time. Finally, test automation is a very broad topic, so when I say test automation I am referring specifically to the automation of an otherwise manual test against a user interface (UI). This does not mean that test tools cannot be used for non-UI testing, or that there are not many other types of testing that are automated by a wide array of approaches and tools, only that they are outside the scope of this article. August 2009 Automated Software Testing Magazine 15
  16. 16. Text Capture/Playback arose around dynamic content, like date and time stamps or frequently changing data fields. Timing synchronization was also problematic, especially as the user community expanded and performance Mainframes dominated the early enterprise landscape. The mainframe fluctuated. Whenever the system changed or ran slower than the was a monolithic environment, closely guarded behind glass walls original recording, context was lost and errors occurred. and accessible only to a few. Computing resources were scarce and expensive, so development processes were formal and highly But the killer issue turned out to be maintenance. If you recorded structured, requiring flow charts and extensive documentation before 100 transactions, there were 100 sets of the keystrokes and screens, being committed to code. The majority of software was custom so any change had a wide ripple effect. It was usually easier to re- developed for back office applications such as accounting and record the test than it was to fix it, which defeated the value premise payroll. Annual release cycles were the norm, usually punctuated by of automation and encouraged reversion to manual testing. emergency patches. The first commercial test tools emerged as a result of the transition So early on, maintenance loomed as a primary obstacle to test from batch to online applications, when multi-user terminal access automation success, with dynamic content and timing close behind. first became available and manual testing became a necessity. These tools recorded the keystroke inputs and character-based screen outputs of a terminal session, then replayed the keystrokes and compared the Text Capture/Playback/Script screens. Computing soon spread to mid-range systems, but the real breakthrough came with the announcement of the IBM PC. Personal This approach was seductive. It sounded so simple: just do the test as computers had been around for a few years, but these 8-bit platforms usual but record it into a script file, then play the test back as needed were generally perceived as truly “personal” computers used strictly and compare the results. Unfortunately it was also deceptive. Issues by hobbyists and a few propeller heads (the early term for geeks). But with the imprimatur of Big Blue, PCs with 16-bit power entered the corporate computing world. Application development continued to expand on the larger platforms, but software for PCs was initially limited either to productivity tools such as spreadsheets or word processors, or to terminal emulators that turned them into smart terminals. The DOS operating system sporting a character-based interface, was single-threaded and severely limited on memory and storage; the first machines didn’t even offer hard drives, so functionality was constrained. The increasing popularity of spreadsheets and word processors propelled PCs into the corporate community, and the use of terminal The emulators allowed them to replace “dumb” terminals. This drove manual testing from the mainframe to the PC. Soon automation tools Archaeological Dig Mainframe domination ibM PC Announced System Evolution Automation Evolutionary Response Text Capture/Playback emulator drivers 16 Automated Software Testing Magazine August 2009
  17. 17. appeared for the PC, capable of driving these emulators and other PC- could use our tool because she was not a programmer. based applications, and these began to replace the mainframe-based capture/playback tools because they were cheaper and more powerful. Until she uttered those words it had never occurred to me that all testers weren’t skilled at scripting. After all, Petroware had been a One of these tools was from my company, AutoTester. Our tool was software company and we wrote code for a living. Although we sold originally developed by my first company, Petroware, when we ported AutoTester as enhanced capture/playback/scripting, this was just a our oil and gas accounting software from an early IBM desktop to euphemism for programming. And the stark fact of the matter is that the PC. Designed as a “session recorder” by my brother to automate most manual testers are subject matter experts, not programmers. his programming tasks, we quickly realized we could use the tool for many purposes including demos, training, task automation and So in the end we traded one set of problems for another: we addressed testing. But the more we used it the more we found we had to enhance the maintenance, dynamic content and timing issues but introduced a it with additional commands to perform logic and branching as well level of skill and complexity that excluded the majority of our target as to handle variable data. users. Before long, we had developed a proprietary language around the I was devastated. Secretly convinced we had developed an essentially recording that enabled the scripts to handle timing and synchronization, defective product that tormented our own customers, we started make decisions based on results, and externalize data into files. Now a project to try to reduce or eliminate the programming aspect by a single script could loop through 100 records from a file and execute generating scripts, and while we made progress it soon became reliably regardless of system speed or dynamic content. Because it academic. Windows was born. also provided more power and control through its scripting language, increasing reliability while reducing maintenance, we consistently replaced older capture/playback tools and handily won competitions GUI Capture/Playback/Script for new customers. The release of Windows brought even more opportunity but also more Yet in spite of our early success, nagging concerns persisted. Even challenges. As a multi-threaded operating system it resolved many though we thought we had addressed the maintenance, dynamic low level issues; the maximum memory limit exploded and resources content and timing issues caused by capture/playback, we still were managed across sessions. Windows also employed an API layer observed customers eventually reverting to manual testing. I was first that converted mouse events into operating system messages that were puzzled then increasingly alarmed. What could be causing this? intelligible and returned screen contents as a mix of objects that could be queried to return text and images. Recorded scripts became more I got my answer from a customer who sent their test manager to our legible and reliable and the extra memory headroom enabled even offices for training. She was ideal for her job: she had over 20 years of more advanced functionality. experience with the application that was being rewritten and knew the functionality inside and out. But after only a few hours of training, she Another dynamic entered the mix as well. Personal computers were began to show signs of discomfort, and on the second day she openly becoming not only ubiquitous but operating platforms in their own burst into tears. Horrified, I asked her what was wrong. She explained right. Instead of just smart terminals for mainframes or personal that she was afraid she would lose her job since there was no way she productivity tools, they offered powerful standalone applications. Client/Server becomes the internet Takes Center windows released norm Stage gui Capture/replay Custom Frameworks Commercial Frameworks August 2009 Automated Software Testing Magazine 17
  18. 18. This transition from mainframes to personal computers caused a slow lane as scripters took over automation. corresponding shift from a formal and ponderous development process to a more flexible, free-wheeling approach. Mainframes were so Custom Frameworks costly that few had access to them; personal computers were so cheap and plentiful that the masses could afford them. This led to software The PC continued to gain computing power, moving to a 32-bit companies being born in garages by kids barely out of high school or architecture, and Windows matured into a more stable and capable operating system that was a virtual monopoly on desktop machines. Client/server application development became the new norm, ...initial apparent success with re- allowing PCs to offload tasks from backend mainframes and servers and provide users with ever-easier yet more powerful interfaces. The cording messages instead of key- battle cry began to replace expensive mainframes with networks of strokes was soon replaced by disap- PCs and servers. pointment and failure... But the early days of client/server application development introduced even more issues for test automation. Commercial development toolkits became available, offering libraries of sexy new graphical controls that augmented the standard Win 32 object class library. college with no patience or need for the constraints of formality. Other component libraries also appeared offering everything from calculators to calendars to spell checkers, allowing programmers to The type of applications changed as well. Whereas previously most quickly assemble rich user interfaces with less and less effort. applications performed back office tasks that were limited to a few internal users, newer applications expanded their reach to front line The problem was that many of these new controls were non-standard, operations such as order entry and customer service. Many of the so the standard Windows messaging layer was no longer applicable. A original custom applications were replaced by commercial, off-the- custom control did not necessarily implement the same messages that shelf systems and new development moved into areas that interacted the standard classes did. Testing tools that relied on the Windows API more directly with core business processes. began to fail, diverting more and more automation time and effort to dealing with custom controls and other non-standard behaviors. As a result, development cycles began to collapse. Market pressure began to dictate functionality and annual delivery cycles gave way to This new complexity raised the stakes. Now automation engineers quarterly releases. More capability was needed faster and faster. This were thrust into the development environment, forcing them to added even more demand for test automation, because manual testing learn and deal with the underlying implementation of the interface. was less and less capable of keeping pace with changes and the risk Whereas previously one had only to be comfortable with using logical profile headed north. constructs and data file processing, now it was necessary to delve into object methods and properties and grapple with installing source This new, more hospitable technical environment and market also hooks or writing test APIs. This effort siphoned time and resources brought new competitors. Frankly, I was hoping that the latest market away from actually automating tests and into a struggle just to interact entrants would know something we didn’t and show us how to assure with the application’s user interface. that our customers were successful over the long term. But the newest tools presented the same problems. Scripting became even more It could not have come at a worse time, as pressure on development sophisticated to deal with ever more robust applications, and initial continued to increase. The new client/server applications penetrated apparent success with recording messages instead of keystrokes was mainstream operations whose delivery schedules were driven by soon replaced by disappointment and failure due to the skills and user demand and stability became even more important. Greater and effort required to write and maintain ever more complex script code. greater functionality was needed and it had to be delivered faster and better. Quarterly cycles soon gave way to monthly. Windows didn’t save test automation. More and more companies invested in automation to meet this demand Presented with this reality, the most successful tool vendors and automation engineers began looking for ways to reduce the time aggressively enlisted partners and consultants who could offload the and effort to code scripts. It became clear that test scripts were no scripting effort from the manual testers. A new class of tester was different than source code and adopting reusable components made created: the automation engineer. Whereas previously manual testers sense as a strategy to reduce development and maintenance. Thus were courted with capture/playback, now they were relegated to the frameworks were born. 18 Automated Software Testing Magazine August 2009
  19. 19. Large companies soon found evolution in thought themselves with multiple Secretly tools and frameworks, some convinced we had of which had been rewritten developed an essentially more than once. defective product that tormented Complicating this was the fact our own customers, we started a that shrinking development project to try to reduce or schedules and expanding functionality meant more and eliminate the more tests were needed. There programming was simply not enough time aspect... and people available to write, maintain and support one or more frameworks while also developing, executing and maintaining the tests themselves. Eventually the overhead became so heavy that companies – again – Originally frameworks provided a basic infrastructure for test reverted to manual testing just to keep up with delivery schedules. execution, handling common tasks such as data file processing, error and test logging and reporting. But they gradually became more Commercial Frameworks robust, providing simplified interfaces that employed spreadsheets and flat files to allow non-technical testers to construct automated By now the Web was taking center stage as a ubiquitous user interface. test cases using pre-built script components. This introduced a role- This was both good and bad. It was good because it provided a basic based approach that allocated tasks based on skill sets and enabled set of standards for access through browser APIs and HTML classes, cooperation between automation engineers and manual testers. but bad because the ability to instantly deploy changes meant delivery cycles could be daily. And, as static pages and controls gave way to A number of framework types emerged, differentiated primarily by dynamically scripted ones, object names deteriorated into gibberish how the functionality was decomposed into components. The most and behavior became unpredictable. prevalent was referred to as “keyword” or “action word” based and it entailed writing script code components around business tasks, Pressure to deliver automation faster soared. Several enterprising such as “Create Order” or “Invoice Customer”, then allowing users automation engineers realized that they were constantly re-inventing to invoke these from a spreadsheet and pass in data values. Others the same framework over and over and decided to commercialize organized their components around screens or windows, such as “Add their solution and market it as a library of pre-built components. This Customer Information” or “Delete Customer Information”. helped to accelerate the implementation curve and reduce the overall costs. But depending on the type of framework, there was still a fair These frameworks succeeded in reducing overall development amount of application-specific code required to be developed, usually and maintenance while allowing non-coders to design and execute complicated by the custom control or dynamic content challenge. automated tests more easily, but still required custom code. Test designers could write linear test cases but could not make decisions One of these framework types – my favorite - was designed around and control the flow of the test based on real-time execution results. object classes. Commonly referred to as “table-driven” or “class- That still required code. action”, this approach sought to reduce or remove application-specific code by using components at the object class level: For example, But frameworks fell victim to their own success. When the architect “Press Button” or “Input Text”. A key advantage was that the same and chief (often only) automation engineer left the company or moved object classes and actions could be used across applications that used to another area, his or her replacement almost inevitably believed the same class library, allowing greater reusability with less code. It it would be easier and better to rewrite the code left behind than it was even possible to share libraries across companies. would be to maintain it. With no de facto standards, almost every automation engineer adopted their own pet approaches and designs. As the amount of code that needed to be written continued to August 2009 Automated Software Testing Magazine 19
  20. 20. decline, the battlefield moved to the objects themselves. Virtually all • Test automation architectures will continue to shift from code modern test tools rely on an inventory of the application’s testable to data. The only way to keep up with the blistering pace of objects. Variously referred to as an “application map”, “GUI map” development will be to minimize or remove altogether the need or “object repository”, this inventory contains a list of each object in for custom code and accelerate or automate the acquisition of the the application’s interface including its name, class and the attributes application map. used to identify it. This map enables an object to be added to a test, • Test automation penetration will continue to decline - unless located and acted upon during execution, and provides a central point something changes. The accepted rule of thumb says that 30-40% of maintenance in the event of changes. or even 50% of a project should be budgeted for testing, but that assumed old-fashioned coding, not snap-together systems. There But as previously noted, object inventories were becoming is no way that hand-crafted test scripts can keep up with high- increasingly dynamic and unpredictable. Building the map was a slow speed application production. Nonsense names, closed controls and unpredictable context will disap- pear when developers are both accountable and rewarded for success- ful test automation. and tedious process, often requiring the engineer to navigate to each So what has to change? In my opinion, our only hope is for test page and objects. But with dynamic pages creating scripted objects automation to be integrated into the development process. No on the fly, using names that weren’t just incomprehensible but also application should be deemed complete without a test harness. Let’s changing depending on user choices, session parameters or a host of make developers responsible for making their software testable instead other factors, this was often an impossible task. Without a predictable of engaging in a constant struggle to compensate for their latest bad and reusable object inventory, automated tests are unusable. Trying behavior. Nonsense names, closed controls and unpredictable context to maintain them with daily releases was crazy enough, but if the will disappear when developers are both accountable and rewarded attributes changed within a release without any predictability, it was for successful test automation. simply impossible. Such a harness would provide a test object inventory and interface So history shows us that every time test tools and implementation with defined methods and properties to be shared by developers and approaches are enhanced to meet the latest challenge, new ones are testers alike. It would be kept current with all software changes and be introduced. Vendors and customers both find themselves in a perpetual part of a holistic test process that includes automated unit, integration, state of catch-up, seemingly doomed to repeat the past by continually system and user acceptance. reverting to manual testing. Test tool vendors needn’t become extinct; just shift their focus to What’s Next managing data instead of code. Automation engineers needn’t become an endangered species; just shift their focus to managing the object inventory and interface metadata. But don’t react just yet; history also The only way to break this cycle seems to be to anticipate the future tells us that changes take decades. Even today there are companies and be ready for it, instead of reacting to it. The harder question is to who believe capture/playback is a viable automation strategy, though know what’s coming. Since I don’t own a crystal ball, all I can do is we have known better for almost 30 years. look at history and current trends to project what’s coming. But until management sees the light and directs development to play History says that: their part, we all need to continue finding ways to reduce the amount of test code that must be written and maintained, work closely with • Development will continue to accelerate. Software will development to correct testability issues and manage the impact continue to insinuate itself into every aspect of our lives and of changes, and educate management that the historical ratio of jobs, driving demand for more functionality faster. Companies development to test effort is outdated but that the extra investment in will adopt packaged applications when possible. Developers automation is worth it. And keep trying. will take advantage of increasing levels of abstraction to keep The Automated Testing institute (ATi) conducted a up, using code libraries, code generators, web services and other middleware technologies to quickly assemble composite podcast interview with linda Hayes. To listen in, visit applications. 20 Automated Software Testing Magazine August 2009
  21. 21. real adventures in automation Submit your Stories! Now’s Your Chance to be Heard Hav e so me thin g to say ? o listen ATI i s here t Ad The Automated Testing institute believes that one of the best ways to improve upon test automation is to listen to the lessons learned by others. So, if you have an interesting story that you’d like to share with the world, contact us at to find out how. August 2009 Automated Software Testing Magazine 21
  22. 22. Navigating Existing Evolutionary Frameworks through Modern Day Technological Evolutions by j.l. Perlin Road 22 Automated Software Testing Magazine August 2009
  23. 23. it is a natural occurrence for products to evolve over time. Advancements in technology and consumer demands spur radical changes in design that make products more efficient, easier to use, and less costly to produce. How can you prepare test automation to best handle these changes? it is becoming increasingly essential to plan for change when deciding upon an automation approach. in order to do this it helps to understand why these changes are occurring so we as test automation engineers can understand the catalysts behind these changes and be more prepared not only for these new technologies and their effects on automation, but also for the interactions we will have with the organizational stakeholders who have introduced these new improvements. Evolution of a Product So, why do products evolve? let’s use the automobile as an example. The first self-propelled road vehicle was a steam-powered military tractor invented by the French engineer Cugnot. it was a ground-breaking invention, but, alas, due to low speeds (2.5 miles per hour), operating dangers, and the need to stop every ten to fifteen minutes to build up steam, it was not an improvement over the horse speed of the vehicle was still slower to the introduction of hybrid vehicles and buggy. than the horse and buggy, the and all-electric cars, with steps The next attempt at making range of the vehicle was very short made to bring us closer to the day a successful automobile was in (about 15 miles before a recharge when vehicles we drive every day the early 1800’s when a dutch was required), and the cost was can run on hydrogen. but, why engineer used an engine that ran on exorbitant and out of reach for most did the motor vehicle evolve in this rechargeable batteries to design the people of that era. manner? what was the catalyst for first electric vehicle. The vehicle was Fast forward to today, and new improvements and changes initially successful due to a smaller we see that over the years cars to the overall design? it is the and lighter weight engine that have become much more fuel same underlying factor that causes allowed the vehicle to travel faster efficient, safer, and easier to drive. other products to evolve, including and further than its predecessor. improvements to electric and computer systems and software, despite the improvements, the top- combustion technologies have led and that factor is market demand. August 2009 Automated Software Testing Magazine 23
  24. 24. How technology Evolution efficiently improves upon the proverbial forward and the number of products that have Affects Test Automation horse and buggy. Changing major or even been made available to development teams to Technologies of the Information minor pieces of an application will most help them create large, complex applications Age are evolving at a rate of speed that likely require changes to other components of in less time. Automated test tools often have THE TRUE CHALLENGE FOR SOFTWARE TEST ENGINEERS IS DETERMINING WHETHER OR NOT THE CHANGING TECHNOLOGY AFFECTS THE WAY IN WHICH THE PRODUCT IS TESTED. increases exponentially with each passing the system. And all of these new and updated issues when interacting with newly day. Applications are growing in size and components need to be tested, along with released third-party custom complexity. One major software component the legacy components that aren’t changing controls and components, that has undergone massive changes is the to ensure that they still perform properly and these have become operating system. Much like the automobile under the new design. The true challenge very popular in recent engine, operating systems have undergone for software test engineers is determining years. Although very useful numerous changes since their inception in whether or not the changing technology to developers, they can be a major order to meet consumer demands. These affects the way in which the product is tested. problem for the demands have led to changes that In addition, we must determine if we can have made operating systems use the same tools we used prior to the increasingly efficient, faster, technology change to test these new intuitive, scalable, and more components, or if we need to look at secure while providing powerful new tools and/or a new approach. features and useful functionality Just like the products themselves, previously unavailable to users. these tools need to be more efficient, It seems pretty clear cut, no? Changes easier to use, and affordable. in market demands spur innovations in But most technology. What, however, does this mean for software t e s t automation engineers. There may also be issues with development technologies of all the like .NET, Windows Presentation automated tools need to Foundation (WPF), ActiveX, etc., that are be able to access the new components being used by your developers to design the of the application and have the ability to bulk of your application. The majority of the engineers perform all of the methods against the test time we cannot find a tool that can address all that test objects that our tests require to fulfill their of our needs, but at times we do find one that these evolving objectives. As technology moves forward, so can handle the majority of the tasks at hand. applications? It means must our testing approaches, processes, and tools. A Modern-Day that we have to lift the Evolutionary Scenario hood, kick the In today’s environment finding a tool tires and take this new that meets your project requirements can be Imagine that you’re working at an technology for a test a difficult task, especially when you consider organization and have been there for a couple drive to ensure it effectively and how quickly technology has been moving of years, and during this time you and your 24 Automated Software Testing Magazine August 2009
  25. 25. team have been building and maintaining a your concern was centered on testability. The posed similar questions on how to automate suite of automated tests to execute against the first few controls presented by development tests for these controls but, unfortunately product that your company markets. When were proposed for use in portions of the lacked any useful responses. Some of these automation was first introduced to the project application that were minimally accessed by threads were several years old, so it was new test automation resources were hired users, and that required only a small set of becoming apparent that the tool vendor (including you) and the first assignment was tests for validation. Worst case scenario: If probably had no plan to release any add-ins to evaluate the AUT to identify the test tool your automated tool couldn’t handle these for this technology. You next contacted the requirements. Most importantly you wanted controls you may have only had a handful of manufacturer of these 3rd-party controls to find a tool that could interact with all of additional tests to execute manually. Then to see if it was possible to further explore the different components of the application, the other shoe dropped. A new grid control automation possibilities with your tool. The allowing you to automate more tests and was presented that would be accessed by all manufacturers’ response was that this was exercise more of the AUT functionality. You of the product’s users and a large portion theoretically possible, but that the controls looked at a number of different tools and of your automated tests. In other words, if do not officially support your automated compared them using a set of pre-determined you’re automated test tool couldn’t handle testing tool. When you asked how to make criteria/requirements. this control; the majority of your automated the controls accessible by the automated tool, Upon completing the evaluation you tests would become useless. You gently you were only informed that a few of their selected the test tool that would provide the but firmly communicated your concerns customers had attempted automating tests most value to your project, identified an regarding the affect that these new controls against these controls but their results weren’t automation approach that best suited your would have on the existing automation readily unavailable. Hmmm… someone was project, and then began automating the smoke suite so the developers agreed to send you trying to sell something here. and regression tests. Eventually your team a sample grid allowing you to evaluate how It was time to find other options for IT WoULD STILL BE UP To THE TEST TEAM, HoWEVER, To DETERMINE WHETHER OR NOT THESE NEW CONTROLS WERE AN IMPRoVEMENT oVER THE PRoVERBIAL ‘HoRSE AND BUggY’ built a large suite of tests that you designed your automated test tool would react to the making your tool adapt to the new technology. with great deference given to maintainability, newly proposed grid controls. and you felt pretty comfortable knowing Option 1: Extensibility – Some automated Researching Solutions testing tools offer extensibility, which means that most updates and fixes made to the application could be addressed easily using that you’re able to integrate the tool with the After receiving the sample grids as your well-planned framework. new controls by executing small snippets of promised you began trying to access the custom code via calls from the test script. After several new versions of your controls using your existing automated This solution is often the last resort, because application were released the development testing tool, and that’s when you learned the it could require extensive setup, coding, and team started looking at new 3rd-party bad news; the tool didn’t fully recognize the almost always requires assistance from the application controls that could be used grid components. It recognized that a new development team. to improve upon the current design. In a control object existed, but couldn’t access meeting, the development team presented any of the rows, columns, menus, or buttons Option 2: Workaround – Another possible the proposed controls, and these controls contained within this new grid control. Was solution might be to identify a workaround appeared to provide required application there a way to solve this issue? The first step in the application. Was there another screen functionality and also looked much nicer than you took was to contact the automated test in the application that you could be used to the previous, less powerful controls that were tool vendor to see if they supported these accomplish the same goals? Was there a presently in use. It would still be up to the test controls. To this they responded “No, but way to navigate around this control while team, however, to determine whether or not you can search our knowledge base to see accomplishing the same test objectives? If these new controls were an improvement over if anyone has done this”. Searching their so, this would allow the automated tests to the proverbial ‘horse and buggy’, therefore knowledge base turned up several threads that execute, but you and the developers would August 2009 Automated Software Testing Magazine 25
  26. 26. shoes. You’ve done your research and have which way to Maximum roi? discovered this cool 3rd-party control that will fulfill the new and existing system requirements and/or is a major improvement over the previous control. You’ve performed your evaluation, probably of proof-of- concept, and have presented this idea to the rest of the development team. Now some automation engineer is walking in saying it shouldn’t be implemented because it will break the majority of the automated tests. That isn’t going to happen without a fight. Offering Options Since it is good policy to avoid fighting in the workplace, the best thing to do would be to walk into the developer’s office not only with a list of constraints, but also a list of options. Keeping a positive demeanor is of utmost importance. If possible, the most effective way to communicate this information to development would be via the Quality Assurance (QA) Manager. Inter- team communication is one of the major roles of a manager, so as an automation engineer you might as well take advantage have to remain aware of the fact that the verify its functionality is worth the effort. of it. Additionally, program management bypassed controls would need to be tested For example, if a new control is being added may be involved and have input regarding manually. but it only affects a small number of tests, you the final decision, so it is best for them to Option 3: Programmatic Solution – Another might want to see if the effort of automating communicate with someone that they are option would be to design a programmatic the tests against the control is cost-effective. familiar and comfortable with. solution; a coded solution offered by It may make more sense to allow the test to remain manual. Conversely, if the new control It is important to include caveats when developers providing a custom way for communicating these options. For example, your tests to bypass this screen while still is going to be used frequently and is touched by a large number of your automated scripts, you might have a programmatic workaround accomplishing the test objectives. For in mind, but this would require developer example, if the new controls were being it might be worth the effort. Ultimately, it boils down to how easy or difficult it will be involvement, add time to the project schedule, implemented on an ‘Add User’ screen, was remove focus from other current automation there a way that the developers could allow for automation to adapt, and if the effort is worth the result. activities, and forego any testing of the new you to bypass this screen using pre-built user control itself since it is being bypassed. You data stored in the user information database Presenting Automation’s Case might have found an add-in that could resolve that your test could access and use? This So you’ve discovered that your automated most of the issues you had documented would again result in the tests for the new testing tool has problems accessing and during the evaluation, but the add-in was controls being reserved for manual execution, manipulating these new 3rd-party controls. expensive, code needed to be re-written to but it would also allow all of your tests that Should you just run to the development team replace the previous code used to manipulate need to access information processed on this and tell them that the new control can’t be the previous controls, and the add-in still screen to continue running properly. automated? Before doing this you need to wouldn’t provide all of the functionality Analyzing Impact keep several things in mind. The developer(s) required to meet the test objectives for this When identifying different options it also who discovered these new controls and are application component. Or you might explain makes sense to look at the impact that pushing for their implementation have not that your automated test tool extensibility the new control has on the application to only a technical stake in this but an emotional features would allow you to handle this new determine if automating tests to exercise and stake, as well. Put yourself in the developer’s control via automation, but it would also 26 Automated Software Testing Magazine August 2009