The future of Test Automation


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The future of Test Automation

  1. 1. ISSN 1866-5705 free digital version print version 8,00 € printed in Germany 16© The Magazine for Professional Testers The Future of Testing December 2011
  2. 2. The Future of Automated Testing by Bernd Beersma A Silver Bullet?The future of testing? Where will we go with testing? Since mymain focus is on test automation, this article will describe my vi- In 2001 I started at the Computer Management Group (CMG) insion on the future of automated testing. the Netherlands as a software tester, adopting the TestFrame™ method as the method for designing test cases. During my firstTo understand where the future in automated testing will be, we weeks at CMG I followed courses on test case design and, auto-first have to understand where it was. So let’s go back to the fu- mated testing with a record and playback tool from a well-knownture… tool vendor. I learned how to use the record and playback tool to automate test cases. After that first encounter I rapidly got ac-We will take a walk down memory lane, and we will do so follow- quainted with other automated testing tools in the field of recording my experiences with test automation from 1999 until now. and playback. All of you that were interested can figure out which tools I used at that moment. Those tools were similar to each oth-First Contact er and could create test cases in a quick and easy to understandI first came in contact with testing and test automation during an way, generating a large number of test cases. But was this theinternship at a hard and software production company in 1999. I silver bullet? No it wasn’t, and the future was not even close atwas on this project for creating a web based interface for config- this moment.uring a Sun Solaris system which was the beating heart of a com- The First Generation, Record and Playbackplex hardware solution. Developed in Perl I created this interface,but being a software engineer I believed that everything I created The so called first generation of automated test tools consisted ofwas working fine. One day during lunch I talked to two of the Java record and playback tools. This basically worked like a VCR (ok, fordevelopers and they told me I should test my just developed soft- our younger professionals a DVD or HDD recorder). A tester needsware, so I could verify if it worked as requested. They introduced to record his or her actions on the system under test (SUT). Theme to a software testing method from The American Department recording tool will record this and create an automated test scriptof Defense (DOD) and showed me how I could use this to test my of those actions. After this session, the tester can play back thissoftware. Ok, now, I wouldn’t think about testing my own soft- just created script and the tool will interact with the SUT simulat-ware, but at that moment it was the best I could do, because there ing the tester and the actions of this tester.were no testers in the company. The biggest problem was the maintenance issue. Because of theSo I tested and tested my software, setting up a test plan, creating static scripts created with the record and playback tools, changestest cases etc. and of course found some defects. During another in the system under test or even test data challenged the testlunch I asked the Java developers how they use testing as part of automation engineers. This meant changing the automated testtheir development process. They said they had to run thousands scripts which is okay for one script, but what if the login sequenceand thousands of tests for their embedded software, but they did for your thousand plus test cases needed changing, or new ob-it automatically. They programmed their test cases within the jects in your SUT are created and they are mandatory? At that mo-embedded software and executed these test cases time and time ment you needed to adapt your automated test scripts to theseagain, guaranteeing the software worked correct and as request- new situations, creating a lot of work for the test automationed. This was my first contact with automated testing, automated engineers.unit testing. So, what was the future for us, the test automation engineers?After that internship I decided that testing would be a part of my Because this time consuming test automation certainly was notlife. Finishing my BA in Software Engineering I always used the the best solution for test automation.DOD standard to test my software. Although it still was manually,it was more testing than most of my fellow students did.116 The Magazine for Professional Testers
  3. 3. The Second Generation, Data Driven Test Automation The Second Generation, a Generic Technology Independent Test Automation FrameworkAs I learned in my next assignment for CMG at a government or-ganization in the Netherlands, the solution was data driven test After leaving this project I was convinced this was the approachautomation. This basically meant that you take your test data for test automation. And so I used this framework and approachout of your automated test scripts and place it outside your tool. for many years at several projects and customers until I started aThis could be an Excel, a CSV or even a plain text file. Storing your project at a big insurance company.test data outside of your tool meant easier maintenance on yourtest data and faster creation of additional test cases. For the test Becoming the lead automation engineer I was responsible forautomation engineers this meant less maintenance on test data, setting up test automation on a large scale and I was convictedbut still the maintenance problem was the biggest challenge for we (me and my colleagues) could improve the framework we hadsuccessful test automation. That’s why the data driven approach been using for years and years. This led me to introducing an ap-wasn’t the future of test automation, we were getting ahead, but proach which I described in my previous article in Testing Experi-still had a long way to go. ence 14, Improving automated regression testing.The Second Generation, the Reuse of Generic Functions This approach is based on Generic Test Automation and GenericIn addition to this data driven approach, the logical next step was Test Case Design and is technology independent. This basicallyto create functions of repeatable or generic parts of your test au- means that it doesn’t matter which system or application youtomation software. The project I worked on did both and it was want to test. Interacting with the SUT is always the same, becausebased on the action or keyword principle of TestFrame. it doesn’t matter if the screen you need to fill out is an HTML ap- plication or an SAP screen. If we look at the business logic, theLet’s take a closer look at creating generic functions as part of path we need to test is described as:test automation. I mentioned earlier the login sequence needed A. Search for the book “Implementing Automated Softwarein a lot of test scripts. This login function is a good example of a Testing” by Elfriede Dustin (Screen A)test step which can easily be transformed to a generic function.If you transform this recorded login sequence into this function B. Place the book in your shopping cart (Screen B)(Function Login(User, Password)), it means that you can use this C. Go to the check out (Screen C)newly created function everywhere a login is needed, just refer to D. Review your order (Screen D)Login(Bernd, Bernd77) and it works with the credentials you sendto the function. Slashing maintenance, because if something E. Confirm your order ( Screen D)changes in this function, we only need to change the functionand the login will work again for all the test steps where it is used. So these are five steps on a business level, but only four steps on a technical level, screen A until screen D, but more importantly, theyAgain, one step closer to the future, but we aren’t there yet. are all screens; screens with different objects, different checks and different actions, but still screens.The Second Generation, the Use of Action WordsBack to the action words: An action word describes an action on As we have been building generic functions for objects like an HT-your system like “Enter field Name” in combination with the val- MLEditbox or an SAPTextField and putting them into libraries forue “Bernd” or “Press button” with the value ”Enter”. These action years, the next step in test automation took shape. Because thewords correspond with a function in your automated test script only difference between HTMLEditBox “Name” and HTMLEditBoxlibrary and are executed on your SUT. “Address” is the technical information needed to steer this object in the SUT. So our libraries where full of functions to steer differ-So these generic functions of action words soon led to a test au- ent applications and environments. The only thing that was lefttomation framework which was used to create automated test to do was to identify the technical information and put this inscripts more easily. The testers would create their test cases in an external file, in our case an XML File. This could easily be doneExcel and the test automation engineers registered those ac- by the automated testing tools we used at that time, commercialtion words in their framework; this resulted in a big decrease in tools from big vendors or open source tools. For a further detailedmaintenance. However, this still wasn’t ideal and we needed to description of this framework, I would like to refer to my previousimprove our framework. This resulted in so called functional ac- article.tion words. Again one step closer to the future, but still not there. I believe this Generic Test Automation framework is a good ap-The Second Generation, the Use of Functional Action Words proach for test automation, and stands between the second gen-The functional action words were action words which consisted eration of test automation and the future. It brings back mainte-of a static part “Fill Screen Personal Information” combined with nance to a minimum and lets testers create test cases in a logicalall arguments used to fill this screen like “name” and “address” and easy to understand way, but still, you need test automationwith the corresponding input values and often an action to per- engineers with technical knowledge and programming skills.form “press next”. So now a tester did not have to use multiple And these are not skills every tester possesses of even wants toaction words to fill in a screen, but could use the action word “Fill possess.Screen XXX”. Again this led to a decrease in maintenance and test The Next Generationcases were also easier to create. If we speak of generations of testautomation, I believe this was the second generation of test au- This leads us to the future of test automation and what does ittomation, record and develop test automation. However, we still look like? So I have been around test automation for several years,use the same tools as before, only we use them in another way, so I worked for a large Dutch test company, a worldwide tool vendorthis is still not the future of test automation. and now a smaller Dutch test company. I had my share of tools and still come across new tools very often. So where is the future? The Magazine for Professional Testers 117
  4. 4. I think it’s in different areas like scriptless automation, userfriendliness, standardized integration between tools and the > biographymindset of combining different tools to fit your needs. Bernd BeersmaFirst of all, I see new, upcoming and promising tool vendors with is competence leader test au- tomation and senior test con-an innovative approach on test automation. The new approach sultant with Squerist. He hasis scriptless automation, so functional testers can set up auto- over 9 years experience withmated test cases without the technical knowledge or the need for different forms of test auto-programming skills. Tools that are born from the need to create mation and performance test-an approach by which testers can setup test automation in a sim- ing for different companies.ple and logical way. I like to use the phrase “Created by testers for Bernd holds a bachelor de-testers”. And I believe there will be more and more vendors who gree in software engineeringwill try to go this way. and became acquainted with software testing during this period.Another thing I hear very often is that automated test tools are During his numerous customer assignments Bernd creat-too complex. The majority of testers would like a test tool that is ed different frameworks and solutions for test automationeasy to use, adaptive and comes with a low learning curve, so they with a broad variety of tools. These different approacheswon’t need months of tool training to get started using this tool. led to creating a generic approach for test automation.So in the future we need tools that have these characteristics. As a result of his knowledge about test automation, he also gives advice on how to implement test automation andFurthermore, what the future of test automation needs is stand- does workshops and trainings.ardized tool integration. Why is it that every vendor creates its Bernd is a member of the TestNet Test Automation work-own interface for test cases, defects, requirements etc.? It would group and co-initiator of the Test Automation Day. His goal is to keep on learning and thus improving his skillsbe easier if we could use tool A for requirement management, on test automation.tool B for automated test execution and tool C for defect man-agement because these are the best tools for the job. Also whenwe decide to use another tool, it’s a pity we need to migrate ourautomated test scripts, because they cannot be used by this newtool. Ok, I must admit a lot of closed source products do interacton a minor scale with each other. One vendor has developed aplatform on which we can use closed source, open source or evenhomebrew. But still, there is a long way to go.Finally, the future of test automation needs the mindset to usemultiple test tools and not wait for the silver bullet, because itwon’t come. I came across a lot of projects and customers whostill think there is one single test automation solution to solveall their problems. I am sorry, but there simply isn’t. I believe youneed to choose the tool which best fits your problem or needs. Ifyou are a web developer and need to run an automated test eachdaily build, maybe an open source webtest tool is the tool for you.If you are a functional tester on an SAP system, you may want toset up your automated regression test with a commercial tool.This is not only applicable for test automation tools, but also forother tools which support your test process like defect manage-ment or requirement management.Looking back, we came far from the early days of test automation,but there is still a long way to go to use test automation on a widescale. And we as testers take part in creating this way, togetherwith tool vendors, open source communities, and automationenthusiasts.So, my final conclusion on the future of test automation:The future of test automation is now, and you’re in it…118 The Magazine for Professional Testers