Testing Experience - Evolution of Test Automation Frameworks


Published on

Implementing automated tests is something that everybody wants to do. If you ask
any tester, test automation is their aim. And while it may be the golden target, very
few testers take pains to assess the required knowledge, under the illusion that a
programming language or expensive tool will suffice to cope with all problems likely
to arise. This is not true. Writing good automated tests is much harder than that,
requiring knowledge this article will make clear

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Testing Experience - Evolution of Test Automation Frameworks

  1. 1. 23 September 2013 The Magazine for Professional Testers
  2. 2. By Łukasz Morawski Evolution of Test Automation Frameworks From Spaghetti to Cucumber Implementing automated tests is something that everybody wants to do. If you ask any tester, test automation is their aim. And while it may be the golden target, very few testers take pains to assess the required knowledge, under the illusion that a programming language or expensive tool will suffice to cope with all problems likely to arise. This is not true. Writing good automated tests is much harder than that, requiring knowledge this article will make clear. 1. Dimensions The first thing that you must consider when setting out test automation efforts is framework or architecture (sometimes called a test harness) in which tests will be placed. Here I mean all code that will support and facilitate not only test cases development and execution, but also the project structure and any management that will take place (e.g. test data management or documentation). Let’s run through various framework concepts. Perhaps you will recognize the path you chose during your own test automation adventure (if you are an experienced tester). Or perhaps, if you have just started down this path, it will help you make some decisions. We will also consider tester development skills and their evolution which accompany the framework evolution.There are many dimensions when thinking about automation framework. I borrowed these dimensions from Brian Marick’s “testing quadrants” concept (business facing, technology facing, supporting the team, critique the product) and used it to create frames, a starting point for our comparisons. We will consider automation frameworks in the following dimensions: Support the knowledge Support the user Support the test Support the code public class LoginPage { public HomePage loginAs(String username, String password) { // ... clever magic happens here } public LoginPage loginAsExpectingError (String username, String password) { // ... failed login here, maybe because one or both of the username and password are wrong } public String getErrorMessage() { // So we can verify that the correct error is shown } } ▪▪ Supporting the tests – we will talk about what else is needed for a framework to be useful and see in what ways context is important. 2. Supporting the Code When reading testing press, blog post comments, or various forum threads on testing, one special question tops all the others in the context of test automation. How much development skill do I need to proceed with test automation? Do I need to be a developer to begin automating tests? This question is usually posed by manual testers lacking technical skills or testing beginners. This is an important question, and answering it through the lens of the supporting the code dimension can clarify tester doubts. Linear Tests We usually start with the simplest approach: record and play. A tool records our actions and then makes it possible to have them played back. Tests usually look like linear code – one line after another, and so on. This is easy to create, but leads us to a maintenance nightmare. Often these linear scripts need to be refactored to repeat. Also, the recording feature can be used to speed up test creation in much more complicated frameworks instead of typing it from scratch each time. This approach is easy to use for a tester who does not possess any development skills. 1. [java] 2. @Test 3. public void linearTest() { ▪▪ Supporting the code – we will see what coding means to automated testing and how code complexity grows with our needs; ▪▪ Supporting the users – shows different styles and approaches of implementing tests that focus on individuals besides the tester (business, customers); 48 Testing Experience – 23/2013 selenium.open(url); 5. selenium.type(text); 6. Figure 1 –Test Automation Dimensions 4. selenium.click(button); 7. selenium.verifyTitle(title); 8. selenium.click(link); 9. selenium.wait(time); 10. selenium.open(url2); 11. selenium.type(text2); 12. selenium.click(button2); 13. selenium.verifyTitle(title2);
  3. 3. 14. selenium.click(link2); 15. selenium.wait(time); 16. } 17. [/java] Example of Linear Test Code Structured Tests With a higher complexity of test cases to implement emerges the need for more sophisticated test scripts. We begin to familiarize ourselves with the programming or script language that the tool supports and to use its features: conditional statements, variables, and loops. We start to call one script from another, creating chains of scripts. With this, our scripts can do more – more extensively and more reliably. On the other hand, lack of development skills can easily lead to creating spaghetti code with a complex and tangled control structure. We still suffer from maintenance issues and excitement over new possibilities, not realizing we are approaching a dead end. Reusable Tests When maintenance issues finally hit us, it might already be too late. We see that some portions of code repeat and understand that we can reduce the maintenance impact by creating reusable blocks of code. We start using functions or procedures to extract code and create libraries to gather it in one place. We share it between tests and projects. Tests get more clear and readable. At this point, the tester has made a big progress with their development skills. They know the basics of the language that they are using and start to explore more advanced features. 1. [java] 2. @Test 3. public void reusableTest() { 4. CQ55 cq = new CQ55(); 5. cq.openWcm(); 6. cq.open(INTEGRATION_AUTHOR_INSTANCE); 7. PresentationTemplatePage presentation = new PresentationTemplatePage(); 8. presentation.runExportPdfWorkflow(); 9. presentation.confirmWorkflowInfoDialog(); 10. presentation.waitUntillWorkflowNotFinish(); 11. } 12. [/java] Example of Reusable Test Code in Java fills up the entity and from then onward, actual values for assertions can be fetched from that object. Using design patterns in automated tests is not necessary but recommended. It requires some dedication from a tester as it can be a giant leap for them in terms of their development skills. It is certainly worth trying. 3. Supporting the User The aforementioned architectures fit best when technical engineers are the individuals that implement tests. But what if we want a customer to be involved in the process of writing tests or we have more non-technical testers? Is such participation possible and what can be derived from it? What if we want to separate a test design from implementation or a code from tests? Keyword-Driven Tests In the keyword-driven framework, tests are created in an external file: spreadsheet, text, or xml outside of code. Tests consist of keywords that correspond to the actions that user can take upon the application under a test, e.g. click, type, drag and drop. These actions can take arguments like objects to be clicked on or typed in. To improve maintainability, one can move to a higher level of actions that can be undertaken and define more complex keywords like “login”, “add document”, “create an invoice” – it only depends on the application under test. The keywords are defined in code, and adding a new one or changing the old one involves fixing the code base. Also, since the test cases are on a different level, one can change the execution tool beneath without changing test cases themselves. Playing this game makes sense only if a non-technical person will be involved in test automation, because this approach has some serious shortcomings. The fact that tests are defined in a text file, outside code, and “outside” a programming language creates problems with debugging. We lose test case abstraction, benefiting only from keyword abstraction. Since tests that we want to implement can be really complicated and GUI testing tends to be flaky, the loss of debugging on the test level can be painful. The concept of the keyword-driven framework may seem complicated, especially its implementation. Fortunately, there are existing tools/ frameworks that can be integrated with yours. For example, Google’s Robot Framework can be easily integrated with Selenium/Webdriver. Many commercial tools also have such capabilities. If not, they provide sets of helper objects that can facilitate this task. Nevertheless, the tester will need to demonstrate development skills in order to get all of these working. keyword Design Patterns Sometimes we can go much further. We might bravely step into the development world and take little bites from anything that can help us. Design patterns are that thing. A design pattern is a general reusable solution to a problem occurring commonly. With design patterns, we can improve some tests capabilities: for example, Selenium’s Page Object pattern reduces maintenance impact and fosters order in source code. In Selenium, Singleton can also be used to provide a driver instance to tests instead of inheritance. If you need to verify a database, entity can be useful to reduce the quantity of database connections. One connection argument 1 argument 2 open www.mypage.com click addButton type editName John type editAge 30 click submitButton verifyText header Thanks, John! Table 1. Example of Keyword-Driven Test in a Spreadsheet Testing Experience – 23/2013 49
  4. 4. Data-Driven Tests 4. Hybrids While keyword-driven tests focus on the logical flow of actions, datadriven tests focus on data and inputs/outputs. Usually, we have a defined test case and a set of test data to be passed into it as arguments. This way we can execute one scenario with different inputs or outputs, when testing, for example, some sort of calculations or algorithms. Test data can be defined in external file; in this way, we separate test data from test code. This is, however, not necessary. The TestNG framework, allowing implementation of unit tests for (not only) Java applications, comes with the @DataProvider feature, letting us create data sets that can be used and shared across test methods. In most cases, our needs go beyond one solution. Usually we need to combine all or parts of them in order to accomplish our work. This is what we call a hybrid. Implementing such a hybrid on your own may require serious development skills from the tester. But tools like Fit/Fitnesse, Robot Framework, or Test Complete all have ready-to-use features available, so there is no need to implement from scratch. A hybrid may consist of a mix of BDD with data-driven tests and page objects, so in the real world we meet conjunctions rather than the exact implementation of one framework. There is nothing excluding the use of both. Using this framework lets us remove redundancy from tests when we have many similar tests differing in data only. Moving the data to external source facilitates users, customers, and businesses to create tests and thereby produces better collaboration. What is the best approach or the golden solution? There is no golden solution, one might say, but this can only be determined by one’s needs. The most important questions are as follows: ▪▪ who will use the system and who will develop the tests? ▪▪ does the user possess the right technical skills? BDD Tests ▪▪ do we need to separate tests from code? If one needs to emphasis a business point-of-view or business requirements, then one could go with behavior-driven development (BDD). Originating from TDD and eXtreme programming, given-when-then- ▪▪ do we need to emphasize collaboration? style tests focus on system behavior people understand, allowing nontechnical individuals to participate in test creation. This style focuses on collaboration between business, QA, and developers, facilitating it with ubiquitous, understandable language. BDD is about examples, and there are tools that help us express them in a useful manner. Tools like FIT, Fitness, or Cucumber allow the creation of examples in spreadsheets, wiki tables, or text files. No development skills are required for creating tests (defining features or scenarios), but either a mediation layer called fixture or step definitions must be implemented in a programming language in order to make them all work. ▪▪ do we have the time and budget to implement it on our own? Or do we emphasize learning by implementing it from scratch? The Hybrid Page Objects BDD DataDriven Reusable Model-Based Tests On the other hand, sometimes we don’t need to support a non-technical user, but we might need to conduct a more technical analysis of the system and therefore test in a more pragmatic way. In such a case, model-based testing (MBT) might be a good choice. In this framework, test cases are derived from prepared models automatically, so changes in the model are reflected in changes in test cases. With such an approach, one can manage requirements via managing models. Like in the aforementioned frameworks, test design is separated from implementation. Models in this case are an abstraction layer. These models can be created and maintained by testers, but also by business analysts or whoever can handle a modeling tool. Since the most popular model is a finite state machine reflecting system behavior, the modeler must understand fundamentals of software engineering at minimum. Besides the model, code that executes tests must be implemented, and this is usually done by devs. When looking for MBT tools, one can find Spec Explorer or MISTA, an MBT tool ready for Selenium integration. MBT evangelists claim that testing is always based on models – if not in a tangible, genuine form, then in a mental one. I think there is something to this statement. Thinking about flows of actions, structure, or behavior is much easier when visualized. And even if we do not transcribe it, we have always such a model in mind in one or another form. 50 Testing Experience – 23/2013 Figure 2. The Hybrid 5. Supporting the Tests Architecture construction is yet complete. While most of the commercial tools provide many helper objects to support tests using tools like Selenium, one must also either develop many themselves or use existing ones like Robot Framework. Nevertheless, this may illustrate that the code supporting tests will be the main task in implementation.“Supporting the tests”means that code must be implemented to help us achieve our testing aims in conducting comparisons, analyzing, setting up, and tearing down. Features supporting the tests generally vary, as do our needs with regard to the application under test. Depending on the context, web applications might need, for example, verification of Javascript errors. Using the Oracle database means we might need to implement a mechanism importing a database dump file before every test runs or developing a set of SQL queries to clear up database tables. When testing analytics, we must have a means to check that proper requests are sent by the application and are extracting the desired properties from it. So as you see, this effort might require not only development skills, but also
  5. 5. uggling Quit str ecision ith the d w joy test and en sign in case de today! e cloud th y maticall aS syste by ker Sa CaseMa st case design ht in ug s te support e techniques ta rogram th ester p covering ertified T he British B® C the ISTQ rdized within t d a lemente d stand 7925. The imp an d BS alence Standar re: equiv heck, error es a techniqu , boundary c g airwise artitionin cision tables, p . p g, de testing guessin -based and risk testing, rfect fit your pe r SaaS is ement seMake irement manag Ca requ t between anagement/tes tm and tes tion. automa u r.e semake saas.ca y ibe toda y Subscr a 14-da y and enjo for free! al period tri tarts at nse s Your lice nth (+ 75 €/mo VAT).
  6. 6. decent domain knowledge (databases, telecommunications, etc). Some issues supporting tests include: ▪▪ logging – to console, to a text file, or to an HTML file with a timestamp, allowing us to analyze and then precisely conduct a root cause analysis of “red” or flaky tests ▪▪ comparisons and checkpoints – comparing images, XML files, text file, database tables, application under test control’s values, strings, or numbers ▪▪ statistics – created at the end of tests: the number of successful or failed tests, execution time, brief of failed tests ▪▪ abstraction layers – at the test run, test suite, test, action (“add document,” “edit user”), or operation (click, type) levels. May be needed to inject probes for better analysis of certain issues during testing ▪▪ email support – creating an email to be sent after tests, connected with an inbox verifying email content ▪▪ working with files ▪▪ working with databases ▪▪ test evidence – integrate with a video or screen-shot capture tool to gather the evidence of conducted tests ▪▪ APIs – use tools/libraries (like DefaultHttpClient) to submit HTTP requests in order to communicate with APIs ▪▪ test execution parallelization – this can be developed as a solution in code, but also can be brought out of code, e.g. by splitting tests to different test suites and running with different jobs in a CI tool (like Jenkins) ▪▪ and many more, according to one’s needs Implementing a test case is always the most important issue. You can have sophisticated code with many design patterns, BDD, keywords, data, and models, but pointless and imprecise tests will ruin all this effort. Hence, implementing well-balanced tests themselves is the most important thing to achieve. 6. Supporting the Knowledge For further reading on the subject, you could consider: 1. Mark Fewster, Dorothy Graham: ”Software Test Automation” 2. Hans Buwalda: http://safsdev.sourceforge.net/ FRAMESDataDrivenTestAutomationFrameworks.htm 3. James Bach: “Test Automation Snake Oil” 4. Brian Marick: ”Agile Testing Quadrants” – http://www.exampler. com/old-blog/2003/08/22/#agile-testing-project-2 5. http://en.wikipedia.org/wiki/Software_design_pattern 6. http://code.google.com/p/selenium/wiki/PageObjects 7. http://en.wikipedia.org/wiki/Behavior-driven_development 8. Aleksander Lipski, Maciej Klepacki: “Specification by Example applied” – http://prezi.com/xsyjezcqcmye/specification-byexample-applied/ 52 Testing Experience – 23/2013 9. http://jbehave.org/ 10. http://blog.mattwynne.net/2012/11/20/tdd-vs-bdd/ 11. http://www.exampler.com/old-blog/2003/08/22/#agile-testingproject-2 12. “How Spotify does test automation” – http://www.slideshare.net/ SmartBear_Software/how-spotify-does-test-management-kristiankarl 13. Testing Experience, “Model-Based Testing”, No. 17 (March 2012) – http://www.testingexperience.com/ 14. Alan Richardson: “Model Based Testing & Webscraping with WebDriver” – http://youtu.be/ATqakgFK-qA 15. MISTA with Selenium – http://youtu.be/FaZpwhasu54 7. The End I understand that these considerations only scratch the surface of automated testing and that there are many more aspects that have not been raised here. The topic is quite big, though. This little story and my conclusions derive from my own experience on the subject. My experience may be different from your own. I was lucky to be given a chance to experience all of it myself, what it means to implement such frameworks from scratch, to have the possibility to explore, experiment and observe. Just science. I hope you will be so lucky too. ◼ > about the author Łukasz Morawski is an engineer with over eight years of experience in the field of software testing. He is a test automator with Selenium/Webdriver, Test Complete, and AutoIt, and an exploratory tester with a passion to test challenges and Testing Dojos. A student at the Miagi-Do School of Software Testing, he is cofounder of the Poznań Testing and Quality Group (PtaQ – www.ptaq.org). Currently employed at Cognifide, he occasionally writes on the Cognifide testing blog (www.cognifide. com/blogs), the Miagi Do blog (miagido.org/blog), and speaks at local testing events.