QA:news – N5 Zbyszek Moćkun and Karol Kujawiak – Cognifide June 2011 Selenium - the way of successIntroFirst, we would like to share with you when the idea of writing this article came to our mind.A few months ago one of my colleagues sent us a link with a note - “it might be aninteresting conference for you”. We opened that link with curiosity - it was a link to the firstconference about Selenium in San Francisco (http://www.seleniumconf.com/). And it wasthe moment when we realized how popular this tool is. It’s not only used by testers withtechnical knowledge but by beginners too. In this article we will try to concentrate onfeatures and share our experience how to introduce and expand automation in projectsusing Selenium.About the conference - If you are using Selenium or want to learn more about the toolplease visit the webpage. You can find there videos from conference. On the webpage youread in how many ways you can use it in automation.Our experience with Selenium I and IIFigure 1 - SeleniumAuthors have been using and extending Selenium for several years. A great influence onhow we use Selenium has our experience from testing tools that we had used before. Whatwe achieved is to moving good practices from them into Selenium - make this tool ideal :).Trying to make automation easier and quicker we develop Selenium extensions and useadvice from Selenium community. Although we achieved a lot of using Selenium I about halfyear ago we reached the point where further improvements were hard to introduce and timeconsuming.It was the moment when we had to decide about future of automation in our company. Aftersome research and labs we found it - the answer was quite easy - Selenium II. After lessthan half of the year using Selenium II we made a big step in next level of automation.“Exploratory Automated Test” - it’s the best description of the idea. The idea was to move exploratory testing into automation (more flexible, lower costs, quick to introduce, coverage most of functionalities without test management costs, concentrate on aspects that are important for users not on specific tests cases, concentrate on finding symptoms of issues rather than verify specific functionality). Please notice that automation in 99% of usage is use for regression tests. Our aim is to achieve automated tests that can be written in “five”
QA:news – N5Selenium IISelenium II in a new web testing framework which provides new opportunities, but also putsnew demands on the QA. In Selenium II test cases are written in high-level object orientedprogramming languages such as C# or Java so HTML is no longer supported. Tester whowould like to write test scripts in this framework has to have knowledge of C# or Javabasics. But high-level object languages give an opportunity to make it fit your needs.Functionality can be automated not only by simple step by step test case. These steps canbe concentrated in parameterized functions, which are simple to maintain.Our goal is to provide high-quality products while keeping costs at an acceptable level. Apartfrom functional tests we also wanted to run layout tests, check if site meets WC3 standardsand also if the analytics engine works as expected. We also wanted to have an ability to runtests on collections of migrated sites (or after backend changes like new version ofHibernate), to check if there are some differences. All of these automatically of course. Ourapproach forced us to create our own framework, which gives us such possibilities. The mainadvantage of our framework is that test cases written in it could catch issues that might beunnoticed by manual tests.Other reason why we created our framework was to have an ability to provide somestandalone testing tools for our clients. Especially when migration of huge collection of sitescomes into play, standalone tools are very helpful. In this case the only effort that clientshave to make is to create a set of original and migrated url’s. Figure 2 - Four aspectsTesting mobile devicesIf there is a client whose main targets are iPhone and Android phone users, website has tomeet some specific requirements. It’s easy to figure out that these requirements could betested only on native environment. Selenium II provides such functionality. Test scripts can
QA:news – N5be run on the device or on the emulator. There is no need to prepare separate test suites torun in mobile device environment. The same piece of code can be used for testingstandalone and mobile browser.Current beta version is partly underdeveloped, so not everything works how it should, but Iam sure that final version will give us possibility to attract potential clients with suchopportunity.Selenium II – imperfectionsReportsReports are one of the most important parts for automation. Selenium I has a built- inreport generator for HTML tests, which by default provides detailed information about thetest progress and failures. Quick debug which give you enough information (no need torerun test, all results saved, all data for debug available). It’s very important, because wewant to run our test in Continuous Integration environment (each commit/daily) so we willcheck them minimum once daily.When we run Selenium tests in Java - we use JUnit framework which means that is case ofany errors we get only stack-trace. It’s very difficult to debug and it consumes a lot of timeto discover what was the cause of error - particularly when issue is not I00% reproductive orwithout specific java knowledge. Bad reports as JUnit only complicates it, additionally inmost of cases require rerun test manually.WebTest is an example of how reporting part should work. Fortunately, it was not difficult toimplement the same for Selenium and effort returned after a few weeks. The mainadvantage of custom report is that it could be well fitted to our demands, and could containsome customization like JSON files generated by firebug, screen-shots, WC3 validationdetails, source code etc.Lack of HTML supportAs mentioned before Selenium II is based on high-level programming languages and nolonger supports html scripts. It could be hard for beginners to learn how to automate usingC# or Java, but you can look at it from the other side, and treat Selenium IDE as firstdegree of initiation. Writing in Selenium II would for sure improve testers’ technical skillsand broaden their horizons.Google contribution to the developmentWorth mentioning is that the main developer of Selenium II framework is Google Inc. Googlelaunched wide accessible bug tracking system, which allows testers from all over the worldto notify about all of the issues and suggestions that they may have. It was also an impulseto grow a huge community of Selenium users, which could communicate with themselvesand share their experience and know-how. That confirms our belief that the final versionSelenium II will be a high quality product with solid support from Google and communityside.
QA:news – N5HTML against javaThe most interesting survey which we have made during last few years was about Seleniumprogramming languages: java vs. HTML (by html we mean test created by Selenium IDE notconverted to other languages). The same teams in almost identical projects were trying toachieve two goals: to cover by automated tests about 20% of functionality and maintainthem during the project life-cycle. The result was a little surprising. Coding in HTML andmaintenance these type of tests takes 50% less time than java. It shows a very importantfact and I think this one decided about Selenium popularity - ease of use, quickness ofcreation, no need of technical knowledge so almost all team members can work on SeleniumHTML tests. Flexibility of Java wasn’t an argument even for Java developers. Of courseplease notice that the aim was to automate about 20% of test cases (about 100-200 tests).If we want to achieve better coverage probably java flexibility dominates html.Continuous IntegrationIntegration with CI environment shows the strength of open source community. There is noproblem with configuration, because it was described on several blogs/pages and thanks thiswas quick and easy. There is also a number of plug-ins which could improve functionality ofCI environment, which already have been written by open source community.Developers could get test reports on demand and instantly. They can run our tests onnightly builds, their development machines after each commit/change using one clickfeature. Reports (rewrote from WebTests) was so clear that give them all information abouterrors. Thanks this they send to QA high quality software with minimum of regressionissues. Less time spend on testing/retesting bugs allowed us to invest more in automation.Please notice that Selenium suite contains Selenium Grid - a tool to run the same test inparallel on different browsers/machines. We don’t want to describe it here, as ourexperience is not big. We have just tried it and when realize that the same you can achieveusing CI tool (CruiseControl/Jenkins) stop researching. It’s better to minimize the number ofused tools. Figure 3 - Continuous Integration Environment