Selenium - The Way Of Success


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Selenium - The Way Of Success

  1. 1. QA:news – N5 Zbyszek Moćkun and Karol Kujawiak – Cognifide June 2011 Selenium - the way of successIntroFirst, we would like to share with you when the idea of writing this article came to our mind.A few months ago one of my colleagues sent us a link with a note - “it might be aninteresting conference for you”. We opened that link with curiosity - it was a link to the firstconference about Selenium in San Francisco ( And it wasthe moment when we realized how popular this tool is. It’s not only used by testers withtechnical knowledge but by beginners too. In this article we will try to concentrate onfeatures and share our experience how to introduce and expand automation in projectsusing Selenium.About the conference - If you are using Selenium or want to learn more about the toolplease visit the webpage. You can find there videos from conference. On the webpage youread in how many ways you can use it in automation.Our experience with Selenium I and IIFigure 1 - SeleniumAuthors have been using and extending Selenium for several years. A great influence onhow we use Selenium has our experience from testing tools that we had used before. Whatwe achieved is to moving good practices from them into Selenium - make this tool ideal :).Trying to make automation easier and quicker we develop Selenium extensions and useadvice from Selenium community. Although we achieved a lot of using Selenium I about halfyear ago we reached the point where further improvements were hard to introduce and timeconsuming.It was the moment when we had to decide about future of automation in our company. Aftersome research and labs we found it - the answer was quite easy - Selenium II. After lessthan half of the year using Selenium II we made a big step in next level of automation.“Exploratory Automated Test” - it’s the best description of the idea. The idea was to move exploratory testing into automation (more flexible, lower costs, quick to introduce, coverage most of functionalities without test management costs, concentrate on aspects that are important for users not on specific tests cases, concentrate on finding symptoms of issues rather than verify specific functionality). Please notice that automation in 99% of usage is use for regression tests. Our aim is to achieve automated tests that can be written in “five”
  2. 2. QA:news – N5minutes. Adding new component/functionality to tests should be as easy as writing one lineof simple code (example open a page with several version of the same component) thatshows where it is only. Scripts that are written by tester are run on that page and shouldlearn themselves about functionality and then use this knowledge in regression tests(compare results with reference patterns).Selenium against other automation toolsThere is a lot of discussion on web which tool is better. Especially the war is betweenSelenium and WebTest. Almost all blogs/discussions begin with or refer to Marc Gillemontblog - and his comparison between WebtTest and Selenium( disagree with Marc opinion (please notice that he wrote it as WebTest committer) atseveral points, its good place to start and compare your thoughts (as different features aremore important for you). Moreover, this blog post was written four years ago andapplications have changed a lot since then.We don’t want to concentrate on that point in this article or try to compare Selenium withother tools. We would rather concentrate on Selenium itself.Selenium IDE - quick and easy to startSelenium IDE is the most powerful part of Selenium. We think (and in our case it’s how itwas) IDE decided about Selenium popularity. Thanks this Selenium is not a tool for narrowpart of testers’ community who has developed skills but it can be used by all, even not atesters. IDE provides a quick start which allows to show business side very quick results ofautomation and gets money for further growing (for example develop programming skills).From IDE to RCFor very beginners there is a record function, which has an ability to store actions like clicksor typing. But only a very simple test could be recorded. When testing target is moresophisticated, a tester has to raise his knowledge and learn some commands and theirusage. After that, designing even a complex test script isn’t so difficult.Selenium IDE has many ready to use extensions which could extend its functionality such asflow control, loops etc. Testers could also write extensions on their own. All that is needed ismore or less JavaScript knowledge.Worth mentioning is that scripts written in Selenium IDE can be fully integrated withcontinuous integration server, and run not only against Firefox but also against InternetExplorer or other browsers. After the test run there is a detailed report generated whichprovides information about test progress and possible errors. This report is not great but isquite easy to read in compare with JUnit reports.What more could we expect from testing tools? From our point of view many companies limittheir tests to which we have described previously. But if you look deeper there are someareas which Selenium IDE or even RC with object oriented languages doesn’t cover. Herecomes Selenium II.
  3. 3. QA:news – N5Selenium IISelenium II in a new web testing framework which provides new opportunities, but also putsnew demands on the QA. In Selenium II test cases are written in high-level object orientedprogramming languages such as C# or Java so HTML is no longer supported. Tester whowould like to write test scripts in this framework has to have knowledge of C# or Javabasics. But high-level object languages give an opportunity to make it fit your needs.Functionality can be automated not only by simple step by step test case. These steps canbe concentrated in parameterized functions, which are simple to maintain.Our goal is to provide high-quality products while keeping costs at an acceptable level. Apartfrom functional tests we also wanted to run layout tests, check if site meets WC3 standardsand also if the analytics engine works as expected. We also wanted to have an ability to runtests on collections of migrated sites (or after backend changes like new version ofHibernate), to check if there are some differences. All of these automatically of course. Ourapproach forced us to create our own framework, which gives us such possibilities. The mainadvantage of our framework is that test cases written in it could catch issues that might beunnoticed by manual tests.Other reason why we created our framework was to have an ability to provide somestandalone testing tools for our clients. Especially when migration of huge collection of sitescomes into play, standalone tools are very helpful. In this case the only effort that clientshave to make is to create a set of original and migrated url’s. Figure 2 - Four aspectsTesting mobile devicesIf there is a client whose main targets are iPhone and Android phone users, website has tomeet some specific requirements. It’s easy to figure out that these requirements could betested only on native environment. Selenium II provides such functionality. Test scripts can
  4. 4. QA:news – N5be run on the device or on the emulator. There is no need to prepare separate test suites torun in mobile device environment. The same piece of code can be used for testingstandalone and mobile browser.Current beta version is partly underdeveloped, so not everything works how it should, but Iam sure that final version will give us possibility to attract potential clients with suchopportunity.Selenium II – imperfectionsReportsReports are one of the most important parts for automation. Selenium I has a built- inreport generator for HTML tests, which by default provides detailed information about thetest progress and failures. Quick debug which give you enough information (no need torerun test, all results saved, all data for debug available). It’s very important, because wewant to run our test in Continuous Integration environment (each commit/daily) so we willcheck them minimum once daily.When we run Selenium tests in Java - we use JUnit framework which means that is case ofany errors we get only stack-trace. It’s very difficult to debug and it consumes a lot of timeto discover what was the cause of error - particularly when issue is not I00% reproductive orwithout specific java knowledge. Bad reports as JUnit only complicates it, additionally inmost of cases require rerun test manually.WebTest is an example of how reporting part should work. Fortunately, it was not difficult toimplement the same for Selenium and effort returned after a few weeks. The mainadvantage of custom report is that it could be well fitted to our demands, and could containsome customization like JSON files generated by firebug, screen-shots, WC3 validationdetails, source code etc.Lack of HTML supportAs mentioned before Selenium II is based on high-level programming languages and nolonger supports html scripts. It could be hard for beginners to learn how to automate usingC# or Java, but you can look at it from the other side, and treat Selenium IDE as firstdegree of initiation. Writing in Selenium II would for sure improve testers’ technical skillsand broaden their horizons.Google contribution to the developmentWorth mentioning is that the main developer of Selenium II framework is Google Inc. Googlelaunched wide accessible bug tracking system, which allows testers from all over the worldto notify about all of the issues and suggestions that they may have. It was also an impulseto grow a huge community of Selenium users, which could communicate with themselvesand share their experience and know-how. That confirms our belief that the final versionSelenium II will be a high quality product with solid support from Google and communityside.
  5. 5. QA:news – N5HTML against javaThe most interesting survey which we have made during last few years was about Seleniumprogramming languages: java vs. HTML (by html we mean test created by Selenium IDE notconverted to other languages). The same teams in almost identical projects were trying toachieve two goals: to cover by automated tests about 20% of functionality and maintainthem during the project life-cycle. The result was a little surprising. Coding in HTML andmaintenance these type of tests takes 50% less time than java. It shows a very importantfact and I think this one decided about Selenium popularity - ease of use, quickness ofcreation, no need of technical knowledge so almost all team members can work on SeleniumHTML tests. Flexibility of Java wasn’t an argument even for Java developers. Of courseplease notice that the aim was to automate about 20% of test cases (about 100-200 tests).If we want to achieve better coverage probably java flexibility dominates html.Continuous IntegrationIntegration with CI environment shows the strength of open source community. There is noproblem with configuration, because it was described on several blogs/pages and thanks thiswas quick and easy. There is also a number of plug-ins which could improve functionality ofCI environment, which already have been written by open source community.Developers could get test reports on demand and instantly. They can run our tests onnightly builds, their development machines after each commit/change using one clickfeature. Reports (rewrote from WebTests) was so clear that give them all information abouterrors. Thanks this they send to QA high quality software with minimum of regressionissues. Less time spend on testing/retesting bugs allowed us to invest more in automation.Please notice that Selenium suite contains Selenium Grid - a tool to run the same test inparallel on different browsers/machines. We don’t want to describe it here, as ourexperience is not big. We have just tried it and when realize that the same you can achieveusing CI tool (CruiseControl/Jenkins) stop researching. It’s better to minimize the number ofused tools. Figure 3 - Continuous Integration Environment
  6. 6. QA:news – N5Selenium II - future of automation and few our thoughtsWhy have we called article - the way of success. It’s because Selenium allows us tointroduce automation successfully in long term and short term project, minimize automationcost and gave us ability to automate almost all aspects that are important for users orcustomers.Thanks Selenium II we are able to think about future of testing. We called it - ExploratoryAutomated Tests. Tests are learning using results from reference run and then only verify itwhen rerun again. The idea is not to write test case for all functionalities separately - ratherto use pages created during manual (functionality) tests and run on them robots specifiedfor application. Robots are looking for symptoms of errors (changes on layouts, JavaScripterrors, performance issues, w3c validation, content comparison, link checker, errors in logs...). Most of errors have some symptoms which can be found by above approach.ABOUT THE AUTHORSKarol Kujawiak and Zbyszek Mockun are working for Cognifide ( Digital Technology Agency where they was engaged in long or short term projects forvariety of clients (worldwide recognized brands). Authors gather their experience aboutautomation in previous companies too - thanks this they have wide recognition ofautomation tools and approaches.Last few years they have been working in Agile environment where clients demandforced approach which is described in this article. If we summary our achievements thisapproach is quicker, give better results and is cheaper for clients than standard one.Karol Kujawiak Zbyszek Moćkun