1. QA:news – N5
Zbyszek Moćkun and Karol Kujawiak – Cognifide
June 2011
Selenium - the way of success
Intro
First, we would like to share with you when the idea of writing this article came to our mind.
A few months ago one of my colleagues sent us a link with a note - “it might be an
interesting conference for you”. We opened that link with curiosity - it was a link to the first
conference about Selenium in San Francisco (http://www.seleniumconf.com/). And it was
the moment when we realized how popular this tool is. It’s not only used by testers with
technical knowledge but by beginners too. In this article we will try to concentrate on
features and share our experience how to introduce and expand automation in projects
using Selenium.
About the conference - If you are using Selenium or want to learn more about the tool
please visit the webpage. You can find there videos from conference. On the webpage you
read in how many ways you can use it in automation.
Our experience with Selenium I and II
Figure 1 - Selenium
Authors have been using and extending Selenium for several years. A great influence on
how we use Selenium has our experience from testing tools that we had used before. What
we achieved is to moving good practices from them into Selenium - make this tool ideal :).
Trying to make automation easier and quicker we develop Selenium extensions and use
advice from Selenium community. Although we achieved a lot of using Selenium I about half
year ago we reached the point where further improvements were hard to introduce and time
consuming.
It was the moment when we had to decide about future of automation in our company. After
some research and labs we found it - the answer was quite easy - Selenium II. After less
than half of the year using Selenium II we made a big step in next level of automation.
“Exploratory Automated Test” - it’s the best description of the idea. The idea was to move
exploratory testing into automation (more flexible, lower costs,
quick to introduce, coverage most of functionalities without test
management costs, concentrate on aspects that are important for
users not on specific tests cases, concentrate on finding symptoms
of issues rather than verify specific functionality). Please notice that
automation in 99% of usage is use for regression tests.
Our aim is to achieve automated tests that can be written in “five”

2. QA:news – N5
minutes. Adding new component/functionality to tests should be as easy as writing one line
of simple code (example open a page with several version of the same component) that
shows where it is only. Scripts that are written by tester are run on that page and should
learn themselves about functionality and then use this knowledge in regression tests
(compare results with reference patterns).
Selenium against other automation tools
There is a lot of discussion on web which tool is better. Especially the war is between
Selenium and WebTest. Almost all blogs/discussions begin with or refer to Marc Gillemont
blog - and his comparison between WebtTest and Selenium
(http://mguillem.wordpress.com/II007/I0/II9/webtest-vs-selenium-webtest-wins-I3-5/).
We disagree with Marc opinion (please notice that he wrote it as WebTest committer) at
several points, its good place to start and compare your thoughts (as different features are
more important for you). Moreover, this blog post was written four years ago and
applications have changed a lot since then.
We don’t want to concentrate on that point in this article or try to compare Selenium with
other tools. We would rather concentrate on Selenium itself.
Selenium IDE - quick and easy to start
Selenium IDE is the most powerful part of Selenium. We think (and in our case it’s how it
was) IDE decided about Selenium popularity. Thanks this Selenium is not a tool for narrow
part of testers’ community who has developed skills but it can be used by all, even not a
testers. IDE provides a quick start which allows to show business side very quick results of
automation and gets money for further growing (for example develop programming skills).
From IDE to RC
For very beginners there is a record function, which has an ability to store actions like clicks
or typing. But only a very simple test could be recorded. When testing target is more
sophisticated, a tester has to raise his knowledge and learn some commands and their
usage. After that, designing even a complex test script isn’t so difficult.
Selenium IDE has many ready to use extensions which could extend its functionality such as
flow control, loops etc. Testers could also write extensions on their own. All that is needed is
more or less JavaScript knowledge.
Worth mentioning is that scripts written in Selenium IDE can be fully integrated with
continuous integration server, and run not only against Firefox but also against Internet
Explorer or other browsers. After the test run there is a detailed report generated which
provides information about test progress and possible errors. This report is not great but is
quite easy to read in compare with JUnit reports.
What more could we expect from testing tools? From our point of view many companies limit
their tests to which we have described previously. But if you look deeper there are some
areas which Selenium IDE or even RC with object oriented languages doesn’t cover. Here
comes Selenium II.

3. QA:news – N5
Selenium II
Selenium II in a new web testing framework which provides new opportunities, but also puts
new demands on the QA. In Selenium II test cases are written in high-level object oriented
programming languages such as C# or Java so HTML is no longer supported. Tester who
would like to write test scripts in this framework has to have knowledge of C# or Java
basics. But high-level object languages give an opportunity to make it fit your needs.
Functionality can be automated not only by simple step by step test case. These steps can
be concentrated in parameterized functions, which are simple to maintain.
Our goal is to provide high-quality products while keeping costs at an acceptable level. Apart
from functional tests we also wanted to run layout tests, check if site meets WC3 standards
and also if the analytics engine works as expected. We also wanted to have an ability to run
tests on collections of migrated sites (or after backend changes like new version of
Hibernate), to check if there are some differences. All of these automatically of course. Our
approach forced us to create our own framework, which gives us such possibilities. The main
advantage of our framework is that test cases written in it could catch issues that might be
unnoticed by manual tests.
Other reason why we created our framework was to have an ability to provide some
standalone testing tools for our clients. Especially when migration of huge collection of sites
comes into play, standalone tools are very helpful. In this case the only effort that clients
have to make is to create a set of original and migrated url’s.
Figure 2 - Four aspects
Testing mobile devices
If there is a client whose main targets are iPhone and Android phone users, website has to
meet some specific requirements. It’s easy to figure out that these requirements could be
tested only on native environment. Selenium II provides such functionality. Test scripts can

4. QA:news – N5
be run on the device or on the emulator. There is no need to prepare separate test suites to
run in mobile device environment. The same piece of code can be used for testing
standalone and mobile browser.
Current beta version is partly underdeveloped, so not everything works how it should, but I
am sure that final version will give us possibility to attract potential clients with such
opportunity.
Selenium II – imperfections
Reports
Reports are one of the most important parts for automation. Selenium I has a built- in
report generator for HTML tests, which by default provides detailed information about the
test progress and failures. Quick debug which give you enough information (no need to
rerun test, all results saved, all data for debug available). It’s very important, because we
want to run our test in Continuous Integration environment (each commit/daily) so we will
check them minimum once daily.
When we run Selenium tests in Java - we use JUnit framework which means that is case of
any errors we get only stack-trace. It’s very difficult to debug and it consumes a lot of time
to discover what was the cause of error - particularly when issue is not I00% reproductive or
without specific java knowledge. Bad reports as JUnit only complicates it, additionally in
most of cases require rerun test manually.
WebTest is an example of how reporting part should work. Fortunately, it was not difficult to
implement the same for Selenium and effort returned after a few weeks. The main
advantage of custom report is that it could be well fitted to our demands, and could contain
some customization like JSON files generated by firebug, screen-shots, WC3 validation
details, source code etc.
Lack of HTML support
As mentioned before Selenium II is based on high-level programming languages and no
longer supports html scripts. It could be hard for beginners to learn how to automate using
C# or Java, but you can look at it from the other side, and treat Selenium IDE as first
degree of initiation. Writing in Selenium II would for sure improve testers’ technical skills
and broaden their horizons.
Google contribution to the development
Worth mentioning is that the main developer of Selenium II framework is Google Inc. Google
launched wide accessible bug tracking system, which allows testers from all over the world
to notify about all of the issues and suggestions that they may have. It was also an impulse
to grow a huge community of Selenium users, which could communicate with themselves
and share their experience and know-how. That confirms our belief that the final version
Selenium II will be a high quality product with solid support from Google and community
side.

5. QA:news – N5
HTML against java
The most interesting survey which we have made during last few years was about Selenium
programming languages: java vs. HTML (by html we mean test created by Selenium IDE not
converted to other languages). The same teams in almost identical projects were trying to
achieve two goals: to cover by automated tests about 20% of functionality and maintain
them during the project life-cycle. The result was a little surprising. Coding in HTML and
maintenance these type of tests takes 50% less time than java. It shows a very important
fact and I think this one decided about Selenium popularity - ease of use, quickness of
creation, no need of technical knowledge so almost all team members can work on Selenium
HTML tests. Flexibility of Java wasn’t an argument even for Java developers. Of course
please notice that the aim was to automate about 20% of test cases (about 100-200 tests).
If we want to achieve better coverage probably java flexibility dominates html.
Continuous Integration
Integration with CI environment shows the strength of open source community. There is no
problem with configuration, because it was described on several blogs/pages and thanks this
was quick and easy. There is also a number of plug-ins which could improve functionality of
CI environment, which already have been written by open source community.
Developers could get test reports on demand and instantly. They can run our tests on
nightly builds, their development machines after each commit/change using one click
feature. Reports (rewrote from WebTests) was so clear that give them all information about
errors. Thanks this they send to QA high quality software with minimum of regression
issues. Less time spend on testing/retesting bugs allowed us to invest more in automation.
Please notice that Selenium suite contains Selenium Grid - a tool to run the same test in
parallel on different browsers/machines. We don’t want to describe it here, as our
experience is not big. We have just tried it and when realize that the same you can achieve
using CI tool (CruiseControl/Jenkins) stop researching. It’s better to minimize the number of
used tools.
Figure 3 - Continuous Integration Environment

6. QA:news – N5
Selenium II - future of automation and few our thoughts
Why have we called article - the way of success. It’s because Selenium allows us to
introduce automation successfully in long term and short term project, minimize automation
cost and gave us ability to automate almost all aspects that are important for users or
customers.
Thanks Selenium II we are able to think about future of testing. We called it - Exploratory
Automated Tests. Tests are learning using results from reference run and then only verify it
when rerun again. The idea is not to write test case for all functionalities separately - rather
to use pages created during manual (functionality) tests and run on them robots specified
for application. Robots are looking for symptoms of errors (changes on layouts, JavaScript
errors, performance issues, w3c validation, content comparison, link checker, errors in logs
...). Most of errors have some symptoms which can be found by above approach.
ABOUT THE AUTHORS
Karol Kujawiak and Zbyszek Mockun are working for Cognifide (http://www.cognifide.com)
- Digital Technology Agency where they was engaged in long or short term projects for
variety of clients (worldwide recognized brands). Authors gather their experience about
automation in previous companies too - thanks this they have wide recognition of
automation tools and approaches.
Last few years they have been working in Agile environment where clients demand
forced approach which is described in this article. If we summary our achievements this
approach is quicker, give better results and is cheaper for clients than standard one.
Karol Kujawiak Zbyszek Moćkun