Automated softwaretestingmagazine december2011

An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.com

Automated .......
S T

A
oftware esting
December 2011 $8.95
MAGAZINE

Automated
M obile
esting
Cost Effective
Test Management
Develop a HomeGrown
Managment Solution

Surviving
Obfuscation
How To Automate
Obfuscated Code

Test Management Getting Crowded: Crowdsourcing Inspires New Tool

A Sotera Defense Solutions Company

If you’re looking to thrive in a diverse and vibrant professional team, we’re looking for you!
Submit your resumé to staffing@spt-inc.com today!

Through the SPT PromiseTM, we’ve made a commitment to value our employees, to be
understanding and supportive of individual needs and to provide opportunities for
career growth. We value and appreciate our employees for the work they do for us
as a company and for the clients’ mission.

SPT partners with client organizations to design, develop, implement and deploy
high-end, custom, technology-based solutions by integrating existing and emerging
concepts. Positions are available for professionals looking for exciting and rewarding
challenges with significant impact to ongoing missions. Openings include:
• Software Engineers • Systems Engineers
• Configuration Managers • Systems Administrators
• Systems/Software Testers

Most positions require an active TS/SCI clearance
with full scope or CI polygraph.
SPT is EOE, M/F/D/V.
.com
t-inc
w.sp
7230 5 • ww
Lee D -221
efores 1-854
tD rive, Suite 6 • 30
100 • Columbia, MD 2104

Our promise is to ensure we are committed to continued exellence and success for our clients’
mission. We are devoted to doing what is right for our company, our partners, and our
employees to best serve our community and country.
© 2011 Software Process Technologies, LLC. (SPT). All rights reserved.

Automated
S T
oftware esting
December 2011, Volume 3, Issue 5

Contents
Cover Story
Mobile Automated Testing 19
For many, mobile devices are fast becoming the computing environment of choice. With so many different devices
and new ones emerging all the time, it’s infeasible to conduct sufficient testing for each, manually. But how will
test automation need to adjust from the PC to mobile? Read this article to find out. By Patrick Quilter

Features
Obfuscation: Avoiding Friendly Fire in the Battle for Security 12
How do we handle application dynamics introduced by security measures such as obfuscation? Read this article to get strategies for
handling this challenge that results in modern computing. By Dion Johnson

“Integrating MS Office For Cost-Effective Test Management 26
Need a cost-effective test management solution? Read this article as Tom Wimsatt identifies key test management features, as well as a
step-by-step approach for using common Windows desktop software for creating a tool customized to your needs. By Tom Wimsatt

Columns & Departments
Editorial 4 I ‘B’Log To U 38
The Mobile Evolution Enjoy a testing comic strip that is funny and informative.
How mobile technology effects test management and
automated test execution. Go On A Retweet 40
Read featured microblog posts from the web.
Authors and events 6
TestKIT is Hot! 42
Learn about AST authors and upcoming events.
The KIT is Koming
Learn what this year’s conference has to offer.
Open Sourcery 8
The Automated Software Testing (AST) Magazine is an Automated Testing
Test Management is Getting Crowded Institute (ATI) publication.
How Crowdsourcing and the Mobile Revolution Have For more information regarding the magazine visit
Inspired The Birth of a New QA Tool http://www.astmagazine.automatedtestinginstitute.com

December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 3

Editorial

The Mobile Evolution
by Dion Johnson
Many computing technologies have been
introduced over the years and with them,
new approaches for test implementation.
Linda Hayes wrote a great article called
the “Evolution of Automated Software
Testing” that was published in Volume
1, Issue 1 of the Automated Software
Testing Magazine; her article chronicled
the evolution in computing from
Mainframe dominance all the way up
to the Internet taking center stage. Each
evolutionary phase was accompanied by
a test automation evolutionary response
that highlighted the changes to how test
automation was conducted in order to
keep pace with the shift in the computing
world at large.

The computing world is at the beginning
of a new computing sea change, and
I have no doubt that there will need to ...efficient test management is no longer
be a major evolutionary response to this
change in the world of test automation. just a luxury.
The sea change will move us further into test management is no longer just a Quilter discusses tools for managing
the world of mobile technology, and the luxury. Many organizations have gone mobile device test cases.
move is likely to happen faster than any without any real approach or tool for test
of the other computing paradigm shifts management, with the excuse that such Next, Tom Wimsatt reveals an approach
to date – evidenced by the faster than an approach or tool would be overkill for for implementing a homegrown test
expected depletion of IPv4 addresses that the way they do testing. With increased management solution in his article
has forced us quickly into the world of test complexity, however, and teams that entitled, “Integrating MS Office for Cost-
IPv6 (see AST Volume 3, Issue 1). may span multiple organizations, and/or Effective Test Management”. In addition
continents, test management is becoming to identification of key test management
Mobile technology has introduced new
increasingly important. Another item features, Wimsatt provides a step-by-step
devices, new types of software and a whole
that we must come to grips with is how approach for using common Windows
host of new challenges for testing and test
automation of our test execution will be desktop software for creating a tool
automation. In the past we were largely
changed by the mobile evolution. This customized to your needs.
concerned about adjusting to a world in
issue of the Automated Software Testing
which setting up lab environments to test In the second feature, Dion Johnson
Magazine is the first of many to come
multiple versions and configurations of a (yours truly) shifts gears slightly to offer
that focuses on these items and more.
handful of browsers was the challenge. strategies for handling another challenge
That’s nothing compared to the 100s, In our cover story entitled, “Mobile that results in modern computing: the
maybe 1000s of mobile devices and Automated Testing”, Patrick Quilter challenge posed by security measures
configurations that may be running your provides a look into how the explosion such as obfuscation. Obfuscation often
organization’s mobile application (app)! of mobile device usage has affected
results in dynamic UI properties that
We need to quickly come to grips with the current day implementation of test
execution automation. He discusses how render an application non-automatable.
what this means for testing and the tools
that aid our testing. traditional tools and strategies can be In this article entitled, “Obfuscation –
used for mobile app automation with a Avoiding Friendly Fire in the Battle for
One thing this technological shift little assistance from simulators, but also Security”, he provides useful mechanisms
means is that we need to come to grips addresses test execution automation on for creating stable automated scripts on
with the fact that structured, efficient the actual mobile devices. In addition, an unstable application.

4 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011

Training That’s Process Focused Yet Hands On
Software Test Automation Training
www . training . automatedtestinginstitute . com

Public Courses
Software Test Automation Foundations
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development

Virtual Courses

Come
participate in a set
of test automation courses
that address both fundamental
and advanced concepts from a
theoretical and hands on perspective.
These courses focus on topics such
as test scripting concepts, automated
framework creation, ROI calculations
and more. In addition, these courses
may be used to prepare for the
TABOK Certification
exam.

Public and Virtual Training Available

Authors and Events

Who’s In This Issue?
Patrick Quilter Jr. has earned a Bachelor
of Science in Computer Science & Mathematics, a
Automated
S T
oftware esting
graduate certificate in Homeland Security Technology
Management, and is in the process of acquiring a
Managing Editor
Master degree in Technology Management. Patrick
Dion Johnson
has been an automation architect for a number of large
companies such as Travelers Insurance, Computer Contributing Editor
Science Corporation (CSC), and GEICO. While at Donna Vance
Travelers Insurance, Patrick received the company’s Edward Torrie
Outstanding Achievement Award for an R&D solution that programmatically
generated business test cases for automated regression execution. Patrick Director of Marketing and Events
has also worked as a test consultant providing tools architecture and process Christine Johnson
guidance for both commercial organizations and federal government
agencies. After 10 years of experience and continued development, he is
in the process of patenting a cloud based, automation framework that’s A PUBLICATION OF THE AUTOMATED
been configured with many popular automation engines. He is currently
employed as a lead automation architect while also assisting organizations
through Quilmont LLC, a test process consulting company and cloud based,
test automation service provider.

Tom Wimsatt is an ISTQB/ASTQB Fully
Certified Advanced Tester with Software Process
Technologies, Inc. A Sotera Defense Company in CONTACT US
AST Magazine
Columbia, Maryland. He has been in the field of
astmagazine@automatedtestinginstitute.com
testing for more than 20 years beginning in the Parts
Analysis labs with NASA at Goddard in Maryland. ATI Online Reference
The last ten years he has been concentrating on testing contact@automatedtestinginstitute.com
software specializing in tools and automated test
development. He can be reached at: TomWimsatt@wimsatt.truepath.com

Dion Johnson has several years of experience
ATI and Partner Events
in providing IT services to both government and
private industry. With a Bachelor of Science degree January 1 2012
in Electrical Engineering, he has spent much of his TestKIT 2012 Conference Call
professional career as a consultant in the areas of quality For Presentations Opens
assurance (QA), quality control (QC), software process
improvements, requirements analysis and software
April 1, 2012
test automation. As a regular conference speaker 4th Annual ATI Automation
and presenter, Dion has delivered award winning and highly acclaimed Honors Awards Nominations
presentations at many of the most prestigious industry conferences, including Begin
www.atihonors.automatedtestinginstitute.com
the StarEast and StarWest International Conference on Software Testing,
Analysis and Review, and the Quality Assurance Institute Conference. He
Fall 2012
also teaches college level classes relative to testing and test automation, and
has several published articles in various IT publications. TestKIT 2012 Conference


K
The KIT is Coming
gnimoC si TIK ehT
gnimoC si TIK ehT

Fall
2012
http://www.testkitconference.com


Open Sourcery

Test Management is Getting
Crowded
How Crowdsourcing and the Mobile Revolution Have Inspired The Birth of a
New QA Tool

A few years ago, the uTest customers seeking
open source alternatives to have their software
for test management tested in diverse,
were pretty sparse. The realistic environments.
bug tracking portion of Matt Johnston, CMO
test management didn’t of uTest, referred to
lack for automated this type of testing as
tools, but not very many “in the wild testing”.
tools filled the needs It was this “in the wild
for features such as test testing” along with
case administration, the large number of
assigning work, status clients involved in
tracking, reporting, and mobile application (app)
other key elements of test development that helped
management. This is far to lead to the decision
from being the case now, to create a new test
as there are several viable management tool.
open source solutions Johnston explained
for test management. that once it was clear that
Among the most popular a test management tool
tools are TestLink and announcements were not only intriguing
was necessary to better administer the
XStudio. XStudio isn’t completely open due to the fact that several open source
testing process, they polled nearly 1000
source, but is free and has some open test case management tools already exist,
of their customers to get a feel for what
source elements. There are many more but also because uTest indicated in these
announcements that they were partnering solutions were being used. The feedback
open source test management tools
with Mozilla to produce this new offering. revealed that the test management
besides these two, however, including
That’s right! They were partnering with market was extremely fragmented.
RTH (Requirements Testing Hub),
the maintainers of Testopia, an already Small companies often didn’t use test
Salome-TMF, and Testopia. Testopia is
available open source test management management tools at all, while larger
an open source test case management
tool. AST reached out to uTest to get companies felt their solutions – whether
extension for Mozilla’s open source bug
some answers. commercial or open source – had serious
tracker, Bugzilla. Given this existing list
deficiencies. Ultimately, while most of
of available tools, we were intrigued uTest describes themselves as “the their customers were happy with the bug
to hear of a new test management tool world’s largest marketplace for software tracking systems they each employed,
known as CaseConductor entering this testing services.” With a community
already crowded space. A company there was little consensus on the most
of 45,000+ professional testers from
known as uTest announced in April that satisfactory test case management tool.
180 countries around the globe, they
they’d be developing a new test case When asked specifically about tools
are founded on a crowdsource testing
management system (http://www.utest. such as TestLink and XStudio, Johnston
business model that involves a number of
com/press/utest-announces-partnership- indicated that his customer base gave
different testers from different places as
mozilla-build-test-case-management- little feedback on these at all.
opposed to in-house consultants. Testers
system), an announcement that came sign-up with uTest and complete a Open source tools such as TestLink
to fruition on October 25, 2011 as technical profile indicating their technical and XStudio were, however, evaluated
they announced the release of the expertise, along with the platforms, by uTest, along with many commercial
beta version of this system known as environments and/or devices they have tools and, according to Johnston, some
CaseConductor (http://blog.utest.com/ access to. These testers are then enlisted key elements necessary for uTest’s
utest-and-mozilla-release-beta-version- for testing assignments as necessary approach for doing business were
of-caseconductor/2011/10/). These when their profiles meet the needs of missing. One of the key elements that


Open Sourcery

was lacking was the ability to adequately the beta release, CaseConductor was regression tests.
handle remote collaboration. For any developed with the following three core 3. Simple interface – The producers
given testing project, uTest may have values: of CaseConductor set out to keep
testers from all around the world plus 1. Community-oriented – Allow the tool as accessible and simple
testers from a client’s company (for which dynamic distribution of test cases as using Word or Excel since those
the testing is being conducted) involved across different user roles. With are the tools being used to manage
in test implementation. Tests or even test CaseConductor, your testing test cases for a big chunk of the
steps may need to be assigned to various team can be a combination of market. Test cases libraries have
testers in different locations, testers may your internal QA members, open the potential to be quite large, but
need to be able to communicate with source testers and uTesters – all the UX has been streamlined to
one another, and status may need to be of whom can execute test cases keep assignments and results on
appropriately tracked. Another missing simultaneously. Essentially, you one screen.
element that the CMO identified was the can create an unlimited testing
ability to appropriately handle a more Currently, there is no integration
team using a single interface. with functional automated test tools, but
fluid, agile style of testing. uTest has a lot
of customers that develop mobile apps, 2. Environment coverage – In according to Johnston, conversations have
which Johnston says requires a different today’s software ecosystems, already begun with organizations that are
testing approach than that employed for almost every product needs to interested in developing integrations with
larger, more monolithic applications. support an enormous testing matrix it. AST will keep an eye out for this and
consisting of OS versions, device will also be listening to the automation
For these reasons and others uTest felt makers, languages, locations community to find out how well you all
that a new tool was necessary, and upon and more. CaseConductor’s receive CaseConductor into the fold of
working with uTest customer Mozilla, the design around environment test management solutions.
makers of the popular Firefox browser, it attributes allows testers to easily
was found that they had many of the same and visually specify complex Wait! There’s More!
test management needs that apparently compatibility test metrics and
weren’t being filled by Testopia or any select those high priority test cases For more information on
other tool. uTest and Mozilla decided for quicker and more targeted CaseConductor, visit the
to work together to create a new test compatibility testing. The goal CaseConductor repository at
management solution that culminated into for CaseConductor was to enable https://github.com/mozilla/
what is now known as CaseConductor. testing managers to improve their caseconductor-platform/
According to the announcement of product coverage when doing full

Training That’s Process Focused Yet Hands On
Software Test Automation Training
www . training . automatedtestinginstitute . com

Public Courses
Software Test Automation Foundations

Obfuscation
Avoiding Friendly
Fire in the Battle
for Security
by
Dion Johnson


“Obfuscation is not the first technique employed to protect
code from those that wish to reverse engineer or tamper with
it, but is instead another weapon used in a long battle between
programmers and those that seek to use or access the programs
in unintended, undesirable ways.”
Obfuscated Code - Obfuscated code is source or machine code that has been made difficult to understand for humans.
Programmers may deliberately obfuscate code to conceal its purpose (security through obscurity) or its logic to prevent tampering,
deter reverse engineering, or as a puzzle or recreational challenge for someone reading the source code.
http://en.wikipedia.org/wiki/Obfuscated_code

C
yber security and the ability to protect code from being reverse engineered have become
increasingly important as the threats of a system being reverse engineered or compromised
have exploded over the years. It is widely accepted that the risk of these threats cannot
be eliminated, but can be greatly reduced by making application code more obscure and
costly to decipher in an effort to deter the act of reverse engineering. Increasingly, a method known
as obfuscation is being employed to make the code more obscure.
In general, obfuscation is the practice of making something more confusing, unclear, and/or
difficult to understand. This same definition applies to obfuscated code as well. Obfuscated
code is code that has been modified, typically by a program known as an obfuscator, to
be more convoluted and difficult for humans to understand and follow its behavior. The
obfuscated code is functionally indistinguishable from the original code, but the details
surrounding the implementation of those functions have changed.
Obfuscation is not the first technique employed to protect code from those that wish
to reverse engineer or tamper with it, but is instead another weapon used in a long
battle between programmers and those that seek to use or access the programs in
unintended, undesirable ways. Unfortunately, it is often the software automator
that gets caught in the cross fire from this never ending battle, particularly when
the obfuscation affects an application’s user interface (UI). When obfuscation
affects an application’s UI, test automation is severally impacted because these
tests depend on a certain degree of consistency and/or predictability in how
UI elements are referenced. Turning to Figure 1 allows us to look at three
commonly exercised elements: a Username textbox, Password textbox and
a Login button. Automated tests identify these three elements by property
values that have been assigned to them in the application code.
For example, each object may have an ID property that defines it. The
username textbox may have an ID equal to ‘uname’, the password
textbox may have an ID equal to ‘pword’, while the Login button may
have an ID equal to ‘Login’. These property names are descriptive
and probably fairly constant, thus helping the page remain relatively
automatable and maintainable. An obfuscated UI will likely change
these IDs in a manner that will negatively affect their ability to be
consistently accessed by automated tests.
Table 1 and Figure 2 reveal how the ID properties may be
affected by obfuscation. Not only do the properties follow no
set pattern, they may change from build to build or as often
as each time the application is invoked.


• Pre-obfuscation automation
• Label-based identification

ID property = ‘uname’ • Obfuscation map utilization

labels • Image-based automation
ID property = ‘pword’
Pre-obfuscation Automation
Probably the most obvious solution to
ID property = ‘Login’ dealing with obfuscated code is to…
well…not deal with obfuscated code.
Since obfuscated code is typically the
product of a baseline set of code being run
through an obfuscator, it stands to reason
Figure 1: Login Screen
that automation be performed on the code
prior to it being obfuscated. Some take
umbrage with this solution because it
is testing of a code base that is different
from that which goes into production.
In his blog, Nick Olivo, frequent ATI
Object ID First Accessed ID Second Accessed contributor, takes aim at this criticism:
User erlwlkjf4 askflaf6 “I would think that obfuscation would
Password pewtlj90 aasksj2 impact any tests that were run after
the obfuscation process. So is it safe
Login alsjfafs3 aljfljlaj5 to say that companies who employ
unit tests must be running those tests
Table 1: Obfuscated Properties prior to the obfuscator running? If so,
should the manager be concerned
that the unit tests were run against
non-obfuscated code?
Obfuscation would also prevent
any other type of post-build white
box testing; so any tools I've written
that call methods directly from dlls
or invoke web service calls would
be stopped dead. So what's the best
ID property = ‘erlwlkjf4’ solution here? Is it to do the bulk of
labels your automated testing against non-
obfuscated code and then do manual
ID property = ‘pewtlj90’
sanity checks against an obfuscated
build? Or is there a better way that I'm
not seeing?”
ID property = ‘alsjfafs3’ (http://autonomicon.blogspot.com/2009/02/
obfuscation-question.html)

Label-Based Identification
Figure 2: Obfuscated Login Screen Obfuscation is typically a process that is
transparent to the user of an application.
Therefore, while much of the code is

T
changed during the obfuscation process,
items that we may reasonably expect to
remain stable are text-based UI labels,
his type of UI instability has brought down many test automation along with their relationship to fields on
efforts, because it makes the test automation un-maintainable, a page. For example, Figure 2 has two
particularly as the automated test bed grows. text boxes, each preceded by a label that
provides information to the user on how
There are several proven techniques, however, that may be those textboxes should be used. The labels
employed to provide cover for test automation when development are not necessarily directly linked to fields
injects obfuscation into the coding battles against hackers and the that they describe, but there is normally a
like, including: pattern for assigning labels to fields. Once


is moved around the screen and/or its
table row 1, table row 1, column 2 properties change.
column 1
Cell-based identification is useful when
the page objects are contained within a
table. Provided that each label/field pair is
contained within cells separate from other
label/field pairs, the cell configuration
table row 2, may be used for uniquely identifying a
column 1 table row 2, column 2 field.
The first step in making this work is to
programmatically locate the label that is
associated with the desired field on the
Figure 3: Cell-based Object Identification screen, and then get the table cell in which
the label exists. Based on the pattern used
the pattern is identified it can be assessed Thus, gathering all of the page’s objects in the application for storing label/field
and used for object identification by an into an array named pageElementArray, pairs, the label cell information will be
automated test at run-time. would produce an array or collection with used for uniquely identifying the desired
the following items: field. For example, if the label typically
Three approaches for using labels to
exists in the same cell as its associated
uniquely identify fields are: pageElementArray[0] = Username label field, then code can be written to get the
1. Location-based identification pageElementArray[1] = erlwlkjf4 textbox child textbox of the same cell element in
2. Cell-based identification pageElementArray[2] = Password label
which the label exists. If the field typically
exists in the same row, but an adjacent
3. Property-based identification pageElementArray[3] = pewtlj90 textbox column of the label, as illustrated in
pageElementArray[4] = alsjfafs3 button Figure 3, code can be written to get the
Location-based identification initially child textbox element of the adjacent cell.
seems closely linked to the volatile Upon looping through the array or
coordinate-based automation approach In a roundabout way, the cell-based
collection of objects and locating the
that we’ve been taught to steer clear of. approach is dependent on location,
desired label, the automation loop can
Despite appearances, however, it is not because you’re expecting the field to be in
be set to return the very next item in the
the same thing, but is instead a much more a location relative to its associated label.
array or collection that is a textbox. So,
stable approach to identifying application The third label-based approach, property-
if the automated test needs to access the
fields. Location-based identification based identification, is the only label-
‘Password’ textbox without using the
using labels is based on the stability of based approach that is not dependent
dynamic ID property that is currently
the label names and the pattern used by on location at all. This approach is most
equal to ‘pewtlj90’, it would simply
programmers for coupling labels with useful when the label element has an
loop through the array of objects and
fields, which is a more robust approach attribute that is directly tied to an attribute
find the ‘Password’ label. Upon find the
than using the actual position of an object of its corresponding field.
‘Password’ label at array item 2, the loop
on a screen. - which has been set up to return the very For example, a label may have a NAME
Location-based identification first requires next array item that is a textbox - will attribute that is identical to the ID attribute
that your test automation approach provide return array item 3, which is the ‘pewtlj90’ of its corresponding field, as illustrated in
a mechanism for obtaining all fields on a textbox. Provided that the label name text Figure 4. In this situation, automated test
page into an array or collection, based on remains constant, the associated field code can be written to programmatically
the order in which those objects appear can be found even if the field (and label) obtain the desired label based on its label
on the screen. For web pages, the “all”
document object model (DOM) method NAME property = ‘erlwlkjf4’
makes this possible. If you are using xpath
for web object identification, “//*” will
do the trick. Once the objects are stored ID property = ‘erlwlkjf4’
in an array, a loop may be established to
traverse that array or collection in search ID property = ‘pewtlj90’
of a label element with the desired label
text. Then, based on the pattern identified
for coupling labels with fields, the desired ID property = ‘alsjfafs3’
field can be identified. For example, the
application screen represented in Figure
2 has its fields preceded by labels that NAME property = ‘pewtlj90’
represent that field, evidenced by the fact
that the ‘Username’ label precedes the Figure 4: Property-based Object Identification
textbox used for entering a username.

text, and then get the NAME property (TABOK) Guidebook describes Image-
associated with that label. The automated based automation as follows:

C
code can then be instructed to find the …often based on Virtual Network
field with an ID property that is equal to
the label NAME property that has been
Computing (VNC), [an image-based
automation tool} relies on image
ontribute
captured. recognition as opposed to object

C
Upon identifying the appropriate label- recognition (Context Sensitive) or
based approach, functions and/or methods
can be created to perform the necessary
coordinate recognition (Content
Sensitive). ontent
computations in a way that is seamless to VNC-based tools follow the two-

T
the design time test automation process. computer system, with the automated
The automators would simply need to
use a field’s associated label text for
tool residing on the client machine and
functioning as the VNC client. The server
oday
identification of that field as opposed to machine on which the AUT is installed
properties directly related to the desired will run a VNC server that communicates
field. with and transmits images to the VNC
client. These tools, therefore, recognize
oBFUSCATION mAPS application objects based on analysis of
the transmitted images.
At times, when code is passed through Community
an obfuscator, in addition to outputting This mode is more closely related to
context sensitive automation than content
Comments Box
the obfuscated code an obfuscation map
may also be produced. This map provides sensitive, because it is not completely
a direct relationship between old property coordinate-based. These images may still
values and the new property values of be located if they are moved to a different
application fields and other elements. If an screen position or if their internal
obfuscation map is not produced, you may properties change. Thus, image-based Announcements &
be able to request that the development automation can be useful for handling Blog Posts
team modify the obfuscator to produce applications that utilize obfuscation.
such a resource, because it could be
invaluable. With an obfuscation map, a cONCLUSION
utility can be written to synchronize the
Obfuscation is a battle between software
object information used by the automated
producers and their enemies, and if testing
scripts with the new object information Automation
suffers, ultimately the software producer
in the application prior to or during script
suffers. Follow the approaches defined Events
execution. in this article to avoid being a casualty
of obfuscation or any other development
IMAGE-Based AUTOMATION approach that injects instability in the
The Test Automation Body of Knowledge application UI field properties.
As a registered user you
can submit content directly
to the site, providing you
with content control and
the ability to network with
like minded individuals.

Learn more today
at

http//www.about.automatedtestinginstitute.com


A Automated
Mobile Automation

M obile
esting
With Traditional Tools & Beyond
by Patrick Quilter

Has consumer demand already chosen mobile devices to replace the personal computer?
Will traditional webpage design with drop downs and text boxes lose ground to mobile
touch screen data entry? Will everything we know about creating automated tests for
applications change because of mobile devices? Don’t abandon your old automation
frameworks just yet. You’ll still need them to expand into the untamed world of automated
mobile testing.
Mary Meeker, a renowned Internet analyst with Morgan Stanley, highlighted the significance
of mobile consumer demand in her latest “State of the Internet” report. Meeker predicted
that within the next five years, “more users will connect over mobile devices than desktop
PCs.” Since 2010 the trend of smartphone shipments outpacing PC shipments continues
to surge and gives credence to Meeker’s forecast (Ingram, 2010). Anyone can see that
a multitude of organizations have already embraced this technology swing by developing
applications explicitly to establish a mobile presence. Recognized companies continue to
offer free or inexpensive apps for both business and entertainment purposes.

While having a mobile presence is significant for organizations,
it is also becoming imperative to have stable applications and mobile
“Simulators on
websites. PhoCusWright conducted an 800 participant survey to evaluate Windows or Macs
user tolerance for buggy mobile software. The survey revealed that 37% of
smartphone users indicated that they would be less likely to use that particular allow teams to employ
application again and 28% indicated that they would seek out competitor
applications (Leggatt, 2010). These facts illustrate the importance for sound traditional automation
application development practices and the ability to automate test cases. With
so many different devices and new ones emerging all the time, it’s infeasible
engines like Quick
to conduct sufficient testing for each, manually. Automation’s luster is the
ability to execute large volumes of repeatable scenarios that a person could
Test Professional or
not undertake manually. Selenium”

To Simulate or Not to Simulate software under development is going device transactions has spurred the growth
to be application based or web based? of application development service
A typical beginning for application
If application based, functionality will providers. A capable service provider will
development teams journeying the path
vary depending on device platform such offer an on-demand testing infrastructure,
of mobile testing is to create automated
as iPhones, Androids, Blackberries, and multiple devices, and automated scripting
scripts that execute on desktop simulators.
Windows. Although some solutions capabilities.
A simulator has the same look-and-
are materializing to allow developers In terms of mobile testing this type of
feel of a mobile application displaying
the ability to create one codebase that service is invaluable because it accounts
on a device, however, it executes on a
deploys over multiple devices (regardless for all major strategic characteristics
desktop operating system. Teams utilize
of iPhone, Android, etc.) the typical under one solution. But what specific
a simulator by constructing business
approach is to maintain a separate service provider existing today offers all
flow, user designs, and verifying basic
codebase per platform. A savvy tester of this?
functionality of their applications.
will see the vulnerability in maintaining
Simulators on Windows or Macs allow
multiple codebases. This implies multiple
teams to employ traditional automation
builds to verify.
Meet Keynote
engines like QuickTest Professional or DeviceAnywhere
Selenium to create test cases. Device manufacturers and operating
A great example of this type of service
systems have the biggest impact on
Although using a simulator adds provider is a company called Keynote
how applications function even across
value to an application development DeviceAnywhere headquartered in
the platform families. For example, the
practice, verifying functionality is San Mateo, California. Keynote
HTC EVO and Samsung Galaxy are both
limited. Neglecting to conduct tests on a DeviceAnywhere helps organizations of
part of the Android family, however, the
physical device causes vulnerabilities to all sizes bring their products, services,
same application installed on them could
application performance. This realization and IT applications to a mobile platform.
function differently. The reason for
has its drawbacks. For one, the list of Their cloud-based solution hosts a phone
this is the manufacturer can manipulate
popular devices is growing rapidly, and bank of over 1000 real devices that are
how processing, GUI rendering, and
testing must occur for all devices on used for mobile application testing, test
form factors affect execution on their
the list. Secondly, how do application automation, monitoring, and development
devices. Web based functionality is more
development teams execute test cases worldwide. This is possible through
straightforward because developers and
outside of the PC-based automation tools? their award winning, end-to-end studio
testers converge on a single deployment
Are there any strategies to handle these platform that cuts operational expenses
of code to a server. Configuration settings

A capable service provider will offer an on-demand testing
infrastructure, multiple devices, and automated scripting
capabilities.
determine functionality based on the
challenges? and time associated with testing mobile
device a user connects with to the mobile
applications (HP/KDA datasheet, 2009).
website. Normally, functionality across
What’s the strategy? product families will work the same.
The most challenging aspect of an Although not a guarantee, testing a mobile Accessing the Environment
automation strategy is devising a practice website with an HTC EVO and Samsung Keynote DeviceAnywhere users first
that optimizes test case execution across Galaxy that is running the same version of gain access by logging into a web-based
multiple devices. Ideally we’d like to Android should produce the same results. portal that provides a management
craft a single test case that can execute Testing just one of them would most likely dashboard, training information, and
on multiple devices, otherwise we would be sufficient. The device carrier is also links to support resources. From there,
have to maintain a separate copy of each significant when considering a strategy for the user downloads the client-side
test case, per device. Test sets (a collection both mobile applications and web based software onto their individual desktop to
of test cases that compose a test plan) software. Network speeds and protocol experiment with the actual devices. This
would become maintenance intensive restrictions are common performance gives them a window into the Keynote
over time without this reuse capability. hinderers for mobile software. The key DeviceAnywhere devices hosted in the
Mobile devices cause a significant amount for any organization will be how to turn cloud. A visual representation of the
of variation for development and testing the strategy into reality. physical device displays once the user
efforts. First, is the question of whether Luckily consumer demand for mobile acquires a device. Users can license any


K
The KIT is Coming

If you thought ATI’s 2011 event was
good, wait until you see 2012.
http://www.testkitconference.com


Most tools rely on open and published APIs to obtain the
discrete properties of a control. Mobile platforms do not
provide the traditional APIs, rendering your favorite tools
useless in mobile
of the hosted Keynote DeviceAnywhere developed for one device it can be cloned script-less actions correspond to these
devices upon request. Acquiring devices to run on multiple devices. It’s script- screen interaction properties.
is an important concept because team less because automation steps are created Walking through an example, a user
members can share and schedule times by dragging and dropping icons which developing a script for an iPhone will first
for when they want to test on particular are preset to perform specific actions. acquire the device and is then presented
devices. Stringing together a series of icons with a window view of the iPhone next
Once a device has been acquired, the user produces a whole test case. to a blank execution grid. This tells the
can control all aspects of the physical For users that are familiar with traditional scripting component that the particular
device through mouse clicks and keyboard web-based automation, the biggest implementation is going to interact with
strokes to mimic the actions as if they had adjustment may be getting accustomed an iPhone device. The users will drag-
the device in their hand. The user can to how objects are represented on mobile and-drop Keynote DeviceAnywhere
also manipulate the features of the devices pages. Tal Barmeir, CEO of Experitest, actions (icons) onto the execution grid
such as buttons, LCD screen, speaker, in an order that corresponds to test case
ringer, battery, power-plug, and data- steps. The actions available to the user
cables. The Keynote DeviceAnywhere can search for some text on the screen,
control sliders allow the user to change Script-less automation: enter text, click on GUI icons, or validate
these setting and pre-condition the
device before conducting a test case.
“ Script-less means no scripting/
a checkpoint area located on the screen.
The actions are separate steps represented
programming in the test automation
Another controller known as the Device as separate icons. During execution, the
tool’s native language. This
Console is also accessible for typing actions perform the key strokes and touch
doesn’t mean there are no scripts
text and performing SMS messaging. screen interaction that a human has with
involved. Explicitly, this means
This Console can monitor HTML traffic the iPhone. The iPhone’s screens are
while automating test cases, there
between webpages and the device. The driven by the actions in the execution grid
is no more need to program scripts
importance of controlling these features until the last one has been reached.
for each test case”
is in the fact that mobile test cases have Once the test case has been created and
some unique verification steps when (Qualitia, 2011).
saved, the test case can then be cloned
compared to PC based applications or for other devices. All of the steps that
webpages. For example, a user may want were created for the iPhone could be
to see what happens to their application enabled for an HTC EVO. The benefit of
when the phone rings or a text message explains that, “existing industry-leading cloning test cases is that for the most part
is received (Keynote DeviceAnywhere, test automation tools are restricted by application flow stays consistent from one
2010). Accessing the devices through object recognition limitations. Most device to the next. All action command
the studio and using the Device Console tools rely on open and published APIs to steps are converted for use on the new
allows a user to set the stage for these obtain the discrete properties of a control. device. In some cases, actions that
types of scenarios. Mobile platforms do not provide the contain specific GUI representations will
To accomplish automation, Keynote traditional APIs, rendering your favorite need updating after the cloning process.
DeviceAnywhere provides a module tools useless in mobile” (Barmeir, 2011). Typically these types of updates are
specifically for managing requirements, Unlike web page objects, mobile objects minimal and identifying where to make
test case steps, execution tracking, and are identified differently. Web-based the adjustment is intuitive to the users.
result reporting. Each of the items are objects are referenced by property tags This is because the user has the advantage
organized within a project and shared such as an object name or ID. A scripter of using the Keynote DeviceAnywhere
between team members for collaborative can use tools to break down the various technology to stay within a consistent
testing. A user can start creating types of objects on a page and associate framework. Users can also mitigate the
automation steps by accessing Keynote data with those objects. With mobile amount of updates by incorporating an
DeviceAnywhere’s device-agnostic, application rendering, property tags are automation development strategy or best
script-less automation technology (HP/ not referenced in this way. Instead a user practices within the framework that fits
KDA datasheet, 2009). The technology is interacts with the screen by anchoring on their applications. Trial and error will
device-agnostic because when a script is specific text, icon recognition, and pixel help identify the best strategy.
locations. Keynote DeviceAnywhere’s


Incorporating Framework an add-in that a user can download for editing complex logic before the compile
Automation this capability. For those that are QTP stage. The Keynote DeviceAnywhere
developers, you know that when launching APIs provide a great way for organizations
For many applications, using the script- QTP you have the option to select which that don’t have expensive commercial
less approach is appropriate especially add-ins are loaded for your development tools or the budget to purchase them
if functionality changes are minimal and execution sessions. After installation, to still get the most out of the mobile
between code updates or if the particular the user just needs to check the box in the automated testing experience.
application doesn’t have an abundance add-in selection screen for the Keynote
of data entry. An application heavy with
links and button pushes would be covered
DeviceAnywhere add-in. Once this step How We Integrate with
is complete, a user can reference any of Keynote DeviceAnywhere
adequately. This is also a useful approach the actions they’ve created to drive the
if the test case suite is small and easily automation capabilities on any of the Here at Quilmont we have a pre-developed
maintainable. devices. Interacting with a specific device framework that comprises a user-friendly
Users that have applications with heavy is managed in the QTP object repository interface for managing application
data entry in many business scenarios will where every device that is going to be properties and creating test case steps.
naturally want to incorporate industry tested is a separate object in the repository. Our out-of-the-box function libraries
standard practices such as creating a data The handshake between the repository integrate with QTP/QC, Selenium,
driven or keyword driven framework. This and the devices are controlled by a unique CodeUI, Rational Functional Tester,
type of approach enhances script reuse, ID property. From this point a user can and yes, Keynote DeviceAnywhere.
quicker maintenance, and offers the ability design a framework and separate the For our Keynote DeviceAnywhere
to create test cases without consuming a automation code from their test case data. implementation, we provide plug-ins for
Keynote DeviceAnywhere license (the As a best practice, a user should create both QTP and JAVA APIs to deliver a
last section provides an example of how a design where QTP does the repetitive framework with keyword and data driven
this can be accomplished). For those well calling of a sequence of Keynote capabilities.
versed in automation design, the script- DeviceAnywhere actions while passing In our setup we employ a three layer
less approach may sound like a variation parameters to those actions. Keystrokes approach. In the first layer we have several
on record and playback. Also restricted and data entry can be controlled in keyword actions packaged at a project level
access to the scripting language may strike spreadsheets or other data harnesses. All for use in the Keynote DeviceAnywhere
automation developers as a hindrance to results and screenshots during execution Studio. Keynote DeviceAnywhere
their effectiveness. In other words there will roll up into the QTP execution report Studio allows projects to be imported
are some things a seasoned programmer (Keynote DeviceAnywhere, 2010). and exported. For us, this means we
can accomplish more effectively with code can import and export our keyword
actions to any Keynote DeviceAnywhere
Keystrokes and data entry can be Studio installation. Our second layer is
a driver script which could be the QTP

controlled in spreadsheets or other or JAVA API version depending on the
customer request. This is the layer that
processes the main execution and calls
data harnesses. the keywords depending on the test case
information it receives from the third
A second option for organizations that do layer. The third layer is our framework
than relying on script-less actions. Not to not have Functional Testing (QTP) and interface. In this layer our customers
worry because Keynote DeviceAnywhere don’t plan to purchase it, is to create a can manage any application properties
offers two ways to interact with their data driven or keyword driven framework (for mobile application this would be
automation engine so that a tradition data by utilizing Keynote DeviceAnywhere’s text, icon objects, pixel information)
framework can be created. JAVA based APIs to control the actions. and create test case steps. The test
For organizations that use HP/Mercury In the same way a user can create case steps call keyword actions in the
Functional Testing (QTP) and Quality reusable code and incorporate a datasheet Keynote DeviceAnywhere studio while
Center, there is a tremendous opportunity to drive the commands. Keynote also passing parameters to these actions
to continue utilizing these tools and DeviceAnywhere provides a JAVA editor through the driver script. This allows our
configure them to work with the Keynote internally to create the API programs and customers to first create all of their test
DeviceAnywhere Studio. This allows conveniently stores all components inside case information outside of the Keynote
organizations to expand upon the familiar the Keynote DeviceAnywhere Studio. In DeviceAnywhere Studio which cuts down
features of QTP and QC as well as enhance addition, the Keynote DeviceAnywhere on license consumption and then log into
their tester’s skillsets. This option allows Studio can interact with other JAVA the Keynote DeviceAnywhere studio
a QTP developer to extend their current editors such as Eclipse to get the full when they are ready to execute their test
framework to invoke existing actions in capabilities of a JAVA environment cases.
the Keynote DeviceAnywhere Studio. (Keynote DeviceAnywhere, 2010). This An example of how the three layers work
Keynote DeviceAnywhere has created is convenient for managing packages and together can be illustrated as follows:


Let’s assume the device we are testing is an HTC EVO and our These three calls will log into the application and drive the screen
application has a login screen. The login screen has objects for to the next page.
a user name, password, and a login button. The goal of our test
Statistically we can see that mobile devices are becoming
case is simply to verify that we can log into the application and
the preferred way of conducting business transactions for
get to the next screen. Starting with layer three (framework
consumers. Touch screens provide a more desirable method
interface) we would represent each of the objects (user name,
for collecting customer information than keying data into web
password, and login button) in the section of the interface that
manages object information. Since these are mobile objects we forms or scrolling through overpopulated dropdown boxes.
would most likely reference the text labels for each object. In Mobile devices have arrived and more companies are embracing
the test case creation section of the interface we would create the the change every year. It’s fair to describe the application
step sequence where each step would reference an object name, development techniques for mobile software as still being in its
value, and keyword action. early stages. However, applying discipline through traditional
application development practices (test automation being one
of them) will assist future standardization. From an automation
developer’s standpoint, learning the tools and techniques in this
early stage will have great implications on a person’s individual
Framework Interface: skillset. It’s an amazing opportunity to learn, discover, and help
Step 1: User Name | user1 | EnterData standardize the untamed world of automated mobile testing.

Step 2: Password | pwd1 | EnterData
References:
Step 3: Login | button | Click
Barneir, Tal (2011), Need for Mobile Test Automation
Tool on Various Platforms (Android, iPhone, Blackberry,
Symbian & WindowsPhone 7), Mobile Application
In the second layer, the driver script would read these steps Testing. Retrieved: November 17th 2011 From: http://
from the framework interface and call the action in the Keynote
DeviceAnywhere Studio that corresponds to the action in the
www.mobileappstesting.com/category/mobile-application-
step sequence. The driver passes the object name and value to testing/mobile-application-testing-and-automation-tools/#
the keyword action.
HP/KDA Datasheet (2009), Test Automation for Mobile
Devices, Hewlett-Packard. Retrieved: July 16th 2011 From:
http://support.openview.hp.com/
Functional Testing (QTP) or JAVA API Driver:
GET Step 1 from interface, CALL EnterData in Keynote Keynote DeviceAnywhere (2010), Keynote
DeviceAnywhere Studio, PASS User Name | user1 DeviceAnywhere Test Center User Manual for Enterprise
GET Step 2 from interface, CALL EnterData in Keynote Deployments, Keynote DeviceAnywhere. Retrieved: June
DeviceAnywhere Studio, PASS Password | pwd1 15th 2011
GET Step 3 from interface, CALL Click in Keynote
DeviceAnywhere Studio, PASS Login | button Ingram, Mathew (2010), Mary Meeker: Mobile Internet
Will Soon Overtake Fixed Internet, Gigaom. Retrieved:
October 25th 2011 From: http://gigaom.com/2010/04/12/
mary-meeker-mobile-internet-will-soon-overtake-fixed-
In the first layer, the pre-developed keywords receive the internet/
parameters from the driver script and use the Keynote
DeviceAnywhere automation engine to translate the action for
use on the HTC EVO. Leggatt, Helen (2010), Travelers display low tolerance for
mobile app, website glitches, Gigaom. Retrieved: October
25th 2011 From: http://www.bizreport.com/2010/08/
travelers-display-low-tolerance-for-mobile-app-website-
Keywords stored in Keynote DeviceAnywhere Studio: glitches.html#
RUN EnterData, FIND User Name, TYPE user1
RUN EnterData, FIND Password, TYPE pwd1
Qualitia (2011), Defining Scriptless, QualitiaSoft. Retrieved:
November 17th 2011 From: http://www.qualitiasoft.
RUN Click, FIND Login, TOUCH Login
com/pdfs/Definition%20of%20Scriptless%20Test%20
Automation.pdf


Are You Contributing
Content Yet?
The Automated Testing Institute relies heavily on the automated testing
community in order to deliver up-to-date and relevant content. That’s why
we’ve made it even easier for you to contribute content directly to the ATI
Online Reference! Register and let your voice be heard today!

As a registered user you can submit content directly to the site, providing you with
content control and the ability to network with like minded individuals.
Community
Comments Box >> Community Comments Box - This comments box, available on the home page of
the site, provides an opportunity for users to post micro comments in real time.
>> Announcements & Blog Posts - If you have interesting tool announcements, or
you have a concept that you’d like to blog about, submit a post directly to the ATI
Announcements & Online Reference today. At ATI, you have a community of individuals that would love
Blog Posts to hear what you have to say. Your site profile will include a list of your submitted
articles.
>> Automation Events - Do you know about a cool automated testing meetup,
webinar or conference? Let the rest of us know about it by posting it on the ATI site.
Automation Add the date, time and venue so people will know where to go and when to be there.
Events
Learn more today at http//www.about.automatedtestinginstitute.com

t
m sat
m Wi
by To

We all have the
necessary components on
our systems to build a
solution of our own.

Test
Management

“A simple system...to coordinate your test procedure development effort.”

Integrating MS Office
F C or-E ost ffective
Test Management
A Homegrown, Desktop Solution for Test Management

A s in any test effort, we write test procedures and eventually
require some way to keep track of all of the documentation.

Many of us become spreadsheet jockeys with workbooks of our tests
and other supporting information tucked away in the labyrinth of
our disk drives. The management becomes increasingly cumbersome
leading to duplication and confusion. We forget if we wrote a
procedure to cover a case, a new member joins the team and does not
know what has been written, or we are unable to locate the needed
procedure. Our typical solution is to write a new test and add it to
the workbooks for safe keeping.

Another complication arises when we want to create a test set for a
particular purpose, such as a user acceptance test or regression test
set. We may also get requests from management about which tests
cover which requirements, or about getting some sort of a test report.
Pulling the different parts together to create the needed documents,
and managing the whole process becomes time consuming and error
prone. In other words, a perfect opportunity for automation!

There are a number of nice commercial solutions available that
answer the task quite well; but, what if we do not have a large
team or a budget that can afford one of the packages? We all have
the necessary components on our systems to build a solution of our
own. What follows is an explanation of a system I developed with
Microsoft (MS) Office, and the same can be adapted for OpenOffice.
Org.


Requirements
Before jumping into the details of the
design, a few requirements need to be
established. After having used a number
of tools over the years, I settled on the
following basic functions:

• Viewing of all tests
• Filter and Sort
• Test Set Production
• Track Status and Author
• Allow Attachments
• Associate Requirements
• Track Team Members
• Allow Team Communication

Figure 1: Test procedure database directory structure
Basic Architecture
Most of us have the Microsoft Office need to share out the location to your team if you choose to store it on your own work
suite on our machines and it makes sense station. This organization is optional, so, feel free to adjust to your own desire.
to develop for something the majority This organization works well, as it helps to keep things organized in a way that seems
has access to. Most of us, though, forget logical. At the root directory is the actual database – in this case it would be under
that Office is more than Word, Excel and the Test Procedures directory. Under that are subdirectories for each of the systems
PowerPoint. At the heart of this system of your particular test effort. If this was an effort to test MS Office, they would be
is the MS Access database. The test Word, Excel, PowerPoint, etc. Under each of the system directories are the subsystem
procedures are in MS Excel spreadsheets directories and under that are the test files themselves.
to make the formatting somewhat
easier, and allow for some additional The naming convention for the files mirrors the above directory structure. It helps
functionality without the complication of
Word document formatting. In addition,
there is a hook into Outlook to provide
some automated communication for the
team.
Behind the scenes, MS Office uses Visual
Basic for Applications (VBA) for the
programming language. This makes it
easy to add more functionality or modify
any of the existing functions.
The database provides a place to store the
details about the test procedures, but the
file system contains the actual spreadsheet
files. Using links in the database, the tests
can be displayed and modified. New tests
are created from the information entered
into the database. The test procedure
files are kept as originals, and modified
as needed and copies are created and
collected to build test sets for the actual
runs.
Refer to Figure 1 below. The database is
best stored on a networked drive shared
out to the team. At a minimum, you will Figure 2: Tables and relationships


locate a file easily as the name contains the
basic structure:

<ProcNum><System><Subsystem>.xlsx

The ProcNum is automatically generated
by Access, and the initial spreadsheets get
the number on the tabs as you will see.
Delving further into the structure of the
database, we have tables that not only
contain the data for our test procedures but
also the various components associated
with the test procedures. To make up a test
procedure, we need to know certain details
such as description, system, subsystem,
status, etc. Some of the information can be
selected from drop-downs that draw from
the data in the tables. Figure 2 shows the
tables and their respective relationships
within the database. Keep in mind when
you modify the structure of the tables (e.g.
add or remove columns), you may need to
remove a relationship and restore it after
Figure 3: Main Page Form
you finish the modification.
The tables are:
• System – contains the names
of the systems and their related
subsystems. Typically you can
use work breakdown structures,
diagrams or other documents to
develop items for this table;
• Members – the team members
and related information such
as telephone numbers, e-mail
addresses, etc;
• Status – stages of the procedure
development from In Progress to
Obsolete;
• Methods – the test methods for
each procedure such as Analysis,
Observation, and Test;
• Requirements;
• Procedures;
• Req Attachments – requirement
attachments;
• Attachments – procedure
attachments;
• ReqtoProcID – the associated
IDs of the requirements and test
procedures.

Forms and Reports
MS Access uses forms and reports to serve
as the interfaces to the tables and data. The Figure 4: Test Procedure Entry/Edit Form


Automated softwaretestingmagazine december2011

Automated softwaretestingmagazine december2011

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Automated softwaretestingmagazine december2011

Similar to Automated softwaretestingmagazine december2011 (20)

Recently uploaded

Recently uploaded (20)

Automated softwaretestingmagazine december2011