3. conditions of merchantability, of fitness for a particular purpose
, of accuracy or completeness of responses,
of results, of workmanlike effort, of lack of viruses, and
of lack of negligence, all with regard to the
document.
THERE IS NO WARRANTY OR CONDITION OF NON‐ INFRI
NGEMENT OF ANY INTELLECTUAL PROPERTY RIGHTS
WITH REGARD TO THE DOCUMENT. IN NO EVENT
WILL GEOMETRIC LIMITED BE LIABLE TO ANY
OTHER
PARTY FOR LOST PROFITS, LOSS OF USE, LOSS OF DATA
, OR ANY INCIDENTAL, CONSEQUENTIAL, DIRECT,
INDIRECT, OR SPECIAL DAMAGES WHETHER UNDER
CONTRACT, TORT, WARRANTY, OR OTHERWISE,
ARISING IN ANY WAY OUT OF THIS DOCUMENT, WHETH
ER OR NOT SUCH PARTY HAD ADVANCE NOTICE OF
THE POSSIBILITY OF SUCH DAMAGES.
Confidentiality Notice
This document is disclosed only to the recipient pursuant to a co
nfidentiality relationship under which the
recipient has confidentiality obligations defined herein
after. This document constitutes confidential
4. information and contains proprietary information belonging to
Geometric Limited, and the recipient, by its
receipt of this document, acknowledges the same. The recipient
shall use the confidential information only
for the purpose defined above for which this document is suppli
ed. The recipient must obtain Geometric
Limited’s written consent before the recipient discloses any info
rmation on the contents or subject matter
of this document or part thereof to any third party which may in
clude an individual, firm or company or an
employee or employees of such a firm or company. The recipien
t acknowledges its obligation to comply
with the provisions of this confidentiality notice.
3
Getting ready for PLM
Contents
Introduction............................................................................
............................................ 4
5. Automation Frameworks..........................................................
.......................................... 4
The Need for a Framework
....................................................................................... ........
..........4
Types of Frameworks...............................................................
...................................................5
Test Automation Life Cycle (TALC).........................................
............................................. 6
Cost of test automation............................................................
.......................................... 9
Cost of the Automation Tool....................................................
...................................................9
Cost of the Automation Effort..................................................
...................................................9
Benefits of Test Automation.....................................................
........................................ 10
ROI on Automation: A Conservative Estimate...........................
....................................... 11
What can be Automated...........................................................
........................................ 12
What to Automate, and When
......................................................................................... 13
Relationship of the TALC to the SDLC.....................................
.......................................... 13
6. Challenges in Automation........................................................
......................................... 13
Automation Best Practices........................................................
........................................ 14
Conclusion..............................................................................
........................................... 14
About the Author.....................................................................
......................................... 15
About Geometric
...............................................................................................
............... 15
4
Getting ready for PLM
Introduction
Of every four dollars spent on software development, at least on
e dollar goes towards testing. Of
this dollar, at
least 70 cents go towards functional testing. Within functional t
esting, test case
design is the key determinant of the quality of testing, but it is t
7. est execution that is the single
largest consumer of effort. This effort multiplies rapidly with th
e number of configurations to be
tested, is required in cycles, and is repetitive to a significant ext
ent. Companies are constantly
looking for ways to ‘rationalize’ this cost and enable comprehen
sive testing, rather than having
to make risky choices about which platforms to test
thoroughly. Test automation offers an
elegant way to tackle this challenge, though it is by no means an
instant panacea. This explains
why worldwide spending on test automation is growing 10% fas
ter than spending on testing as a
whole.
Test automation refers to the development and usage of
tools to determine the success or
failure of pre‐ specified test cases, against the Application Und
er Test (AUT), without human input.
The primary objective of test automation is to reduce repetitive
manual tasks.
Several commercially available test automation tools allow the r
ecording of user actions taken on
the AUT. The recording session generates scripts that can later
be replayed without human input.
8. This paper attempts to outline the what, why, how and when of t
est automation. It focuses more
on automation for the QA team, as compared to that for the Dev
elopment team.
Automation Frameworks
The Need for a Framework
An automation test suite can be built by simply recording variou
s test cases. However, it is often
possible to significantly enhance the reusability and maintainabi
lity of such suites, by developing
frameworks for automation.
Reusability
•
Consider an AUT that has a combo with five possible values. Fu
rther, assume there are five
separate test cases, each using a different value, but identical in
other respects. If we use the
‘record and replay’ approach, we would have to record five diff
erent test cases. Instead, we
can abstract the combo as an argument to be passed to the script
, and then call the same
script five times with different arguments.
• There are certain user actions – log on, for instance –
9. that might be common to several test
cases. Again, these actions (sequences) can be abstracted
out and reused in several
automated test cases, rather than recording the same sequence m
ultiple times. Short action
sequences can also be used to compose long ones.
5
Getting ready for PLM
Maintainability
Consider a website (the AUT) that has a set of commonly neede
d links appearing in several web
pages. Further assume that we have automated the testing of this
website.
In the absence of any frameworks, the
identification of the links will be arbitrary: it could
be indexed on the current page layout, for
instance. In a future version of the website, these
indices could change – say, due to the addition of
10. other links. In this case, each test script involving a
web page that has this set of links will be broken,
and will need rectification. This could mean a
significant rework effort, as there could be many
such pages and perhaps the whole website.
Additional benefits of a framework
A framework makes it easy to provide
exception handling and wait
mechanisms, both of which further
increase robustness.
_________________________________
A well designed set of utilities in the
framework makes it easy to create a
dashboard of the test results.
Instead, if named references are provided to these links and onl
y these names are referred by
the test scripts; then changes will be needed only to (re)map the
names to the actual links.
Conceptually, a framework eliminates ‘hard coding’ and provide
s ‘modularization’. Based on our
experience, we estimate that in the long run, up to 50%
of the script development and
maintenance effort can be saved by investing in creating an auto
11. mation framework.
Types of Frameworks
Automation frameworks can broadly be classified into three typ
es.
• Data driven: In this approach, variables are used to hold
test data. At runtime, these
variables could be loaded from an external data source. This app
roach reduces the problem
of hard coding. Note that identification of GUI elements is still
hard coded; for instance, the
script might contain instructions that effectively mean ‐
Select the image whose index
number is 3.
•
Keyword driven: In this approach, the input, user actions and ex
pected output are encoded
using keywords that are typically independent of the AUT. A tes
t case is encoded as a record
composed of these keywords. Test suites composed of such test
cases are typically stored in
tables. As part of the framework development, scripts are writte
n to translate these records
to a specific AUT.
This approach reduces the problem of hard coding and also prov
ides modularization.
12. •
Hybrid: This approach combines the two approaches outlined ab
ove, and brings in benefits
derived from both. Over a period of time, hybrid frameworks ha
ve emerged as the de facto
standard for automation requirements. Based on our
experience, we recommend serious
evaluation of the hybrid approach during framework design.
6
Getting ready for PLM
Test Automation Life Cycle (TALC)
Broadly, the Test Automation Life Cycle can be depicted as foll
ows. It should not be assumed that
efforts (person weeks) will be proportional to the schedules (cal
endar weeks) indicated.
Release Cycle n n+1
Month 1 2 3 4 5 6 7 8 9 10 11 12
Study
Framework
Design
13. Scripting
Execution &
reporting
Script
Maintenance
Scripting
Legend
Activities related to Release n‐ 1
Activities related to Release n
Activities related to Release n+1
Release independent activities
The automation project should ideally be divided into five distin
ct phases as described below. It
is assumed that test case design is already done, and is not treat
ed as part of the automation
effort.
Study
Typically, the study phase would involve the following steps:
•
Identification of a few (could be just two) test cases that are fair
ly complex
•
14. Identification of candidate tools. Several (say six) tools may be
compared and a few (say, two
or three) may be tagged as candidates. This short‐ listing
should be based on a multiple
criteria, some of them being support for the AUT’s
underlying technology, scripting
languages, cost etc.
•
PoC (Proof of Concept) automation of the selected test cases usi
ng the candidate tools. The
main intent is to discover problems that a particular automation
tool may have with the AUT,
and solutions to those problems. At this point, the focus is not o
n elegance in scripting, script
reuse etc.
•
Broad assessment of the long term returns from the automation i
nitiative
7
Getting ready for PLM
• If the above assessment confirms sponsorship for the
automation initiative, then
recommend the tool to be used, and the test cases to be taken up
15. in the first automation
cycle.
Deliverables from this phase
• Broad assessment of the long term returns from automation
• Recommendation on the tool to use
• List of test cases to be taken up in the first automation cycle
• Data for estimation of subsequent phases
Framework Design
This phase is similar to the high level design phase in the SDLC
(Software Development Lifecycle)
and involves deciding the type of framework to create, how to o
rchestrate the execution, and so
on.
Deliverables from this phase
• Architecture for the framework
•
High level flowchart for the way test case execution will be orc
hestrated
• Format for reporting test results
• Test management strategy if not already decided
16. • List of the main keywords and functions that will be required
Scripting
The scripting phase is similar to the construction (coding) and u
nit testing phase in the SDLC. It
includes verification to ensure that an automatically executed te
st case returns the exact same
result as the manually executed test case.
Deliverables from this phase
• Automated test cases
• Test result reporting mechanism
In order to precisely estimate the effort required for
scripting, it is recommended that the
following points are identified for each test case to be automate
d
•
Action points: They represent the number of user actions (clicks
, selections, etc.) that are
required while executing the test case. Conceptually, they are si
milar to ‘function points’ in
application development, but are at a much finer level of granul
arity.
• Verification points: These are of two types
• Checks required for the automation itself to work
17. correctly. For instance, has a link
appeared before we attempt to click it?
• Checks that are deliberately inserted to validate the
behavior of the AUT. These are
conceptually similar to ‘asserts’ used in application developmen
t.
8
Getting ready for PLM
A verification point often takes more efforts to develop than an
action point.
As a guiding indicator, scripting effort is linearly proportional t
o the number of action points and
verification points. The other factors that significantly impact t
he scripting effort are
•
Recording support: While most tools provide support for recordi
ng, some don’t (e.g. Unified
TestPro). In such cases, higher efforts are needed.
• Level of reuse: Higher the level of reuse of action and
verification points, the lower the
effort needed.
•
Wait points: More the wait points and their complexity, higher t
18. he effort needed.
•
AUT(s): Higher the complexity of the AUT, the higher the effor
t needed. Also, greater effort
is typically needed when more than one AUT is involved. The ar
chitecture of the AUT also
impacts the scripting effort needed (e.g. client‐ server versus w
eb).
Execution and reporting
This phase involves the actual execution of the automated test c
ases, and reporting the results.
This is an ongoing phase and is almost completely automatic.
Deliverable from this phase
• Test results (summary and detailed)
Maintenance
Maintenance involves modifying and updating the automated tes
t cases as required, to ensure
that they remain true to the intent of the test. Typical reasons ne
cessitating maintenance are–
• A GUI change in the AUT
• Enhancement of the test case
Deliverable from this phase
19. • Updated / enhanced automated test cases
While good framework design will minimize the need to
rework the scripts in response to a
change like GUI enhancement in the AUT, it is usually difficult
to eliminate this need completely.
9
Getting ready for PLM
Cost of test automation
Investment in automation has two key components
• Investment in the automation tool
•
Investment in the effort required for developing, executing and
maintaining the automated
test suite
Cost of the Automation Tool
Typical single user, perpetual licenses cost in the range of USD
5,000 for a named user license,
and USD 10,000 for a floating
license. AMCs are usually 20% of the license costs. Some of the
most successful commercial test automation tools are provided b
20. y vendors like HP, IBM, Borland
and AutomatedQA.
Cost of the Automation Effort
Estimates discussed here are a broad representation and are base
d on our experience with test
automation of CAx and PLM systems. Actual estimates
would primarily depend on the AUT,
automation tools and the actual test cases to be automated
• Study would take four‐ eight person weeks of effort.
•
Framework design would take two‐ four person weeks of effort.
•
Scripting: As indicated earlier, we recommend point based esti
mation. However, if a rough
estimate is needed in the early phases of the automation project,
then the number of web
pages / forms, controls embedded in them, etc. can be used to ar
rive at an estimate of the
number of test cases, which can then be translated into an effort
estimate.
•
Reporting: Creating the result reporting mechanism would take
one person week of effort.
21. 10
Getting ready for PLM
Benefits of Test Automation
Shorter test cycles
A ‘test cycle’ refers to a single run of a set of test cases. For ins
tance, a system test cycle means
that all system test cases have been run. Automation significantl
y reduces the time required to
execute a test cycle through
•
Faster action triggering: Typically, a script can trigger actions o
n the AUT much faster than a
human. However, actual actions triggered (UI interactions done)
during a manual test might
be different from those during an automated test due to a limitat
ion of the automation tool.
• 24x7 replays: Since it is rare for manual testing to be
done for more than eight working
hours in a day, the fact that automated replays can
run continuously, offers a significant
reduction in the test cycle time.
22. A 50‐ 70% reduction in the test cycle time is common.
Saving of manual testers’ time
The saving in a test engineer’s time is almost linearly proportio
nal to the number of test cases
automated, and the number of test cycles to be executed.
Repeatability
In complex test environments (those involving several
application components, platforms,
environment variables, etc.) human error can creep into
manual test execution. Automation
ensures 100% repeatability and hence greater predictability in e
xecution.
Enabling non‐ functional testing –
synergy with other quality tools
With additional tools and effort, it is often possible to configure
special runs of the automated
test cases in order to perform non‐ functional testing, for examp
le:
• Performance testing
• Scalability/ load testing
• Memory profiling
• Code coverage and impact analysis
23. Depending on the project’s priorities, the above benefits can be
translated into higher quality,
lower costs or lesser time to market.
11
Getting ready for PLM
ROI on Automation: A Conservative Estimate
Based on the benefits outlined above, we recommend that autom
ation should be seen as an
overall quality and productivity improvement initiative,
rather than merely as a cost saving
exercise.
At the same time, automation is more amenable, as
compared to some other quality
improvement initiatives, to a return on
investment (ROI) analysis. The monetary values of the
following can be compared:
•
Return: Saving of manual testers’ time, converted to a monetary
value
• Investment
24. • Cost of the automation tool
• Cost of the automation effort
For automation of functional test cases, a typical ROI profile mi
ght look like the one below.
0
5
10
15
20
25
1 2 3 4
Release Cycle
K
U
SD
‐ 100%
‐ 50%
0%
50%
100%
25. 150%
200%
%
ag
e
Cost of the Tool Cost of the Automation Effort
Cumulative %age test cases automated ROI %age
The above ROI assessment can be termed conservative, as it onl
y considers cost elements, and
does not take into account the benefits resulting from shorter cy
cles, higher repeatability, etc.
12
Getting ready for PLM
What can be Automated
There is a wide spectrum of tasks that can be automated
• Test data generation (particularly useful for load testing)
• Unit testing
• Integration testing
26. • System testing involving a single AUT
• System testing involving multiple AUTs and their integrations
• AUT installation
Non‐ textual input/ output
Consider a simple 2D CAD AUT that has a graphics area, in whi
ch the user can pick objects such
as circles, lines, etc. and drag them with the mouse. Or, conside
r an AUT that converts text to
speech. Commercially available automation tools are usually un
able to handle such AUTs out‐ of‐
the‐ box. However, with additional effort, it
is possible to automate testing of AUTs with non‐
textual input/ output as well.
Pixel based automation and journal based automation, two com
mon solutions to this problem
are described in more detail in the table below. While the CAD
AUT has been used as an example,
variations of the approaches described can be used with
other types of AUTs – the text‐ to‐
speech application, for instance.
Challenges
27. Non‐ textual input Non‐ textual output
A
p
p
ro
ac
h
Record the user actions of picking the circle, dragging the
mouse, etc. just like other user actions
Save a screenshot of the output to a
bitmap file
Then compare this with the reference
bitmap
GUI based, leading to high fidelity with respect to manual testin
g
P
ro
s
No extra coding required
P
ix
el
b
as
ed
a
u
28. to
m
at
io
n
C
o
n
s
Automation tends to be brittle. This can be mitigated by fine co
ntrol over the environment, by not picking
close to the edge, validating output at the right precision level,
etc.
A
p
p
ro
ac
h
Journal inputs (Pick Circle#15, Apply Move [10, 15], etc.) to
a file (one‐ time activity)
Write a test app that mimics the user by using this file to
drive the API of the AUT
Commercial automation tool not used at all
Record the test case as usual
Journal outputs (NewCirclePosition [25,70])
to a file during test execution
Compare this output file with the reference
29. file
P
ro
s Highly robust automation ‐ not affected by environment,
precision, etc. and even by changes to the GUI
Robust automation ‐ not affected by
environment, precision, etc.
Extra coding required in the AUT for journaling
GUI used neither for input nor for output ‐ leading to a
significant loss of fidelity with respect to manual testing
GUI used for input but not for testing
output ‐ leading to a loss of fidelity with
respect to manual testing
So
lu
ti
o
n
s
Jo
u
rn
al
b
as
ed
a
u
30. to
m
at
io
n
C
o
n
s
Significant extra coding required to create the test apps
13
Getting ready for PLM
What to Automate, and When
We do not recommend making automation decisions solely
on the basis of (costs only) ROI.
However, given that automation needs a non‐ trivial
effort, it is advisable to undertake
automation only after some parts of the AUT have become fairly
stable. This ensures that the
automated test cases will get executed enough number of times t
o offset the automation effort.
A few consequences of this rationale are as follows:
31. • Automation is much more attractive in product
development, as compared to services
projects.
•
In product development, it would be prudent to start automation
after a couple of releases
of the product. This ensures that chances of a GUI overhaul are
minimal.
• It is advisable to first automate those test cases that
involve functionalities that are fairly
stable. New features could typically be taken up for automation
in the next release. This also
means that automation is most effective for regression testing.
Relationship of the TALC to the SDLC
Since much automation is targeted towards regression testing, it
is not essential to synchronize
the Test Automation Lifecycle with the Software Development
Lifecycle. An automation initiative
can be started at any point in the SDLC, given that some parts o
f the AUT have reached stability.
However for convenience in planning and tracking, it
is common to synchronize these two life
cycles, and to then treat the TALC as a sub‐ project of the main
SDLC. This is depicted in the figure
below.
32. Challenges in Automation
•
The choice of the tool is often restricted by the technology und
erlying the AUT. For instance,
a popular cross‐ platform development framework Qt
is not supported well by one of the
leading automation tools.
• In web applications, multi‐ window test cases are usually
difficult to automate. Pop‐ ups,
single child windows are not a problem.
•
Integrations between AUTs are sometimes difficult to automate.
14
Getting ready for PLM
Automation Best Practices
We recommend the following best practices for automation.
• Since automation frameworks are essentially about
abstraction, an important set of best
practices deals with ensuring loose coupling between –
• The test data and the test scripts,
33. • Test scripts themselves,
• The automation framework and the AUT(s),
• The test cases and the automation framework, and
• The automation framework and the automation tool.
The entities listed above will essentially have to reference
one another to form a complete
working test automation system. It is important that these refere
nces be through well defined
interfaces only.
• Keyword names should be carefully chosen, so that
human readability is also high. This
enables gradual transitioning from manual testing to automated
testing.
• Avoid duplication in scripts. Any duplication should be
investigated to check whether a
separate unit (say, function) can be created.
•
Verification points should be judiciously inserted into the script
s. In case of test case failure,
these points accelerate the process of zeroing in on the reason o
f the failure.
• The development of an automation framework is similar
to the development of an
application in several respects, and hence should be planned and
tracked as a (sub) project
34. in itself. It should be noted that framework creation and
test case design are distinct
activities (and require different skills).
•
Simpler test cases should be automated before complex ones. Th
is makes it easy for later
scripts to build on earlier ones. There is an exception to this app
roach though: fairly complex
test cases should be taken up in the study phase, as the
objective there is to discover
problems in automation (rather than achieving a high level of re
use).
Conclusion
Test automation offers a promising way of quality and productiv
ity improvement in the area of
software testing – particularly in products. While manual
testing is required and also desired
(except perhaps for a product that is purely in sustenance mode)
, the time and cost required for
it can be significantly reduced. Moreover, a part of this saving c
an be invested for better quality.
15
35. Getting ready for PLM
Commercial tools and a rapidly growing body of knowledge hav
e led to a reduction in the time
needed for monetary returns to be seen, thus accelerating the ad
option of test automation in
the industry.
We recommend that for all product development, at least
the study phase of the test
automation lifecycle should be undertaken.
About the Author
Prashant Anaskure is the Head of the Quality Assurance
and Support Practice, part of the
Software Product Engineering group, at Geometric.
Prashant has a very strong background in
product development lifecycle having worked for over ten years
with different busines units in
Geometric. Prashant has lead teams in product development and
quality assurance activities for
Geometric’s Enterprise Products, Desktop Products and also for
our key customers.
About Geometric
Geometric is a specialist
36. in the domain of engineering solutions, services and technologie
s. Its
portfolio of Global Engineering services and Digital
Technology solutions for Product Lifecycle
Management (PLM) enables companies to formulate, implement
, and execute global engineering
and manufacturing strategies aimed at achieving greater efficien
cies in the product realization
lifecycle.
Headquartered in Mumbai, India, Geometric was
incorporated in 1994 and is listed on the
Bombay and National Stock Exchanges. The company recorded
consolidated revenues of Rupees
4.86 billion (US Dollars 121.6 million) for the year ended Marc
h 2008. It employs close to 3000
people across 10 global delivery locations in the US,
France, Romania, India, and China.
Geometric is assessed at SEI CMMI Level 5 for its software ser
vices and ISO 9001:2000 certified
for engineering operations. For further details, please visit www
.geometricglobal.com.
http://www.geometricglobal.com/IntroductionAutomation
FrameworksTest Automation Life Cycle (TALC)Cost of test
automationBenefits of Test AutomationROI on Automation: A
Conservative EstimateWhat can be AutomatedWhat to
37. Automate, and WhenRelationship of the TALC to the
SDLCChallenges in AutomationAutomation Best
PracticesConclusionAbout the AuthorAbout Geometric
This assignment is not required but is offered for those who
wish to add to their point totals. In order to get these points,
you must submit the assignment on or before November 10 at
11:00 pm CENTRAL TIME. Your paper per must be at least
550 words in length and must address the issues below.
The assignments consist of finding a newspaper article that is
published between October 5, 2021 and November 10, 2021
either in a local paper or in an on-line newspaper such as the
New York Times, Los Angeles Times, Chicago Sun Times,
Washington Post, Greenwood Commonwealth or any
other United States or English language newspaper. This means
CNN, yahoo, Fox News, MSNBC, Huffington Post, tv station
websites, etc. are not eligible for these assignments. ONLY
NEWSPAPERS/NEWSPAPER websites. Each article must have
some discussion of, or allusion to, African American history or
issues.
1. The article must come from newspapers, not websites
unconnected to a print paper. Don't use opinion/commentary
articles. Ask your instructor if you don't know what this is.
2. Cite the article [Title of the article, author [if given], name of
the newspaper, date of publication] AT THE TOP OF YOUR
PAPER. Put your name at the top of the paper, as well. No
repeat headers allowed in your papers. You lose 5 points if you
don’t follow the citation and repeat header direction. For this
course, the only repeat header that should appear on your pages
is a page number.
3. You are to summarize the article and the point of view of the
author.
4. Then indicate whether your textbook sheds any light on the
subject---use the index and cite the textbook pages.
5. Comment on how history is “used” by contemporary authors
to support their points of view or illustrate subject matters they
38. deal with.
6. Have you learned anything from the article?
7. Finally, and most importantly, what are the specific sources
the author(s) cite(s)/uses to support the article? That is, how
does the author prove to you, their reader, that they have facts
to back up their writing? They won't have a bibliography, but
they normally are letting their readers know where they are
getting their information. Don't use opinion/commentary
articles.
Your assignment must be turned in to Canvas by 11:00 pm
CENTRAL TIME on the date it is due. It must be at least 550
words in length, must address the seven numbered issues above,
and the article must have been published in the designated
dating range for the assignment in which you are turning in the
paper. The subject matter of the articles must be on a topic
appropriate for the course. You are not guaranteed the full 25
points if you don't follow these directions.
Another object of the assignment, as with other assignments in
the course, is to have you follow specific directions, as you
would be asked to do in any job situation.
The point of these assignments is for you to think about how
history relates to what is happening now, so think about this
issue as you write your essay.
REMEMBER: SUBMIT YOUR PAPER ONLY IN MS WORD
FORMAT.