Functional Testing with Domain-Specific Languages

W9
Test Automation
10/15/2014 1:45:00 PM
Functional Testing with Domain-
Specific Languages
Presented by:
Tariq King
Ultimate Software
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ·∙ 904-278-0524 ·∙ sqeinfo@sqe.com ·∙ www.sqe.com

Tariq King
Ultimate Software
Tariq King is a software test architect at Ultimate Software, a leading cloud provider of people
management solutions. With more than a decade of experience in software testing, Tariq
provides technical leadership, strategic direction, and staff training in quality assurance across
the organization. Tariq currently supports more than fifteen cross-functional teams working in an
agile environment and manages a team that develops internal tools. His testing research
appears in several IEEE and ACM publications, and includes collaborations with Microsoft and
IBM. Tariq also serves as an adjunct professor at Florida International University, teaching
courses in software engineering and project management. Contact Tariq via LinkedIn
at linkedin.com/in/tariqking.

OCTOBER 15, 2014, ANAHEIM, CA
unctionalF
9
stingT
52
e with
Tariq King
Domain-Specific Languages

Functional Testing with Domain-Specific Languages
Tariq King
2
9 52

Tariq King
3

Tariq King
4
DOMAIN

Tariq King
5
It’s hard to wreck a
nice beach
It’s difficult to
recognize speech

Tariq King
6

Tariq King
7

Tariq King
8

Tariq King
9

Tariq King
10
UMLSQL

Tariq King
11
Cryptol
SQL

Tariq King
12
Cryptol
SQL
SELECT * FROM EMPLOYEES
WHERE LastName = ‘Smith’

Tariq King
13
Cryptol
SQL
<html>
<title>HTML</title>
<body>
<h1>DSL</h1>
</body>
</html>
DSL
HTML

Tariq King
14
Cryptol
SQL
Cryptol > [0 1 2 3]
[0x0 0x1 0x2 0x3]
Cryptol > :set base=10
[0 1 2 3]

Tariq King
15
Cryptol
SQL
$ awk '$4 ~/Technology/' employee.txt
200 Jason Developer Technology $5,500
300 Sanjay Sysadmin Technology $7,000
500 Randy DBA Technology $6,000’

Tariq King
16
Cryptol
SQL
describe User do
context 'with admin privileges' do
before :each do
@admin = Admin.get(1)
end
it 'should exist' do
expect(@admin).not_to be_nil
end

Tariq King
17
Cryptol
SQL
.background {
height: 480px;
width: 320px;
font-size: 10px;
font-family: Arial
color: #000000;
}

Tariq King
18
Cryptol
SQL
begin{document}
title{Functional Testing with
Domain-Specific Languages}
author{Tariq M. King}
institute{Ultimate Software Group, Inc.}
address{2000 Ultimate Way,
Weston, Florida 33326, USA}
email{tariq_king@ultimatesoftware.com}
url{www.ultimatesoftware.com}

Tariq King
19
Cryptol
SQL
public class CannonFire {
var gun;
var target;
var reloadSpeed;
function Update() {
if(target!= null) {
var bomb:GameObject=(Resources.Load(“Bomb”));;
gun.transform.rotation = Quaternion.Slerp(…);;
…

Tariq King
20

Tariq King
21

Tariq King
22

Tariq King
23

Tariq King
24

Tariq King
25

Tariq King
26

Tariq King
27

Tariq King
28

Tariq King
29
9 52
Application Domain Model
User Interface Data Access Macros
Test Automation Framework
Test Fixture Test Commands
Internal DSL Library
Support Types

Tariq King
30
public class SearchPage extends PageObject
{
private final String pageTitle = "Google";
private final String searchFieldName = "q";
public SearchPage(WebDriver driver)
{
super(driver, pageTitle);
URL = "http://www.google.com";
}
public void enterSearchCriteria(String text)
{
driver.findElement(By.name(searchFieldName))
.sendKeys(text);
}
public SearchResultsPage submitRequest()
{
driver.findElement(By.name(searchFieldName))
.submit();
return new GoogleSearchResultsPage(driver);
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

Tariq King
31
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public class GoogleSearchTest
{
private SearchPage googleSearch;
private SearchResultsPage results;
@BeforeClass
public void setUp() {
googleSearch = new SearchPage(driver);
googleSearch.open();
}
@AfterClass
public void tearDown() {
googleSearch.quit();
}
@Test
public void simpleTextSearchTest() {
googleSearch.enterSearchCriteria("Pi");
results = googleSearch.submitRequest();
assertTrue(results.contains("3.1415"));
}
}
Pi

Tariq King
32
[TestMethod]
public void GoogleSearchStory()
{
new Story("Google Search").Tag("Sprint 1")
.InOrderTo("find public information")
.AsA("user")
.IWant("to search the Web for documents")
.WithScenario("simple text search")
.Given(IOpenGoogleSearch)
.When(IEnterSearchCriteria, "Pi")
.And(ISubmitTheRequest)
.Then(TheResultsPageContains, "3.1415")
.ExecuteWithReport(GetCurrentMethod());
}
Story is Google Search
In order to find public information
As a user
I want to search the Web for documents
With scenario simple text search
Given I open google search
When I enter search criteria Pi
And I submit the request
Then the results page contains 3.1415
Passed
Passed
Passed
Passed
#Sprint 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14

Tariq King
33
Domain-Specific Language
Specification
Specification Interpreter
Behavioral Scenarios
Programming Language
Test Automation
Test Runner
Test Methods
map to
invokes
9 52

Tariq King
34
Domain-Specific Language
Specification
Test Automation
Test Runner
Test Methods
<SpecFlowTest> (2 tests) Success
〈〉 SpecFlowTest (2 tests) Success
GoogleSearchFeature (2 tests) Success
SimpleTextSearch Success
AddressSearch Success
map to
invokes
9 52

Tariq King
35
DSLs
clear

Tariq King
36

Tariq King
37

Tariq King
3838
19 teams
225 people
115 engineers
60 testers
6,179,551 lines of code
232,394 documented tests
229,138 automated tests

Tariq King
39

Tariq King
40
CODED UI
Specifies tests
Executes tests
Coordinates

Tariq King
41

Tariq King
42
Suite Information
Declarations
UltiPro UltiPro
Setup
Given I launch UltiPro
Tests
FILE EDIT VIEW PROJECT BUILD DEBUG TEAM TOOLS TEST LEGEND RESHARPER
Debug x86
Scenario: Search for Employee under My Team
Given I login to UltiPro
By setting the LoginPage UserNameTextBox to @username
And setting the PasswordTextBox to @password
And clicking the LoginButton
Purpose System should correctly filter employees
Authors Dionny Santiago, Tariq King
Stories ULTI-126101
Tags Employees, Finding
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
26
27
28
Test1.legend

Tariq King
43

Tariq King
44

Tariq King
45

Tariq King
46

Tariq King
47

Tariq King
48

Tariq King
49

Tariq King
ankTh
90
ouY
39
?
!
50

Functional Testing with
Domain-Specific Languages
Tariq M. King
Ultimate Software Group, Inc.
2000 Ultimate Way, Weston, Florida 33326, USA
tariq king@ultimatesoftware.com
www.ultimatesoftware.com
Abstract. Developing high-quality software requires e↵ective commu-
nication among various stakeholders including customers, business an-
alysts, domain experts, developers, and testers. Misunderstandings be-
tween di↵erent stakeholders can introduce defects into software, reducing
its overall quality and threatening the project’s success. Domain-specific
languages (DSLs) are special purpose languages created to describe tasks
in a particular field. DSLs provide stakeholders with a common vocabu-
lary for describing various aspects of the application, while focusing on
the real-world problem being solved. This paper describes how DSLs can
be leveraged during functional testing to identify potential issues early,
reduce misunderstanding, and improve the automation process. A case
study on the evolution of functional testing DSLs in a large-scale agile
project is also presented.
Keywords: Software Testing, Test Automation, Domain-Specific Lan-
guages, Agile Software Development
1 Introduction
After more than seven decades since the invention of the first programmable com-
puter, software continues to revolutionize the world. Software has become per-
vasive throughout our lives, existing in home appliances, handheld devices, mo-
tor vehicles, and more. Whether it be through shopping, entertainment, paying
bills, or connecting with friends, software makes everyday life more convenient.
Advances in software have brought innovation to several industries including
healthcare, travel, manufacturing, education, and human capital management.
A key part of successful software engineering is understanding the real-world
problem being addressed. Problem domain analysis allows software engineers to
know what product to build, and helps them identify with the customer’s needs
and expectations. In addition, the complexity of real-world phenomena is a pri-
mary reason why software engineering is challenging [1]. Real-world complex-
ity can cause misunderstandings among stakeholders, which typically produces
costly defects, delays, rework, and budget overruns. Domain expertise is there-
fore an essential ingredient for enabling the delivery of high quality software,
and doing so in a timely, cost e↵ective manner.

2 Tariq M. King
Domain-specific languages (DSLs) are special purpose languages created to
describe tasks in a particular field. DSLs provide stakeholders with a common
vocabulary for describing various aspects of the application, while focusing on the
real-world problem being solved. Using DSLs for software engineering activities
can therefore reduce communication errors, and raise quality and productivity
levels through domain expert involvement. However, creating a DSL involves
upfront investment and maintenance costs. Deciding whether to use a DSL for a
given software engineering task therefore requires careful cost-benefit analysis.
Testing continues to be the de-facto standard used in industry to control soft-
ware quality. In particular, functional testing provides feedback on whether or
not the system meets user requirements. The automation of functional tests can
therefore be viewed as the codification of user requirements [2]. As executable
requirements specifications, functional tests should be easy to read, write, exe-
cute, debug, validate, and maintain. Incorporating DSLs into functional testing
activities can help achieve these goals.
This paper describes how DSLs can be leveraged during functional testing
to identify potential issues early, reduce misunderstanding, and improve the au-
tomation process. It also presents a case study on the evolution of functional
testing DSLs in a large-scale agile project. The rest of this paper is organized
as follows: the next section provides background material on DSLs. Section 3
describes DSLs for automated functional testing. Section 4 is the case study and
Section 5 concludes the paper.
2 Domain-Specific Languages
This section answers three fundamental questions on DSLs for software engi-
neering: what are they? why use them? and how are they implemented?
2.1 Language Classification
Software languages can be categorized as general-purpose or domain-specific. A
general-purpose language (GPL) is a language designed for specifying, construct-
ing, documenting and/or visualizing software for a wide range of application
domains. GPLs may be textual or graphical. Examples of textual GPLs include
programming languages such as Java, C#, C, C++, Python, Ruby, among oth-
ers. In contrast the Unified Modeling Language (UML) provides a graphical no-
tation for creating software blueprints [3]. Whether textual or graphical, GPLs
are useful for tackling several classes of real-world problems.
On the other hand, a domain-specific language (DSL) targets a particular
problem area or aspect of an application. The term domain refers to the body of
knowledge surrounding the real-world problem for which the language is being
built [4]. As such, DSLs are based on abstractions that are consistent with the
real-world problem, application, or system. Table 1 lists a cross-section of DSLs
used in software engineering along with associated problem domains.

Functional Testing with Domain-Specific Languages 3
Name Problem Domain
Requirements
LOTOS Protocol specification.
WebSpec Specification of web navigation and interaction.
Design
CSS Formatting and styling web pages.
Cryptol Designing cryptographic algorithms.
OCL Expressing constraints on UML models.
Implementation
AWK Textual data manipulation.
ESP Writing device firmware.
HTML Creating web pages.
SQL Querying structured data.
UnityScript Video game development.
Testing
Selenese Web browser test automation.
RSpec Test automation for the Ruby language.
Documentation LaTeX Typesetting of technical documents.
Table 1: Examples of domain-specific languages used in software engineering.
2.2 Benefits and Challenges
DSLs have become increasingly popular in software engineering because of the
benefits they bring to project stakeholders. These benefits are summarized as
follows [4–7]:
– Clarity and Communication. Recall that essential complexity in software is
caused by the real-world problem [1]. Using domain abstractions helps the
development team stay focused on learning, understanding, and solving the
problem under investigation. Misunderstandings between stakeholders can
be cleared up or avoided because everyone is speaking the same language.
In this way, a DSL acts as a ubiquitous language that facilitates stakeholder
communication and collaboration.
– Domain Expert Involvement. Software engineers are not necessarily experts
in the domain of the software they are developing. Using a software lan-
guage that aligns with the problem area allows domain experts to easily
read, understand, create, or validate software artifacts. Whether they are
creating the software artifacts with the DSL on their own, or collaborating
with technical stakeholders, keeping domain experts engaged in the software
development process is critical to the success of the project.
– Quality and Productivity. On their own accord, the previously mentioned
benefits can lead to improvements in product quality and team productivity.
However, additional productivity benefits are realized from having a dedi-
cated language and toolset tailored to the problem domain. A well-designed
DSL with several reusable functions leads to less code, better syntax and
error checking, and more meaningful tool support than a GPL alternative.
Manual checking is also easier since the language is clear and concise.

4 Tariq M. King
– Model and Platform Independence. Domain-specific modeling produces ab-
stractions that are independent of the underlying implementation techniques
and technologies. As a result, domain models can be transformed into other
representations if necessary, which makes it easier to migrate artifacts and
tooling to new platforms. In other words, DSLs improve portability and
maintainability by separating concerns among aspects of the domain, imple-
mentation, and technological platform.
– Comparable Runtime Overhead. DSLs can be constructed in a way that allow
benefits to be reaped without added performance overhead. DSL translators
can be designed to generate e cient code, making them as performant at
runtime as compiled general-purpose solutions.
DSLs clearly provide several benefits to the software engineering process.
However, these benefits are not without upfront investment costs and challenges.
Building a DSL is a non-trivial task, which requires significant e↵ort and skilled
expertise. Once created, the language and associated tooling must be actively
maintained and evolve with the software. The decision as to whether or not to
use DSLs should be based on careful cost-benefit analysis.
2.3 Implementation Methods
Regardless of their implementation method, DSLs consist of the following parts:
1. Abstract Syntax. Defines the language structure in terms of primitive ele-
ments, and describes how those elements are combined. The abstract syntax
is independent of any particular language encoding or representation.
2. Concrete Syntax. Describes the specific representations of the language, in-
cluding its encoding and/or visual appearance.
3. Semantics. Ascribes meaning to the language elements and di↵erent ways of
combining them. Common forms of language semantics include:
(a) Denotational. Each phrase in the language has an abstract, conceptual
meaning. These are often represented as mathematical objects.
(b) Operational. Describes language execution by defining the meaning of
phrases as a set of state transitions or syntactic transformations.
(c) Translational. Ensures that the meaning of the language can be preserved
when it is transformed into some other target language.
A DSL implementation may be categorized as internal or external [4, 5]. An
internal DSL is one embedded within the context of a host language, typically
a GPL. Internal DSLs therefore influenced by the syntax of the host language,
and inherit any IDE and tool support. However, this support is limited because
the host language tools are generally not aware of the DSL grammar, rules, and
other constraints. In practice it is common for internal DSLs to be developed
using fluent interfaces [8].
External DSLs are standalone languages designed to be independent of other
languages. The language definition along with any tools for editing, checking,

translation, and execution must therefore be developed separately. As a result,
external DSL development is typically more complex than internal DSL develop-
ment. Compiler construction techniques such as lexical analysis, syntax analysis,
semantic analysis, optimization and code generation are usually applied to this
type of DSL engineering [4]. Modern compiler tools and language workbenches
have therefore made it easier to develop external DSLs [9, 10].
3 DSLs for Functional Testing
Functional testing plays an important role in software engineering by providing
feedback on whether or not a system meets user requirements. Automated func-
tional tests can therefore be viewed as a means of codifying user requirements,
allowing them to be executed against the system for quality assurance purposes.
However, prior to automation, requirements typically go through multiple trans-
lations among stakeholders. From the customer, end users, and domain experts
through the business analysts, and then to the developers and testers. Each
communication exchange is a potential point of failure, especially when various
stakeholders are using their own jargon.
Leveraging DSLs for functional testing can mitigate risks associated with
translation errors by allowing: (1) stakeholders to communicate e↵ectively using
a common vocabulary centered around the problem domain; and (2) customers,
end users, business analysts, and other domain experts to express requirements
and validate system behaviors directly. The former reduces the likelihood that
misunderstandings will occur among stakeholders during communication ex-
changes, while the latter avoids translating requirements into functional tests,
and engages domain experts in software testing activities.
As executable requirements specifications, functional tests should be easy to
read, write, execute, debug, validate, and maintain. These goals directly align
with the benefits of DSLs outlined in Section 2.2, further motivating the use of
DSLs for functional testing. However, since DSL implementation may require
significant investment, understanding and comparing the approaches used to
develop these types of DSLs is important. This section describes the current
state-of-the-art in design approaches for enabling DSL-based functional test au-
tomation. To set the stage for the case study, design descriptions focus on au-
tomated testing in agile projects, and provide examples of tools that implement
those designs in practice. The section starts with an overview of test automation
under agile development, and then presents typical approaches used to create
internal and external DSLs for functional testing. Observations on the potential
costs and benefits of the di↵erent approaches are also discussed.
3.1 Agile Test Automation
Agile software development promotes the use of a lightweight software process
to allow teams to be adaptive, flexible, and responsive to change [11]. Software
testing under agile is continuous and occurs throughout the development process

6 Tariq M. King
[11, 12]. In other words, testing is viewed as an integral part of software develop-
ment, as opposed to a separate phase like in the traditional waterfall approach.
Automation is a key contributing factor in determining the ability of an agile
team to carry out continuous testing.
A number of agile practices have been established to support functional test-
ing throughout the software lifecycle [7, 11–13]. Business analysts capture re-
quirements and acceptance criteria as user stories written in domain language.
Developers engage in story-driven modeling and behavior-driven development
(BDD) to design and implement the desired features. Test engineers create test
cases to validate those features by elaborating on the acceptance criteria, which
can also be automated using a BDD framework. For test automation to be ef-
fective in agile projects, the process and tooling should facilitate [2, 14]:
– Testing Early and Frequently. The test automation e↵ort should start before
or in parallel with other software development activities. Once the automated
tests have been created, they should be run often to provide continuous
feedback on software quality.
– Test Comprehensibility, Availability, and Collaboration. At a minimum, func-
tional automated tests should be readable by stakeholders who are familiar
with the domain. Such tests should also be readily-available and accessible
so they can easily be reviewed and executed. In other words, test tooling
should foster stakeholder communication and collaboration.
– Test Abstraction, Implementation, and Self-Documentation. There should be
a way to separate the essential details that reflect the goals and objectives
of the test, from the incidental details that result from its implementation.
Additionally, there should be IDE support for test automation activities.
Making test code intent-revealing can lower documentation overhead.
– Test Traceability, Categorization, and Selection. To be responsive to change,
agile teams must be able to determine the e↵ects of that change. From a
software engineering perspective, this involves establishing and maintaining
traceability relationships among requirements, code, documented tests, and
automation. Tool support for categorizing, selecting, and executing a strict
subset of tests aid change impact analysis.
– Test Reliability, Maintainability, and Standardization. The way an auto-
mated test is written can impact its ability to provide consistent results,
and be maintained throughout its lifetime. Automation guidelines, stan-
dards, and coding conventions should be put in place to ensure tests are
reliable, modifiable and transferable across teams.
3.2 Internal DSLs for Functional Testing
A common approach to developing DSLs for automated functional testing is to
define an application domain model, and expose that model to an automation
framework that resides in the context of a programming language.

Application Domain Model Test Automation Framework Internal DSL Library
Test Fixture Test CommandsUser Interface Data Access Macros Support Types
Fig. 1: Typical design of internal DSLs for automated functional testing.
As shown in Figure 1, the application domain model (left) encapsulates
any implementation details needed for automated tests to: access the GUI el-
ements of the system under test; and add, remove, or verify persistent data
values. Test abstraction is raised by developing a set of reusable test routines
as part of the model. These routines, referred to as macros in this paper, model
and automate domain-level user interactions, e.g., HireEmployee, RunPayroll,
VerifyPaycheck. The DSL is derived by combining test fixture and command
constructs from a test automation framework (middle) with the application
model. To ensure that the language is business readable, application elements
and behaviors are expressed using domain-specific terms.
Many software design practices support the implementation of internal DSLs
for functional testing. Object-oriented programming constructs are generally
used for defining application models. The page object pattern provides a design
for encapsulating UI implementation details and, although described in terms of
web pages, may be applied to any UI technology [15, 16].
Internal DSL implementation may be realized with the help of a support
library (right of Figure 1). Such libraries provide data types and objects to aid
structuring the language, and enhance the editing experience in strongly typed
host languages. StoryQ is a good example of an open-source internal DSL support
library developed for the .NET platform [17]. It is designed as a lightweight,
behavior-driven DSL with a fluent interface, and comes with a conversion tool
that transforms plain-text stories into the DSL syntax. StoryQ does not parse,
link to, or depend on those stories outside of the conversion process. Instead,
after conversion the plain-text stories are meant to be discarded.
3.3 External DSLs for Functional Testing
Although there is no standard approach for creating external DSLs for functional
testing, the state-of-the-art in existing tools have adopted the design shown in
Figure 2. Under this design, a DSL (left) is developed to allow the author to
create specifications in the form of plain-text, story-based behavioral scenarios.
Each line in a behavioral scenario is mapped to a method implemented using a
test automation framework (right). The test automation framework lies outside
of the DSL, and mappings between the two are realized through annotations
on the test methods. Annotations contain regular expressions that are matched
against the contents of the plain-text story. As a result, the story files are main-
tained with the DSL and treated as the source of truth.

8 Tariq M. King
Domain-Specific Language Programming Language
Test Automation
Test Methods
Test Runner
Specification
map to
invokes
Fig. 2: Typical design of external DSLs for automated functional testing.
Execution of the behavioral specifications is generally carried out through a
process of code generation and interpretation. The DSL tooling is first configured
for use with a test automation framework and test runner of choice. As the spec-
ification is being authored, the DSL tooling generates driver code for the target
automation platform. During execution, steps in the specification are interpreted
line by line, and the interpreter relays calls to the test method associated with
each step. Upon completion a test report is generated showing which specifica-
tion steps passed or failed. Debugging support is achieved by instrumenting the
driver code with line numbers that correspond to specification steps, and using
the instrumented file as input to an IDE or standalone debugger.
JBehave, SpecFlow, and Cucumber are open-source BDD frameworks that
implement the aforementioned design [13, 18, 19]. Table 2 provides a summary
of these frameworks including their supported programming languages, test au-
tomation platforms, IDEs and other tools. Proprietary BDD tools and extensions
are also available. For example, Twist is an o↵-the-shelf, functional testing DSL
solution [20]. It facilitates writing test specifications in natural language, and im-
plementing test scripts in Java or Groovy. Twist provides an IDE that supports
manual, automated or hybrid testing [20]. SpecFlow+ is a set of commercial ex-
tensions for SpecFlow aimed at providing better stakeholder collaboration and
more scalable test execution [19].
Programming Languages and Test Platforms IDE and Tool Support
JBehave Java, JUnit, TestNG, Spring Test.
Eclipse, IntelliJ IDEA,
Ant Tasks, Maven Goals
SpecFlow
Microsoft.NET, Mono, Silverlight, Windows
Phone 7, NUnit, MbUnit, MSTest, xUnit.
Visual Studio, MonoDevelop,
MsBuild, TeamCity
Cucumber
Ruby, Java, .NET, Flex, Javascript, Python,
Groovy, PHP, Scala, and more.
Eclipse, IntelliJ IDEA, Rake,
Ant, Maven, and more.
Table 2: Popular open source external DSLs for automated functional testing.

3.4 Cost-Benefit Observations
Both internal and external DSLs for functional testing promote writing test cases
before the application or feature has been developed. Test authors only need to
know the functional requirements, and the stubs of the automated test methods
that will be implemented. Support for tagging tests with user-defined categories
has been incorporated into tools that realize either design [17, 19]. Tags can be
leveraged to improve traceability and for selective regression testing.
One consideration to be made if opting for an internal testing DSL, is that
creating application models and macros requires di↵erent technical skills than
those needed to maintain test scripts. This could lead to changes in the sta ng
profile, and/or having developers take on these additional modeling tasks. Using
an internal DSL for writing test cases exposes multiple stakeholders to the tools,
features and constructs of the host language. In some cases that exposure can
have a positive e↵ect, and result in technical growth across the team. However,
there is a risk that non-technical stakeholders fail to adopt the DSL, or introduce
automation defects due to their lack of programming knowledge.
Recall from Section 2.3 that external DSLs are generally more di cult to
develop than internal DSLs. However, building an external DSL for functional
testing has the benefit of full control over the language syntax and semantics.
This control facilitates designing tools that are more e↵ective at supporting agile
testing goals than their internal DSL counterparts. Open source and o↵-the-shelf
BDD frameworks are viable alternatives to homegrown external testing DSLs.
Such tools can be set up with little to no upfront cost, and provide a quick
and easy way to get business-level stakeholders reading and writing behavioral
specifications. However, one drawback with these tools is that automated tests
still reside in a technical programming language, which can deter non-technical
stakeholders from using them. Some tools attempt to circumvent this problem
through record and playback features [20].
Another drawback of the state-of-the-art in BDD frameworks is that the
same natural language text is duplicated in the specification and automation.
Therefore, when a requirement changes, the same information must be updated
in multiple places. This may not be a significant maintenance cost in small
projects, but becomes noticeable during large-scale agile development. Lastly,
the interpreter-based test execution used by these tools yields some performance
overhead in comparison to compiled test execution.
4 Case Study
Ultimate Software uses an agile software development process based on Kanban
to develop UltiPro, a cloud-based people management solution [21, 22]. UltiPro
integrates all aspects of human resources, payroll, and talent management. Re-
quirements and tasks are captured as user stories in JIRA Agile, which serves
as a centralized system of record for multiple stakeholders [23].
UltiPro’s development process emphasizes continuous delivery of high qual-
ity software. To facilitate this, automated tests are run against continuous in-

10 Tariq M. King
tegration environments, and the exit criteria for work items states that: (1) all
acceptance criteria must be automated prior to signing o↵ on a story or feature;
(2) regression tests must run and pass; and (3) manual exploratory testing must
be performed on impacted product areas.
Due to the large size and complexity of the human capital management
problem space, testing UltiPro relies heavily on domain expertise. The majority
of domain knowledge on development teams is distributed among the business
analysts and domain testers. A domain tester is a subject matter expert who
has been trained in black box software testing methodologies. Since UltiPro is
domain-intensive, engaging business analysts and domain testers during valida-
tion has always been important. Many of these key stakeholders are functional
rather than technical, which has served as motivation for developing novel ap-
proaches to domain-specific, functional test automation [14, 24–26].
4.1 Motivation
In 2011, functional testing of UltiPro involved 19 cross-functional agile devel-
opment teams and approximately 230K tests. Functional tests were primarily
automated using FitNesse and the Simple Web Automation Toolkit (SWAT)
[26, 27]. FitNesse provided non-technical stakeholders with a wiki for specifying
and executing acceptance tests. The FitNesse wiki markup acted as a test au-
thoring DSL while automation was driven by SWAT, a cross-browser web testing
framework created in-house. A small percentage of UltiPro remained in legacy
code validated using WinRunner test scripts.
Although combining FitNesse and SWAT reaped benefits with respect to
stakeholder engagement, it brought several maintenance challenges. The loose
grammar of the FitNesse wiki led to a variety of test case formats and authoring
styles being used within and across development teams. Another problem was
that SWAT did not provide mechanisms for abstracting the UI implementation
details from the tests. As such the locators for identifying objects on the web
pages were hard-coded in the test automation. A change to a single element in a
web page could therefore require updates to a large number of automated tests.
Key goals for development teams in 2012 included creating UltiPro Mobile,
and focusing on enhancing the overall user experience of the product [22]. It was
apparent that in order to achieve these goals, the test authoring, automation, and
tooling strategies would have to be revisited. This led to research on developing
internal and external DSLs for automated functional testing [14, 24, 25]. The
remainder of this section describes the major contributions of that research.
4.2 Echo: Middleware for Domain-Specific Functional Testing
Developing a new test automation strategy to replace FitNesse and SWAT in-
volved formulating and analyzing several design goals. Browser compatibility was
an important factor in selecting technologies to drive web UI testing. Selenium
was chosen as a primary underlying UI automation framework because the Web-
Driver API had the support of major browser vendors [15]. The new automation

Framework Interaction
Framework Abstraction
Command Execution
Test Abstraction
Common
Application
Model
Application
Under Test
specifies tests using
executes tests against
Selenium Sikuli Appium Coded UI SQL Server
coordinates
« »New Framework
Layered Middleware Testing FrameworkModel Interface
Fig. 3: A Layered Middleware Architecture for Domain-Specific UI Automation
strategy also needed to be extensible to other UI testing frameworks and ap-
proaches, e.g., JQuery, Sikuli [28, 29]. Other immediate requirements called for
interacting with persistent data stores and native Windows UIs, while future
plans targeted mobile test automation. The results of our analysis and design
was codenamed Echo, an internal DSL and middleware framework for automated
functional testing [24].
Architecture. Figure 3 presents the design of the Echo framework. As an
internal DSL for functional testing, Echo raises test abstraction via an API
that integrates with one or more application domain models (top-left). In Echo,
the page object pattern is applied across all application models regardless of
UI technologies. As a middleware testing framework, Echo raises framework
abstraction through a layered test architecture (right). Starting from the top-
right of Figure 3, the layers of the Echo framework are:
– Test Abstraction Layer. Defines a fluent API that allows users to specify
automated tests by combining high-level test commands, application domain
models, and macros.
– Command Execution Layer. Executes test action and assertion commands
according to timing characteristics that can be configured for a given test
project. This layer provides mechanisms for handling any exceptions thrown
during test command execution.
– Framework Abstraction Layer. Provides abstractions that serve to decou-
ple the execution of test commands from the testing tools and interaction
frameworks that will carry them out. Framework abstraction reduces the
e↵ort required to change underlying tools and frameworks.

12 Tariq M. King
– Framework Interaction Layer. Realizes test command execution through
calls to a set of underlying testing tools, frameworks, and libraries. This
layer coordinates multiple low-level frameworks that directly interact with
the application under test.
– Common Layer. Responsible for cross-cutting concerns such as event logging,
exceptions, among others. This layer contains libraries and data structures
that need to be shared across the four horizontal layers.
Technology Stack. Echo’s fluent API is written in C# and can be configured
to use MbUnit, MSTest, or NUnit [30–32]. Selenium, JQuery, Sikuli, Coded UI
and SQL Server have been integrated into the framework, and support for mobile
automation with Appium is in progress. [15, 28, 29, 33–35]
Experience Report. The Echo framework has been successfully incorporated
into the agile testing process at Ultimate Software, where it continues to play a
pivotal role in functional test automation. Maintenance on automated test scripts
and the testing framework itself have been greatly reduced. Some of these bene-
fits have been a direct consequence of moving to Selenium. In addition to provid-
ing a highly robust underlying Web testing framework, the Selenium project has
allowed the responsibility for browser automation support to be shared through
community contributions. Echo developers have therefore consumed and con-
tributed Selenium bug fixes and reports over the last two years.
Test readability improved significantly over the previous approach, and test
engineers learned how to write Echo tests without much training. Defining appli-
cation models and centralizing macros have greatly increased the reuse of domain
functions across teams, and virtually eliminated unnecessary redundancy. How-
ever, as with any transition a number of obstacles were encountered along the
way. Some challenges were inevitable and expected due to the large scale of the
testing process. For example, the task of migrating automated test scripts from
SWAT to Echo required a significant amount of manual e↵ort. Initially a SWAT-
to-Echo conversion tool was developed, but had limited success due to structural
and content variations in the source scripts.
There were also some unexpected challenges and lessons learned during this
part of the case study. These are summarized as follows:
– Lack of non-technical user adoption for test authoring. Though non-technical
business analysts and domain testers could easily read and understand Echo’s
fluent syntax, learning how to write Echo tests proved di cult for these types
of users. This is because authoring required knowledge of programming in the
host language including variables, methods, enumerations, and parameters.
– Limited stakeholder access to test documentation. With the DSL test code
being used as living test documentation, stakeholders now needed technical
knowledge of source control repositories to be able to modify test documen-
tation. Similar to test authoring, these technical aspects hindered adoption

by non-technical stakeholders. View-only access was not a major concern
since web-based solutions for version control browsing were freely available.
– Fragile test implementations due to non-determinism. Early automation re-
views revealed one of the risks of using an internal DSL for functional test-
ing. Test authors started using programming constructs such as conditionals,
loops, and sleeps within test scripts. This caused automated tests to exhibit
di↵erent behavior on di↵erent test runs, even though the system under test
had not changed.
– Insu cient tool support for enforcing standards. Even though there was
much improvement in the uniformity of test scripts, Echo still fell short
in some areas of standardization. Perhaps the most important of these is en-
forcing requirements-to-test traceability. Code templates and style checking
tools provided some help, but were limited since they could not ensure that
authors used valid requirements identifiers in their tests.
– Miscategorization or incorrect test selection. Test tagging features were non-
restrictive since they were inherited from the underlying test runners. Users
could therefore enter any value for a given tag. This was problematic in
that it led to tests being excluded from selection if: (1) di↵erent terms were
used to refer to the same concept, i.e., synonyms were treated as di↵erent
categories; and (2) typographical errors were made when tagging test cases,
i.e., slight variations in spelling were also treated as di↵erent categories.
To address these shortcomings, a range of Echo tools and workarounds were
developed including: test templates and code insertion snippets to enhance the
test authoring experience; and static analysis tools to check that test scripts
conformed to a predefined standards. Building these types of tools motivated
research on using Echo as a platform for creating Legend, an external DSL and
toolset for automated functional testing in agile projects [14, 25].
4.3 Legend: Agile DSL Toolset for Functional Testing
Legend is the product of rethinking the way external DSLs for functional test-
ing are designed and used to support agile development [14]. It combines test
specification and automation into a single, non-technical, authoring experience.
To achieve this, Legend blends the architectural styles for internal and external
testing DSLs presented in Sections 3.2 and 3.3 respectively.
Architecture. The approach to designing the Legend DSL is illustrated in
Figure 4. Similar to Cucumber and SpecFlow, Legend is an external DSL that
facilitates the specification of behavioral scenarios (top-left) using a BDD style
syntax. However, Legend di↵ers from these other languages in that test automa-
tion is intrinsic to the DSL (top-right). Abstract, English-like constructs for
writing automated tests against one or more application models are built into

14 Tariq M. King
Domain-Specific Testing Language
Specification Test Automation
Abstract Test
Commands
Behavioral
Scenarios
Compilation
Application Model
Interface
Abstract Test
Fixture
Execution
Middleware Testing Framework
Test Automation Tools and Frameworks
Fig. 4: A DSL Architecture for Functional Test Specification and Automation
the DSL itself. Both specification and automation statements in the language
are compiled, and therefore go through syntax and semantic analysis.
Legend interfaces with the Echo middleware framework for integrating ap-
plication models and driving test execution [24]. Application model elements
and macros defined in Echo can appear in Legend test scripts. During compi-
lation, Legend checks that all automation commands and model references are
supported by Echo, and generates object code that can be executed by the un-
derlying testing tools and frameworks. In addition to the DSL, Legend provides
a toolset to support functional testing in agile projects.
Technology Stack. A prototype of Legend was developed in C# as a Visual
Studio extension. Project workspace and UI components were implemented using
the Visual Studio SDK and Windows Presentation Foundation [31, 36]. Language
translation and code generation services were realized using the Roslyn .NET
compiler platform [10].
Toolset Features. A screenshot of the Legend Visual Studio project workspace
is presented in Figure 5. The workspace consists of three major UI components:
(a) DSL Editor – a test authoring environment that facilitates text editing, and
includes features such as syntax coloring, error highlighting, intelligent sugges-
tions, block outlining, tool tips, and breakpoints (b) Data Manager – a grid
control for adding, removing, and updating di↵erent types of test data; and (c)
Test Explorer – a navigable tree control for browsing, searching, organizing, and
executing tests. These components collaborate to provide the following features:
– Story Linking and Navigation. The DSL editor provides the user with the
ability to link one or more business requirements to a test suite. This is
achieved by entering a list of user story identifiers into the Stories sec-
tion of the Suite Information block (top of Figure 5a). Upon inputting

Fig. 5: Legend UI components: (a) DSL Editor, (b) Data Manager, (c) Test Explorer.
a story identifier, Legend queries the requirements inventory to ensure it is
valid, and transforms the identifier text into a clickable hyperlink. Clicking
the hyperlink opens a tab within the project workspace and automatically
navigates to the user story. To promote requirements traceability, Legend
requires that each test suite be linked to at least one user story.
– Test Template Generation. As part of the test authoring experience users can
create a test from an existing story. This feature prompts the user to enter the
unique identifier of the story under test. Legend then generates a template
to aid the user when writing automated tests against the acceptance criteria
defined in the story. The result of test template generation is a syntactically
correct Legend test file, pre-populated with the following information: a set
of behavioral scenarios copied from the acceptance criteria of the given user
story; the author name initialized to the currently logged in user; and a
clickable hyperlink to the story under test.
– Model-Based Automation. The steps of behavioral scenarios can be refined
with automated test commands in a stepwise fashion. On the line immedi-
ately following a scenario step, the test author can use the By keyword to
add a block of automated test commands. Like scenario steps, automated
test steps are English-like, but incorporate abstract test commands and ap-
plication model elements. Any data values specified in tests may be replaced
with formal parameters, defined by prefixing a parameter name with the @
symbol, e.g., @username. Macros are integrated into Legend as executable

16 Tariq M. King
phrases that can also be associated with formal parameters. The actual val-
ues of parameterized tests and macros are maintained in the Data Manager
(Figure 5b). Tests can be executed and debugged via the DSL editor.
– Test Inventory Synchronization. To facilitate searching, reviewing, and or-
ganizing documented tests outside of the IDE and code repository, Legend
seamlessly integrates with test case inventory/management systems, e.g.,
Microsoft Test Manager, Zephyr [23, 37]. A context menu within the DSL
editor provides the author with an option for publishing the active test file
to a test inventory. Legend then parses the test suite and inserts the test
scenarios into the inventory via API calls. A mechanism for directly viewing
test inventory artifacts from within the project workspace is also provided.
The test author can use this subfeature to view any comments that reviewers
post on the test cases in the inventory.
– Centralized Test Tagging. Challenges associated with test categorization and
selection are tackled by incorporating a centralized taxonomy of tags into
Legend. The DSL editor presents test authors with a list of valid test cate-
gories that can be inserted into the Tags section of the Suite Information
block (Figure 5a). Misspellings are flagged as invalid tags and displayed as
errors on the editor. Furthermore, tag synonyms have been incorporated
into intelligent suggestions. The Test Explorer (Figure 5c) supports view-
ing test cases by Tag, Author, or Project. While browsing tests in any of
these logical views, users can navigate directly to individual test scenarios
or selectively execute a subset of tests.
Demo. The following video showcases key features of the Legend DSL toolset:
To watch the demo, scan the quick response (QR) code on the left
with a mobile device or visit http://youtu.be/Z5K6plr7cNA
Presented by Dionny Santiago, Ultimate Software (6m 15secs).
5 Conclusion
Although the use of domain-specific languages for automated functional testing
is not new, there have been few significant advances to the state-of-the-art in
recent years. The framework for integrated test (FIT) in 2002 was one such
advancement, followed a year later by the first ever behavior-driven development
(BDD) framework [7, 18]. Since the introduction and rise in popularity of BDD,
tools have been expanded to support various programming languages and test
runners. However, the fundamental design and features of BDD frameworks have
generally remained the same.
Research on DSLs for functional testing provides insight into the communi-
cation and validation needs of project stakeholders. Comparing these needs with

existing tools helps to identify gaps in the current state-of-the-art. Echo and
Legend are the products of research directions aimed at closing gaps in language
expressiveness, IDE support, and process integration. It is my hope that shar-
ing these ideas, designs, prototypes, and experiences will stimulate additional
research and innovation in these areas.
Acknowledgments
The author would like to thank Adam Cando, Cody Mack, Dionny Santiago,
Francisco Montas, Gabriel Nunez, Jorge Martinez, Matthew Wallick, Michael
Mattera, Robert Vanderwall, Rui Lin, Sang Nguyen, and Troy Thomas for their
contributions to this work. Any opinions, findings, conclusions, or recommenda-
tions expressed in this material are those of the author, and do not necessarily
reflect the views of the Ultimate Software Group, Inc.
References
1. Brooks, Jr., F.P.: No silver bullet essence and accidents of software engineering.
Computer 20(4) (April 1987) 10–19
2. Hendrickson, E.: Agile Friendly Test Automation Tools and Frameworks
(2008) http://testobsessed.com/2008/04/agile-friendly-test-automation-
toolsframeworks/ (Sept. 2014).
3. OMG: Unified Modeling Language, Superstructure, Version 2.4.1 (August 2011)
4. Voelter, M., Benz, S., Dietrich, C., Engelmann, B., Helander, M., Kats, L.C.L.,
Visser, E., Wachsmuth, G.: DSL Engineering - Designing, Implementing and Using
Domain-Specific Languages. dslbook.org (2013)
5. Fowler, M.: Domain Specific Languages. Addison-Wesley Professional (2010)
6. Vernon, V.: Implementing domain-driven design. Addison-Wesley, Harlow (2013)
7. North, D.: Introducing BDD. Better Software Magazine (March 2006)
8. Evans, E., Fowler, M.: Fluent Interface (December 2005) http://
www.martinfowler.com/bliki/FluentInterface.html (September 2014).
9. Steinberg, D., Budinsky, F., Paternostro, M., Merks, E.: EMF: Eclipse Modeling
Framework 2.0. 2nd edn. Addison-Wesley Professional (2009)
10. Microsoft Corporation: Roslyn Compiler (April 2014) msdn.microsoft.com/en-us/
vstudio/roslyn.aspx.
11. Beck, K.: Manifesto for Agile Software Development (2001) http://
agilemanifesto.org (April 2014).
12. Shore, J., Warden, S.: Art of Agile Development (1st Ed.). O’Reilly Media, Se-
bastopol, California (November 2007)
13. Hellesoy, A., Wynne, M.: The Cucumber Book: Behaviour-Driven Development
for Testers and Developers. Pragmatic Programmers. Pragmatic Bookshelf (2012)
14. King, T.M., Nunez, G., Santiago, D., Cando, A., Mack, C.: Legend: An agile
dsl toolset for web acceptance testing. In: Proceedings of the 2014 International
Symposium on Software Testing and Analysis. ISSTA 2014, New York, NY, USA,
ACM (2014) 409–412
15. Huggins, J., Stewart, S., Hanrigou, P., Kasatani, S., Lightbody, P., Chai, H.B.: Se-
lenium: Browser Automation (2004) http://docs.seleniumhq.org/ (Sept. 2014).

18 Tariq M. King
16. Fowler, M.: Page Object (September 2013) http://martinfowler.com/bliki/
PageObject.html (September 2014).
17. Fonseca-Ensor, R.: StoryQ (July 2010) storyq.codeplex.com/ (April 2014).
18. North, D.: What is JBehave? (2003) http://jbehave.org/ (Sept. 2014).
19. Nagy, G., Bandi, J., Hassa, C.: SpecFlow: Pragmatic BDD for .NET (November
2009) http://www.specflow.org/specflownew/ (Sept. 2014).
20. ThoughtWorks: Twist Agile Testing (April 2014) http://www.thoughtworks.com/
products/twist-agile-testing.
21. Ultimate Software: UltiPro Enterprise: Human Resource Information Systems So-
lutions (April 2014) www.ultimatesoftware.com/HRIS.
22. Ultimate Software: UltiPro Mobile: Keep track of HR and Payroll Info, Anytime,
Anywhere (March 2013) www.ultimatesoftware.com/Mobile-Technology.
23. Atlassian: JIRA Agile - Go agile with ease (June 2008) https://
www.atlassian.com/software/jira/agile.
24. Martinez, J., Thomas, T., King, T.M.: Echo: A middleware architecture for
domain-speciﬁc ui test automation. In: Proceedings of the 2014 Workshop on Join-
ing AcadeMiA and Industry Contributions to Test Automation and Model-Based
Testing. JAMAICA 2014, New York, NY, USA, ACM (2014) 13–15
25. Santiago, D., Cando, A., Mack, C., Nunez, G., Thomas, T., King, T.M.: To-
wards Domain-Speciﬁc Testing Languages for Software-as-a-Service. In: ACM-
IEEE MODELS 2013: Workshop on Model-Driven Engineering for High Perfor-
mance and Cloud Computing. Volume 1118., CEUR (2013) 43–52
26. Solomon, J.: SWAT: Simple Web Automation Toolkit (2007) http://
sourceforge.net/projects/ulti-swat/ (Sept. 2014).
27. Martin, R.C., Martin, M.D., Wilson-Welsh, P.: FitNesse: Wiki and acceptance
testing framework http://www.fitnesse.org/ (Sept. 2014).
28. Resig, J.: JQuery (August 2006) http://jquery.com/ (Retrieved Sept. 2014).
29. Yeh, T., Chang, T.H., Miller, R.C.: Sikuli: Using GUI Screenshots for Search
and Automation. In: Proceedings of the 22nd ACM Symposium on User Interface
Software and Technology. UIST ’09, New York, NY, USA, ACM (2009) 183–192
30. Brown, J.: MbUnit Test Framework http://mbunit.com/ (July 2013).
31. Microsoft Corporation: Visual Studio (April 2014) msdn.microsoft.com/en-us/
library/bb166441.aspx.
32. Poole, C., Terrell, J., Busoli, S.: NUnit http://www.nunit.org/ (Sept. 2014).
33. Microsoft Corporation: Verifying Code by Using UI Automation http://
msdn.microsoft.com/en-us/library/dd286726.aspx (Sept. 2014).
34. Microsoft Corporation: SQL Server (May 2014) www.microsoft.com/sqlserver.
35. Cuellar, D.: Appium: Automation for Apps http://appium.io (Sept. 2014).
36. Microsoft Corporation: Windows Presentation Foundation (April 2014)
msdn.microsoft.com/en-us/library/ms754130.aspx.
37. Microsoft Corporation: Team Foundation Server (April 2014) msdn.microsoft.com/
en-us/vstudio/ff637362.aspx.

Functional Testing with Domain-Specific Languages

Recommended

Recommended

More Related Content

Similar to Functional Testing with Domain-Specific Languages

Similar to Functional Testing with Domain-Specific Languages (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Functional Testing with Domain-Specific Languages