The world is changing. Everything in enterprise software is moving to the Web. The economic
advantages of browser software delivery (no installs, instant availability everywhere, updates applied
centrally) create great demand for this type of enterprise application. But, these new Web applications
are difficult and expensive to test. In fact, most Web testing is currently hand built.
While there are many point testing tools on the market today, they are too hard to use and are
breathtakingly expensive. With the shift to SOA and Web 2.0, the industry requires an integrated,
collaborative, affordable environment that allows users to quickly and easily test Web applications and
SOA will fail if a new testing paradigm does not arise that can handle the complexity of real-world Web
What do we mean by real world? Web applications and services made up of thousands of messages
and events, supporting tens of thousands of users, that are based upon a variety of protocols…SOAP,
REST, HTTP, Ajax and so on.
This is the problem that SOASTA was founded to solve. SOASTA is delivering the industry’s first
browser-based Web testing service. SOASTA Concerto is Web native and purpose built to support the
testing of any Web application or service…from the message layer to the user interface.
SOASTA simplifies today’s complex world of software testing by substantially reducing the effort and
technical expertise necessary to test Web applications and services. This will make Web testing more
productive, while enabling companies to deliver higher quality software to market more quickly.
While there are many acronyms that have been invented to describe the different variations in
approach, such as SOA, and Web 2.0, there is deep consensus that XML messaging and Web
Services are the basic building blocks of these new Web applications. Proper and complete Web
testing is difficult because:
• The architecture of a Web application or Web service involves more technology (browser,
Web server, application server, database server, etc.) than in a client/server architecture.
While it is a better, less expensive, and easier delivery platform for end users, it is also more
complex and offers many more technology choices for the developer. These technologies
must also be handled by the tester as well. This means that the testing task is bigger and
more difficult overall.
• Web standards. We have many standards. We also have many things that are not standards
but operate like standards (i.e. HTTP, HTML, SOAP, XML, REST, XML-RPC, Ajax, JSON,
CSS, etc.). This diversity gives testers a wide universe of protocols to test. One might argue
that testers can standardize on a smaller set of technologies, but one of the major drivers of
Web applications and services is the ability to integrate an organization’s applications with
those of customers, partners, and suppliers. In doing so, however, the testing scenario
becomes even larger and more complex.
• The Web is not easy to test. There are no standard connectors to testing environments.
• There are many types of testing:
- Functional/regression testing (does my application work?)
- Performance testing (is my application fast enough?)
- Load testing (how many users can I support with decent performance?)
SOASTA Concerto™ A New Approach To Automated Web Testing 1
- Stability testing (how long will my application run without a reboot?)
- Platform testing (will my application work across operating systems?)
- Web UI / Ajax testing (will my application work across all browsers?)
This new Web-oriented approach provides enterprises with a flexible and powerful architecture that
promises better, easier scalability, while preserving investments in existing software. Further benefits
of this approach include code reuse, business agility and the technical ability to launch processes
“ON DEMAND”. Every leading software application vendor (SAP, Oracle, salesforce.com, etc.) is
moving rapidly to deliver versions of their products that expose web services to support this
The analyst community is unanimous in supporting this architectural movement and is forecasting its
success in the Automated Software Quality (ASQ) category over the next five years:
• One predicts that, “By 2008, SOA will provide the basis for 80 percent of new development
• One states, “Quality and software testing, a longtime stepchild of the software market, will get
hot. This will be especially true in the context of ensuring that business services perform the
functions they are designed for in a predictable manner.”
• Another reports, “Although SOA once was used primarily by large enterprises to lower
integration costs by using web services, it is now reaching broader implementation in the
United States, regardless of company size or vertical industry designation.”
• A third analyst states, “SOA is as big a transition as the movement from host-based
architectures to client/server and from client/server to Web applications. Each of these
generations has required different approaches.”
• A fourth analyst indicates, “The continual shift toward service-oriented architecture will drive
demand for new ASQ capabilities to assess and certify Web services and components.”
• A fifth analyst remarks, “There is still a real lack of solutions for doing any real testing of Web
services in the context of SOA” and, “. . . for SOA implementations, testing is essentially being
done in a custom way.”
While these new application architectures deliver the ability to quickly integrate and customize
applications, they are necessarily complex. A full SOA implementation of an application system in the
enterprise will expose all of its transaction types as operations. These operations are then invoked by
sending the correct message. A complex application like ERP can support tens of thousands of
message types. This complexity makes the task of testing and certifying applications far more difficult
than in the past.
SOASTA believes there is a large market requirement for easy-to-use integrated testing solution that
simplifies and drastically reduces the effort necessary to build, modify, demonstrate, test, certify, and
re-certify Web services and applications. In 2006, SOASTA will implement and bring to market a
suite of visual software products that directly and easily support the creation and playback of complex
Traditional Testing Approaches
An examination of the marketplace is necessary to understand why current solutions do not
adequately meet the testing requirements of new Web applications and services. There are three
product approaches to software/services testing.
2 SOASTA Concerto™ A New Approach To Automated Web Testing
• GUI Testing Tools
• Capture/Replay Testing Tools
• Perl Scripts/Hand-Coded XML/In-House Tools
GUI Testing Tools
This is the largest category of existing ASQ tools representing over 80 percent of the current market.
Mercury Interactive (now part of Hewlett-Packard) pioneered this approach to automated testing in
1991 when it introduced Windows technology to “capture user actions” in the user interface and
automatically generate reusable scripts. Those scripts are then replayed in WinRunner™ to test
Windows UIs and in LoadRunner™ to simulate hundreds and thousands of simultaneous Windows
end users. This was a revolutionary product approach at the time because it allowed business
analysts and end users (instead of software engineers) to participate in creating and running tests to
certify Windows-based client/server applications. Tests could be “captured” and reused electronically.
IBM Rational’s TestStudio™ product is very similar to Mercury’s. These are great products for GUI
Testing of client/server applications. They do not address the needs of developers, sales engineers,
business analysts and testers to create message scenarios for testing complex, composite web
Capture/Replay Testing Tools
A very popular product approach is to capture messages from the existing application and playback
those messages for regression testing. SOASTA believes that this capture capability is one
necessary feature in a full solution to testing a message-based application. While capturing
messages is the easiest and fastest way to generate message volume, this approach does not
ensure the completeness and validity of the captured test. In addition, messages cannot be captured
while an application is under construction. SOASTA Concerto supports message capture, creation,
Perl Scripts/Hand-Coded XML/In-House Tools
Testing, certifying, and demonstrating service-based applications is time consuming, complex,
tedious, error-prone, and expensive. Existing ASQ tools are NOT addressing this problem, and as a
result, the testing of service-based applications is predominantly done using hand-coded messages
that are “played” using Perl scripts or in-house tools. SOASTA’s conversations with industry
practitioners confirm this and SOASTA believes that this is by far the current market-leading
SOASTA Concerto™ A New Approach To Automated Web Testing 3
SOASTA Concerto is the industry's first browser-based service for automating Web testing.
With five powerful visual editors working as one, SOASTA Concerto’s seamless integration of test
design, editing, and output offers everything users need to test and deliver high quality Web
applications and services.
SOASTA’s design combines a drag-and-drop visual interface with the ability to access the underlying
message complexity. Through an easy and powerful UI, users create test scenarios that generate
complex, parallel message streams. Task specific visual editors support the easy and efficient
creation of targets, messages, and test compositions.
Based upon the premise that sending a message to a target is the same as playing a musical note,
SOASTA Concerto is modeled after today’s popular digital media creation tools such as Apple’s
GarageBand and iMovie or Adobe’s Premiere. Building and executing a test or an event is as easy
as creating and editing a piece of music or a movie from a series of film clips. This new metaphor
requires a new taxonomy. As such, SOASTA Concerto has adopted the following naming convention:
• Message – raw XML or text for the protocol being used in the test
• Test Clip – equivalent to a unit test
Test Composition – a combination of test clips arranged on “tracks” to make up a complete
• Playlist – multiple test compositions to be played all at once or one after another.
Test compositions are “played” on a multi-track, timeline-based mixing board. Test results are
captured, compared to expected results, and then certified or rejected. Test results are available in
real-time and historically. SOASTA Concerto’s repository automatically maintains all messages, test
compositions, and other system objects as well as a complete historical archive of test results. All
tests are reusable.
SOASTA Concerto supports the following types of testing:
• Functional Testing
Modern Web applications and services contain multiple levels of technical complexity that
makes building and executing valid Web application test suites more difficult than ever before.
SOASTA Concerto handles this complexity, substantially improving testing productivity.
SOASTA Concerto has built in support for testing Web applications and services using SOAP,
REST, XML-RPC, POX as well as support for testing at the Web server using HTTP messages.
Cross browser and Ajax testing completes SOASTA’s ability to test the complete Web
application technology stack at every layer (separately and simultaneously). SOASTA provides
a single integrated testing service for all levels of Web testing.
SOASTA’s visual approach supports the drag-and-drop creation of unit tests and is self-
documenting. These unit tests may be reused many times as part of test compositions to
simulate very complex testing scenarios. SOASTA’s unique Results Viewer allows the test
designer instant access to errors in test results. No more digging through text logs to decipher
the source of errors. Productivity is enhanced and tester fatigue is reduced.
4 SOASTA Concerto™ A New Approach To Automated Web Testing
• Load/Capacity Testing
SOASTA Concerto has a number of features to support load testing at any point in the
development cycle. Any number of test compositions can be run (or played) in parallel in
SOASTA Concerto’s multi-track timeline, allowing for the precise control of timing and testing
volume. No more 1 ... 2 ... 3 ... ENTER. Testers can run many parallel tests with one click of the
PLAY button. Virtual tracks and virtual test clip support allow testers to quickly multiply the
number of virtual users in a test. This is a very fast, easy way to create high-volume real-world
load test scenarios.
SOASTA’s Testing Analytics provide load charting that quickly summarizes load test results,
combining performance statistics from both the test client and Web application server.
• Performance Testing
In many real-world load test scenarios performance issues are identified. This requires that
developers and testers dig into the behavior of an individual component. SOASTA Concerto
allows testers to quickly and easily create tests that stress specific components in a Web
application or service. Tests can be repeated continuously and performance statistics are easily
captured from the target application or service. Whether testers need to target the browser, the
Web server, the Web service, or every layer at once, SOASTA Concerto supports the fast, easy
creation of specific unit tests.
• Web UI / Ajax Testing
Not that long ago it was good enough to focus on Internet Explorer (IE) as the browser for a
Web application or service. Today’s rich Browser UIs must run across IE, Firefox, Safari, and
Opera. Ajax user interface approaches add another dimension that must be tested and certified.
Browser actions and browser messages must be captured, simulated, validated, and certified
across all commonly used browsers. Performance data and load testing becomes extremely
important. UI testers need to validate the performance of many Ajax UIs running at once, as
well as the performance of the Web UI when many Ajax messages arrive in a short time.
SOASTA Concerto uniquely supports for all Web UI/Ajax testing requirements.
SOASTA Concerto™ A New Approach To Automated Web Testing 5
Complexity is a huge issue in testing and certifying service-based applications. In a scenario
involving an enterprise application (SAP, Siebel, Oracle, Peoplesoft, etc.), the number of operations
available through the many web services can number in the tens of thousands. Even more daunting,
a complete functional regression test intended for certification of that application will contain
hundreds of thousands of messages. SOASTA Concerto provides the capability to organize and
manage this innate complexity through its support of versioning and the ability to create hierarchical
taxonomies of test cases.
SOASTA Concerto is the best and easiest way to test Web applications and services. The products
deliver unparalleled ease-of-use and productivity through an intuitive and unique visual interface.
This allows complex message streams to be composed by any developer, sales engineer, business
analyst, or tester. Enterprises are now able to create, test, and certify services-based applications in
days rather than months.
6 SOASTA Concerto™ A New Approach To Automated Web Testing
SOASTA Concerto is available as a hosted service, installed on customers’ in-house servers as a
subscription, or as an appliance. All interaction between users and SOASTA occurs over secure
connections to the SOASTA online service. SOASTA requires that every user have a unique user-id
SOASTA Concerto™ A New Approach To Automated Web Testing 7
SOASTA Central provides access to all SOASTA Concerto features and components. The objective
is to allow the user to easily navigate and manipulate objects while supporting fast drill down to the
message detail of a test.
8 SOASTA Concerto™ A New Approach To Automated Web Testing
The SOASTA Target Editor allows users to create a connection to the destination Web application or
service to be tested. Targets are defined items in the SOASTA Repository that contain a specification
of the information needed in order to properly format messages to send to that type of external
application or system. A target location is the definition of a particular instance of a target. It contains
a complete, concrete specification of the physical location of an implementation of the target so that
messages can actually be sent to the target. Any number of target locations may be defined for each
SOASTA Concerto™ A New Approach To Automated Web Testing 9
The SOASTA Clip Editor is where users arrange and organize messages within a message clip.
Similar to PowerPoint’s slide sorter, the SOASTA Clip Editor supports drag-and-drop and cut and
paste for easy manipulation of individual messages and groups of messages. New messages may be
added, deleted, and duplicated. Timing constraints may also be added in this editor.
10 SOASTA Concerto™ A New Approach To Automated Web Testing
The SOASTA Message Editor enables users to directly edit the data contents of individual test
messages. Users are provided the flexibility of doing so at the message level, the results level, or at
the core XML level.
SOASTA Concerto™ A New Approach To Automated Web Testing 11
The SOASTA Composition Editor is a timeline-based mixing board where testers play final test
compositions. Test clips are assembled on the timeline to control sequence, tempo, and timing. This
approach to building a complex message stream is similar to UI approaches found in professional
media creation tools like Apple’s Final Cut Studio™ and GarageBand™.
12 SOASTA Concerto™ A New Approach To Automated Web Testing
The Results Viewer enables users to easily view and navigate test results by flipping through a
jukebox style graphic. Errors are clearly marked in red as “failures” and users can drill down to the
exact message or messages that failed and get detail on what went wrong.
SOASTA Concerto™ A New Approach To Automated Web Testing 13
Real World Example
The following is a real world, step-by-step scenario for creating a functional test against the
Salesforce.com SOAP-based Web service. Your web testing is sure to be more complicated than this
Step - Create a target called Salesforce.com Enterprise API v7 (na4) in the SOASTA Concerto
Target Editor. This SOAP-based target is automatically created from the Web service description.
14 SOASTA Concerto™ A New Approach To Automated Web Testing
Step - Create test clips in the SOASTA Concerto Clip Editor to test specific Salesforce.com
functional scenarios. An example of an Account Creation functional test is shown below. This
test was created in SOASTA Concerto’s powerful Ajax interface allowing for easy drag-and-drop to
quickly create tests.
SOASTA Concerto™ A New Approach To Automated Web Testing 15
Step - Create a composition in the SOASTA Concerto Composition Editor by dragging and
dropping the test clips onto the multi-track timeline. The example below shows a test composition
with seven different test clips. The first test clip (highlighted in green) logs into the Salesforce.com
service and acquires the necessary login credentials. The remaining test clips all use these
credentials to interact with the service and test a specific function. Having these test clips on different
tracks in the timeline allows multiple user scenarios to be executed in parallel simultaneously.
16 SOASTA Concerto™ A New Approach To Automated Web Testing
Step - Play the test composition. Users can easily execute the test composition by merely clicking
on the play button located on the timeline s control bar. The orange playhead moves across the test
composition to indicate which part of the test composition is being executed at any given time.
SOASTA Concerto™ A New Approach To Automated Web Testing 17
Step - Review test results. After the test composition is complete, the results can be reviewed in
the SOASTA Concerto Results Viewer. The Results Viewer makes it very easy to review test results
and quickly determine what errors (if any) occurred in the test. Once these errors are pinpointed they
can be resolved quickly.
18 SOASTA Concerto™ A New Approach To Automated Web Testing
Enterprise software applications are moving to the Web. Based on services and messaging, the
objective is to make the transaction assets of an enterprise available as web services. Orchestration
tools allow these services to be integrated into new processes within days. This new software
generation delivers on the promise of the Agile Enterprise, but it is far more complex and has more
moving parts than previous software generations. As a result, testing service-based applications is
also more complex and requires a new approach. SOASTA’s vision is to provide a complete solution
to automated web testing that addresses the unique requirements of this new generation.
SOASTA Concerto™ A New Approach To Automated Web Testing 19