• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Functional test automation tools
 

Functional test automation tools

on

  • 1,813 views

 

Statistics

Views

Total Views
1,813
Views on SlideShare
1,813
Embed Views
0

Actions

Likes
0
Downloads
44
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Functional test automation tools Functional test automation tools Document Transcript

    • Yphise Software Assessment Report Functional test automation tools EXECUTIVE volume This volume is a component of the Yphise Software Assessment Report (see chapter entitled “Yphise Software Assessment Reports”). Yphise is ISO 9001-certified in software assessment
    • Yphise P.O. Box 350281 Westminster, CO 80035-0281 USA T 303 410-7753 F 303 410-4980 Yphise 6 rue Beaubourg F-75004 PARIS T (33) 1 44 59 93 00 F (33) 1 44 59 93 09 Email : yphise@yphise.com www.yphise.com This report is produced and published by Yphise. Yphise is an analysis and consulting company absolutely independent of any computer or software vendor. The opinions and results put forth are based on a thorough analysis founded on sources of information known to be reliable. Still, Yphise cannot be held responsible for the use made of the opinions and results provided. No part of this document may be reproduced or transmitted in any form or by any means, without the express written consent of Yphise. © Technology Transfer. March 2002. Yphise is a registered trademark. ©Yphise
    • Get the full report This EXECUTIVE SUMMARY condenses the results of Yphise in-depth research published in the REQUIREMENTS and DETAILED EVALUATION volumes. These provide you with a comprehensive assessment of the evaluated products according to our ISO 9001-certified method. We propose you the REQUIREMENTS and DETAILED EVALUATION volumes for $500 instead of $770. FULL DETAILED REPORT "Functional test automation tools" Please return to Yphise - PO Box 350281 - Westminster, CO 80035-0281 - USA Phone 303 410-7753 - Fax 303 410-4980 - yphise@yphise.com or Yphise - 6 rue Beaubourg - 75004 Paris - FRANCE Phone (33) 1 44 59 93 00 - Fax (33) 1 44 59 93 09 - yphise@yphise.com I order : The REQUIREMENTS (80 pages) and DETAILED EVALUATION (200 pages) volumes $500 Billing address ___________________________________________________________________ Company _______________________________________________________________________ Phone/Fax ______________________________________________________________________ Address ________________________________________________________________________ ______________________________________________________________________________ Your name______________________________________________________________________ Title ___________________________________________________________________________ Service_________________________________________________________________________ Purchase Ref____________________________________________________________________ ‰ AMEX ‰ VISA ‰ MASTERCARD Authorized signature Card number __________________________________ Card Holder __________________________________ Expiry date __________________________________
    • ©Yphise
    • Contents FOREWORD OVERALL PRESENTATION FUNCTIONAL TEST AUTOMATION TOOLS MARKET SEGMENTATION ASSESSMENT DIMENSIONS OVERALL RANKING RANKING BY ASSESSMENT DIMENSION 1. TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS 2. PROVIDING FUNCTIONAL RICHNESS FOR TESTING 3. FACILITATING THE TEST DEFINITION AND USE 4. ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION 5. PROVIDING A CLEAR VIEW OF THE TEST RESULTS THE ASSESSED SOFTWARE PRODUCTS AUTOTESTER QARUN ROBOT SILKTEST WINRUNNER YPHISE SOFTWARE ASSESSMENT REPORTS ©Yphise
    • ©Yphise
    • Foreword Objective of the EXECUTIVE volume This volume is a component of the Yphise Software Assessment Report (see chapter entitled “Yphise Software Assessment Reports”). It is designed to monitor developments of the software product market and to select a short list. • It assesses the maturity and opportunities of the products available. • It highlights the list of software products that we consider of interest to large companies. Our experience has shown that products do not always satisfactorily provide the functions that we expect, based on vendor positioning. As a result, many improper selections of software products are made. • It highlights the strengths and weaknesses of each software product, based on our detailed assessment conducted according to our ISO 9001-certified methodology. It provides a top- down ranking of the products and it outlines our opinion. The charts The charts in this volume aggregate all the results, criterion by criterion, established in the corresponding DETAILED ASSESSMENT volume. • Sorted Aggregated Table. The products (lines) and chapters (columns) are both sorted so that the best notes appear at the top left as much as possible. This chart provides a ranking of the products in descending order from top to bottom, and a best-to-worst ranking of the chapters from left to right. • Ranking chart. The best product on a dimension has the highest ranking. ⇒ This does not mean it is excellent. This is the reason the icon representing the note obtained by the best product is drawn at the end of each dimension. ⇒ An identical gap between products does not have the same significance for each dimension. On a dimension where the best application is average, a product can be ranked as being clearly better than another one just because it has a “little extra”. • Strength and Weakness List. ©Yphise
    • Position of the product Best product note Average note Worst product note ©Yphise
    • Functional test automation tools Added value When the information system changes due either to a new application or to a new version of an existing application, the IT department needs to test the added functions. Since there are many use cases to test, the test needs to be automated. The application being integrated into the information system and the data flow exchanged with other applications must be tested. Since the data flows from many sources, this test must also be automated. When introducing a new version of an application, the IT department must ensure the nonregression of the application and its integration into the information system. It must verify that the functions already tested have not changed. It must test, for each release, the functions that were tested previously. As the applications become more complex, the nonregression tests also become more complex, since the testers must reproduce the same initial state and flow. The functional test automation tool makes it possible to record a test and to reproduce it easily. It can automate the test replay based on specific use cases. It can also chain the tests and collect the test results. Once a test is completed, the functional test tool can compare the test results to a reference state. This makes it possible to detect any error and, where there is none, to validate the application. Functional architecture The functional test automation tool provides the following functions: • Definition of tests. The definition of tests is done by recording an interaction with the application to test. The record outputs a test script that can be edited by using the integrated script editor. In order to handle data-driven tests, the functional test automation tool provides data access capabilities that help select the data source accurately. In order to facilitate the test result analysis, the tool can be used to define control points that can be set either on graphical objects or on data. • Execution of tests. The execution of tests is automated. Various test cases are executed by reproducing the recorded user interaction. Data-driven tests are executed based on the data access that was set during the definition period. • Reporting on test results. When the test is finished, the test results compare the test execution to the reference state (e.g. based on the control points set during the definition period). Accurate details can be obtained on each difference. ©Yphise
    • In order to facilitate the test definition and execution and the reporting of test results, the functional test tool relies on a repository where are stored the following elements: • Functional library. This is the list of available functions for defining complex test scripts. • Object library. This is the list of recognized objects, depending on the development environment used to develop the application and the platform on which it is installed. • Test scripts. The test records output test scripts that can be edited. These scripts are used to replay the test. • Test results. The test results data are in open format in order to facilitate reuse by third- party analysis tools. Functional test automation tool Definition of tests Test repository Test recorder Script editor Function library Control points Data access Object library Execution of tests Test scripts Test results Reporting on test results ©Yphise
    • Market segmentation Positioning Functional test automation tools differ from test plan and campaign management tools. Test management tools do not allow the recording of tests. They provide capabilities for integration into other test tools (including functional test automation tools) in order to manage defined test plans and campaigns. Examples of test plan and campaign management tools include AutoAdviser (AutoTester), QA Director (Compuware), QCIT (QIS), SilkPlan (Segue Software), Test Director (Mercury Interactive) and Test Manager (Rational Software). Functional test automation tools differ from test evaluation tools. They focus on “black box” tests, while test evaluation tools are used for “white box” test. Functional test automation tools do not inspect the application source code. Examples of test evaluation tools include PureCoverage (Rational) and Test (McCabe). Functional test automation tools differ from load and performance test tools. They do not measure the response time and the ability of an application to respond to a workload increase. Examples of load and performance test tools include AutoController (AutoTester), LoadRunner (Mercury Interactive), QA Load (Compuware) and SilkPerformer (Segue Software). Software products assessed Yphise has assessed the following products, which can be grouped into two groups depending on their commercial packaging: • Standalone product: QARun (Compuware) and WinRunner (Mercury Interactive). • Product bundled with complementary tool: AutoTester bundled with AutoAdviser (AutoTester), Robot bundled with Test Manager (Rational Software) and SilkTest bundled with TestPlan (Segue Software). Yphise did not assess WinEZ (QIS), since its vendor could not provide significant worldwide references. Yphise did not assess e-Tester (Empirix), since its distributor did not provide resources for the evaluation. Yphise policy is to recommend products only if sufficient skills and documentation are available. Market trends Functional test automation tools are becoming specialized components within test suites. This is reinforced by the move, from functional test automation tool to test plan and campaign management tools, of functions not required for the test recording. ©Yphise
    • • A scheduler was previously integrated into Robot (Rational Software). The recent versions lack this feature, since the scheduling functions are now provided by the vendor’s test plan and campaign management tool (Test Manager). • A reporting module (based on Crystal Reports) was previously integrated into WinRunner (Mercury Interactive). The recent versions lack this feature, since reporting is now handled by the vendor’s test plan and campaign management tool (Test Director). • Integration of functional test automation tools into test plan and campaign management tool is also evidenced by the recent addition, in the Segue Software catalog, of a product called SilkPlan (based on T-Plan’s Professional). This product provides advanced planning functions, compared to those provided by TestPlan, which is included in SilkTest. New versions or new products of functional test automation tools demonstrate a user- friendliness that the previous ones lacked. Products such as QuickTest Professional (Mercury Interactive) and Test Partner (Compuware) provide a user interface that facilitates use by business users, with no need to know coding test scripts. ©Yphise
    • Assessment dimensions Overview • Testing the applications in various environments. The applications are installed on various platforms and accordingly provide different user interfaces. When testing the nonregression, the functional test automation tool can test the various applications according to the specificity of each platform on which they are installed. • Providing functional richness for testing. The applications to test are functionally rich, and the test cases are complex. The functional test automation tool makes it possible to define complex and accurate tests. It takes into account all the interactions with the application. When an object or action is not recognized, the tool can extend the available object library. • Facilitating the test definition and use. The tests can be defined by the end-users, who are more familiar with the application to be tested. The functional test automation tool provides user-friendly functions in order to facilitate the definition and execution of tests. It can capture the user interactions and replay them. • Ensuring the technical robustness of test execution. The test cases are numerous. This requires a large number of runs for a single test. Because multiple tests can be executed at the same time, the functional test automation tool can rely on the technical robustness of the platforms on which it is installed to ensure the completeness of the test execution. It also makes it possible to control the test execution. • Providing a clear view of the test results. Once the tests are completed, the functional test automation tool provides reports on the results. These results are then used to compare the test cases to a reference state. They can be combined with other results in order to point out recurrent errors or to verify the nonregression of an application from an earlier stable version to the newly developed version. Coverage of the requirements • The best-covered dimension is “Providing functional richness for testing”. This dimension is a critical one. It constitutes the core of functional test automation. Its coverage ensures the support of the applications’ functional richness in order to facilitate the definition of complex tests. • The least-covered dimension is “Ensuring the technical robustness of test execution”, since most of the products do not handle the concomitant execution of numerous tests. This task is reserved for test plan and campaign management tools. We recommend relying on these, since a large number of tests must first be planned before they are defined and executed. ©Yphise
    • Robot QARun SilkTest AutoTester WinRunner PR OV ID IN G TE FU Sorted Aggregated Table ST NC EN IN TIO VI G NA RO TH LR NM E A IC FA EN P P HN TS LIC ES CI AT LIT IO SF AT NS OR IN G IN TE ST PR TH VA E RI IN OV ID TE OU G IN ST S G DE EN A FIN CL EX SUR EA ITI EC ING R ON UT VI AN TH EW D IO E N TE OF US CH TH E NI E CA TE LR ST OB RE Functional test automation tools US SU LT TN ES S SO FT ES T ©Yphise
    • Overall ranking Comments • Robot stands out on “Facilitating the test definition and use”, since it provides a user- friendly interface that helps hide the complexity of test recording from end-users. The steps of the recording are classified and shown graphically with an icon specific to each element. Screenshots of the application are shown for each test step. The product also facilitates test definition by testers, since its script language is based on Visual Basic. • Robot stands out on “Providing a clear view of the test results”, thanks to its ability to provide statistical information on the testing of successive versions of an application. Without requiring a complementary tool, as the other products do, Robot allows a comparison of the test results of one version with those of an earlier version of the same application. It provides limited customization capabilities but does allow the use of a third- party analysis and reporting tool, for which it provides a specific interface. • Robot stands out on “Ensuring the technical robustness of test execution”, thanks to its ability to distribute the test execution among various stations, based on criteria such as the volume of test cases and the number of independent test sequences. The product makes it possible to control the execution of tests by taking into account the iterations and the workload schedules. These features are useful for organizing the execution of numerous tests. • WinRunner stands out on “Providing functional richness for testing”, thanks to its ability to facilitate the definition of control points in the tests. The properties of a recognized object are collected automatically. The product also allows the collection of information on new objects. This is useful for refining control points accurately, even for objects not initially recognized. • WinRunner stands out on “Testing the applications in various environments”, thanks to integration into the vendor’s other test tools, such as QuickTest Professional aimed at Web- based interfaces and SAP R/3 applications, and XRunner specialized in testing X-Window interfaces on Unix platforms. The product also integrates into Robot by using its Script Runner conversion tool. • The ranking of AutoTester on “Providing functional richness for testing” is explained by its inability to recognize specific Web-based objects such as iFrames and layers, since it does not support Version 4 of HTML. This support should be provided by another product, AutoTester ONE, announced for the first quarter of 2002. • The ranking of AutoTester on “Testing the applications in various environments” is due to its lack of support for Java environment. It is the only product that does not allow testing in this environment. Also, the product provides no specific support for development languages and development tools. It only recognizes Windows-based objects, based on their visible attributes. ©Yphise
    • 1. TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS NS IO AT IC PL N IO AP AT EN IC WE PL AP ET PB OF HI ITY NS FIC TIO CI LA ES SPE RE AC HE E TH T TE ING G IN RF ST ST TE TE IN WinRunner QARun Robot SilkTest AutoTester ©Yphise Sorted Aggregated Table
    • 1. Testing the applications in various environments Overview • Testing the specificity of application interfaces. The applications used in the enterprise provide various interfaces. These interfaces are available on various platforms. The functional test automation tool recognizes all available interfaces in order to ensure the testing of all possible interactions with the application. It takes into account the variety of the application interfaces and the platforms on which these are installed . • Testing the relationship between applications. When integrating a new (version of an) application into an existing information system architecture, the administrators want to ensure that the data flow between the applications is handled correctly. The functional test automation tool makes it possible to test not only the application itself but also its integration into other applications. When the integration is based on data flow, the functional test automation tool checks data availability on the various DBMSs. Comments • QARun and WinRunner stand out on “Testing the specificity of application interfaces”, thanks to their support for VisualAge, Uniface and Magic development tools. The products also provide specific support for Baan’s ERP System. These features help in taking into account the specificity of the development tool or the ERP application. • Robot stands out on “Testing the relationship between applications”, thanks to its specific support for Oracle DBMS, which helps in the testing of data-based relationships between applications. This makes the product suitable for projects where this DBMS is used to store data shared by various applications. • The ranking of AutoTester on “Testing the specificity of application interfaces” is low, because it does not allow the testing of Java-based interfaces. Also, it provides no interfaces to application development tools. These interfaces are useful for retrieving the hidden attributes of the graphical objects and facilitating their recognition and testing. • The ranking of AutoTester on “Testing the relationship between applications” is low, because it does not interact directly with the DBMS. An interface with the DBMS is needed when testing the relationship based on a data exchange between applications. • The ranking of QARun on “Testing the relationship between applications” is due to its inability to test a relationship between applications that is based on data or messages. This lack of support for data and message exchange makes the product unsuitable for integration tests. ©Yphise
    • 2. PROVIDING FUNCTIONAL RICHNESS FOR TESTING X LE MP CO F E SO FIN ES DE HN TO IC NS LR TIO NA NC TIO FU NS U N C CH F RI E TH G IO IN G AT TE VID IN IC ST S O ST PL PR TE AP WinRunner QARun SilkTest Robot AutoTester ©Yphise Sorted Aggregated Table
    • 2. Providing functional richness for testing Overview • Testing the functional richness of applications. Applications are complex and provide various functions. This makes testing of all the use cases more difficult. The functional test automation tool helps test these applications by recognizing their graphical objects and supporting their functions in order to ensure the testing of all the use cases. • Providing rich functions to define complex tests. In order to define accurate tests, the functional test automation tool provides a functional richness that takes into account all the interaction of an application. This ensures that all the functions and test steps are performed correctly. It also allows control of the functions that are tested or not. Comments • WinRunner stands out on “Providing rich functions to define complex tests”, thanks to its ability to assist the testers in defining the control points on the graphical objects of the application to test. The product collects all available properties on recognized objects and allows the collection of information on unrecognized objects. Once this information is collected, the product makes it possible to refine these properties in order to define accurate control points. • The ranking of AutoTester on “Testing the functional richness of applications” is due to its lack of support for specific Web-based objects such as JavaScript, ActiveX and multimedia objects. These objects are now widely used on Web sites. The product supports only Version 3.2 of HTML. The vendor announced, for the first quarter of 2002, a new product, AutoTester ONE, targeted to supporting the Web-based application interfaces. ©Yphise
    • 3. FACILITATING THE TEST DEFINITION AND USE S ST TE US S RO ST S ME TE ST NU TE OF OF OF E NC N ON IO NA ITI UT TE FIN EC IN MA DE EX E E E TH TH TH G G G IN IN IN AT AT AT LIT LIT LIT CI CI CI FA FA FA Robot SilkTest QARun AutoTester WinRunner ©Yphise Sorted Aggregated Table
    • 3. Facilitating the test definition and use Overview • Facilitating the definition of tests. Testers have numerous tests to define. They want to define them easily and rapidly. The functional test automation tool helps define the tests easily, thanks to the use of task recording. It allows the users interact as usual with their application and then records all their actions. The record generally outputs a script that must be readable by the testers. The functional test automation tool also allows testers to build test scripts through the use of assistants and then to debug these scripts easily. • Facilitating the execution of numerous tests. Testers manage a large number of tests. They want to organize these tests logically in order to facilitate use. The functional test automation tool allows the accurate handling of numerous tests and the reuse of a specific test in numerous and various other cases. It helps define variables within the test script and use external data to facilitate the execution of tests corresponding to the various cases. • Facilitating the maintenance of tests. One test is defined for a specific version of an application. When a new version needs to be tested, the testers do not want to redefine the test. The functional test automation tool facilitates the test script update and the building of new tests based on existing ones. It also allows testers to manage various versions of their tests. This facilitates test maintenance. Comments • Robot stands out on “Facilitating the definition of tests”, thanks to the user-friendliness of its interface. The record steps are classified and shown graphically with an iconography specific to each element and screenshots of the application displayed for each test step. The product also facilitates the test definition, since its script language is based on Visual Basic. • SilkTest stands out on “Facilitating the definition of tests”, thanks to its ability to automate the porting of a test from one platform to another. The tester only needs to recompile the test script on the new platform after the GUI map has been updated separately. This saves time in terms of test redefinition, especially when the test is complex. • QARun stands out on “Facilitating the maintenance of tests”, thanks to its ability to manage versions of the test elements. The product creates a new version each time a test script is modified. It also manages the versions for control points and test results. This makes the product suitable for test teams with numerous testers intervening on the test elements. • The ranking of AutoTester on “Facilitating the definition of tests” is due to its lack of graphical buttons that facilitate the control of test recording by end-users. The product does not provide stop or suspend buttons. Such user-friendly elements facilitate the test recording. ©Yphise
    • 4. ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION S ST TE S ST OF TE N OF IO UT N IO EC UT EX EC E EX TH E G TH IN LL NG RO RI NT SU CO EN Robot SilkTest AutoTester QARun WinRunner ©Yphise Sorted Aggregated Table
    • 4. Ensuring the technical robustness of test execution Overview • Ensuring the execution of tests. Tests may take a long time to execute. The functional test automation tool ensures the completeness of test execution by relying on the technical robustness of the IS infrastructure of the enterprise. It helps avoid any test execution interruption due to technical incidents or human intervention. When an interruption does occur, the tool allows continuation of the tests, resuming at the interruption point. • Controlling the execution of tests. When numerous tests must be executed simultaneously, the functional test automation tool allows load distribution among various stations. Also, when various cases must be tested for a single test, the tool helps define execution sets that can be processed simultaneously. This helps reduce the total execution time. To ensure execution of the tests, the functional test automation tool provides monitoring functions in order to intervene accurately when a problem happens. Comments • Robot stands out on “Controlling the execution of tests”, because it can control the execution of tests by taking into account the tests iterations and the workload schedules. It also helps estimate the test execution load, based on criteria such as the requirements and the network use of the master station and the playback agents. These features are useful for organizing the execution of numerous tests. • Robot stands out on “Ensuring the execution of tests”, thanks to its ability to distribute the test execution on various stations, based on criteria such as the volume of test cases and the number of independent test sequences. The product provides playback agents for various platforms, including Windows and Unix (AIX, HP-UX, Linux, Solaris and Dynix). This facilitates the deployment of test replay on these various platforms. • The ranking of WinRunner on “Ensuring the execution of tests” is low because it does not rely on a DBMS to store its repository. When the technical robustness of a DBMS is needed, another product, Test Director, could be used. The complementary product used for test plan and campaign management is not bundled with WinRunner. ©Yphise
    • 5. PROVIDING A CLEAR VIEW OF THE TEST RESULTS ST TE E TH N SO RT S LT PO SU RE RE TE ST RA TE CU E AC TH G G IN IN RE V I D S YZ LT AL O SU PR AN Robot SilkTest AutoTester WinRunner QARun ©Yphise Sorted Aggregated Table
    • 5. Providing a clear view of the test results Overview • Providing accurate reports of the test results. When a test or a group of tests is finished, testers want to know the results. The functional test automation tool provides testers with accurate reports on the test results. The information collected helps the testers view the positive and negative tests and detect the errors that generated negative tests for each application. • Analyzing the test results. Testers want to use the test result data to build customized indicators. The functional test automation tool provides them with test result data in open formats. This allows the testers to handle the data by using third-party tools. Testers can build a test result history per application in order to track the application dysfunction across versions. This is useful in order to check the nonregression of an application from one version to another. Comments • Robot stands out on “Providing accurate reports of the test results”, thanks to the customization capabilities it provides. It allows testers to select whether the information is displayed graphically or not, to select the graphic type, to customize the graph axis labels, and to add a custom description to the graphics. Furthermore, the product provides a specific interface for Crystal Reports. • WinRunner stands out on “Providing accurate reports of the test results”, because it helps the testers select the information to display in the reports. The product allows testers to select the information by using check boxes. The information available relates to the tester, the test, the control points, the date and time of test execution, and even the total runtime of the execution. • The ranking of AutoTester on “Providing accurate reports of the test results” is due to its inability to let the testers select the information to include in the reports. It also does not allow report customization. The reports are predefined and provide test results in summarized or extended formats. • The ranking of QARun and WinRunner on “Analyzing the test results” is due to their inability to provide statistical information on test results and a historical comparison for nonregression tests of different versions of an application. This information can be provided by the vendors’ respective test plan and campaign management tools, QA director and Test Director. ©Yphise
    • AutoTester ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION FACILITATING THE TEST DEFINITION AND USE PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS Software Ranking Best software evaluation Average evaluation Worst software evaluation ©Yphise Ranking Chart
    • AutoTester Opinion • One of the strengths of AutoTester is its object-oriented recognition of the elements of the application. The graphical elements are first recognized and labeled. Only the name of each element is inserted into the test script. A click on the name brings up a dialog box with the captured properties that can be modified. This feature is useful for updating the test scripts by taking into account the evolution of the object attributes. • One of the weaknesses of AutoTester is its lack of support for specific development languages and toolkits. This support is needed to ensure the replay of the test despite changes in the interface. It helps in retrieving the necessary information (especially hidden attributes such as ID and logical name) on the objects to test from the development environment used for building the application. • AutoTester is not suitable for testing any Java-based interfaces. It does not provide a specific support for this environment. Vendor presentation • Developer AutoTester 10300 N. Central Expressway, Suite 550 Dallas, Texas 75231, USA Tel.: +1 (214) 368-1196 Fax: +1 (214) 750-9668 • Vendor contacted Adar 2, rue Louis Pergaud 94700 Maisons Alfort, France Tel.: +33 148 93 71 71 Fax: + 33 143 68 37 68 • Web site www.autotester.com www.adar.fr • Development laboratory Dallas, USA • Hotline Maisons Alfort, France Software product presentation • Version number 5.01 • Date of first sale 1993 • Price indication AutoTester for Windows: 5,500 Euros AutoTester for SAP R/3: 8,600 Euros • Maintenance 18% of purchase price • Training Rapid development of tests: 6,000 Euros for 10 users and 5 days • Setup services 600 Euros to 1,000 Euros per day ©Yphise
    • • User group No • Main environments Support for Windows, terminal emulation and Web environments Main modules • AutoTester for Windows. Used to perform regression and functional testing on MFC-based applications. • AutoTester Client/Server (for SAP R/3). AutoTester’s specialized R/3 command set allows tests to interact with each R/3 object at the system level, thus reducing the maintenance and enabling the tests to be easily migrated to new versions of R/3. • Test Station. Automated testing solution designed specifically for character-based PC and host applications. • AutoAdviser. Version 4.0. Used for the management of test scripts. AutoAdviser provides a central repository for organizing and managing the business requirements, tests and associated files, and test results. It allows testers to plan and organize the tests. It controls the test schedules and execution, and it handles the reporting of test results. ©Yphise
    • AutoTester Functional test automation tools TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS TESTING THE SPECIFICITY OF APPLICATION INTERFACES TESTING THE RELATIONSHIP BETWEEN APPLICATIONS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE FUNCTIONAL RICHNESS OF APPLICATIONS PROVIDING RICH FUNCTIONS TO DEFINE COMPLEX TESTS FACILITATING THE TEST DEFINITION AND USE FACILITATING THE DEFINITION OF TESTS FACILITATING THE EXECUTION OF NUMEROUS TESTS FACILITATING THE MAINTENANCE OF TESTS ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION ENSURING THE EXECUTION OF TESTS CONTROLLING THE EXECUTION OF TESTS PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING ACCURATE REPORTS ON THE TEST RESULTS ANALYZING THE TEST RESULTS ©Yphise Strengths and Weaknesses List
    • QARun ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION FACILITATING THE TEST DEFINITION AND USE PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS Software Ranking Best software evaluation Average evaluation Worst software evaluation ©Yphise Ranking Chart
    • QARun Opinion • One of the strengths of QARun is its support for Web-based development languages used by application servers for dynamically generating pages. It is the only assessed product that provides specific support for CFML, ASP and JSP development languages. This helps in recognizing the functions associated with Web-based objects in order to test the currently database-driven Web sites. • QARun is suitable for testing Web pages with JavaScript-based events. It is the only assessed product that supports mouse-based events such are those associated with the “mouse over” status. This helps in testing the so-called Dynamic HTML events that can, for example, bring up a navigation menu once the mouse points to a parent object without being clicked on it. • One of the weaknesses of QARun is its inability to define controls points on data. This makes it inappropriate for tests where the control points relate to data (e.g. for batch operations). Defining the controls points on data helps verify that the application handles the input data correctly and outputs the correct values. Vendor presentation • Developer Compuware 31440 Northwestern Hwy Farmington Hills, MI 48334, USA Tel.: +1 (248) 737-7300 • Vendor contacted Compuware France 1 avenue de la Cristallerie 92310 Sèvres, France Tel.: +33 141 14 20 00 Fax: +33 146 23 88 89 • Web site www.compuware.com www.compuware.fr • Development laboratory Detroit, USA • Hotline Detroit, USA Sèvres, France Software product presentation • Version number 4.7 • Date of first sale 1991 • Price indication 5 users: 4,250 Euros • Maintenance 16% of purchase price • Training Testers, developers, users: 3 days • User group No ©Yphise
    • • Recent developments Integration of Java Siebel certification • Announced functions Automated recognition of graphical objects • Main environments Support for Windows, terminal emulation, Java and Web environments Main modules • QARun. Allows the testing of character-based, client-server and graphical applications. The client module is installed on a Windows platform. ©Yphise
    • QARun Functional test automation tools TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS TESTING THE SPECIFICITY OF APPLICATION INTERFACES TESTING THE RELATIONSHIP BETWEEN APPLICATIONS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE FUNCTIONAL RICHNESS OF APPLICATIONS PROVIDING RICH FUNCTIONS TO DEFINE COMPLEX TESTS FACILITATING THE TEST DEFINITION AND USE FACILITATING THE DEFINITION OF TESTS FACILITATING THE EXECUTION OF NUMEROUS TESTS FACILITATING THE MAINTENANCE OF TESTS ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION ENSURING THE EXECUTION OF TESTS CONTROLLING THE EXECUTION OF TESTS PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING ACCURATE REPORTS ON THE TEST RESULTS ANALYZING THE TEST RESULTS ©Yphise Strengths and Weaknesses List
    • Robot ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION FACILITATING THE TEST DEFINITION AND USE PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS Software Ranking Best software evaluation Average evaluation Worst software evaluation ©Yphise Ranking Chart
    • Robot Opinion • Robot is suitable for testers skilled in the Visual Basic development language. The product uses this language to code the test scripts. This gives the testers the ability to extend the functional richness of the tool in order to define complex tests. • Robot is suitable for testers who already use Crystal Reports. The product includes a slimmed down version of the reporting tool. Personalization of predefined reports is allowed for those testers who have already installed Crystal Reports. Integration is seamless, since a specific interface is provided. Robot is the only assessed product that provides this kind of integration into an analysis and reporting tool. • The commercial package of Robot represents an opportunity, since a Web site checking module (SiteCheck) and a test plan and campaign management tool (TestManager) are bundled with the functional test automation tool. This makes the product suitable for the management and the execution of numerous tests. Vendor presentation • Developer Rational Software 20 Maguire Road Lexington, MA 02421, USA Tel.: +1 (781) 676-2400 • Vendor contacted Rational Software 1, Place Charles de Gaulle 78180 Montigny le Bretonneux, France Tel.: +33 130 12 09 50 Fax: +33 130 12 09 66 • Web site www.rational.com • Development laboratory Lexington, USA • Hotline Amsterdam, Netherlands Software product presentation • Version number 2002 • Date of first sale 1994 • Price indication Robot bundles in two packages: • - TeamTest: 7,235 Euros - TestStudio Suite: 4,825 Euros • Maintenance 20% of purchase price • Training Test engineering: 3 days TeamTest: 3 days TestStudio Suite: 3 days QuickStart packages ©Yphise
    • • Setup services 1,350 Euros • User group Test-forum@rational.com Rational Developer Network (www.rational.net) • Announced functions Support for Java and MS .Net environments • Main environments Support for Windows, terminal emulation, Java and Web environments Main modules • Robot. Automates the functional tests for the development languages (e.g. C++, Java, Delphi, Visual Basic, and HTML), IDEs (VisualStudio, VisualAge, JBuilder, and VisualCafé) and the ERP systems (SAP, PeopleSoft, and Oracle). • TestManager. Automates the test processes by providing some planning, management and execution functions. It also allows the monitoring of test execution and the collection of test results. • SiteCheck. Automates the testing, management, and maintenance of Web sites. Its features (e.g. link parsing and extensive link) make it possible to validate the health of a Web site. Other products that integrate into Robot are available within the TeamTest package: • TestFactory. Generates the tests scripts automatically from the application code or binary in C++; Java and Visual Basic. These scripts are executed by Robot. • PurifyPlus. Detects the execution errors, analyzes performance, detects bottlenecks and analyzes the code coverage in an application developed in C or C++. Its integration into Robot facilitates the testing of the nonregression of an application, helps verify that performance is not lowered during the tests and allows the verification of the code coverage of a tested application. ©Yphise
    • Robot Functional test automation tools TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS TESTING THE SPECIFICITY OF APPLICATION INTERFACES TESTING THE RELATIONSHIP BETWEEN APPLICATIONS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE FUNCTIONAL RICHNESS OF APPLICATIONS PROVIDING RICH FUNCTIONS TO DEFINE COMPLEX TESTS FACILITATING THE TEST DEFINITION AND USE FACILITATING THE DEFINITION OF TESTS FACILITATING THE EXECUTION OF NUMEROUS TESTS FACILITATING THE MAINTENANCE OF TESTS ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION ENSURING THE EXECUTION OF TESTS CONTROLLING THE EXECUTION OF TESTS PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING ACCURATE REPORTS ON THE TEST RESULTS ANALYZING THE TEST RESULTS ©Yphise Strengths and Weaknesses List
    • SilkTest ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION FACILITATING THE TEST DEFINITION AND USE PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS Software Ranking Best software evaluation Average evaluation Worst software evaluation ©Yphise Ranking Chart
    • SilkTest Opinion • SilkTest is suitable for testers or developers using the C++ development language. 4Test, the proprietary metalanguage used to record the tests, generates the test scripts in C++. This makes the editing and updating of the generated scripts easier for testers or developers familiar with this language. • One of strengths of SilkTest is its ability to facilitate the porting of tests from one platform to another. This facilitation is provided by the externalization of the GUI map (TestFrame file) from the test script. Only this GUI map is therefore ported automatically from the original platform to any supported platform, by recompiling the test script on the new platform. This makes the product suitable for test campaigns involving applications that are ported from one platform to another. • SilkTest is suitable for projects involving the test of CRM package of Chordiant, Amdocs (formerly Clarify) and E.piphany. Vendor presentation • Developer Segue Software 201 Spring St. Lexington, MA 02421, USA Tel.: +1 (781) 402-1000 Fax: +1 (781) 402-1099 • Vendor contacted Expertilog 67 Avenue Maréchal Joffre 92000 Nanterre, France Tel.: +33 146 97 12 01 Fax: +33 145 06 25 31 • Web site www.segue.com www.expertilog.com • Development laboratory Lexington, USA • Hotline Belfast, Ireland Nanterre, France Software product presentation • Version number 5.5.2 • Date of first sale 1992 (QA Partner was the previous name of SilkTest) • Price indication SilkTest: 7,283 Euros SilkTest International: 9,530 Euros • Maintenance 18% of purchase price • Training SilkTest Foundations: 2 days, 1,458 Euros 4Test language: 2 days, 1,458 Euros ©Yphise
    • Advanced Object oriented 4Test Testing custom objects Testing Java objects • Setup services On demand • User group Yes • Recent developments SilkTest v5.5.2 with: - Support for Microsoft Windows XP, including themes - Remote file operations - Enhancements to 4test - Improved online help system - Support for Internet Explorer 6.0 with the Virtual Object extension and Document Object Module extension • Announced functions Release of SilkTest 6.0 planned for the third quarter 2002 • Main environments Support for Windows, X-Window, terminal emulation and Web environments Main modules • SilkTest. Functional regression tool. • SilkTest Agent. Agent which allows multiuser functional testing on several PCs at the same time. • TestPlan. Formerly QA Organizer, module within SilkTest to organize the tests. • SilkTest International. Unicode-enabled version of SilkTest. Allows the testing of multiple language versions of an application. ©Yphise
    • SilkTest Functional test automation tools TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS TESTING THE SPECIFICITY OF APPLICATION INTERFACES TESTING THE RELATIONSHIP BETWEEN APPLICATIONS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE FUNCTIONAL RICHNESS OF APPLICATIONS PROVIDING RICH FUNCTIONS TO DEFINE COMPLEX TESTS FACILITATING THE TEST DEFINITION AND USE FACILITATING THE DEFINITION OF TESTS FACILITATING THE EXECUTION OF NUMEROUS TESTS FACILITATING THE MAINTENANCE OF TESTS ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION ENSURING THE EXECUTION OF TESTS CONTROLLING THE EXECUTION OF TESTS PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING ACCURATE REPORTS ON THE TEST RESULTS ANALYZING THE TEST RESULTS ©Yphise Strengths and Weaknesses List
    • WinRunner ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION FACILITATING THE TEST DEFINITION AND USE PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS Software Ranking Best software evaluation Average evaluation Worst software evaluation ©Yphise Ranking Chart
    • WinRunner Opinion • One of the strengths of WinRunner is its ability to handle accurately the definition of control points. The definitions of the control points are grouped within a separate file. All the graphical element of the application to be tested are captured. The testers are allowed to select the elements to control. The values of the selected elements can be modified in order to fit the testers’ needs. They can even be changed into variables in order to reproduce the numerous test cases to use. • Another strength of WinRunner is the extensibility of its object and function libraries. The product provides a GUI_add function that allows the capture of the label of a new object as well as other information compared to the information available in the object library. The scripting function library can also be extended by either defining new functions or customizing the existing ones. These functions are organized within existing folders or new customized folders. A newly defined function can be set as default. • One of the weaknesses of WinRunner is its inability to organize the execution of numerous tests. This feature is provided by the vendor’s test plan and campaign management tool, Test Director. Unfortunately, the two products are not bundled in a common commercial package. We recommend including Test Manager when considering WinRunner for projects involving a great number of tests. Vendor presentation • Developer Mercury Interactive 1325 Borregas Avenue Sunnyvale, CA 94089, USA Tel.: +1(408) 822-5200 Fax: +1(408) 822-5300 • Vendor contacted Mercury Interactive 1, bis rue du Petit Clamart BP 277 78147 Vélizy-Villacoublay Cedex, France Tel.: +33 140 83 68 68 Fax: +33 140 83 68 50 • Web site www.merc-int.com www.mercuryinteractive.fr • Development laboratory Tel Aviv, Israel • Hotline Tel Aviv, Israel London, UK Software product presentation • Version number 7.01 ©Yphise
    • • Date of first sale January 1992 • Price indication 6,400 Euros to 8,100 Euros per seat • Maintenance 18% of purchase price • Training Engineer: 2 days Consultant: 4 days 500 Euros per day and per user • Setup services 1,150 Euros per day • User group MIUG (Mercury Interactive User Group) • Recent developments Support of ActiveX Support of MS .Net platform Support of ERP systems • Announced functions Web-based user interface New protocols • Main environments Support for Windows, terminal emulation, Java and Web environments Main modules • RapidTest Script Wizard. Helps create functional test scripts automatically. • Data-driven Wizard. An assistant that automates the replacement of data with variables within the test scripts. • Database Checkpoints. Verifies the data in databases during an automated nonregression test. ©Yphise
    • WinRunner Functional test automation tools TESTING THE APPLICATIONS IN VARIOUS ENVIRONMENTS TESTING THE SPECIFICITY OF APPLICATION INTERFACES TESTING THE RELATIONSHIP BETWEEN APPLICATIONS PROVIDING FUNCTIONAL RICHNESS FOR TESTING TESTING THE FUNCTIONAL RICHNESS OF APPLICATIONS PROVIDING RICH FUNCTIONS TO DEFINE COMPLEX TESTS FACILITATING THE TEST DEFINITION AND USE FACILITATING THE DEFINITION OF TESTS FACILITATING THE EXECUTION OF NUMEROUS TESTS FACILITATING THE MAINTENANCE OF TESTS ENSURING THE TECHNICAL ROBUSTNESS OF TEST EXECUTION ENSURING THE EXECUTION OF TESTS CONTROLLING THE EXECUTION OF TESTS PROVIDING A CLEAR VIEW OF THE TEST RESULTS PROVIDING ACCURATE REPORTS ON THE TEST RESULTS ANALYZING THE TEST RESULTS ©Yphise Strengths and Weaknesses List
    • ©Yphise
    • Yphise Software Assessment Reports About Yphise Large companies Yphise is committed to providing the most relevant and accurate advice to assist large companies in embracing information management change. We assist them in • identifying how to generate more business value through information management • developing the new skills and methods necessary to achieve this value • conducting innovative projects • making appropriate technical decisions IT consultants We advise IT consultants in selecting their partnerships and choosing the best software to recommend and use in each mission. Software vendors and We advise software vendors and investors in strategy development. investors Our clients appreciate : • our independence, • the accuracy and rigor of our methodologies, as we are ISO 9001-certified, • our in-depth understanding of IT management challenges, as we advise many large companies in defining IT strategy, organization and architectures, • our coverage of all areas and technologies required for developing and managing complex information systems, • the unique expertise we have developed since 1985. Research programs Yphise conducts research programs that focus on IT management challenges in large companies. Our research programs cover three aspects. ©Yphise
    • Monitoring technological development to create more value from information Establishing the best strategies and management practices Assessing the software products for developing the information systems Software product assessment Yphise assesses over 160 software products each year. IT departments have made thousands of software selection decisions based on Yphise expertise since 1985. Yphise research covers all areas of interest to large IT departments. The results are available at our Yphise.com SoftAssessment Portal. Various licenses provide you with the reports that you need. Each software product assessment is conducted according to our ISO 9001-certified methodology. To our knowledge, no other company in the world produces expert and independent comparative assessments of software products based on ISO 9001-certified methodology guaranteeing accuracy and independence. Our lab is in Paris (France) across from Pompidou Center. The numerous vendors who visit us appreciate this very convenient location. Components of the Yphise Software Assessment Reports REQUIREMENTS vol EXECUTIVE vol DETAILED SELECTION ASSESSMENT vol ASSISTANT List of Requirements Opportunities Assessing the products ©Yphise
    • REQUIREMENTS vol It lists the functional and technical requirements to meet the needs of large companies in the evaluated area. This list is set according to our ISO 9001-certified methodology in consultation with large companies, in order to be accurate, practical and proactive. Our experience is that many errors when selecting software products are due to improperly defined requirements. It is indeed very difficult to establish the correct list of criteria. We have unique expertise in this field, as we work daily with large companies, monitor new developments from vendors, and cover all areas of information management within large companies. The REQUIREMENTS volume provides you, at very low cost, with the results of our expert analysis. It sets the standard to base decisions on in the evaluated area. EXECUTIVE vol It is designed to monitor developments of the software product market. It assesses the maturity and opportunities of the products available. It highlights the list of software products that we consider of interest to large companies. Our experience has shown that products do not always satisfactorily provide the functions that we expect, based on vendor positioning. As a result, many improper selections of software products are made. It highlights the strengths and weaknesses of each software product, based on our detailed assessment conducted according to our ISO 9001-certified methodology. It provides a top-down ranking of the products and it outlines our opinion. The EXECUTIVE volume is ideal for an initial review of the market, for defining your strategy, planning your investments and selecting a short list. DETAILED ASSESSMENT vol It provides a detailed assessment of each software product, based on the criteria of the requirements. The assessment is conducted according to our ISO 9001-certified methodology. The results are provided in tables for quick access and accurate detailed comparison of the software products. The assessed software products are those that we consider of interest to large companies in the evaluated area. SELECTION ASSISTANT This is a Windows application that you can download. It contains all data from the corresponding DETAILED ASSESSMENT volume. It allows you to customize all the data and weights, in order to arrive at your own ranking of the software products according to your own opinion and specific needs. You can then print your personalized DETAILED ASSESSMENT volume. List of Requirements structure The List of Requirements has a three-level structure : main issues (chapters), first analysis level of each issue (subchapters) and assessment criteria. Each criterion is composed of items. ©Yphise
    • Chapter Subchapter Criterion • items • EXECUTIVE volume aggregations and SELECTION ASSISTANT The charts of the EXECUTIVE volume aggregate all the results established at item level in the corresponding DETAILED ASSESSMENT volume. Most of these charts aggregate results on several hundred items. The rankings and overall assessment obtained are robust and reliable. However, you may not necessarily agree with all the notes attributed by Yphise. You may also be interested in a product that provides fewer functions and is less expensive. Some criteria and items are also more important for you. The EXECUTIVE volume charts do not take your specific requirements into account. They are very useful for a first overview of the market, but you need more information to make a purchase decision. The SELECTION ASSISTANT allows you to customize Yphise notes and criteria weights according to your own specific needs. It then adapts the ranking, and you can print your own customized report. How to use Yphise Software Assessment Reports Analyzing opportunities The EXECUTIVE volume is designed to assess the opportunities of the market. It of the market highlights the list of software products that we consider of interest to large companies. Obtaining your list of The REQUIREMENTS volume enables you to work from a reference list that you customize requirements according to your own needs. Some criteria may be ignored, while others are essential. Selecting a short list The EXECUTIVE volume provides you with a brief overview of the market. Then use the SELECTION ASSISTANT. Identify the few criteria that, for you, are essential functions. The assistant then adapts the ranking of the products according to these criteria : you get valuable information to select the best short list. Selecting a software You must make a decision based on your environment, context and issues. Focus your product effort on the essential criteria for you. The DETAILED ASSESSMENT volume enables you to work from a reliable reference. Then use the SELECTION ASSISTANT. Input your own notes when they differ from Yphise, and use the editing functions to obtain a first overview of ©Yphise
    • the results. Then use the DETAILED ASSESSMENT volume to compare the products on the basis of criteria that are less important for you. Negotiating with the The EXECUTIVE and DETAILED ASSESSMENT volumes provide you with the strengths vendors and weaknesses of each product. You have valuable information to use when negotiating with the vendors. ©Yphise