Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Automated Software Testing using Open Source Testing Tools By Elfriede Dustin
  2. 2. Agenda <ul><li>Who is IDT </li></ul><ul><li>State of Software Testing </li></ul><ul><li>What is Automated Test and Re-test (ATRT) </li></ul><ul><li>Approach to ATRT </li></ul><ul><li>Advantage of open source </li></ul><ul><li>ATRT and open source – Our IDE and solution </li></ul><ul><li>ATRT Challenges </li></ul>
  3. 3. Who is IDT? <ul><li>IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions </li></ul><ul><ul><li>Deliver turn key automated test suite </li></ul></ul><ul><ul><ul><li>Automated test strategy and identification of highest payoff areas to apply automation for your project </li></ul></ul></ul><ul><ul><ul><li>Selection of best automation tools for your project </li></ul></ul></ul><ul><ul><ul><li>Using your existing test cases or develop test cases for you </li></ul></ul></ul><ul><ul><ul><li>Test results documented in requirements traceability matrix </li></ul></ul></ul><ul><ul><li>Training and pilot project implementation with companies </li></ul></ul>
  4. 4. Who is IDT? <ul><li>IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions </li></ul><ul><ul><li>Deliver turn key automated test suite </li></ul></ul><ul><ul><ul><li>Automated test strategy and identification of highest payoff areas to apply automation for your project </li></ul></ul></ul><ul><ul><ul><li>Selection of best automation tools for your project </li></ul></ul></ul><ul><ul><ul><li>Using your existing test cases or develop test cases for you </li></ul></ul></ul><ul><ul><ul><li>Test results documented in requirements traceability matrix </li></ul></ul></ul><ul><ul><li>Training and pilot project implementation with companies </li></ul></ul><ul><li>We are currently hiring Java Developers </li></ul><ul><li>Plus we’d like to hire 2 or 3 grad students to work for us part time ( i.e. 20 hours week ) </li></ul><ul><li>must be US citizen </li></ul>
  5. 5. State of Software Testing - Why ATRT? ?
  6. 6. State of Software Testing - Why ATRT? Utilize Automated Testing Strategies and Technology to Improve Productivity and Quality Manual Automated Reduction in Test Days IDT Provides Automated Software Testing Solutions Increased Test Coverage Manual vs. Automated Spiral / Release 1 Spiral / Release 2 Spiral / Release 3 Spiral / Release 4 Size & Complexity of SW Baseline Test Days Test Days Test Days Test Days 50% or More of Overall Development Cost is Typically Spent on Testing
  7. 7. What is ATRT? <ul><li>Our definition of ATRT is: </li></ul><ul><li>Application and implementation of software technology throughout the entire Software Testing Life Cycle (STL) and QA lifecycle; with the goal to improve STL efficiencies and effectiveness </li></ul>
  8. 8. ATRT Spans the Software Development Life Cycle – Visual Modeling Use cases Design Build Assemble Round-Trip Engineering Test <ul><li>Modeling and test case generation </li></ul>Development Tools Components
  9. 9. ATRT Spans the Software Development Life Cycle – Requirements <ul><ul><li>Organizes, tracks, & controls requirements </li></ul></ul><ul><ul><li>Requirements Traceability Matrix (RTM) </li></ul></ul>Development Tools Components Visual Modeling Requirements Management and Process Automation
  10. 10. ATRT Spans the Software Development Life Cycle – Automated Test Tools Automated Software Testing Visual Modeling Execution Management Development Automated Testing Tools Development Tools Components Requirements Management and Process Automation <ul><ul><li>Automates test cases using vendor-provided, </li></ul></ul><ul><ul><li>open-source tools or in-house development </li></ul></ul>
  11. 11. ATRT Spans the Software Development Life Cycle – Integrated Suite of Tools <ul><li>Additionally: </li></ul><ul><li>Middleware </li></ul><ul><li>Infrastructure </li></ul><ul><li>Defect Tracking </li></ul><ul><li>Configuration Management </li></ul><ul><li>Memory Leak Detectors </li></ul><ul><li>Performance Testing Tools </li></ul><ul><li>Documentation Tools </li></ul><ul><li>others </li></ul>Visual Modeling Requirements Management and Process Automation Infrastructure Development Tools Components Automated Testing Tools Software Configuration Management Defect Tracking Middleware
  12. 12. Types of Software Test Life-cycle Support Tools <ul><li>Which tools are you familiar with? </li></ul>
  13. 13. Types of Automated Test Tools Create use cases Use Case Generators Verify syntax, semantics, and testability Requirements Verifiers Manages and organizes requirements; allows for test procedure design and test progress reporting Requirements Management Requirements Definition Phase Automate document generation Documentation Generators Facilitates communication, while automating the technical review/inspection process Technical Review Management Manages system life-cycle defects Defect Tracking Baselines important data repositories Configuration Management Records definitions of user needs and automates rapid construction of flexible, graphical, client-server applications Business Modeling Business Analysis Phase Tool Description Type of Tool Life-Cycle Phase
  14. 14. Types of Tools (Cont’d) Depict quality and structure of code Static and Dynamic Analyzers Verifies maintainability, portability, complexity, and standards compliance Source Code Testing Detects runtime errors and memory leaks Memory Leak and Runtime Error Detection Programming Phase Perform syntax checking and have debugging capability; usually available with built-in programming language compiler Syntax Checkers/ Debuggers Generate test procedures from requirements, design, or data and object models Test Procedure Generators Manage processes Structure Charts, Flowcharts, and Sequence Diagrams Develops second generation enterprise client-server systems Database Design Analysis and Design Phase Tool Description Type of Tool Life-Cycle Phase Provide usability testing as conducted in usability labs Usability Measurements Identify untested code and support dynamic testing Code (Test) Coverage Analyzers or Code Instrumentors Metrics Tools
  15. 15. Types of Tools (Cont’d) Extract Data from various formats into various formats Data Extraction Tool Performs security testing and vulnerability scanning at the application or network level; plus debuggers will allow to check for security coding errors (source code checkers) Security Testing Conducts load/performance and stress testing Load/Performance Testing Conducts automated GUI tests; capture/playback tools record user interactions with online systems so they may be replayed automatically GUI Testing (Capture/Playback) Monitors, measures, tests, and diagnoses performance across the entire network Network Testing Tests management Test Management Simulates application to measure for scalability, among other tasks Simulation Find discrepancies between files that should be identical in content File Compare Utilities Generate test data Test Data Generators Other Testing Life-Cycle Support Tools Tool Description Type of Tool Life-Cycle Phase
  16. 16. Deployment Approach – Adapted ATLM <ul><li>Collect/Create: </li></ul><ul><li>Requirements </li></ul><ul><li>Test Cases </li></ul><ul><li>Test Procedures </li></ul><ul><li>Expected Results </li></ul><ul><li>Interface Specifications </li></ul><ul><li>System/Component </li></ul><ul><li>Configuration Description </li></ul><ul><li>Identify or Develop </li></ul><ul><li>& Pilot </li></ul><ul><li>Test Manager </li></ul><ul><li>Test Scripts </li></ul><ul><li>Data Comparators </li></ul><ul><li>Performance Test Tools </li></ul><ul><li>GUI Record/Playback </li></ul><ul><li>Storing/Retrieving Results </li></ul><ul><li>Populate Test Manager with Test Cases and Requirements </li></ul><ul><li>Develop Test Scripts from Test Procedures </li></ul><ul><li>Test Results: </li></ul><ul><li>Pass/Fail by Test Case </li></ul><ul><li>Pass/Fail Summary </li></ul><ul><li>Test Time Required </li></ul>
  17. 17. Why Open Source <ul><li>? </li></ul>
  18. 18. Why Open Source? <ul><li>Advantage of Open Source: </li></ul><ul><ul><li>no licensing issues; i.e. license cost; maintenance; etc. </li></ul></ul><ul><ul><li>easily modifiable </li></ul></ul><ul><ul><li>adaptable </li></ul></ul><ul><ul><li>lightweight </li></ul></ul><ul><ul><li>flexibility - not tied to one vendor </li></ul></ul>
  19. 19. ATRTs IDE - Eclipse <ul><li>Eclipse – this site lists all the plugins </li></ul><ul><li>http://www.eclipse-plugins.info/eclipse/plugins.jsp?category=SCM&pager.offset=0&firstItem=1 </li></ul><ul><li>Eclipse/TPTP – Open Source - Developed by IBM </li></ul><ul><ul><li>Eclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions: </li></ul></ul><ul><ul><ul><li>Requirements Management http://sourceforge.net/projects/osrmt/ </li></ul></ul></ul><ul><ul><ul><li>Configuration Management - Subversion </li></ul></ul></ul><ul><ul><ul><li>Testing Tools (http://www.nabble.com/Eclipse-TPTP---Testing-Tools-f2262.html) </li></ul></ul></ul><ul><ul><ul><li>Profiling: default Java applications, but can be extended to other apps </li></ul></ul></ul>
  20. 20. ATRTs IDE - Eclipse <ul><ul><li>Eclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions (cont): </li></ul></ul><ul><ul><ul><li>Static Analysis: default for C++ and Java, but can be extended to other apps </li></ul></ul></ul><ul><ul><ul><li>Application monitoring and log analysis </li></ul></ul></ul><ul><ul><ul><li>Build tool – ant </li></ul></ul></ul><ul><ul><ul><li>Xml input and output - using an open source Java package http:// jakarta.apache.org/ecs / , which is a Java library that has an API for constructing and outputting XML.  </li></ul></ul></ul><ul><ul><ul><li>Test Manager – STAF/STAX </li></ul></ul></ul>
  21. 21. Current Test Tool Implementation – Example and Challenges <ul><li>Vendor-provided (Capture/Playback) Tool: </li></ul><ul><li>Automated test tools mimic actions of the test engineer. </li></ul><ul><li>During testing, the engineer uses the keyboard and mouse to perform some type of test or action. </li></ul><ul><li>Testing tool captures all keystrokes and subsequent results, which are baselined in an automated test script. </li></ul><ul><li>During test playback, scripts compare latest outputs with previous baseline. </li></ul><ul><li>Testing tools have built-in, reusable test functions. </li></ul><ul><li>Most test tools provide for non-intrusive testing; i.e., they interact with the “application-under-test” as if the test tool was not involved. </li></ul>
  22. 22. Current Testing Tools – Challenges (cont) <ul><li>In-House Software Development required: </li></ul><ul><li>Capture/Playback tools generate hard-coded values; test scripts are not reusable, nor do they implement software development best practices right out of the box; scripts need to be modified. </li></ul><ul><li>Capture/Playback tools don’t necessarily provide all testing features required; code enhancements are often required to meet testing needs. </li></ul><ul><li>Capture/Playback tools are not necessarily compatible with system engineering environment, and software testing scripts need to be developed in-house. </li></ul><ul><li>Developed testing scripts are not cross-vendor compatible, even if the same scripting language is used </li></ul>
  23. 23. Linux Capture/Playback Tools IDT Findings Replay Xcessory http:// www.scl.com/products/ics/motif/testing/replay / www.ics.com QF-Test http://www.qfs.de/en/qftestJUI/index.html Froglogic http://www.froglogic.com/pg?id=Google&category=squishjava Redstone Software Eggplant KD Executor http://www.kdab.net/?page=products&sub=kdexecutor Jameleon http://sourceforge.net/projects/jameleon/ ShUnit2 for shell scripts testing GNU Linux Desktop Project http:// ldtp.freedesktop.org /wiki/Home Cantana http://www.ipl.com/pdf/p0003.uk.pdf crontab Part of Unix   crontab, cruisecontrol, www.adminschoice.com/docs/crontab.htm Dogtail http:// people.redhat.com/zcerza/dogtail/about.html http://www.cyberciti.biz/tips/linux-automated-gui-testing-with-python-based-dogtail-tool.html Expect http://expect.nist.gov/
  24. 24. Automated Test and Re-Test (ATRT) <ul><li>ATRT includes a common framework comprised of re-usable software services </li></ul><ul><ul><li>Open Source Software Test Automation Framework (STAF) </li></ul></ul><ul><li>Provides automated test procedures, test inputs, and test results for SW components </li></ul><ul><ul><li>Suitable for use by 3 rd parties not associated with development of the component </li></ul></ul>
  25. 25. What is STAX? <ul><li>STAX is an automation system </li></ul><ul><li>With STAX you can automate and monitor your entire test environment </li></ul><ul><ul><li>System/product setup </li></ul></ul><ul><ul><li>Testcase distribution </li></ul></ul><ul><ul><li>Testcase execution </li></ul></ul><ul><ul><li>Testcase results collection and analysis </li></ul></ul><ul><li>STAX consists of </li></ul><ul><ul><li>A programming language designed for automation </li></ul></ul><ul><ul><ul><li>Programs written in this language are called jobs </li></ul></ul></ul><ul><ul><li>An execution engine (similar to an interpreter for other programming/scripting languages) which manages the execution and runtime behavior of the job </li></ul></ul><ul><ul><li>A GUI application, called the STAX Monitor, which provides a dynamically updated view of your executing jobs </li></ul></ul><ul><ul><li>A tool (STAXDoc) that is used to generate documentation for your STAX jobs </li></ul></ul>
  26. 26. STAF can run on:     * Windows 95     * Windows 98 (and 98 SE)     * Windows Millenium Edition     * Windows NT Version 4.0     * Windows 2000     * Windows XP     * Windows Server 2003 (IA32, IA64, AMD64)     * Windows Vista (IA32, AMD64)     * Linux (Intel32/64, AMD64, PPC32/64)     * Linux on zSeries (31-bit, 64-bit)     * AIX Version or higher (32-bit, 64-bit)     * Solaris (Sparc) 2.6 and higher     * Solaris AMD Opteron 64-bit     * HP-UX 11.00 and higher (PA-RISC, IA64 32-bit and 64-bit)     * OS/400 V5R2 or higher     * z/OS UNIX V1.4 and higher
  27. 27. STAF Internal Services <ul><li>DIAG Provides diagnostics services Internal (&quot;DIAG&quot;) </li></ul><ul><li>DELAY Provides a means to sleep a specified amount of time Internal (&quot;DELAY&quot;) </li></ul><ul><li>ECHO Echos back a supplied message Internal (&quot;ECHO&quot;) </li></ul><ul><li>FILE SYSTEM Allows you to get and copy files across the network Internal (&quot;FS&quot;) </li></ul><ul><li>HANDLE Provides information about existing STAF handles Internal (&quot;HANDLE&quot;) </li></ul><ul><li>HELP Provides Help on STAF error codes Internal (&quot;HELP&quot;) </li></ul><ul><li>MISC Handles miscellaneous commands such as displaying the version of STAF </li></ul><ul><li>that is currently running Internal (&quot;MISC&quot;) </li></ul><ul><li>PING Provides a simple is-alive message Internal (&quot;PING&quot;) </li></ul><ul><li>PROCESS Allows you to start, stop, and query processes Internal (&quot;PROCESS&quot;) </li></ul><ul><li>QUEUE Provides a network-enabled IPC mechanism for STAF Programs Internal (&quot;QUEUE&quot;) </li></ul><ul><li>SEMAPHORE Provides network-enabled named event and mutex semaphores Internal (&quot;SEM&quot;) </li></ul><ul><li>SERVICE Allows you to list services available on a machine and to examine the </li></ul><ul><li>Requests that have been submitted on a machine Internal (&quot;SERVICE&quot;) </li></ul><ul><li>SHUTDOWN Provides a means to shutdown STAF and register for shutdown </li></ul><ul><li>Notifications Internal (&quot;SHUTDOWN&quot;) </li></ul><ul><li>TRACE Provides tracing information for STAF services Internal (&quot;TRACE&quot;) </li></ul><ul><li>TRUST Interfaces with STAF's security Internal (&quot;TRUST&quot;) </li></ul><ul><li>VARIABLE Provides a method for maintaining configuration and runtime data </li></ul><ul><li>(variables) Internal (&quot;VAR&quot;)             </li></ul>
  28. 28. STAF External Services <ul><li>The executable code for external STAF services resides outside of STAFProc, for example in a Java jar file, a C++ DLL file, or a Rexx script file. </li></ul><ul><li>CRON Calls into STAF services at a specified time interval External (Java) </li></ul><ul><li>EMAIL Allows you to send email messages External (Java) </li></ul><ul><li>EVENT Provides a publish/subscribe notification system External (Java) </li></ul><ul><li>EVENTMANAGER Allows you to call STAF services when a specified Event occurs External (Java) </li></ul><ul><li>HTTP Allows you to make HTTP requests which can be grouped together in a session External (Java) </li></ul><ul><li>LOG Provides a full-featured logging facility External (C++) </li></ul><ul><li>MONITOR Allows a testcase to publish its current running execution status for others to read External (C++) </li></ul><ul><li>RESOURCE POOL Allows you to manage exclusive access to pools of elements, e.g. VM UserIDs or Software Licenses External (C++) </li></ul><ul><li>STAX Provides an XML-based execution engine External (Java) </li></ul><ul><li>ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives External (C++) </li></ul>
  29. 29. Technologies used in STAX <ul><li>STAF </li></ul><ul><ul><li>STAF provides the infrastructure on which STAX builds. The full power of STAF and its services is exposed for use within STAX jobs. </li></ul></ul><ul><li>XML </li></ul><ul><ul><li>The STAX programming language is based on XML. This provides built-in structure to your jobs, as well as providing a set of existing tools for use in constructing your jobs, such as XML (aware) editors and XSLT. </li></ul></ul><ul><li>Python </li></ul><ul><ul><li>The STAX programming language builds on Python to provide a rich and accessible data model </li></ul></ul><ul><ul><li>STAX's Python integration also allows you to access the wealth of existing Python libraries </li></ul></ul><ul><li>Java </li></ul><ul><ul><li>The STAX programming language allows you to access existing Java classes/libraries, providing another source for reuse </li></ul></ul>
  30. 30. Automation Tasks Execution Synchronization Monitoring Resource Management Automation Completion Testcase Output Analysis Results Notification Testcase Cleanup Testcase Execution System Setup Automation Startup
  31. 31. End-to-End Automation with STAF and STAX Event, EventManager, Cron FS, Process Process, Monitor, Log, Variable, Queue, ResPool Log, Process Process Email, HTTP S T A X J o b <function> <parallel> <sequence> <process> <stafcmd> <testcase> <timer> <block> <loop> <message> <log> <import> <job> <iterate> <paralleliterate> <hold> <release> <terminate> Automation Completion Results Notification Testcase Cleanup Testcase Output Analysis Testcase Execution System Setup Automation Startup
  32. 32. Key ATRT Architecture Requirements <ul><li>Support applications running on multiple computers </li></ul><ul><li>Support applications developed in different languages </li></ul><ul><li>Support applications running on different types of OS’s </li></ul><ul><li>Support applications which have GUI and those which do not (for example Interface testing) </li></ul><ul><li>Support applications which use different types of network protocols such as TCP/IP, DDS, etc </li></ul><ul><li>Support integration of multiple commercial testing tools from different vendors ( allowing as new or better products emerge in the market they can be utilized ) </li></ul><ul><li>Support testing w/o having to install ATRT on the same computers as the application under test and be able to be distributed across computers </li></ul>
  33. 33. Display Application Display Application Analysis & Display Application Simulation of Application Processing Messages Simulation of Application Processing Messages System Under Test Test Manager RTM Bugzilla ATRT Eggplant R/P VNC Robot R/P STAF/STAX Prototype – Proof of Concept Windows Based Linux Based
  34. 34. <ul><li>Able to successfully execute ATRT for 48 hours without operator intervention </li></ul><ul><li>During the tests: </li></ul><ul><ul><li>VNCRobot - Approximately 25,000 operator key strokes were replayed </li></ul></ul><ul><ul><li>STAF/STAX - More than 1,000,000 messages processed and verified by ATRT </li></ul></ul>ATRT Prototyping Effort
  35. 35. Additional Requirement <ul><li>Pristine test environment required – automated testing tool cannot be installed on Application-Under-Test PC </li></ul><ul><ul><li>Eggplant – allows for remote capture/playback via VNCserver and KVM switch </li></ul></ul><ul><ul><li>VNCRobot </li></ul></ul>
  36. 36. Automated Software Testing <ul><li>Software testing paradox </li></ul><ul><li>What is it? </li></ul>
  37. 37. Effective Automated Software Testing <ul><li>Current State: </li></ul><ul><ul><ul><li>Developing software in order to test software </li></ul></ul></ul><ul><ul><ul><li>Automate the test support development: automated IDL to code generation; test data generation (using FireEye from NIST); compare utilities; log file analyzers; </li></ul></ul></ul><ul><li>Our Automated Software Testing Goal: </li></ul><ul><ul><ul><li>Minimize Human interaction </li></ul></ul></ul><ul><ul><ul><ul><li>Possibly get on par with Hardware Automated Testing? </li></ul></ul></ul></ul>
  38. 38. Hardware Testing: current state <ul><li>Many automated test generators for manufacturing/ logic devices, circuit testing, etc., available i.e. SEI/ATG, uMaster, etc. </li></ul><ul><li>Automated testing without human interaction is the norm for hardware testing </li></ul>
  39. 39. Hardware vs. Software AT <ul><li>How is hardware testing different? </li></ul><ul><li>Software Challenge: </li></ul><ul><ul><li>Changing requirements </li></ul></ul><ul><ul><li>GUI interfaces </li></ul></ul><ul><ul><li>Code access/availability </li></ul></ul><ul><ul><li>Defects uncovered earlier in the software development lifecycle are cheaper to fix </li></ul></ul><ul><li>Solution flexibility required – problem dependent </li></ul>
  40. 40. Proposed solutions <ul><li>Focus on Reusable Components </li></ul><ul><ul><li>Test Data / Expected Results </li></ul></ul><ul><ul><li>Interfaces </li></ul></ul><ul><ul><li>Middleware </li></ul></ul><ul><li>Assuming code and models are available: </li></ul><ul><li>Model based testing; automated test case generators based on models </li></ul><ul><li>Eclipse – effective ATRT IDE </li></ul><ul><li>Probes – see Aprobe – Code instrumentation </li></ul><ul><li>Building self-testable components </li></ul><ul><li>Using standard language MOF to Text to generate automate test cases </li></ul><ul><li>GUI Testing tool ideas </li></ul><ul><li>New tools - other </li></ul><ul><li>Working with NIST to produce additional tools </li></ul>
  41. 41. GUI Testing <ul><li>Jemmy - java library to test UI </li></ul><ul><li>http://jemmy.netbeans.org/ </li></ul><ul><li>http://abbot.sourceforge.net/doc/overview.shtml </li></ul><ul><li>Perl modules </li></ul><ul><li>run via VNCServer? </li></ul>
  42. 42. Questions <ul><li>? </li></ul><ul><li>Email [email_address] with any follow-up questions </li></ul>
  43. 43. IDT <ul><li>We are hiring……. </li></ul><ul><li>Please come see me now if interested or send </li></ul><ul><li>email [email_address] with any follow-up </li></ul>