Eclipse Test and Performance Tools Platform (TPTP)


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Eclipse Test and Performance Tools Platform (TPTP)

  1. 1. Eclipse TPTP <ul><li>TPTP = “Test and Performance Tools Platform”, </li></ul><ul><ul><li>Part of the open-source Eclipse project </li></ul></ul><ul><ul><ul><li>Project created in August 2004 </li></ul></ul></ul><ul><ul><ul><li>Successor of the Eclipse “Hyades” tools sub-project (2002) </li></ul></ul></ul><ul><ul><li>Web site: http:// </li></ul></ul><ul><li>Four major components / sub-projects: </li></ul><ul><ul><li>Platform </li></ul></ul><ul><ul><li>Monitoring tools </li></ul></ul><ul><ul><li>Testing tools </li></ul></ul><ul><ul><li>Tracing and Profiling tools </li></ul></ul><ul><li>Most recent release: TPTP (Sept. 2007) </li></ul><ul><ul><li>Java 6 is not yet fully supported   </li></ul></ul>
  2. 2. TPTP Objectives <ul><li>Aims to bring software test and performance tools into the Eclipse environment in a consistent way that maximizes integration with tools used in the other processes of the software lifecycle </li></ul><ul><li>Reduce the cost and complexity of implementing effective automated software quality control processes </li></ul><ul><li>Share data through an OMG-defined test profile model, Common Base Event (CBE) Model and trace model implemented via the Eclipse Modeling Framework (EMF) </li></ul>
  3. 3. TPTP Project Structure <ul><li>TPTP Platform Project </li></ul><ul><ul><li>Covers the common infrastructure in the areas of user interface, EMF based data models, data collection and communications control, remote execution environments and extension points </li></ul></ul><ul><li>TPTP Monitoring Tools Project </li></ul><ul><ul><li>Collects, analyzes, aggregates and visualizes data that can be captured in the log and statistical models. </li></ul></ul><ul><li>TPTP Testing Tools Project </li></ul><ul><ul><li>Provides specializations of the platform for testing and extensible tools for specific testing environments, initially 3 test environments: JUnit, manual and URL testing. </li></ul></ul><ul><li>TPTP Tracing and Profiling Tools Project </li></ul><ul><ul><li>Extends the platform with specific data collection for Java and distributed applications that populate the common trace mode, also viewers and analysis services </li></ul></ul>
  4. 4. TPTP Architecture Presentation System Statistical Trace EMF Data Models Log Test Test Platform TPTP Workbench Standard Widgets Reference Perspectives Target System Data Collection Application Data Collection Interface Execution Environment Log Collection Trace Collection Performance Monitor JVMPI Data Loader Distributed Data Collection Framework Controller Interface Controller Interface Distributed Control Framework Agent Controller Eclipse Platform JVMTI Test Execution Monitor Trace
  5. 5. TPTP Architectural Components (1) <ul><li>On the controller system: </li></ul><ul><ul><li>User interface: the graphical view from which test execution is directed. </li></ul></ul><ul><ul><li>Data models: based on the Eclipse Modeling Framework (EMF) </li></ul></ul><ul><ul><ul><li>Includes models for: </li></ul></ul></ul><ul><ul><ul><ul><li>test execution data </li></ul></ul></ul></ul><ul><ul><ul><ul><li>logs </li></ul></ul></ul></ul><ul><ul><ul><ul><li>traces </li></ul></ul></ul></ul><ul><ul><ul><ul><li>statistics </li></ul></ul></ul></ul><ul><ul><li>Eclipse platform: used for windows, editors, etc. </li></ul></ul><ul><ul><li>Interfaces to the agent controller, and data collection engine. </li></ul></ul>
  6. 6. TPTP Architectural Components (2) <ul><li>On the target system: </li></ul><ul><ul><li>the application to be tested </li></ul></ul><ul><ul><li>the application’s execution environment: </li></ul></ul><ul><ul><ul><li>includes the Java virtual machine and additional monitoring via virtual machine tool interfaces </li></ul></ul></ul><ul><ul><li>the test engine, to start the application and its environment </li></ul></ul><ul><ul><li>data collection </li></ul></ul>
  7. 7. Models implemented by TPTP (1) <ul><li>Includes a reference implementation of the UML2 test profile (MOF). </li></ul><ul><li>Test profile: definition model for the creation of: </li></ul><ul><ul><li>test suites </li></ul></ul><ul><ul><li>test cases </li></ul></ul><ul><ul><li>data pools </li></ul></ul><ul><ul><li>invocation </li></ul></ul><ul><ul><li>loops </li></ul></ul><ul><ul><li>synchronization </li></ul></ul>
  8. 8. Models implemented by TPTP (2) <ul><li>Definition model for the definition, creation, management and persisting of test executions over time including: </li></ul><ul><ul><li>Tests to be executed. </li></ul></ul><ul><ul><li>Deployments. </li></ul></ul><ul><ul><li>Locations. </li></ul></ul><ul><ul><li>Verdicts. </li></ul></ul><ul><ul><li>Messages and console output from the test execution. </li></ul></ul><ul><li>A collection of test execution traces and results : the test log . </li></ul>
  9. 9. Agent Controller <ul><li>A daemon process that resides on each deployment host and enables client applications to launch host processes and interact with agents that coexist within host processes </li></ul><ul><li>Contains a server that has the capability to launch and manage local or remote applications from a local TPTP workbench </li></ul><ul><li>The Java profiling engine included uses the Java Virtual Machine Tool Interface to profile local or remote Java applications from a local TPTP workbench </li></ul><ul><li>Launch and profile local or remote Java applications and import local or remote logs </li></ul><ul><li>Platform support : Windows, Linux, Solaris, HP-UX, AIX, Z/OS, OS/400 </li></ul>
  10. 10. Agent Controller Eclipse Workbench Agent Controller Java Profiling Agent Host process
  11. 11. Agent Controller Architecture Image © 2006 IBM, made available under the EPL
  12. 12. Agent Controller Terminology <ul><li>Host process </li></ul><ul><ul><li>The process that contains the application under test </li></ul></ul><ul><li>Agent </li></ul><ul><ul><li>A reusable binary file that provides services to the host process, and more importantly, provides a portal by which application data can be forwarded to attached clients </li></ul></ul><ul><li>Client </li></ul><ul><ul><li>A local or remote application (e.g. Eclipse Workbench) that is the terminal destination of host process data that is externalized by an agent </li></ul></ul><ul><li>The Agent Controller </li></ul><ul><ul><li>A daemon process that resides on each deployment host and provides the mechanism by which client applications can either launch new host processes, or attach to agents that coexist within existing host processes. The client can reside on the same host as the Agent Controller, or it can be remote. The Agent Controller can only interact with processes on the same node, and it is required on the same machine the targeted JVM is </li></ul></ul>
  13. 13. Virtual Machine (VM) Interface <ul><li>JVMTI , the Java Virtual Machine Tool Interface </li></ul><ul><ul><li> </li></ul></ul><ul><li>Provides interface for tools that monitor or control the virtual machine: debuggers, profilers, thread monitors, coverage analysis </li></ul><ul><li>Capabilities: observation / control of </li></ul><ul><ul><li>memory </li></ul></ul><ul><ul><li>threads </li></ul></ul><ul><ul><li>stack frame </li></ul></ul><ul><ul><li>memory heap </li></ul></ul><ul><ul><li>local variables </li></ul></ul><ul><ul><li>break points </li></ul></ul><ul><ul><li>classes and object instances; methods </li></ul></ul><ul><ul><li>timers </li></ul></ul>
  14. 14. Versions of VM interfaces and the Agent Controller <ul><li>In versions of Java up to 1.4.2, there were two interfaces called JVM P I and JVM D I, where P = “profiling” and D = “debugging” </li></ul><ul><ul><li>These were deprecated in Java 5 and removed in Java 6. </li></ul></ul><ul><li>Both JVMPI and JVMDI have been replaced by JVM T I. </li></ul>
  15. 15. Testing with TPTP <ul><li>JUnit tests can be run using TPTP, as well as the regular JUnit test runner. </li></ul><ul><li>JUnit tests must be imported to TPTP. </li></ul><ul><ul><li>The .java file remains unchanged </li></ul></ul><ul><ul><li>An additional .testsuite file is created. This is the file to be run by TPTP. </li></ul></ul><ul><li>Caveat : Only test case classes that inherit from junit.framework.Testcase will be detected by the import function. At present, only JUnit 3 tests are handled properly by TPTP. </li></ul>
  16. 16. TPTP test import
  17. 17. Opening the .testsuite file Methods in .java file.
  18. 18. TPTP Testing with JUnit <ul><li>This type of JUnit test generates an execution history from which a report can be generated. </li></ul><ul><li>Creating JUnit tests You can create a TPTP JUnit test in one of two ways: manually or from an existing JUnit test case. </li></ul><ul><li>Editing JUnit tests The JUnit Test Suite Editor can create and remove methods on a JUnit test, and control how those methods are invoked. </li></ul><ul><li>Choose an option to run a test: </li></ul><ul><ul><li>Running a test There are a number of options available to run a test. </li></ul></ul><ul><ul><li>Running a JUnit test remotely If you want to run a TPTP JUnit Test remotely, you must create and configure a test deployment. At a basic level, test deployments are entities which relate other TPTP entities, artifacts and locations in pairs. </li></ul></ul><ul><li>Analyzing test results When you run a script, the test results for the events are saved in a test log. You can use the Test Log viewer to view the results and events in the test log. </li></ul><ul><li>Generating reports on JUnit tests and JUnit plug-in tests After you have run your tests and generated execution histories, you can generate a report on those test results to summarize the data. </li></ul>
  19. 19. Test Log Overview (1) <ul><li>Test run details: </li></ul><ul><li>Start, end times of run </li></ul><ul><li>Which file was run </li></ul><ul><li>Host on which run occurred </li></ul>
  20. 20. Test Log Overview 2 <ul><li>Results report summary: </li></ul><ul><li>Pie chart </li></ul><ul><li>Select verdict type </li></ul>
  21. 21. Test Log Event Detail Details for this verdict
  22. 22. Test Report Generator <ul><li>After you have run your tests and generated execution histories, you can generate a report on those test results to summarize the data. </li></ul><ul><li>To generate the report: </li></ul><ul><ul><li>Right-click on a TPTP JUnit test in the Test Navigator and select Report... from the pop-up menu. </li></ul></ul><ul><ul><li>The New Report dialog launches. </li></ul></ul><ul><ul><li>Select the Time Frame Historic option then click on Next . </li></ul></ul><ul><ul><li>Provide the parent folder and a name for the report, then click on Next and provide the report time frame Start and End dates. </li></ul></ul><ul><ul><li>Click on Finish to generate the report. </li></ul></ul>
  23. 23. Time Frame Historic Report – Overview
  24. 24. Time Frame Historic Report - Test Results (1)
  25. 25. Time Frame Historic Report – Test Results (2)
  26. 26. Profiling Tool <ul><li>Broadly useful for performance analysis and for gaining a deeper understanding of a Java program </li></ul><ul><li>Consists of the Profiling and Logging Perspective and a number of graphical and tabular views </li></ul><ul><li>Enables you to profile and interact with your applications, to work with profiling resources, and to examine your applications for performance and memory usage problems </li></ul><ul><li>Help you to visualize and understand your program execution, pinpoint the operations that take the most resources, as well as to explore patterns of program behavior </li></ul><ul><li>Enables you to test your application's performance early in the programming development cycle for improvements </li></ul>
  27. 27. Profiling <ul><li>The process of generating a statistical analysis of a program that shows processor time and the percentage of program execution time used by each procedure in the program. </li></ul><ul><li>Analyze the application execution and look for </li></ul><ul><ul><li>Execution bottlenecks </li></ul></ul><ul><ul><li>Memory consumption </li></ul></ul><ul><ul><li>Memory leaks </li></ul></ul><ul><ul><li>High level view of your application execution </li></ul></ul>
  28. 28. TPTP Java Profiling Tool <ul><li>A library that attaches to a JVM to capture and record the Java application's behavior </li></ul><ul><li>Is a type of agent managed by the Agent Controller </li></ul><ul><li>Runs in the JVM (Java Virtual Machine) process and receives notifications of JVM events, based on the JVMTI (Java Virtual Machine Tool Interface). This agent is best used to identify performance details such as classes or methods responsible for the poor execution performance, also be used to analyze application heap and find memory leaks </li></ul><ul><li>Output from the profiling agent is in the form of XML fragments </li></ul><ul><li>Can be launched from the TPTP workbench, Applications can be in workbench's workspace or binaries that are on the file system </li></ul><ul><li>Can also be invoked using the -Xrun JVM option in command line </li></ul>
  29. 29. Profiling and Logging Perspective <ul><li>The profiling tools available in the Profiling and Logging perspective provide comprehensive information about the performance of an application </li></ul><ul><li>The profiling tool (in this scenario i.e. – Java Profiling) provides information pertaining to </li></ul><ul><ul><li>JVM performance </li></ul></ul><ul><ul><li>Object allocations and references </li></ul></ul><ul><ul><li>Garbage collection </li></ul></ul><ul><ul><li>Object methods performance </li></ul></ul><ul><ul><li>Object->Object interactions </li></ul></ul><ul><ul><li>Thread interactions </li></ul></ul><ul><ul><li>to name a few </li></ul></ul>
  30. 30. Profiling and Logging Perspective Java process associated with a host Time of profile run Monitors
  31. 31. Profiling and Logging Perspective <ul><li>Profiling and Logging perspective provides resources to administering and managing profiling </li></ul><ul><li>Profiling resources </li></ul><ul><ul><li>Project: Make a project of your profiling effort </li></ul></ul><ul><ul><li>Monitor: Aggregate different processes and agents </li></ul></ul><ul><ul><li>Host: The host you are profiling </li></ul></ul><ul><ul><li>Process: The executing program </li></ul></ul><ul><ul><li>Agent : Provides services to a process, a mechanism by which process data can be sent to (attached) clients </li></ul></ul><ul><ul><li>Profiling Type: Group profile data collection </li></ul></ul>
  32. 32. Profiling Monitor View <ul><li>Project: </li></ul><ul><ul><li>Container of the profiling resources </li></ul></ul><ul><li>Monitor: </li></ul><ul><ul><li>A logical container for the profiling information that is collected from a group of agents. The views at the monitor level show data from these agents. Monitors are useful for aggregating processes and agents from a distributed application </li></ul></ul><ul><li>Host: </li></ul><ul><ul><li>Owns the processes that are profiled. A host runs processes. You can specify a host either by its name or by its IP address </li></ul></ul><ul><li>Process & Agent: </li></ul><ul><ul><li>In the Agent Controller architecture model, an agent is a binary file that provides services to the host process by which application data can be transferred to attached clients </li></ul></ul>
  33. 33. Profiling Views <ul><li>Number of views to visualize and organize profiling data </li></ul><ul><ul><li>Memory Statistic view </li></ul></ul><ul><ul><li>Execution Statistic view </li></ul></ul><ul><ul><li>Coverage Statistic view </li></ul></ul><ul><ul><li>Object Reference view </li></ul></ul><ul><ul><li>Execution Flow view and table </li></ul></ul><ul><ul><li>Method Invocation view and table </li></ul></ul><ul><ul><li>UML2 Trace Interactions view </li></ul></ul>
  34. 34. Execution Statistics View