Choosing the Right Testing Tools and Systems Under Test (SUTs) for Practical Exercises in Testing Education

1. Choosing the Right Testing Tools and Systems Under Test (SUTs) for Practical Exercises in Testing Education Vahid Garousi, PhD, PEng Software Quality Engineering Research Group (SoftQual) Schulich School of Engineering, University of Calgary Alberta, CANADA

2. Contents of this talk • A brief analysis on the choices of testing tools and SUTs used in seven selected software testing courses delivered in North American universities. • Observations are made w.r.t. teaching and students learning based on – the choices of size and complexity of SUTs and also – The complexity and strength of testing tools used by students. • Experiences from using realistic exercises and

3. Background • We need to use realistic and relevant Systems Under Test (SUT), and • realistic / commercial testing tools. • Otherwise, students will not be ready to test large-scale industrial software systems after they graduate. • The need to effectively teach software engineering students how to solve real-world problems… • In the context of software testing education, testing tools, exercises, projects and assignments should be practical and realistic.

4. Disclaimer • The analyses made in this talk are purely objective, have no personal bias, and are only intended to help us design better testing exercises and assignments. • No personal judgment is intended w.r.t. the instructors of the courses being analyzed. • The data have been gathered to the maximum accuracy possible. • But there is no guarantee for the accuracy of data and thus the derived analyses based on them.

5. Different Alternatives currently in use and Observations Various open-source or commercial systems (one SUT per student), e.g.: RUBiS SIP Communicator Jarnal: a PDF annotation program jEdit Apache Tomcat JFreeChart Various open-source or commercial tools (one tool per student), e.g.: IBM Rational Functional Tester IBM Rational Manual Tester Parasoft Jtest Parasoft C++test Parasoft .TEST University of Calgary (by the author) [7] A small library book loan systemTests were automated using Unix command-line scripts Queen's University [6] A small-scale office suite called TerpOffice (developed at UMD) JUnitUniversity of Maryland [5] A 49-LOC C programProteumIM-2.0 (mutation tool) www.google.comJMeter A hypothetical library checkout systemTelcordia AR Greenhouse Efficient Test Case Generation Service (AETG) A 73-LOC C programxSuds (coverage analysis tool) A simple Triangle classJUnitPurdue University [4] A VoIP application developed by students in a previous course JUnit, CPPUnitUniversity of Waterloo [3] Presentation tool, part of the Open Office suiteNone (students apparently used the black-box techniques manually). Florida Institute of Technology [2] A simple hypothetical landing gear safety system, and an Automatic Teller Machine JUnitCarleton University [1] SUT(s)Testing tool(s)Testing course at …

6. Different Alternatives currently in use and Observations Various open-source or commercial systems (one SUT per student), e.g.: RUBiS SIP Communicator Jarnal: a PDF annotation program jEdit Apache Tomcat JFreeChart Various open-source or commercial tools (one tool per student), e.g.: IBM Rational Functional Tester IBM Rational Manual Tester Parasoft Jtest Parasoft C++test Parasoft .TEST University of Calgary (by the author) [7] A small library book loan systemTests were automated using Unix command-line scripts Queen's University [6] A small-scale office suite called TerpOffice (developed at UMD) JUnitUniversity of Maryland [5] A 49-LOC C programProteumIM-2.0 (mutation tool) www.google.comJMeter A hypothetical library checkout systemTelcordia AR Greenhouse Efficient Test Case Generation Service (AETG) A 73-LOC C programxSuds (coverage analysis tool) A simple Triangle classJUnitPurdue University [4] A VoIP application developed by students in a previous course JUnit, CPPUnitUniversity of Waterloo [3] Presentation tool, part of the Open Office suiteNone (students apparently used the black-box techniques manually to derive the test suites). Florida Institute of Technology [2] A simple hypothetical landing gear safety system, and an Automatic Teller Machine JUnitCarleton University [1] SUT(s)Testing tool(s)Testing course at …

12. Tradeoff - Pros and Cons of Different Choices • Some of the courses use small-scale (“toy”) or hypothetical SUTs in their exercises. • For example, in the Purdue University’s course, students learn code coverage through a lab exercise by using a less-known tool called xSuds to test a 73-LOC C program (SUT). • It seems to the author that this scale of SUTs will not prepare students well enough for real-world industrial settings. • In the Queen's university’s testing course, students automated their test cases using ad-hoc Unix command-line scripts. • According to the project guidelines in that course, it seems that it is possible and better to use industry-standard testing frameworks (e.g., JUnit) instead of ad-hoc Unix scripts.

13. Tradeoff - Pros and Cons of Different Choices • Also some of the courses (such as Purdue’s) often use simple research- oriented testing tools, e.g., ProteumIM-2.0 (a mutation tool). • Perhaps, it is a better idea to adopt and use more recent and powerful tools instead, e.g., MuClipse a mutation testing plug-in for the Eclipse IDE.

14. Breadth of Tools Used • The courses at Purdue University and the University of Calgary (taught by myself) seem to be more comprehensive than the other five courses. • Purdue’s course involves the students in working with five different testing tools, each for a specific type of testing, e.g., JMeter for performance testing, and xSuds for coverage analysis. Various open-source or commercial systems (one SUT per student), e.g.: RUBiS SIP Communicator Jarnal: a PDF annotation program jEdit Apache Tomcat JFreeChart Various open-source or commercial tools (one tool per student), e.g.: IBM Rational Functional Tester IBM Rational Manual Tester Parasoft Jtest Parasoft C++test Parasoft .TEST University of Calgary (by the author) [7] A 49-LOC C programProteumIM-2.0 (mutation tool) www.google.comJMeter A hypothetical library checkout systemTelcordia AR Greenhouse Efficient Test Case Generation Service (AETG) A 73-LOC C programxSuds (coverage analysis tool) A simple Triangle classJUnitPurdue University [4] SUT(s)Testing tool(s)Testing course at …

15. The approach (choices) taken in the University of Calgary’s Course • Was first offered in Winter 2008. • Unlike the other six courses which are undergraduate-level, this course is graduate-level • 9 students + 3 people from the SW industry • Each student spent the entire three-month term to install, learn and use a commercial testing tool to test a real-world large-scale SUT (details next…). • Students handed in three comprehensive reports in three milestones of the project: – 1: proposal – 2: interim report – 3: final report

16. The University of Calgary’s Course - Goals • The goal was to incorporate both breadth and depth in students’ learning • Depth in learning each tool in detail by each student • Breadth in learning by having project presentations and in-depth discussions, tool comparisons, after each presentation among students

17. The University of Calgary’s Course: The Testing Tools and Systems Under Test (SUTs) Various open-source or commercial systems (one SUT per student), e.g.: jEdit Apache Tomcat JFreeChart RUBiS: a web-based ebay-like application written in J2EE A large-scale commercial web application for an oil & gas government agency. Jarnal: a PDF annotation program BlogEngine.NET SIP Communicator: an open-source java- based internet phone protocol A commercial USB camera control system for robot vision Various open-source or commercial tools (one tool per student), e.g.: IBM Rational Functional Tester IBM Rational Manual Tester IBM Rational AppScan IBM Rational Test RealTime IBM Rational Performance Tester IBM Rational TestManager Parasoft Jtest Parasoft C++test Parasoft .TEST JUnit CUnit NUnit JMeter Fitnesse Acceptance Testing Framework SUTsTesting tools

18. The University of Calgary’s Course - Results • Formal course evaluations have confirmed the course success in using realistic exercises for its project. • For example, 92% of students believed that they “learned a lot in this course”. • 91% believed the “support materials and tools were helpful”. • Only after one offering, the course has become very popular in the department • There is already lots of student and industry interest in taking it in Winter 2009 term. (it is in full capacity this term)

19. Summary and Recommendations: Choosing the right SUTs and Testing Tools • It is hard to generalize the findings and observations… • But the author would recommended that testing educators should align the choices of SUTs and tools with the following criteria… – the ultimate goal of the particular course at hand, e.g., learning goals and the end product (the type of testers) needed at the end of the course – the type and background of the students – and also the time and resources available in the given course, e.g., are tool licenses free, or are their costs within budget?

20. Summary and Recommendations: Choosing the right SUTs and Testing Tools • For example, since the author’s course was graduate-level, • and that Calgary has a very active testing industry, • and it was expected that the students were going to work in the industry soon after the course, • also, the students in that course were senior and had some prior knowledge/experience with testing fundamentals. • Thus, it was decided to use industry-strength testing tools and large-scale SUTs in his course.

21. Further work is needed • To provide better and more precise recommendations on the subject, • systematic surveys and comparative analysis of testing tools for the purpose of education on different types of testing tools. • There are surveys such as the one in [10], but it is only a survey of one specific type of testing tools (only coverage measurement) and, also, • that survey does not compare the tools from the educational standpoint and the type of learning outcomes gained using those tools. • Last but not least, more experience and systematic studies by testing educators on the choice of tools and SUTs to draw generalizable conclusions and external validity on the subject.

22. Thanks for your attention. Q/A period… (LAWST / WTST style!)

Choosing the Right Testing Tools and Systems Under Test (SUTs) for Practical Exercises in Testing Education

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Choosing the Right Testing Tools and Systems Under Test (SUTs) for Practical Exercises in Testing Education

Similar to Choosing the Right Testing Tools and Systems Under Test (SUTs) for Practical Exercises in Testing Education (20)

More from Vahid Garousi

More from Vahid Garousi (20)

Recently uploaded

Recently uploaded (20)

Choosing the Right Testing Tools and Systems Under Test (SUTs) for Practical Exercises in Testing Education