Vahid Garousi, An Open Modern Software Testing Laboratory Courseware: An Experience Report, Proceedings of the 23rd IEEE Conference on Software Engineering Education and Training, Pittsburgh, USA, March 2010

1. Vahid Garousi, CSEE&T 2010 1
An Open Modern Software Testing Laboratory
Courseware – An Experience Report
Vahid Garousi, PhD, PEng
Software Quality Engineering Research Group (SoftQual)
Schulich School of Engineering
University of Calgary, Canada
CSEE&T 2010, Pittsburgh, PA, USA (March 11, 2010)
(some of my students:)

2. Vahid Garousi, CSEE&T 2010 2
Background
The speaker believes that we have a few good textbooks on SW
testing, e.g.:
by Aditya P. Mathur (2008)
by Paul Ammann, Jeff Offutt (2008)
by Sagar Naik and Piyu Tripathy’s (2008)
But frankly speaking, there are not many publicly-available testing
courseware for testing educators to adapt and customize
The drawbacks of majority of the existing testing lab exercises:
They are not kept up-to-date with the most recent testing
tools and technologies, e.g., JUnit
are not built based on real-world large-scale SUTs
A brief comparison next…
A survey on this topic has been done by the speaker and
is available from his website

3. Vahid Garousi, CSEE&T 2010 3
A small library book loan systemTests were automated using Unix
command-line scripts
Queen's
University [6]
A office suite called TerpOffice
(developed internally at UMD)
JUnitUniversity of
Maryland [5]
A 49-LOC C programProteumIM-2.0 (mutation tool)
www.google.comJMeter
A hypothetical library checkout
system
Telcordia AR Greenhouse Efficient
Test Case Generation Service (AETG)
A 73-LOC C programxSuds (coverage analysis tool)
A simple Triangle classJUnitPurdue
University [4]
A VoIP application developed by
students in a previous course
JUnit, CPPUnitUniversity of
Waterloo [3]
Presentation tool, part of the Open
Office suite
None (students apparently used the
black-box techniques manually).
Florida Institute
of Technology [2]
A simple hypothetical landing gear
safety system, and an Automatic
Teller Machine
JUnitCarleton
University [1]
SUT(s)Testing tool(s)Testing course at
Several Software Testing Lab Exercises
Currently in Use in North American Universities

4. Vahid Garousi, CSEE&T 2010 4
Motivations
The above needs were the main motive for the speaker to
work with a new graduate of our SW engineering program
to …
The goal was: to modernize and re-design the labs of an
undergraduate testing course, i.e., Practical Exercises
Real-world testing tools
Realistic/large-scale SUTs

5. Vahid Garousi, CSEE&T 2010 5
We came up with Five Lab Exercises
1. Introduction to Testing and Defect Tracking
2. Requirements-Based Test Generation
3. Code Coverage, Adequacy Criteria and Test Case Correlation
4. Functional GUI Testing
5. Mutation Testing and Analysis
6. Performance, Security and Stress Testing
• Was in the original plan.
• But we felt that there would not be enough time for it during the
course.
• Was thus pitifully removed from the labware
Details about the five labs, next…

6. Vahid Garousi, CSEE&T 2010 6
This Cab Courseware is Available
Online

7. Vahid Garousi, CSEE&T 2010 7
Lab 1.
Introduction to Testing and Defect Tracking
Goals:
Getting hands-on experience in ad-hoc, manual and regression
testing
Using industrial defect tracking practices and tools (Bugzilla)
Experiencing that ad-hoc testing is not always very effective
SUT: An ATM simulation system
Testing support tool: Bugzilla
Realistic defects were manually injected in the SUT

8. Vahid Garousi, CSEE&T 2010 8
Lab 1.
Introduction to Testing and Defect Tracking
Lab document (instructions)…
Extensive effort was spent to
prepare clear, concise and
“interesting” lab exercises…
Involving a recent good student
helped a lot in this aspect

9. Vahid Garousi, CSEE&T 2010 9
Lab 1. Introduction to Testing and Defect Tracking
Marking Scheme
10%Number of defects found: Note that not all defects need to be
found. But if it appears that not enough effort was made in
finding defects, marks may be deducted.
10%Clarity and adherence to defect reporting guidelines
10%Correctness
Bugzilla Defect Reports (30%)
• Bugzilla Defect Reports (30%)
Lab Report (70%)

10. Vahid Garousi, CSEE&T 2010 10
Lab 1. Introduction to Testing and Defect Tracking
Marking Scheme
10%
6. Any difficulties encountered, challenges overcome, and lessons
learned from performing the lab
10%
5. A discussion on how the team work/effort was divided and
managed. Any lessons learned from your teamwork on this lab?
10%4. Notes and a discussion of the peer reviews of defect reports.
20%
3. A comparison of ad-hoc and manual functional testing (based on
the provided test suite) from several perspectives (e.g., benefits,
tradeoffs, effectiveness, efficiency, etc.)
10%
2. The test cases performed during ad-hoc testing (in a table similar to
in Appendix E would be best)
10%1. A high-level description of the ad-hoc testing plan
Lab Report (70%)

11. Vahid Garousi, CSEE&T 2010 11
Lab 1.
Introduction to Testing and Defect Tracking
A template for lab reports
(to be filled out by
students)…
Made it easier for the
students, the instructor and
the TA

12. Vahid Garousi, CSEE&T 2010 12
Lab 1.
Introduction to Testing and Defect Tracking
An example of students’ works…

13. Vahid Garousi, CSEE&T 2010 13
Lab 1.
Introduction to Testing and Defect Tracking
Students’ works…

14. Vahid Garousi, CSEE&T 2010 14
Lab 1.
Introduction to Testing and Defect Tracking
Feedback from students…
“This was a very realistic lab. It clearly involves
works and concepts useful outside of school.
This lab has shown the complications involved in
conducting software testing, and that it is no trivial
task.
This lab was quite the learning adventure.
It also gave me the exposure to challenges with
testing and learning the system efficiently (i.e.,,
unclear requirements), which prepared me to deal
with the future ones [systems].”
And much more…

15. Vahid Garousi, CSEE&T 2010 15
Lab 1.
Introduction to Testing and Defect Tracking
Feedbacks for improvements from students…
“One could say that the instructions were too detailed and
reading it over and trying to keep track of all the points were
frustrating.
vs. I have to admit that contents of the lab manual were
extremely organized which helped me overcoming how
much information was provided
I think the only thing I did not like about the lab was how
contrived the defects were in the SUT.”

16. Vahid Garousi, CSEE&T 2010 16
Five Labs
1. Introduction to Testing and Defect Tracking
2. Requirements-Based Test Generation
3. Code Coverage, Adequacy Criteria and Test Case
Correlation
4. Functional GUI Testing
5. Mutation Testing and Analysis

17. Vahid Garousi, CSEE&T 2010 17
Lab 2. Requirements-Based Test Generation
Goals:
Deriving unit test from Javadoc API Specifications
Using JUnit
SUT: JFreeChart
Testing tool: JUnit

18. Vahid Garousi, CSEE&T 2010 18
Lab 2. Requirements-Based Test Generation
Students applied the equivalence partitioning technique based
on Javadoc API Specifications

19. Vahid Garousi, CSEE&T 2010 19
Lab 2. Requirements-Based Test Generation
Some of the feedback from students…
“Overall, the whole experience working with unit testing and
bug reporting was great.
It also allowed us to deal with inexplicit requirements and
helped prepare ourselves for future unit testing challenges.
We learned the importance of peer review in creating effective
test suites and bug report.
It allowed us to put the idea of equivalence class analysis into
real situation practice. .”
And much more …

20. Vahid Garousi, CSEE&T 2010 20
Lab 2. Requirements-Based Test Generation
Feedbacks for improvements from students…
“The lab could be improved by also showing, or at
least explaining, the different between JUnit 3 and
JUnit 4.
Because of the number of tests that needed to be
done, this lab was very long and repetitive.
The first important lesson that I got from this lab is
that we should avoid the testing department in our
career paths.
It is pretty tedious and copy/pasting and changing
method names slightly all day is unimaginative.”

21. Vahid Garousi, CSEE&T 2010 21
Five labs
1. Introduction to Testing and Defect Tracking
2. Requirements-Based Test Generation
3. Code Coverage, Adequacy Criteria and Test Case
Correlation
4. Functional GUI Testing
5. Mutation Testing and Analysis

22. Vahid Garousi, CSEE&T 2010 22
Lab 3.
Code Coverage, Adequacy Criteria and Test Case
Correlation
SUT: JFreeChart (the test suites of lab 2)
Testing tools:
JUnit
CodeCover
CoverLipse
In this lab, students enhanced the code coverage of
the test suites they had developed in lab 2.

23. Vahid Garousi, CSEE&T 2010 23
Lab 3.
Code Coverage, Adequacy Criteria and Test Case
Correlation
Some of the feedback from students…
“Our knowledge of testing has been greatly enhanced while
performing this lab.
Getting to know testing in a more structured and systematic
way is more effective in numerous aspects.
Performing code coverage definitely has a great impact in
enhancing or creating more effective test suites.
This lab perfectly replicates an industry situation where
inadequate time is available for testing.
Hence, careful consideration was devoted in prioritizing which
test cases to develop.”
And much more …

24. Vahid Garousi, CSEE&T 2010 24
Five labs
1. Introduction to Testing and Defect Tracking
2. Requirements-Based Test Generation
3. Code Coverage, Adequacy Criteria and Test Case
Correlation
4. Functional GUI Testing
5. Mutation Testing and Analysis

25. Vahid Garousi, CSEE&T 2010 25
Lab 4. Functional GUI Testing
SUT: Plazma ERP + CRM
Enterprise Resource Planning (ERP)
Customer Relationship Management (CRM)

26. Vahid Garousi, CSEE&T 2010 26
Lab 4. Functional GUI Testing
Testing tool: IBM Rational Functional Tester
(version 7)

27. Vahid Garousi, CSEE&T 2010 27
Lab 4. Functional GUI Testing
Students recorded GUI test cases based on the
requirements given in the UML use cases of the SUT
They then executed (played back) them on the same
version of the SUT.

28. Vahid Garousi, CSEE&T 2010 28
Lab 4. Functional GUI Testing
Feedbacks for improvements from students…
“For the purpose of this lab, the test suite we derived was
ineffective at benefiting our testing experience;
What this indicates to me is that this lab needs a second part
to be more useful.
If the suite was created, and then a second version of the GUI
was used for a re-run of the test suite, it would have been far
more effective.
This would have shown us the effectiveness and use of the
test suite.”

29. Vahid Garousi, CSEE&T 2010 29
Five labs
1. Introduction to Testing and Defect Tracking
2. Requirements-Based Test Generation
3. Code Coverage, Adequacy Criteria and Test Case Correlation
4. Functional GUI Testing
5. Mutation Testing and Analysis

30. Vahid Garousi, CSEE&T 2010 30
Lab 5. Mutation Testing and Analysis
SUT: JFreeChart (test suites created in labs 2 and 3)
Testing tool: MuClipse plug-in for Eclipse
Arithmetic Operator
Replacement for Binary
operator

31. Vahid Garousi, CSEE&T 2010 31
Known uses

32. Vahid Garousi, CSEE&T 2010 32
Summary and Lessons Learned
Our experience showed that the labware was (for the most part) a
successful leaning experience for students…
The feedback given by students for improvement are being taken
into account to enhance the labware.
Overall, the author recommends that testing educators should
align the choices of SUTs and testing tools with the following
criteria…
The ultimate goal of the particular course at hand, e.g.,
learning goals and the end product (the type of testers)
needed at the end of the course
The type and background of the students
and also the time and resources available in the given
course, e.g., are tool licenses free, or are their costs within
budget?

33. Vahid Garousi, CSEE&T 2010 33
This Cab Courseware is Available
Online

34. Vahid Garousi, CSEE&T 2010 34
Thanks for your attention.
Q/A period…