Software Testing in the Computer Science Curriculum -- A Holistic Approach Edward L. Jones Florida A&M University Tallahassee, FL USA 32307 firstname.lastname@example.orgAbstract environment . This characterisation shows the breadthAlthough testing accounts for 50% of the cost of software, and depth of testing concerns.it receives little treatment in most curricula. This paperpresents some approaches to giving all students multiple, Table 1. Three Dimensions of Software Testingincremental exposures to software testing throughout thecurriculum. A unifying framework is presented which Purpose Granularity Strategyidentifies a minimal set of test experiences, skills andconcepts students should accumulate. The holistic approach Acceptance Function Specification Drivencombines common test experiences in core courses, an Defect Object (black box)elective course in software testing, and volunteer Reliability Component Implementation Drivenparticipation in a test laboratory. Performance Application (white-box) Usability System Usage Driven (operational)1 Introduction Random (statistical)Software testing is a major challenge in industry, Intuitive (ad hoc)accounting for 50% of the software costs. As softwarecomplexity and legal ramifications of poor quality increase, The definitive work on software testing is Glenford Myerssoftware testing will play an even more prominent role. book , The Art of Software Testing. Myers identifiesIn the remainder of this section we assess the state of attitude (psychology) and economics as major obstacles topractice in software testing, and review approaches taken testing. His oft-forgotten principles should be posted as theby educators to address the software quality issue. 13 Commandments of Testing Practice. Besides attitude, testers must also possess superior technical skills.1.1 State of Practice Whittaker  points out the need for testers to have technical sophistication:There are many reasons to test. The classical statements ofpurpose by Dijkstra, "testing demonstrates the presence "Testers must not only have good developmentof bugs, not their absence," and Myers, "testing is the skills--testing often requires a great deal of coding--process of executing a program with the intent of finding but also be knowledgeable in formal languages,an error," belie the diversity of motivations for testing. graph theory, and algorithms."Table 1 depicts three dimensions of the software testing Technical approaches to testing have been publishedproblem: purpose, granularity and strategy. Other widely but, according to Osterweil , there isdimensions exist, e.g., technology/architecture, or the target "… a yawning chasm separating practice from research that blocks needed improvements in both communities." Three parties have a role to play to spanning this chasm. Companies must recognize that testing is a hard problem that is becoming harder as software becomes more complex. Researchers must produce approaches that are less academic and more practical (palatable) to industry consumers . Finally, educators must do a better job equipping students with skills and attitudes for dealing effectively with software quality concerns.
The work of Jones  and Reek  take a testers1.2 Testing in the Curriculum approach to grading student programs. The assignment specification is the basis for test case design. Execution ofSoftware testing is ostensibly absent from computer science programs against the test cases is automated using testcurricula. A premise of this work is that students who learn harnessing techniques. Students learn quickly the mostto test become better software developers because testing important lesson in software quality, that the specificationforces the integration of concepts and practice. Students is king! Failure to satisfy the specification is failure.who have been exposed to testing enter the workforcebetter equipped to deal with the engineering aspect ofsoftware. 2 The Holistic ApproachAlthough testing has been a part of classical computer It is impractical to teach every student all there is to knowscience literature for decades , the subject matter is about testing. It is possible, and desirable, however, to giveseldom incorporated into the mainstream undergraduate every student a common set of testing experiences thatexperience. That so few articles on teaching software apply essential principles inherent in the practice of qualitytesting have been published suggests that software testing assurance. We believe the SPRAE framework identifiesis an afterthought. In the article "Software Quality: A these essential principles, and that experience applyingCurriculum Postscript?" Hilburn makes a case for a these principles transfers to new test situations.comprehensive, end-to-end treatment of software quality by The holistic approach has the following objectives.employing Watts Humphreys team software process (TSP)and personal software process (PSP) throughout the • Every student has multiple testing experiences.curriculum [4,5]. • Each experience incorporates SPRAE principles.Carrington  describes a testing course intended to give • Each course provides at least one test experience.students practical skills along with theoretical knowledge • Students are afforded opportunities for advancedof testing techniques. Students gain experience deriving experiences..test cases from formal specifications (black-box) and fromcode (white-box). Carrington reports that students come The holistic approach has the following components.face to face with the labor-intensive drudgery of testing, • The SPRAE framework defining essential principlesand see firsthand the need to automate mundane tasks using and practices.scripting. • Common testing experiences integrated throughout theJones  presents the SPRAE framework that identifies curriculum.essential testing principles students should learn. TheSPRAE principles include: • Testing methods applied to grading student programs.• Specification. A specification is essential to testing. • Elective course in software testing.• Premeditation. Testing requires a systematic process. • Software test lab for advanced application of testing.• Repeatability. The testing process and its results must • Test lab products absorbed into the academic effort. be repeatable and independent of the tester. • Test lab provides testing services.• Accountability. The test process must produce an audit We believe that a holistic approach is more consistent with trail. the way students learn life-long skills:• Economy. Testing must not require excessive human • SPRAE provides a small set of simple principles. or computer resources. • Repetition of similar experiences occurs in differentThese principles are used to design a set of test experiences contexts (courses).students need to acquire. • Experiences are accumulated over two or three years.1.3 Quality Emphasis in the Curriculum 3 Implications of the SPRAE FrameworkMuch has been published about computer-based evaluation In this section, we identify a minimal set of test experiencesof student work. When programs are being evaluated, the that cover the SPRAE-compliant test life cycle shown inevaluation process is an instance of the testing problem (or Table 2. Student experiences begin at the end of the lifesoftware quality in the broader sense). For most students, cycle, and move backward toward the front as the studentthis is the only time quality becomes important, because gains experience. A minimal sequence of test experiencestheir grade is at stake. Our holistic approach capitalizes on is shown below.program grading events as opportunities to expose studentsto software testing. 1. Debug a program, given the test cases, and a test log. 2. Develop a test log from test results, given test cases.
3. Run tests, given test script and test data. keeping testing requires. Their experience vacillated4. Design test cases, given a specification. between boredom with record keeping, to extreme difficulty with the programming and analytical tasks.5. Refine a specification. Our conclusion is that a separate course in software testing6. Develop a program, given its specification. should not be the only place to incorporate testing into the7. Create a test driver. curriculum. Students who wait to take the elective courseRecall the various dimensions of the software testing lose numerous opportunities to learn and reinforce testingproblem. The set of experiences listed above may be skills in contexts other than "testing class." Students arerepeated for instances of the testing problem that occur in notorious for compartmentalizing knowledge to a singledifferent courses. Each repetition reinforces techniques and course, and not transferring it to new situations. Theprinciples learned earlier. preferred use of an elective course is to prepare students for advanced study in software testing.4 Elective Course in Software Testing 5 Automated Program GradingThe first approach we attempted was to teach an elective Our initial effort to incorporate testing into the curriculumcourse in software testing. The major benefit was that was to automate the grading of student programs . Thestudents received in-depth treatment of concepts and automated program grading system (APGS) brings studentspractices, preparing them for further study or application. face to face with quality assessment: poor quality results inThe major disadvantage was that the course was not a poor grade. Many students are surprised to learn that theyavailable to all students. Most students enrolled in the really must follow the specification. This realization alonecourse only because they needed it to graduate, not because justifies the additional effort the teacher must invest tothey were interested in testing. Regrettably, students who specify a testable assignment. Conscientious students soonare genuinely interested in testing nay not be able to take begin to anticipate how their programs will be graded. Thatthe course because it is not offered regularly. is, they do intuitive test case design based on the assignment specification. They begin to think like testers! Table 2. Test Life Cycle A SPRAE-compliant process is followed to transform an initial assignment specification into a testable specification. Phase Activity => Products Each new assignment is added to a testbed of examples Analysis Define test purpose and strategy. showing faculty (or teaching assistants) how to follow the Refine specification. process. => Test Plan + Refined Spec 1. Develop initial assignment specification A0. Design Systematically derive test cases. 2. Transform A0 into a testable specification A1. Organize test effort. 3. Design test cases from assignment specification A1. => Test Cases + Test Data 4. Develop assignment grading plan P. Implementation Develop test "machinery". 5. Design assignment grading scripts based on plan P. 6. Grade programs using grading scripts. => Test Harness + Test Scripts. Execution Run test as a controlled experiment. 7. Distribute grading logs and scores to students. Capture test results. The initial version of APGS is being used in an upper level => Test Results elective course. We plan to move backwards through the Evaluation Verify results. curriculum, applying APGS to one additional course each Take follow-up actions. year. We also plan to make the test design process and its Debug artifacts more visible to students. Initially, we will make => Test Log + Modified Software the test cases (for interactive programs, usage scenarios) available to students. Next, we will make the test plan available to students. Additionally, we will invoke theWe gained important insights from teaching the course. testing process at the time students submit their programs,First, we discovered that testing is a programming intensive as a way for the student to preview the quality of theactivity. The students whose programming skills were rusty program. The student will be permitted unlimited dry runshad a difficult time with the programming aspects of of submitted programs.testing. We also discovered that testing requires analyticalskills and facility with mathematical reasoning tools such 6 Experience Factory - The Software TestLabas decision tables and graphs. Again, since these topics had The TestLab provides a SPRAE-compliant environment inbeen covered several semesters beforehand, students had which students can learn the art and science of softwareforgotten. Students were shocked by the amount of record
testing. TestLab students work on specific testing tasks to mission of the TestLab. Competent TestLab students mayproduce and disseminate results in the form of completed be outsourced to faculty members to perform testing tasksartifacts, tutorials, experience reports and research such as automated grading or tutoring.publications. We seek to motivate students towards careersand graduate study in software testing, while transferring 7 Conclusionslessons and artifacts from the TestLab into courses in theundergraduate curriculum. Although software testing is a difficult subject, students should acquire experience in fundamental aspects ofThe TestLab is funded by corporate partners and by an testing. To date, our experience is that, when given in smallNSF grant. The NSF support targets students who plan to doses, students absorb testing concepts and practices.attend graduate school. Corporate sponsorship extends Software testing forces students to integrate concepts, skillsstudent involvement to all students. Corporate partners are and practices. The holistic approach makes this integrationalso encouraged to offer students internships in software a continuous process.testing/quality assurance.TestLab students are required to take the elective course insoftware testing to acquire the background needed to be 8 Future Workproductive on TestLab assignments. We have formulated a This paper describes results from the first year of a five-training ladder reflecting the accumulation of testing skills year project. The work for the second year includes theand experiences from the course and TestLab. Table 3 following.describes the competencies TestLab students will acquire. 1. Develop web-based resources to support the softwareNote that certain competencies focus on a specific phase of testing course. Provide ad hoc access to all students.the test life cycle, while others (like Test Specialist) span Design course-specific entry points to support futuremultiple phases. courses. 2. Implement the TestLab experience ladder as a part of Table 3. TestLab Competency Categories the TestLab management process. As the number of TestLab students increases, the experience ladder sets well-defined competency goals, and enables student- Competency Duties to-student mentoring. Test Practitioner Perform a defined test within an 3. Extend automated program grading to one additional established test environment and course. Outsource competent TestLab students as document results. technical support for refining assignment Test Analyst Handle front-end of testing process. specifications and developing grading programs. Determine testing needs. Assure that the specification is an adequate 9 Acknowledgements basis for starting the testing process. The author thanks the TestLab students who worked on ill- Test Designer Given a specification, design test defined, yet groundbreaking tasks. These corporate cases using published and TestLab sponsors provided moral and financial support: Lucent techniques. Technologies, Dell Computer, Lockheed Martin, 3M, and Test Builder Construct the machinery needed to Abbott Laboratories. This work was partially funded by run the test cases. National Science Foundation grant EIA-9906590. Test Inspector Verify that a testing task has been performed correctly according to References TestLab procedures and standards.  Carrington, D. Teaching Software Testing. Test Technician. Set up and support the Proceedings of Second Australasian Conference on Environmentalist test environment. Computer Science Education (1996), 59-64. Test Specialist Perform a series of testing tasks  Dahl, O.J, Dijkstra, E.W., and Hoare, C.A.R. spanning multiple phases of the test Structured Programming. APIC Studies in Data life cycle. Processing 8. Academic Press, 1972.  Hilburn, T.B., and Towhidnejad, M. Software Quality:The training ladder reflects progression through levels of A Curriculum Postscript? Proceedings of the 31stcompetency within a category. Competency certification is SIGCSE Technical Symposium on Computer Scienceearned by the completion of testing tasks representative of Education (May 2000), 167-171.the skill level. As TestLab students progress, they become  Humphrey, W. S., The Discipline of Softwarecapable of performing tasks that support the dissemination Engineering, Addison-Wesley, 1995.
 Humphrey, W. S., Introduction to the Team Software Process, Addison-Wesley, 2000. Jones, E.L. The SPRAE Framework for Teaching Software Testing in the Undergraduate Curriculum. Proceedings of Symposium on Computing at Minority Institutions, (June 1-4, 2000). Jones, E.L. Grading Student Programs - A Software Testing Approach. Journal of Computing in Small Colleges, (November , 2000), to appear. Myers, G.J. The Art of Software Testing. John Wiley & Sons, New York, 1976. Osterweil, L., et al, Strategic Directions in Software Quality. ACM Computing Surveys 28,4 (December 1996), 738-750. Reek, K.A. The TRY System – or – How to Avoid Testing Student Programs. Proceedings SIGCSE Bulletin vol. 21, 1 (February 1989), 112-116. Whittaker, J.A. What is software testing? And why is it so hard? IEEE Software 17, 1 (Jan 2000), 70-79.