Projects in Software Testing


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Projects in Software Testing

  1. 1. Projects in Software Testing Software Testing, ETS 200 Programvarutestning, 5p. Version 1.3, 2007, Lp Vt2 Carina Andersson, Telecom, LTH 1. Introduction This document gives the practical details regarding the project in the course Software Testing. The project in the course is equivalent to 2 weeks of full time studies (2 credit points). The main objective of the project is to gain a deeper understanding of a specific area within verifi- cation and validation (V&V) of software products. A project group consists of 4-5 persons, who perform the project together. All project members should be involved and the total effort should be evenly distributed among participants. 2. Learning Objectives The objective of the project is to learn a specific area of software testing. Furthermore, the structure of the project makes it possible to learn important parts that will be practiced later on in the education. Hence, the main learning points of the project are: • Learn a specific area of software testing • Collect and summarize research information • Critical thinking beyond the written information • Present information in a structured way There are several types of projects to choose among. Examples of project types are: • Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc. • Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis tool. • Practical: Use an existing technique to test a system or design and implement a prototype for a system. 1
  2. 2. 3. Activities The main activities in the project are: 1. Decide on a subject. There are some subjects specified in this document. Other sub- jects may be chosen, but has to be decided together with your project supervisor. 2. Find literature. In each subject area, there is a suggestion of research reports to start reading. Other research literature can be found on (within the domain of LU, or by using your STIL password). For research projects, 7-10 reports should be chosen. For evaluation and practical projects, fewer reports are needed. 3. Outline the report. Specify heading one and two and write in short sentences what will be included in the sections. 4. Read; perform evaluation and practical work. Read and understand the new area of software testing; perform evaluation and practical work if these kind of projects have been chosen. 5. Write the report. The report is preferable written in English. The report shall be writ- ten in the IEEE template, see the homepage of the course. 6. Present the report. Communicate the area to other people who have knowledge of software testing, but not the specific area. This is done during the presentation session. You have about 15 minutes to describe your work. The presentation should either be in Swedish or English. 2
  3. 3. 4. Schedule Week Activities Hand in to supervisor Deadline 11 Decide on a subject Decided subject Thu 15/3 Search literature 12-13 Search literature Read literature Outline the report Outline Tue 27/3 16-17 Read literature Write report Meeting with supervisor Book time 18 Write report 19 Report to supervisor Tue 8/5 Prepare presentation 20 Presentation Session Tue 15/5 Update report If needed, updated report to supervisor Wed 23/5 5. Assessment See the course program. 6. Report The report shall be written using the IEEE template. The report shall consist of: • Abstract: summary of all parts in the report. The purpose of the abstract is to attract peo- ple to read your report. • Introduction: introduce the chosen area. The purpose is to give an introduction to the reader who is not familiar to the specific area, but knows software engineering and testing generally. • Description of the chosen area: summary of the chosen area. The purpose is to describe why, how and for which purpose the area is used. • Analysis: analysis, critical thinking and future plans within the chosen area. For research projects, the purpose is to describe the benefits, drawbacks, what research the area needs in the future. For evaluation and practical projects, the work should be described here together with a shorter analysis. • Conclusion: main conclusion in the report. The purpose is to discuss the main points in the report again. Another structure can be chosen after discussion with your project supervisor. 3
  4. 4. 7. Presentation The project will be presented in the project presentation sessions at the end of the course. Each project will be given 15 minutes to present their chosen area. The presentation should cover the main points of summary, analysis and conclusion. 8. Project areas There are several types of projects to choose among. For research projects, 7-10 reports should be chosen. For evaluation and practical projects, fewer reports are needed. Examples of project types are: • Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc. • Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis tool. • Practical: Use an existing technique to test a system or design and implement a prototype for a system. If you would like to choose another area, describe the area to your project supervisor before starting the work. The keywords in Section 9 can be used when you chose an area and when you search for literature. 8.1 How do you know your test cases are correct? Test cases are developed to check whether a software system is implemented correctly accord- ing to the requirements specification. However, if the test cases are badly chosen, they will not detect the failures. There is a method, called Mutation testing, which can be used to check whether the test cases are correct. Mutation Testing: A. M. R. Vincenzi, J. C. Maldonado, E. F. Barbosa and M. E. Dela- maro, “Unit and Integration Testing Strategies for C Programs using Mutation”, Software Testing, Verification and Reliability, 11(3):249–268, 2001. 8.2 How can you measure that the software system works? Software testing is performed to detect the failures in the software. There are different met- rics to use in order to measure whether the software is correct. One such measure is reliability, which is defined as: “The probability for a failure-free operation of a program for a specified time under a specified set of operating conditions.” (IEEE 610.12-1990). There are several models and techniques that can be used, for example, reliability growth models, Markov models, statistical usage testing, usage-based testing and operational profile testing. Reliability growth models: C. Stringfellow, A. A. Andrews, “An Empirical Method for Selecting Software Reliability Growth Models”, Empirical Software Engineering, 7(4): 319- 343, 2002. Markov models: J. A. Whittaker and M. G. Thomason, “A Markov Chain Model for Sta- tistical Software Testing”, IEEE Transactions on Software Engineering, 20(10):812-824, 1994. 8.3 What techniques are most effective to check the requirements and design documents? Static verification is often used in the beginning of the development of software. There are several different techniques and methods used to check the static representation. The com- 4
  5. 5. mon feature of these is that they have to be manually checked by reviewers. Software inspec- tion, reviews, walkthroughs are common techniques, which are used together with reading techniques. Software inspection and reading techniques: A. Aybuke, H. Petersson, C. Wohlin, “State- of-the-art: Software Inspections after 25 Years”, Software Testing, Verification and Reliability, 12(3):133-154, 2002. 8.4 How can you perform testing on an object-oriented software system? A special branch of software testing is application-based testing. Object-oriented is one such application, which needs special treatment and special methods to test the system. Object-oriented testing: Y. Labiche, P. Thévenod-Fosse, H. Waeselynck, M.-H. Durand, “Testing Levels for Object-Oriented Software”, Proceedings of the 22nd International Confer- ence on Software Engineering, pp. 136-145, 2000. 8.5 How can you perform testing on a distributed system? A special branch of software testing is application-based testing. Testing distributed sys- tems need special treatment and special methods. Testing of distributed systems: S. Goeschl and H. M. Sneed, “Case study of testing a dis- tributed internet-system”, Software Testing, Verification and Reliability, 12(2):77 - 92, 2002. 8.6 What techniques can be used to know when to stop testing? Stopping criteria are used in the testing phase to determine when a certain quality level has been achieved. The quality can for example be defined as the reliability or just the number of faults left in the system. There are several techniques to use as a stopping criterion. One could, for example, estimate the number of faults left after an inspection or to estimate the number failures left in testing. Fault content estimation; L. C. Briand, K. E. Emam, B. G. Freimut, O. Laitenberger, “A Comprehensive Evaluation of Capture-recapture Models for Estimating Software Defect Content”, IEEE Transactions on Software Engineering, 26(6):518-540, 2000. Stopping criterion in testing: J. D. Musa and A. F. Ackerman, “Quantifying Software Vali- dation: When to Stop Testing?”, IEEE Software, 6(3):19-27, 1989. 8.7 How is testing performed for Agile processes? Test-first is a principle that is often used in Agile processes. The principle is based on that test cases are first derived and then these are used to specify how the system should work as well as used as test cases when the system is developed. There are some questions, which are of interest within this area, for example, what is the relation between requirements and test cases, what is the final quality of such a system and how easy is it to maintain such a system? Test-first principle: M. M. Muller, and O. Hagner, “Experiment about Test-first Program- ming”, IEE Proceedings Software, 149(5):131-136, 2002. 8.8 How can different testing techniques (methods) be compared? One research methodology is to use empirical methods in order to evaluate which testing methods are best to use. The methodologies are often divided into experiments, surveys and cases studies. All these methods are important in order to help software organizations to choose the right testing technique for their purpose. 5
  6. 6. Experiment: Sun Sup So, Sung Deok Cha, Timothy J. Shimeall and Yong Rae Kwon, “An Empirical Evaluation of Six Methods to Detect Faults in Software”, Software Testing, Verifica- tion and Reliability, 12(3):155–171, 2002. Survey: P. Runeson, C. Andersson and M. Höst, “Test Processes in Software Product Evo- lution - A Qualitative Survey on the State of Practice”, Journal of Software Maintenance and Evolution, 15(1):41-59, 2003. Case study: T. Berling and T. Thelin, “An Industrial Case Study of the Verification and Validation Activities”, International Software Metrics Symposium, pp. 226-238, 2003. 8.9 What quality models are used for the testing purpose? There are many general quality models developed within software engineering, and spe- cific models for testing. The quality models try to capture the essential parts of software devel- opment (testing) so the final quality will be assured. Some of the general models are CMM, SPICE, ISO 9000, and some of the specific models are TPI and TIM. CMM: M. Paulk, B. Curtis, M.B. Chrissis, C.V. Weber, “Capability Maturity Model, Ver- sion 1.1”, IEEE Software, pp. 18-27, 1993. SPICE: J-M. Simon, “SPICE: Overview for software process improvement”, Journal of Sys- tems Architecture, 42(8):633-641, 1996. TIM: T. Ericson, A. Subotic, S. Ursing, “TIM – A Test Improvement Model”, Software Testing, Verification and Reliability, 7(4):229-246, 1997. 8.10 How is the software of a nuclear plant tested? Safety is concerned with the property of a system that it will endanger human life or the environment. A safety-critical system is, hence, one by which the safety of equipment or plant is assured. An example of such a system is an aeroplane or nuclear plant system. Safety-critical software needs special attention when it comes to testing. Safety: N. G. Leveson, S. S. Cha and T. J. Shimeall, “Safety Verification of ADA Programs using Software Fault Trees”, IEEE Software, 8(4):48-59, 1991. 8.11 How can test cases be derived and generated? Software test cases are often derived from documents produced during the software devel- opment. There are different techniques and languages that can be used. Use cases can be used to derived test cases and two methods that use this approach are usage-based testing and oper- ation profile testing. Furthermore, there are techniques that are automated and use UML dia- grams. Usage-based testing: B. Regnell, P. Runeson and C. Wohlin, “Towards Integration of Use Case Modelling and Usage-Based Testing”, Journal of Systems and Software, 50(2):117-130, 2000. Operational profile testing: J. Musa, “Operational Profiles in Software-Reliability Engi- neering”, IEEE Software, 10:(2):14-32, 1993. Specification-based testing: J. Offutt, S. Liu, A. Abdurazik and P. Ammann, “Generating Test Data from State-based Specifications”, Software Testing, Verification and Reliability, 13(1):25–53, 2003. 8.12 How can process simulation be used for project planning and learn- ing? Software process simulation is used in a variety of fields in software engineering, for exam- ple, to support process improvement, software project management training, decision support 6
  7. 7. and to understand empirical research results. This project can either be practical by focusing on developing a simulation model of a software development/testing project with the pupose to assist in the project planning, or it could be focused on evaluating existing models’ useful- ness in the decision-making. Simulation models for decision support: Donzelli, P., “A Decision Support System for Software Project Management”, IEEE Software, 23(4): 67-75, 2006. Why using process simulation models: Kellner, M., Madachy, R., Raffo, D., "Software Process Simulation Modeling: Why? What? How?", Journal of Systems and Software, 46(2- 3):91-105, 1999. 8.13 How can tools for software testing be evaluated? Tools are important in order to implement software testing effectively in an software organization. However, although tools are needed, they do not solve the problems in the test- ing phase. Several tools exist and they have to be evaluated before a software organisation pur- chases one. Evaluation of testing tools: Poston, R. M. and Sexton, M. P., “Evaluating and Selecting Testing Tools”, IEEE Software, pp. 33-42, 1992. 8.14 How can predictions of the software development project be useful during the testing process? The project management is often interested in predicting the outcome of the ongoing soft- ware development projects. Both predictions of how much resources that will be required to deliver on time, and the quality of the delivered software product are valuable for the plan- ning process. What metrics are useful for predictions during the test process, and how can these be used? One model to use for the whole development project is COCOMO, but other approaches could be applied when focusing on the test process. COCOMO and other models: Boehm, B., Abts, C., Chulani, S., “Software Development Cost Estimation Approaches – A Survey”, Annals of Software Engineering, 10(1-4):177-205, 2000. What is needed for making measurements: Schneidewind, N. E., “Body of Knowledge for Software Quality Measurement”, IEEE Computer, 35(2):77-83, 2002. 8.15 How can web applications be tested? The quality of web applications mostly relies on the skill of the individual developer. In order to meet the quality decided by end-users, the processes of developing and testing web applications need to be formalized. Web application testing: Di Lucca G. and Fasolino, A., “Testing Web-based Applications: The State of the Art and Future Trends”, Information and Software Technology, 48(12):1172- 1186, 2006. 8.16 How do you effectively regression test your system? Regression testing is the process of validating modified software to detect whether new errors have been introduced into previously tested code. Because of time and resource con- traints for testing, regression test selection techniques have been proposed, to reduce the expenses. Regression testing: Rothermel, G. and Harrold, M.J., “Analyzing regression test selection techniques”, IEEE Transaction on Software Engineering, 22(8):529-551, 1996. 7
  8. 8. 9. Keywords in Software Testing The below keywords can be used if you want to decide on a subject not specified above. Another source that is appropriate to use is the Software Engineering Book of Knowledge. The latest version can be downloaded on Black-box testing Gray-box testing White-box testing Equivalence partitioning Reference models from code- Boundary value analysis based testing (flow-graph, call- Decision table graph) Finite-state machine-based Control flow-based criteria Testing from formal specification Data-flow based criteria Error guessing Mutation testing Random testing Coverage measures Operational profile SRET Application testing Testing in Software Process Test Methods Object-oriented testing Connection between require- Statistical usage testing (usage- Component-based ments and testing based) Web-based Architecture Specification-based software GUI testing Testability testing Testing of concurrent programs Code-based, Fault-based Protocol conformance testing Testing from formal specifica- Testing of distributed systems tion Testing of real-time systems Random testing Testing of scientific software Testing metrics and measure- Objectives of Testing Evaluation of the testing ments Acceptance testing Coverage measures Attitudes and egoless program- Installation Fault seeding ming Alpha, beta Mutation score Test process Conformance, functional, correct- Comparison and relative effec- Test documentation and test- ness tiveness of different techniques wares Regression Stopping criteria Test organisation vs. company Performance Size Stress Cost/effort estimation and Back-to-back other metrics Recovery Termination and stop criterion Configuration Test reuse and test patterns Usability Empirical methods in software Quality models and software Estimations in software testing testing testing Reliability Compare testing techniques CMM Reliability growth models, Compare inspection and testing SPICE types and classification of faults Testing with SQA Remaining faults and faults density Testing and certification Testing on the management Inspections level Testing targets Process Unit Reading techniques Integration System Alpha Beta 8
  9. 9. Automated software testing Simulation of the testing proc- Testing connected to develop- (tools) ess ment process Tools for different techniques XP Cleanroom Risk-based 10. Journals and Conferences Below, there are some journals and conferences listed which publish results of software testing research. Journals Conferences •ACM SIGSOFT software engineering notes / Soft- •International Conference on Software Engi- ware engineering notes neering •ACM transactions on software engineering and •International Conference & Workshop on the methodology Engineering of Computer-Based Systems •Automated Software Engineering •International Symposium on Empirical Soft- •Empirical Software Engineering ware Engineering •IEEE Software •International Symposium on Software Metrics •Information and Software Technology •International Conference on Automated Soft- •Journal of Systems and Software ware Engineering •Software - Concepts & Tools •International Symposium on Software Reliabil- •Software Engineering Journal ity Engineering •IEEE Transactions on Software Engineering •Software Engineering. IEE Proceedings •Software Quality Journal •Software Testing, Verification and Reliability 11. Test-Related References Below, there are references to test and process-related conferences. These are almost the same references as in Appendix I in the book by I. Burnstein. Software Testing: Related Conferences Test-Oriented Web Sites • • • • • • • • • • Software Process and Quality Information • • • • • • • • • • • • • • 9