Published on

1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. A CASE STUDY ON SOFTWARE TESTING AUTOMATION Katja Karhu katja.karhu@lut.fi Lappeenranta University of Technology Department of Information Technology Software Engineering Research Group 1 OVERVIEW • Software Testing Automation • Research Problem • Research Process • Results • Conclusions • References 2 1
  2. 2. SOFTWARE TESTING AUTOMATION • Automated software testing – The automation of software testing activities, to include the development and execution of test scripts so as to verify testing requirements, using an automated test tool – Is most efficient when test scripts/cases are repeated • Motivation for automating software testing: - Manual testing is time consuming - Increased efficiency of regression testing (Dustin et al, 1999) 3 THE RESEARCH PROBLEM 1/2 • In the survey conducted previously in the research project: – The respondents evaluated personnel costs as the highest cost item (29 organizations of 30) in software testing. (Taipale et al., 2006) – Testing automation costs were evaluated as the second highest cost item (27 organizations of 29). (Taipale et al., 2006) • Project website: http://www.it.lut.fi/project/anti/ 4 2
  3. 3. THE RESEARCH PROBLEM 1/2 - When is it suitable to automate software testing? - How widely is testing automation used in our case organizations? - Why / Why not? 5 RESEARCH PROCESS • We chose to use the grounded theory research method outlined by Glaser and Strauss (1967) and later extended by Strauss and Corbin (1990) • Observations on the benefits of software testing automation in different types of organizations has not been systematically covered in the literature • We followed the process of building a theory from case study research described by Eisenhardt (1989). 6 3
  4. 4. RESEARCH PROCESS: Data Collection 1/3 • The population of the survey consisted of 30 organizational units (OUs). • From this sample, five were selected for the case study • An organizational unit deploys one or more processes that have a coherent process context and operates within a coherent set of business goals (ISO/IEC 15504-1). • An organizational unit is typically part of a larger organization, although in a small organization, the organizational unit may be the whole organization. • An OU was used as the assessment unit because it normalizes the effect of the company size 7 RESEARCH PROCESS: Data Collection 2/3 Company size1 / OUs Business Operation All 30 OUs including Automation or The average size of an OU cases from A to E telecommunication domain was 75 persons. Case A MES producer and integrator Large / International Software producer and Case B Small / National independent testing agency Process automation and Case C information management Large / International provider Case D Electronics manufacturer Large / International Case E Independent testing agency Small / National 8 4
  5. 5. RESEARCH PROCESS: Data Collection 3/3 Interview Case A Case B Case C Case D Case E Round 1. R&D and testing Testing Testing manager Testing manager Testing Structured/ manager manager R&D manager R&D manager manager Semi-structured R&D and testing 2. Testing Testing manager Testing manager Testing manager Semi-structured manager R&D manager manager Testing manager 3. Tester Tester Tester Tester Tester Semi-structured 4. Developer Developer Developer Developer Developer Semi-structured 9 RESEARCH PROCESS: Data analysis 1/3 • Grounded theory has three data analysis steps: – Open coding, where categories of the study were extracted from the data. • The seed categories were formed from the research question as well as the results of the pre-study and the survey – Axial coding, where connections between the categories were identified. – Selective coding, where the core category was identified, described and related to other categories. 10 5
  6. 6. RESEARCH PROCESS: Data analysis 2/3 • The objective of the axial coding was to further develop categories, dimensions, and causal conditions (connections between categories). • The categories were further developed by defining their dimensions. • The dimensions represent the locations of the property or the attribute of a category along a continuum. • We used the tactic of selecting dimensions, and then looking for within-group similarities coupled with inter-group differences. 11 RESEARCH PROCESS: Data analysis 3/3 • The objective of the selective coding was to identify the core category, systematically relate it to other categories, and generate the theory. • According to Strauss and Corbin (1990) the core category is sometimes one of the existing categories, or a central phenomenon, which cannot be expressed as a single category • In this study the core category was software testing automation 12 6
  7. 7. RESEARCH PROCESS: Categories 1/5 • Business orientation – The ratio between products and services of the OU’s turnover. – The construct was derived from Sommerville (1995) who divides software products into two broad classes, generic products and customized products. – We complemented and widened Sommerville’s (1995) classification with a finer granulation and added a purely service- oriented dimension (consulting and subcontracting). • Tested products – What types of products are developed/tested in the OUs 13 RESEARCH PROCESS: Categories 2/5 • The knowledge management strategy describes whether an OU was more inclined towards the personalization or the codification strategy. According to Hansen et al. (1999) there are two primary knowledge management strategies: – Personalization: • Emphasis on tacit knowledge, such as domain knowledge • Shared by person-to-person contacts – Codification • Emphasis on explicit knowledge, such as documentation, code, test cases, test plans • Easy to access and transfer • The amount of domain knowledge that was needed in testing is described by the domain knowledge category 14 7
  8. 8. RESEARCH PROCESS: Categories 3/5 • Use of testing automation describes what type of automated testing is currently utilized in the OUs • Benefits of testing automation describes the benefits that testing automation would provide according to the interviewees • Reusability of testing automation describes whether there is a possibility to reuse test cases or automated testing systems 15 RESEARCH PROCESS: Categories 4/5 Case OU Case A Case B Case C Case D Case E Category Business Half service and Mostly service Two thirds service Purely product Purely service orientation half product oriented. and one third product oriented. oriented. oriented. oriented. Knowledge Codification less Codification more Codification less Codification Codification more management emphasized, emphasized in the emphasized, growing. more emphasized in the strategy growing. Domain customers’ projects. Domain knowledge emphasized. customers’ knowledge Domain knowledge emphasized in testing Domain projects. Domain emphasized in more emphasized customized systems. knowledge less knowledge testing customized when testing own emphasized. emphasized. systems. software. Knowledge of the Testing of Testing tasks customers’ Testing activities automation and are clearly software Testing MES Domain require only general information specified and do development systems requires knowledge level knowledge of the management systems not require processes is viewed domain knowledge system to be tested. requires domain specific domain as more important knowledge knowledge than domain knowledge. 8
  9. 9. RESEARCH PROCESS: Categories 5/5 Case OU Case A Case B Case C Case D Case E Category Use of Regression testing of Unit testing when Some tests Testing automation Not used Testing the development testing own products. automated widely used Automation environment Not utilized in providing testing services Benefits of Testing Shorter testing times Some tasks are Improves the Helps to stay in the Testing environments enable in own products worth automating quality of testing schedule Automation testing in own premises Reusability Repeatable test cases Reuse in own Reusable parts in the Reuse is a Very little chances if Testing in integration testing products, not in testing environment significant factor for reuse Automation providing testing in development services and testing RESULTS 1/3 • Three distinct groups were identified among the – Product oriented software development • Testing automation widely utilized – Development of customized systems • Some testing automation, non-systematic – Testing service providers • No significant utilization of testing automation 18 9
  10. 10. RESULTS 2/3 Product-Oriented SW Development of Customized Category Testing Service Providers Development Systems OUs in the group D A,C B,E Business Orientation Products Both products and services Mainly services Tested products Small and mainly standardized Large and complex Depends on the customer Knowledge management Personalization with codification Codification with Codification strategy (overlapping strategies) personalization (little overlap) Domain Knowledge in Less need for domain Domain knowledge important Less need for domain knowledge Testing knowledge Use of Testing Widely used Non-systematic Not used Automation Benefits of Testing Testing environments enable Automation Quality improvement testing without going to the Shorter testing times customers premises Reusability of Testing Automation High degree of reusability Test cases Less reuse RESULTS 3/3 - How the OUs plan to develop testing automation: - Case A - Automating service applications may be possible - Case B - Further development of unit testing regarding their own product development. Fully automated testing is not seen possible at least in providing testing services - Case C - Development of testing automation would require more work than testing without automation. Lack of motivation and resources - Case D - Automation and testing tools are constantly being developed - Case E - No need for developing testing automation at the moment 20 10
  11. 11. DISCUSSION AND CONCLUSIONS 1/2 • When to automate software testing? – Codification is the primary knowledge management strategy – Systems are standardized and independent – Need for domain knowledge in testing is low – Test cases and testing environments can be reused 21 DISCUSSION AND CONCLUSIONS 2/2 • Organizations developing generic and independent products are more inclined towards the codification strategy – More potential for codification (Cowan et al.) – Codification enables systematic reuse (Cowan et al.; Hansen et al.) • Organizations making customized systems are more inclined to the personalization strategy – Tacit domain knowledge important – Reusing test cases becomes more difficult • Testing service providers need to follow their customers’ knowledge management strategies – Because each customer has a different system to be tested, systematic reuse between different customer projects is challenging (Hansen et al.) 22 11
  12. 12. REFERENCES • R. Cowan, P.A. David, and D. Foray, The Explicit Economics of Knowledge Codification and Tacitness, Industrial and Corporate Change, 9(2): p. 211-253. • E. Dustin, J. Rashka, J. Paul: Automated Software Testing: Introduction, Management, and Performance, Addison-Wesley, Boston, 1999. • K.M. Eisenhardt, Building Theories from Case Study Research, Academy of Management Review, 14(4): p. 532-550. • B. Glaser and A.L. Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research., Aldine, Chicago, 1967. • M.T. Hansen, N. Nohria, and T. Tierney, What's Your Strategy for Managing Knowledge?, Harvard Business Review, 77(2): p. 106-116. • A. Strauss and J. Corbin, Basics of Qualitative Research: Grounded Theory Procedures and Techniques, SAGE Publications, Newbury Park, CA, 1990. • O. Taipale, K. Smolander, and H. Kälviäinen, A Survey on Software Testing, 6th International SPICE Conference on Software Process Improvement and Capability dEtermination (SPICE'2006), Luxembourg, 2006. 23 12