ICTSS 2010 - Iterative Software Testing Process for Scrum and Waterfall Projects

Uploaded on

Experience in run an iterative and incremental test process for different projects platform

Experience in run an iterative and incremental test process for different projects platform

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. 22nd IFIP International Conference on Testing Software and Systems – ICTSS 2010QUALITY Federal University of Amazonas - UFAM VALIDATION Nokia Technology Institute - INdT Bug MANTIS INdT Iterative Software Testing Process for Scrum and Waterfall Projects with Open Source Testing Tools AGILITY Experience SELENIUM Verification TestLink Eliane Collins & Vicente Lucena SCRUM Natal – RN , Nov 2010
  • 2. Summary 1. Introduction 2. Environments Context 3. Software Testing Process 4. Execution Results 5. ConclusionCompany Confidential
  • 3. IntroductionCompany Confidential 3
  • 4. INTRODUCTION Different projects with different development process request Test activities. The Software Testing process must be agile for all projects. Project 1 used Scrum. It is an agile development process focusing on teamwork . The test activities must follow the scrum and they may be executed efficiently and fast .Company Confidential 4
  • 5. INTRODUCTION The Project 2 used Waterfall software development process. The system test activities must be done for the whole system in the test phase. Software Testing automation is one of the main approaches that has been applied to decrease testing costs and time. Test Case Test Case Test Case Test Case Test Planning Test Planning Defect Report Defect Report Specification Specification Execution Execution Sequential software testing processCompany Confidential 5
  • 6. Environments ContextCompany Confidential 6
  • 7. Environments Context Nokia Technology Institute – INdT It is an independent and nonprofit institute committed to conducting research and development of technological solutions through application development, new technologies and concepts. The INdT area where the experiment was designed is called Product Manufacturing Technologies (PMT).Company Confidential 7
  • 8. Environments Context Project 1 A system to provide an online customer satisfaction survey result to the development team by e-mail. The system registers the projects, leaders, customers, team members and the customers’ answers. The system sends the reminders to customers online to answer the questions on the survey and forward the results to the development team. The project was developed under the Scrum process. Web Platform: MVC Architecture, Ruby and Java languages, Rails framework and MySQL database.Company Confidential 8
  • 9. Environments Context Project 1 - Scenario In the schedule there wasn’t any time planned for test activities. No formal Software Testing process was done. Just one test analyst. Short time for test execution. No financial resources available for this activity.Company Confidential 9
  • 10. Environments Context Project 2 A software project to provide a graphic analysis of factory production tests in real time in order to advise professionals to support manufacturing on the quality trends for production line failures. The project development used the Waterfall process approach. Software platform : Client-Server, Java Swing, IDE Eclipse environment and Oracle database. The System has twelve screens, where six of them were used to show and analyze graphic charts and the other six had the data inputs, reports and user configuration.Company Confidential 10
  • 11. Environments Context Project 2 - Scenario In the schedule there wasn’t any time planned for test activities. No formal Software Testing process was done. Outdated documentation of the system requirements. Functionalities implemented but not tested No financial resources available for this activity. 4 weeks for a tester and a test analyst to do all test activities for the whole systemCompany Confidential 11
  • 12. Software Testing ProcessCompany Confidential 12
  • 13. Software Testing Process Iterative Software Test Process Test Inputs: Project Plan, Requirements or Backlog, Development Environment. Test Deliveries: Test Plan, Test Cases Specification, Test Case Execution Report and Summary Defect Report. Functional Test Automation for basic test cases. Exploratory Testing. Create Test Cases derived from Backlog stories or Requirements. The tester was a scrum team member. Interaction with the Development TeamCompany Confidential 13
  • 14. Software Testing Process Iterative Software Test Process Start Environment Environment Criation/ Update Criation/ Update Planning Config. Test Cases Test Cases Config. NO Summary Summary End of End of Automated Automated Defect Report Defect Report Iterations? Iterations? YES Test Scripts Test Scripts End Regression Test Test Tests Execution Execution New System New System Defect Report Defect Report Version Version Report Report Generation GenerationCompany Confidential 14
  • 15. Software Testing Process Open Source Automated Test Tools TestLink to automate the managing, planning, test cases specification and test execution reports. Mantis for Defect Tracking tool, friendly user interface to interact with developers and can be integrated with TestLink.Company Confidential 15
  • 16. Software Testing Process TestLink: Specification: Reports:Company Confidential 16
  • 17. Software Testing Process Web Functional Open Source Automated Test Tools Selenium IDE and Selenium Core. Aproach rec-and-play for Mozila Firefox and Internet Explorer browsers. Test scripts and Test Suite created in HTML code.Company Confidential 17
  • 18. Software Testing Process Automated Testing Tools: For functional test automation a new open source tool was chosen to test the java swing desktop application: MARATHON. Approach rec-and-play recording the user actions creating test scripts in Jython language and run them later in the tool .Company Confidential 18
  • 19. Execution and ResultsCompany Confidential 19
  • 20. Execution– Project 1 Sprint Test Activities in First week: Test Environment Configuration Automated test scripts creation. The test scripts were implemented in the browser Mozilla Firefox. These scripts were edited and gathered in a test suite, and reused in other test cases; the test cases were done according to the acceptance criteria emphasizing test cases creation for exceptions. The system analyst should review the test cases to ensure coverage of all functionalities;Company Confidential 20
  • 21. Execution– Project 1 Sprint Test Activities in Second week: The test scripts were executed for Firefox and Internet Explorer browsers. Manual Exploratory tests were also executed by the tester. At Mantis and TestLink, the defects were registered, described and assigned to the developer who was notified by e-mail. The Reports were automatically generated by TestLink. When the defects had been fixed, the entire test suite was executed by Selenium doing the regression test. It ensures that another part of the system was not affected by code changes. If the acceptance criteria were met, the same activities of the process will begin again for the next Sprint.Company Confidential 21
  • 22. Execution– Project 2 Goal: To ensure in a short time the main features stressed. Same test process for the scrum, but with some adaptations: A meeting with the development team to understand the project and verify the requirements documentation updates. The testers asked for the prioritized list of the project features where the most important features had higher priority (As a backlog list). The test iterations were planned having the most priority features being tested first. Test Activities executed in 4 iterations (4 weeks to test all applications).Company Confidential 22
  • 23. Execution– Project 2 Test Activities for iteration: Plan the test cases for the priority features in the TestLink Tool. Automated test scripts creation for test cases in Marathon. Execute the automated tests and the exploratory manual tests too. The defects found are registered in Testlink and Mantis. When the developer fixes the defect, the code changes, so all automated tests are executed to do the regression tests. The Automated reports are generated and sent to the Project leader. The development team releases another system version and the same activities of the process will begin again for the next test iteration.Company Confidential 23
  • 24. Results General Results: 100% of the functionalities from backlog were covered in Project 1. 65% of the system defects were from browser incompatibilities in Project 1. 77 defects were registered in Mantis, where 30% were interface defects and 70% were functional defects in Project 2. The regression tests showed defects in the functionalities that were considered done. Good evaluation of the projects.Company Confidential 24
  • 25. Results Waterfall Results Comparison Test Test Process Coverage Regression cicles Defects Cases Manual and 65% 40 0 27 Sequential Iterative and 100% 75 3 77 Automated ProcessCompany Confidential 25
  • 26. ConclusionCompany Confidential 26
  • 27. Conclusion Lessons learned: Plan Test Activities and engage the test team from the beginning of the project. Automated test scripts had to be reworked and updated with every backlog change for each sprint. The scrum developer team had to estimate more time to defects fixing because of the large amount of problems found during the regression tests. Exploratory tests found the usability and performance problems. The test script creation has become faster because of the reuse of the script code.Company Confidential 27
  • 28. Conclusion The test activities broken into iterations made it possible to stress the most important features, making the system attend to what was requested by the client. Search for new automation tools for other development platforms for further projects in the company. Search for new techniques to improve agile test process as Test based on Models to reduce the rework.Company Confidential 28
  • 29. THANK YOU!Company Confidential 29