T-76.613 Software Testing and Quality Assurance


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

T-76.613 Software Testing and Quality Assurance

  1. 1. T-76.5613 Software Testing and Quality Assurance 3.10.2007 Testing in Agile Software Development Juha Itkonen SoberIT HELSINKI UNIVERSITY OF TECHNOLOGY
  2. 2. Overview Agile Testing and Testing in Agile Software Development Challenges of Testing in Agile Development How Agile Quality Practices Work? Agile Unit Testing and TDD Agile Acceptance Testing with FIT Juha Itkonen 2 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  3. 3. Process models from waterfall to agile Functionality Requirements Design Time Implementation Testing Waterfall Incremental, Agile - XP e.g. RUP Redrawn from Beck. Embracing change with Extreme Programming, 1999. Juha Itkonen 3 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  4. 4. V-model Acceptance Requirements testing Functional System specification testing Architecture Integration design testing st Bu Te ild Module Unit design testing Coding Intuitive and easy to explain Quite adaptable to various situations Makes a good model for training people Beats the code-and-fix approach on any larger project Juha Itkonen 4 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  5. 5. Problems in applying V-model Document driven Relies on the existence, accuracy, completeness, and timeliness of development documentation Communicates change poorly Ignores the fact that, in practice, software is developed in a series of handoffs Does not fit into modern iterative development processes Juha Itkonen 5 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  6. 6. Agile Software Development Manifesto In agile development we value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan + 12 more detailed principles How do these values affect testing? http://www.agilemanifesto.org/ Juha Itkonen 6 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  7. 7. Plan-Driven vs. Agile Planned result Wanted result ? Juha Itkonen 7 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  8. 8. Agile Software Development – What About Testing? Agility in software development Working software is most important deliverable Responding to change – not resisting change Tight development and release cycle Iterative and incremental process Fast feedback, frequent adjustments Efficient collaboration and communication Within the team and with the customer Agility from the testing viewpoint Places many challenges Testing must also be agile in agile development You can learn a lot from the agile methods Juha Itkonen 8 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  9. 9. Two views of agile testing eXtreme Testing Exploratory Testing Automated unit testing Utilizes professional testers’ Developers write tests skills and experience Test first development Optimized to find bugs Daily builds with unit tests Minimizing time spent on always 100% pass documentation Functional testing Continually adjusting plans, Customer-owned re-focusing on the most Comprehensive promising risk areas Repeatable Following hunches Automatic Freedom, flexibility and fun Timely for testers Public Focus on automated verification – Focus on manual validation – enabling agile software making testing activities development agile Juha Itkonen 9 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  10. 10. Some Agile Principles Satisfy the customer through The most efficient and early and continuous effective method of conveying delivery of valuable software. information to and within a Working software is the development team is face-to- primary measure of progress. face conversation. Deliver working software Business people and frequently, from a couple of developers must work weeks to a couple of together daily throughout the months. project. Welcome changing Simplicity--the art of requirements, even late in maximizing the amount of development. work not done--is essential. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. http://www.agilemanifesto.org/ Juha Itkonen 10 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  11. 11. Challenges that agile principles place on testing Agile Principle Challenge Frequent deliveries of - Short time for testing in each cycle valuable software - Testing must be time-boxed, too Responding to change even -Testing cannot be designed late in the development beforehand based on specifications - Tests must not prevent change Relying on face-to-face - Getting developers and business communication people actively involved in testing Working software is the - Quality information is required early primary measure of progress and frequently throughout development Simplicity is essential - Testing practices get easily dropped for simplicity’s sake Juha Itkonen 11 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  12. 12. Contradictions with traditional testing principles Testing principle Contradicting practices in agile methods Independency of - Developers write tests for their own code testing and - Developers concentrate on constructive QA destructive attitude practices, i.e., building quality into the product and showing that features work Testing requires - Developers do the testing as part of the specific skills development - The customer has a very important and collaborative role and a lot of responsibility for the resulting quality Oracle problem - Relying on automated tests to reveal defects Evaluating achieved - Confidence in quality through tracking quality conformance to a set of good practices Juha Itkonen 12 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  13. 13. How Do the Agile Quality Practices Work? Juha Itkonen 13 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  14. 14. Organizing testing (waterfall vs. agile) Waterfall model Agile models (XP) Programmers Tester Customer Programmer Programmer Idea: Testing in collaboration Testers Juha Itkonen 14 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  15. 15. Quality Assurance in Agile Development Plan-driven quality assurance does not work in agile context There is no V-model and waterfall The role of documentation and specifications is secondary The rhythm of development is fast and tight Roles and responsibilities are assigned differently Cornerstones of agile quality assurance Constant and tight rhythm Collaboration and communication Rigorous low level (quality) practices Test-Driven Software Engineering Juha Itkonen 15 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  16. 16. Rhythm Continuous unit-level integration and test cycle Short incremental release cycles Completing features in short cycles and small increments Complete, integrated, tested Building quality and tracking the quality level in short cycles Iteration Heartbeat Release t Juha Itkonen 16 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  17. 17. Challenges - Short time for testing in each cycle - Testing must be time-boxed, too - Testing cannot be designed beforehand 1 Rhythm - Quality information is required early and frequently – viewpoint of testing - Testing practices get easily dropped Feedback is faster and more frequent between development and testing from customers and users Defects and issues are found earlier Feel the rhythm - tackle tasks before they bunch up Testing features in small increments is easier Less new defects to hinder testing Less open defects to distract development Real quality level and real progress better visible Risks of testing and deployment are smaller Easier to estimate and plan testing phase (or tasks) Juha Itkonen 17 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  18. 18. 2 Communication Communication in agile development does not work in a traditional way Traditionally others communicate to testers with requirements and specification documents and testers respond with defect reports and test reports “So we can let free of the illusion that documents will save us. We can view them as they are: interesting texts, partly fictional, often useful.” - Brian Marick Juha Itkonen 18 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  19. 19. Collaboration Collaboration is effective way of sharing information, knowledge, and understanding Improves common understanding and makes communication easy and efficient Less guesswork and proceeding based on wrong assumptions Information moves faster and with less overhead Especially effective in the home ground of agile development When developing new, unfamiliar things When requirements are vague and changing When customer is easily available for collaboration Juha Itkonen 19 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  20. 20. Communication and collaboration – viewpoint of testing There are testers also in agile teams Tester’s role is to help the development team to create good enough quality software Testers bring expertise and skills as well as the testers destructive viewpoint in the team By working together with developers and the customer the reliance on comprehensive documentation is not needed Challenges - Tests must not prevent change - Getting developers and business people actively involved in testing - Developers write tests for their own code as part of the development - Developers concentrate on constructive QA practices - The customer has a lot of responsibility for the resulting quality - Relying automated tests to reveal defects Juha Itkonen 20 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  21. 21. 3 Low Level Quality Practices Quality practices are tightly integrated into the daily heartbeat rhythm of the development Testing each individual feature as part of its development Building quality using constructive practices Quality is part of development – not delayed any further E.g. automated unit tests, pair-programming, continuous integrations, refactoring, … Goal is to achieve high quality level before the code leaves the developer’s desk Juha Itkonen 21 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  22. 22. Some practices of an agile developer Integrate early integrate Keep it simple often Don’t fall for the quick hack Keep your project releasable Write code to be clear not at all times clever Automate acceptance testing Warnings are really errors Use automated unit tests Use it before you build it – Provide useful error TDD messages Write code in short Be a mentor edit/build/test cycles Share code only when it’s Emphasize collective ready ownership of code Review code Keep a solutions log Keep others informed of the … status of your work … V. Subramaniam & A. Hunt, “Practices of an Agile Developer”, The Pragmatic Bookshelf, 2006. Juha Itkonen 22 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  23. 23. Challenges - Short time for testing in each cycle Low level quality practices - Testing must be time-boxed, too - Tests must not prevent change – viewpoint of testing - Testing practices get easily dropped Technical quality of the software is high Less trivial defects – easier to test hard issues Boring, routine, checking procedures are automated Testers and developers have a common goal of creating high quality software Testers help developers <–> developers help testers Testers can help developers Pair tester, who helps a developer to test each functionality Tester writes tests for the current functionality under development Tests help developer in implementation and provide fast feedback already during implementation Juha Itkonen 23 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  24. 24. 4 Test-Driven Software Engineering Use it before you build it! Agile test-driven approach can be applied also on the level of customer requirements and acceptance testing Tests can help programmers, testers, and business people (customer) communicate Tests can by used to guide development work Tests act as the true measure of real progress Juha Itkonen 24 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  25. 25. Test-Driven Approach Testers, programmers, and customer design together good tests based on the customer requirements How, exactly, the features and core business logic should work Major problems in the implementation (risks) Documenting the important details and potential problem areas beforehand Tests guide the programmers in implementation work What exactly must be done; how special cases should behave; when the feature is done What are the risks of this feature; which issues to pay attention to 30 25 Tracking the amount of passing tests 20 15 communicates the progress of development 10 5 0 1 2 3 4 5 6 7 8 9 10 As agile values and principles state Juha Itkonen 25 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  26. 26. New Roles for Tests in TDSE In Test-Driven Software Engineering tests have a clear role in the early requirements and design phases Tests help developers, testers, and customer to communicate Common vocabulary and understanding Tests document and concretize the details of requirements Concrete examples of the functions and logic Special cases and exceptions Easy to extend and update during the development Tests guide the implementation work Fast feedback to developers when automated Tests provide an honest and agile metric of true progress in terms of working software Juha Itkonen 26 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  27. 27. Agile Unit Testing and Test-Driven Development Juha Itkonen 27 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  28. 28. No manual tests! Chapter titled ”Manual Tests” in Testing Extreme Programming (Crispin & House 2003) ”No manual tests.” Juha Itkonen 28 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  29. 29. Unit testing ”A unit is the smallest possible testable software component” Various interpretations exists Procedure / function Class / object / method Small-sized component Goal is to ensure that software units are functioning as the developer intended E.g. a function returns correct value when exercised with valid parameters E.g. function behaves correctly when given an invalid input Automated unit test is a piece of code that exercises another piece of code Juha Itkonen 29 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  30. 30. Document driven vs. agile unit testing ”Testing should be carried out by a group that is independent of the development group” Course book presents a unit testing viewpoint which appreciates this principle Unit testing should be a separate task conducted after the development is done Unit test design is based on written specifications Modern ideas of unit testing represent slightly different viewpoint Unit tests are written by developer Automation is essential part of unit testing Unit tests might even be written before the actual program code This lecture is about agile unit testing Read the course book Ch. 6 and understand the difference The same tools are used by both disciplines Juha Itkonen 30 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  31. 31. Unit testing - a vital testing level Writing software is difficult and every programmer makes mistakes There will be defects in all software you write Unit testing lays the foundation for good quality software Unit testing lays foundation for effective testing on higher levels Defects that slip to upper levels (integration, system or acceptance testing) More difficult to spot Time consuming to debug Uses organizational resources — Writing, prioritizing and managing test and defect reports Bad unit level quality slows down integration and system testing Juha Itkonen 31 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  32. 32. How do you know that your code works correctly? Do you just trust your luck? Do you use “printf” or ”System.out.println()” outputs to see what happens in your code? Do you use debugger to find out how your code really works? Do you ask your friend/colleague to check through your code? Do you create small (temporary) widget to access the piece of code directly runtime? Do you use some ugly, cumbersome, time consuming practice that leaves trash into your code and is thrown up to the user’s face when you least expect it… “Getting here is not possible, something is really messed up in this code!!!” “Fatal error 9981: Assertion failed : av_d_dx > 0” Juha Itkonen 32 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  33. 33. Agile unit testing Test-Driven Development Writing and running tests is integral part of daily development rhythm Tests are developers’ QA and programming tool All unit tests are automated Automated tests are run as often as possible Gather all your tests to a test suite and use it for regression testing Integration, nightly, milestone During development; after every change; new code or bug fix Juha Itkonen 33 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  34. 34. Test-Driven Development Writing test before code to be tested ”a little test, a little code, a little test, a little code, ...” Tests are added gradually during implementation – not in large lump afterwards Process of writing tests drives low-level design and programming Tests specify what code should do Tests validate that code does what it should Actually, a design and coding practice One of the core practices of Extreme Programming Developers have been applying TDD for several decades Juha Itkonen 34 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  35. 35. TDD episode (1/2) Proceeds step by step Write a test. Write test Design and implement just enough to make the test Write code Pass pass. Fail Repeat. Testing and coding alternate in Run tests very small steps Duration of one cycle should be a few minutes Small steps – difficult to make mistake Juha Itkonen 35 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  36. 36. TDD episode (2/2) Episode is over when you can’t write a failing test anymore Write test for each requirement of the code Write test for each point that can possibly break One cycle at a time Don’t write a bunch of tests at once Refactor if you ever see the chance to make the design simpler Run all tests after finishing episode Make sure you did not brake anything else Juha Itkonen 36 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  37. 37. Test-Driven Development – claimed benefits (1/2) Close feedback loop TDD cycle is very short – know if code is working right after you programmed it Task-orientation Encourage programmer to decompose problem into manageable programming tasks Helps to maintain focus Low-level design Programmer is forced to think which classes and methods to create how they are used, how to name them, what arguments are needed, what is the return value Juha Itkonen 37 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  38. 38. Test-Driven Development – claimed benefits (2/2) Results better code If the test is too hard to write, the code being tested is too complicated Results testable code Programmer can’t end up with code that cannot be tested Effect on quality Testing becomes part of the development process and gets done Side effect of TDD is that code gets thoroughly unit tested Juha Itkonen 38 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  39. 39. Is Test-Driven Development testing? What is the goal and attitude of a developer? Do the tests reveal defects? What are the tests based on? How is the quality of the unit tests assured? What information the tests actually produce? Juha Itkonen 39 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  40. 40. Unit testing frameworks Convenient frameworks for writing unit tests as part of the programming task In the same environment with the same language Tools of xUnit family are the most famous ones JUnit CppUnit NUnit … The unit testing frameworks provide a convenient way of writing and executing unit tests Standard and convenient way to write tests, log results, group tests into suites Juha Itkonen 40 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT
  41. 41. Basic building blocks of a unit testing framework A way of writing test cases for each class, method, function, … Test case is a function or a class – a piece of code A way of managing suites of test cases A data structure that is used to group test cases into logical collections Assert methods/functions that enable easy logging and reporting test results The framework provides standard functions that are used to check the test results Automated result logging and reporting provided by the framework A way of executing test cases and reporting the results GUI and command-line based execution tools (Integrated to IDEs) Convenient report generators (Xml, html) Initializing and cleaning up The framework ensures that init and clean-up functions are properly executed Initialization (or set up) function to initialize the variables, test data, etc. before executing each test case Clean up (or tear down) function to release the used resources and clean up after executing each test case Juha Itkonen 41 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT