Testing for Agility: Bringing Testing into Everything

  • 1,009 views
Uploaded on

A testing focus can drive agility in every aspect of software development. Topics include: the cost of waiting to test, how waterfall can be transformed into agility with feedback, how testing can …

A testing focus can drive agility in every aspect of software development. Topics include: the cost of waiting to test, how waterfall can be transformed into agility with feedback, how testing can drive requirements discovery, testing your vision, testing during development, testing as part of integration and deployment, testing for customer acceptance, agile ratios of different types of tests, agile testing for bug fixes and maintenance, management concerns for agile testing and code coverage.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,009
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
32
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. CamillesCareer@gmail.com 1 Testing for Agility: Bringing Testing into Everything Camille Bell Agile Coach & Trainer CamillesCareer@gmail.com Twitter @agilecamille TDD User Story: Vetting Sighting In order to show confirmation of a sighting, As a US-CERT analyst, I want to mark a sighting a vetted • XYZ sighting is viewed by Charley US-CERT analyst • Charley marks sighting as vetted • Tracy from Treasury searches for sightings vetted today • Tracy sees XYZ as a confirmed sighting
  • 2. CamillesCareer@gmail.com 2 About Me •  Agile Coach, Trainer and Developer •  Over 4 decades of IT. All sorts of companies, domains, technologies & IT roles. •  Business Email: CamillesCareer@gmail.com •  Twitter: @agilecamille •  Slideshare: camille_bell
  • 3. CamillesCareer@gmail.com 3 • Waiting to test • Transforming Waterfall into Agility using Feedback • Agile testing and requirements • Agile testing and development • Agile testing and integration and deployment • Agile testing and acceptance • Agile testing ratios • Agile testing, bug fixes and maintenance work • Agile testing and management • Code Coverage Agenda
  • 4. CamillesCareer@gmail.com 4 The Trouble with Waiting to Test
  • 5. CamillesCareer@gmail.com 5 Agility puts more focus on testing to improve quality and time to market Which is great assuming 21st Century Agile testing and complementary practices
  • 6. CamillesCareer@gmail.com 6 But many shops bolt on Waterfall phased testing to Agile development While still better than no Agility, it is costly in time and money
  • 7. CamillesCareer@gmail.com 7 Release Testing vs. Continuous Testing Ref: ”Lean from the Trenches” by Henrick Kniberg
  • 8. CamillesCareer@gmail.com 8 End Testing vs. Continuous Testing Ref: ”Lean from the Trenches” by Henrick Kniberg
  • 9. CamillesCareer@gmail.com 9 • It doesn’t guarantee customer acceptance ! • It’s extremely labor intensive ! • It’s error prone ! • It’s often ambiguous ! • It doesn’t stop fixed bugs from reappearing in the code ! • Regression testing is a nightmare ! • It doesn’t prevent “but it works on my machine” thinking ! • It’s sooner, but still after the fact ! (code exists) • It kills your test team ! Some other Waterfall phased testing (there are many more)
  • 10. CamillesCareer@gmail.com 10 Agile Development practices turn Waterfall completely around with Feedback
  • 11. CamillesCareer@gmail.com 11 Design High Level Requirements Requirements Analysis Code Business Vision Deploy Traditional Phased Waterfall No feedback – Assumes perfection! Plan
  • 12. CamillesCareer@gmail.com 12 Code The Waterfall Ideal Writes All the Code and then Tests it Ref: “BDD in 5 minutes” video by Corey Haines "How test-driven development works" blog by J. B. Rainsberger
  • 13. CamillesCareer@gmail.com 13 Code bugs But that never works creating the code, test & fix cycle
  • 14. CamillesCareer@gmail.com 14 Code Writing the Test First Prevents Coding Bugs
  • 15. CamillesCareer@gmail.com 15 Code Test First Feedback Loop Short cycles make it work
  • 16. CamillesCareer@gmail.com 16 Code Test-First, while better isn’t enough Design untestable design
  • 17. CamillesCareer@gmail.com 17 Code The Test-Code Cycle can provide feedback to design Design
  • 18. CamillesCareer@gmail.com 18 Code Heavy refactoring of design is part of Test Driven Development (TDD) Design Again short cycles are key
  • 19. CamillesCareer@gmail.com 19 Code Design Requirements Analysis Now Test, Code & Design all provide continuing feedback for improvement, but …. Not what the customer really wants
  • 20. CamillesCareer@gmail.com 20 The Test Driven Development Cycle provides feedback to analysis Requirements Analysis Code Design
  • 21. CamillesCareer@gmail.com 21 Code Adding in Feature Analysis to TDD creates Behavior Driven Development (BDD) Design Short cycles again Requirements Analysis
  • 22. CamillesCareer@gmail.com 22 Code Design Requirements Analysis High Level Requirements Not exactly what the Market Needed (even though customer rep thought it was) Now Behavior Driven Development provides continuing feedback for improvement, but ….
  • 23. CamillesCareer@gmail.com 23 Code Design Requirements Analysis The Behavior Driven Development Cycle, when deployed to the field provides feedback to marketing Deployed High Level Requirements
  • 24. CamillesCareer@gmail.com 24 Continuous Deployment further improves feedback Code Design Requirements Analysis Deployed High Level Requirements
  • 25. CamillesCareer@gmail.com 25 But what if our initial vision was flawed? Code Design Requirements Analysis Deployed High Level Requirements Business Vision Product Concept wrong !!! Entire effort fails at great expense !!!
  • 26. CamillesCareer@gmail.com 26 Lean Startup’s Build-Measure-Learn feedback loop tests the ideas behind the business vision
  • 27. CamillesCareer@gmail.com 27 Agility Testing starts with Testable Agile Requirements
  • 28. CamillesCareer@gmail.com 28 Idea Test Cases Software Conversation Agile Requirements Evolve from Feedback Ref: Rachel Davies & Liz Sedley – “Agile Coaching”
  • 29. CamillesCareer@gmail.com 29 Customer Collaborates with Dev Team on Confirmation of Stories User Story: Vetting Sighting In order to show confirmation of a sighting, As a US-CERT analyst, I want to mark a sighting a vetted • XYZ sighting is viewed by Charley US-CERT analyst • Charley marks sighting as vetted • Tracy from Treasury searches for sightings vetted today • Tracy sees XYZ as a confirmed sighting Ref: Ron Jeffries 3 Cs of User Stories Card Confirmation Collaboration
  • 30. CamillesCareer@gmail.com 30 User Stories Confirmed Through Automated Tests During Iteration • XYZ sighting is viewed by Charley US-CERT analyst • Charley marks sighting as vetted • Tracy from Treasury searches for sighting vetted today • Tracy sees XYZ as a confirmed sighting User Story: Vetting Sighting In order to show confirmation of a sighting, As a US-CERT analyst, I want to mark a sighting a vetted describe ”Imports TEWI sightings" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe ”CERT analyst finds a recent TEWI sighting" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe ”Vets a sighting" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe ” Trusted partner finds un-vetted sighting" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe "Trusted partner finds vetted sighting" do context ”a vetted sighting should be clearly vetted" do before(:each) do new_sighting.build_from_TEWI() cert_user.new(‘default_cert’) new_sighting.vet(cert_user) new_partner.new(‘default_partner’) end it "should have a tag of ’vetted'" do new_sighting.tag.should_include == ’vetted' end it "should be viewable by partner" do new_sighting.accesable_by(new_partner).should == true end end
  • 31. CamillesCareer@gmail.com 31 Incrementally turn High Level Requirements (features, epics) into Tests
  • 32. CamillesCareer@gmail.com 32 In order to conduct banking when the bank is closed As a bank customer I want to use an ATM Why? Who? What? Why does your user want this? Your idea probably needs to be broken down into smaller stories. Who is this user? How is he/she different from other users. What exactly does your user want? Start with an idea for an App (like ATM banking) These ideas are called User Stories.
  • 33. CamillesCareer@gmail.com 33 Break Down Big Stories into Smaller User Stories In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM In order to deposit my checks when the bank is closed As a bank customer I want to deposit checks at the ATM In order to earn interest even when the bank is closed As a bank customer I want to transfer money from checking to saving at the ATM In order to not overdraw my account when the bank is closed As a bank customer I want to transfer money from savings to checking at the ATM
  • 34. CamillesCareer@gmail.com 34 Choose a Story (to become an automated test) In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM Feature: Cash Withdrawal In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM
  • 35. CamillesCareer@gmail.com 35 Define Some High Level Scenarios for that Story Feature: Cash Withdrawal In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM • Successful Withdrawal • Withdrawal Failed Because Cash Machine Doesn’t Have Dollar Bills • Withdrawal Failed Due to Insufficient Funds • Withdrawal Failed Because Account Closed
  • 36. CamillesCareer@gmail.com 36 • Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Choose a Scenario and Define the Steps in Testable Detail
  • 37. CamillesCareer@gmail.com 37 • Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Use Given, When, Then format. OK to add And and But for readability Test steps must be unambiguous. Quantities are especially testable.
  • 38. CamillesCareer@gmail.com 38 With Minor Changes it can become an Automated Test (Cucumber example) • Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Scenario: Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80
  • 39. CamillesCareer@gmail.com 39 Functional & Unit Tests Behavior Driven Development & Test Driven Development
  • 40. CamillesCareer@gmail.com 40 Behavior Driven Development (aka Specification by Example) •  Builds upon and formalizes the best practices of Test Driven Development •  Works outside-in •  Starts with a failing customer acceptance test •  Test describes needed behavior from a customer viewpoint •  Tests must be readable by anyone on team
  • 41. CamillesCareer@gmail.com 41 So To Understand BDD, First Understand TDD
  • 42. CamillesCareer@gmail.com 42 3 Rules of Test Driven Development 1.  You are not allowed to write any production code unless it is to make a failing unit test pass. 2.  You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures. 3.  You are not allowed to write any more production code than is sufficient to pass the one failing unit test. With tests you are in control; without tests you aren’t! Bob Martin - The Three Rules of TDD
  • 43. CamillesCareer@gmail.com 43 The Test Driven Development Cycle Write a failing test for new functionality Write just enough code to pass test Refactored code must also pass tests, no new functionality CamillesCareer@gmail.com 43
  • 44. CamillesCareer@gmail.com 44 JUnit 4 Tests for Movie List Story Starting Test An empty list should have a size of zero. import static org.junit.Assert.*; import org.junit.Test; public class MovieListTest { @Test public void testEmptyListSize() { MovieList emptyMovieList = new MovieList(); assertEquals("Size of movie list should be 0.", 0, emptyMovieList.size()); } } Ref: Java code examples from “Test-Driven Development: A Practical Guide” by David Astels
  • 45. CamillesCareer@gmail.com 45
  • 46. CamillesCareer@gmail.com 46 Class Code for Movie List Story Test An empty list should have a size of zero. public class MovieList { public int size() { return 0; } } Couldn't be simpler, but that is the point!
  • 47. CamillesCareer@gmail.com 47
  • 48. CamillesCareer@gmail.com 48 Old Test An empty list should have a size of zero. New Test Adding a movie to an empty list should result in a list with the size of one. public class MovieListTest { @Test public void testEmptyListSize() { MovieList emptyMovieList = new MovieList(); assertEquals("Size of movie list should be 0.", 0, emptyMovieList.size()); } @Test public void testSizeAfterAddingOne () { Movie starWars = new Movie(); MovieList oneItemList = new MovieList(); oneItemList.add(starWars); assertEquals("Size of movie list should be 1.", 1, oneItemList.size()); } } JUnit 4 Tests for Movie List Story
  • 49. CamillesCareer@gmail.com 49
  • 50. CamillesCareer@gmail.com 50 Class Code for Movie List Story public class Movie { } public class MovieList { private int numberOfMovies = 0; public int size() { return numberOfMovies; } public void add(Movie movieToAdd) { numberOfMovies = 1; } } Baby Steps: Change as little as possible to get to GREEN New Test Adding a movie to an empty list should result in a list with the size of one.
  • 51. CamillesCareer@gmail.com 51
  • 52. CamillesCareer@gmail.com 52 •  Programmers restructure system by improving the design of existing code without changing its behavior –  to remove duplication, improve communication, simplify, or add flexibility •  Refactoring is not random change, instead it’s driven by learning from our tests and from identifying and fixing code smells •  DRY (Don’t Repeat Yourself) •  Refactoring can occur just prior or just after writing new code •  Test, code, refactor, re-test –  Kent Beck suggests short cycle (10 minutes) Refactoring When fixing bugs, write a Red test, BEFORE fixing the bug. Test will run Green when bug fixed. When working with legacy code make sure to write automated tests BEFORE refactoring code, If tests didn’t exist.
  • 53. CamillesCareer@gmail.com 53 Refactored Class Code after 4th Test import java.util.ArrayList; public class Movie { } public class MovieList { private int numberOfMovies = 0; private ArrayList<Movie> movies = new ArrayList<Movie>(); public int size() { return movies.size(); } public void add(Movie movieToAdd) { numberOfMovies++; movies.add(movieToAdd); } public boolean contains(Movie movieToCheckFor) { return movies.contains(movieToCheckFor); } } New Test If we add a movie to a list, we should be able to ask if it’s there and receive a positive response. Delete Delete Change
  • 54. CamillesCareer@gmail.com 54
  • 55. CamillesCareer@gmail.com 55 BDD & Cucumber Stories to Tests
  • 56. CamillesCareer@gmail.com 56 Parts of a Cucumber Feature File Feature: Cash Withdrawal As a bank customer I want to withdraw cash at the ATM So that I can get money when the bank is closed Scenario: Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Scenario Title Scenario Feature Title Feature Steps cash_withdrawal.feature Ref: Cucumber code examples from “The Cucumber Book” by Matt Wynne and Aslak Hellesøy
  • 57. CamillesCareer@gmail.com 57 The Three Parts of Cucumber
  • 58. CamillesCareer@gmail.com 58 The Structure of Cucumber Features Feature: [ feature title ] As a user [ role ] I want [ feature ] So that [ business value ] Scenario: [ scenario title ] Given [ a pre-condition ] And [another pre-condition ] When [ event ] Then [ post-condition outcome ] And [ another post-condition outcome ] Plain TextKey Words From User Story Ref: Cucumber code examples from “The Cucumber Book” by Matt Wynne and Aslak Hellesøy
  • 59. CamillesCareer@gmail.com 59 Example Cucumber Feature
  • 60. CamillesCareer@gmail.com 60 Step Definitions
  • 61. CamillesCareer@gmail.com 61 Cucumber Command Line
  • 62. CamillesCareer@gmail.com 62 Plus the Evolving Code Under Test
  • 63. CamillesCareer@gmail.com 63 The Cucumber BDD Cycle
  • 64. CamillesCareer@gmail.com 64 Create Cucumber Feature File from BDD Test Definition
  • 65. CamillesCareer@gmail.com 65 Run Feature from the Command Line
  • 66. CamillesCareer@gmail.com 66 Copy Snippet from the Command Line Output
  • 67. CamillesCareer@gmail.com 67 Create Step Definition File and Pasted Snippet
  • 68. CamillesCareer@gmail.com 68 Run Cucumber Again (Notice the Pending and Skipped Steps)
  • 69. CamillesCareer@gmail.com 69 Replace the Pending in the First Step with Test Code You Need to Implement
  • 70. CamillesCareer@gmail.com 70 Run Cucumber Again (Notice the Failed and Skipped Steps)
  • 71. CamillesCareer@gmail.com 71 Implement Just Enough Code to Make the Test Step Pass
  • 72. CamillesCareer@gmail.com 72 Run Cucumber Again (Notice the Passed, Pending and Skipped Steps)
  • 73. CamillesCareer@gmail.com 73 Repeat Cycle Until All Test Steps Pass Write Scenario Create Step Definitions from Snippet Turn Pending Step into Test Watch it Fail Make it Pass by Evolving Code Pending Steps? YesNoDone with Scenario Start with User Story
  • 74. CamillesCareer@gmail.com 74 Example Scenario Outline
  • 75. CamillesCareer@gmail.com 75 Fit, Selenium examples
  • 76. CamillesCareer@gmail.com 76
  • 77. CamillesCareer@gmail.com 77 Browser Testing with Selenium Example Selenium Testing with jUnit and Java in Eclipse Selenium IDE (Firefox only) Record/Playback Recorded tests don’t scale, Only run test individually or single test suite – good for first cut tests Selenium Remote Control cross browser tests with Java, Ruby, Python, Perl, PHP or .Net
  • 78. CamillesCareer@gmail.com 78 Agility Testing Ratios
  • 79. CamillesCareer@gmail.com 79 Dysfunctional Test Ratio
  • 80. CamillesCareer@gmail.com 80 Healthy Test Ratio
  • 81. CamillesCareer@gmail.com 81 Healthy Test Ratio • Very, very, very Slow • Usability testing • Exploratory testing • Labor intensive • Automate exploration • Very slow • Validate UI functionality • Brittle, if using capture playback • Very, very fast • Uses mocks • Validate class and function/method code • Some fast, some slow • Data and other integrity • Validate at higher level than unit tests, but not directly user facing
  • 82. CamillesCareer@gmail.com 82 Agile Testing and Integration & Deployment
  • 83. CamillesCareer@gmail.com 83 3. Update Build Workspace Continuous Integration Cycle Developer Deploy Server Version Control Repository 1. Check-in CI Server2. Monitor changes 4. Build 5. Run Tests 6. Send Build & Test Report 7. Deploy App at Release
  • 84. CamillesCareer@gmail.com 84 3. Update Build Workspace Continuous Integration Cycle with Performance Testing Developer Deploy Server Version Control Repository 1. Check-in CI Server2. Monitor changes 4. Build 5. Run Tests 6. Send Build & Test Report 7b. Deploy App at Release Performance Test Server Frequently 7a. Performance Test App
  • 85. CamillesCareer@gmail.com 85 Only Check-in Working Tested Code Version Control Repository
  • 86. CamillesCareer@gmail.com 86 Continuous Deployment Cycle Builds on Continuous Integration Developer Deploy Server Version Control Repository 1. Check-in 2. Monitor changes 3. Update Build Workspace 4. Build 5. Run Tests CI Server 6. Send Build & Test Report 7. Clean Build & Tests Pass Deploy App Immediately
  • 87. CamillesCareer@gmail.com 87 Tools for the Deployment Pipeline Configuration Management Continuous Integration Testing Strategy Deployment Capistrano
  • 88. CamillesCareer@gmail.com 88
  • 89. CamillesCareer@gmail.com 89 Build-Measure-Lean with Lean Startup Split Tests Customer Interviews Customer Development 5 Whys Root Cause Analysis Customer Advisory Board Falsifiable Hypotheses Product Owner Accountability Customer Archetypes Cross-functional Teams Semi-autonomous Teams Smoke Tests Unit Tests Usability Tests Continuous Integration Incremental Deployment Free & Open-Source Components Cloud Computing Cluster Immune System Just-in-time Scalability Refactoring Developer Sandbox Split Tests Clear Product Owner Continuous Deployment Usability Tests Real-time Monitoring Customer Liaison Funnel Analysis Cohort Analysis Net Promoter Score Real-Time Altering Predictive Monitoring Measure Faster Learn Faster Code Faster
  • 90. CamillesCareer@gmail.com 90 Examples of Testing of Ideas with Lean Startup • Split A/B Testing – controlled experiments – two versions of product released simultaneously – test metric comparison ($ earned, # of signups, etc.) – Netflix huge user of A/B testing • Crowd Testing: Validating the existence of a customer base – bypassing investors with Kickstarter, etc. (e.g. Torment) – creating dummy site to sign up and buy (e.g. Zappos)
  • 91. CamillesCareer@gmail.com 91
  • 92. CamillesCareer@gmail.com 92 Bug Growth on Non-Agile Projects T i m e B u g s code fix code codecode codecodecode fixfixfixfix fix release 1 release 2 release 3 release 4 release 5 release 6 release 7
  • 93. CamillesCareer@gmail.com 93 TDD and BDD prevents a Great Many Bugs, but … a few will probably still occur Write a failing test for new functionality Write just enough code to pass test Refactored code must also pass tests, no new functionality CamillesCareer@gmail.com 93
  • 94. CamillesCareer@gmail.com 94 Create a RED test BEFORE fixing the bug, … so you are sure you really fixed it, and the bug never sneaks through the tests again. CamillesCareer@gmail.com 94 Write a failing test that exposes the bug Write just enough code to fix the bug Refactor if needed, must also pass tests, no new functionality
  • 95. CamillesCareer@gmail.com 95
  • 96. CamillesCareer@gmail.com 96 • By acting as a servant leader and removing obstacles • By providing whole team training in agile processes • By providing technical training in automated test tools to current employees • By requiring automated test experience for new hires • By making testability a must have for languages & tools (many of the best are open source) • By providing the automated test tools themselves • By providing continuous integration and deployment servers • By tracking progress differently Management Needs to Support Agility
  • 97. CamillesCareer@gmail.com 97 •  Agile processes like Scrum, Lean, Kanban and XP: •  often make short near term predictions •  track actual progress •  determine if the predictions, if made, were correct •  use actual progress metrics to project future outcomes •  adapt and adjust based on real outcomes • Managers must ensure that agile practices: •  help the team improve projections •  never become a stick Agility Organizational Processes Have Built-in Feedback Loops and Validation
  • 98. CamillesCareer@gmail.com 98 Running Tested Features Definition 1.  The  desired  so,ware  is  broken  down  into  named  features   (requirements,  stories),  which  are  part  of  what  it  means  to   deliver  the  desired  system.     2.  For  each  named  feature,  there  are  one  or  more  automated   acceptance  tests  which,  when  they  work,  will  show  that  the   feature  in  quesAon  is  implemented.   3.  The  RTF  metric  shows,  at  every  moment  in  the  project,  how   many  features  are  passing  all  their  acceptance  tests.     Ron Jeffries, A Metric Leading to Agility
  • 99. CamillesCareer@gmail.com 99 Calculating RTF is Simple •  No weighting - No partials credit: A Feature is either all running & passing all tests or it isn’t. •  So a feature either counts as 0 or 1. •  Add up the 1s. •  Run all tests for each measurement. •  Track over time.
  • 100. CamillesCareer@gmail.com 100 RTF Works on any Project, even Waterfall RTF on Agile software development projects should steadily increase over time RTF on Waterfall development only increases at project end
  • 101. CamillesCareer@gmail.com 101 Example 1: Buggy New Feature Breaks Old Monday 6 running tested features. Tuesday no new features added, but existing features still pass tests. On Wednesday, new feature added which fails test and causes side effect that causes pre- existing feature to also fail tests. On Thursday, new tests added to old feature and new feature fixed. All tests pass.
  • 102. CamillesCareer@gmail.com 102 Example 2: New Feature Breaks Everything Monday 6 running tested features. Tuesday no new features added, but existing features still pass tests. On Wednesday, new feature added which breaks the build, crashes system, corrupts DB or something similarly catastrophic. On Thursday, new tests added and new feature fixed. All tests pass.
  • 103. CamillesCareer@gmail.com 103 CUMULATIVE BURNUP StoryPointsorFeaturesComplete(RTF) Time Burnup of Story Points or Running Tested Features metrics can be more informative than burndowns. New Stories when added caused existing stories to fail automated tests Stories fixed, tests added, all tests now pass Stories Removed to meet Release DateStories Added New Baseline New Baseline Baseline ReleaseDate Complex Code needed refactoring Ref: Alistair Cockburn http://alistair.cockburn.us/Earned-value+and+burn+charts Ron Jeffries http://xprogramming.com/articles/jatrtsmetric/
  • 104. CamillesCareer@gmail.com 104 Camille Bell Agile Coaching & Consulting Retrospectives Agile Boot Camps Agile Training Updated Slides or just to chat about things agile CamillesCareer@gmail.com
  • 105. CamillesCareer@gmail.com 105
  • 106. CamillesCareer@gmail.com 106 Code Coverage C0, C1 and C2 Coverage C0 – Line coverage analysis measures which lines of code have been executed. C0 coverage is typically used it to find the areas of your program that have not been sufficiently tested, i.e. those that were not run by any your test cases. C1 – Branch coverage analysis measures which of the different possible branches of conditional statements have been tested. It is easy to have 100% C0 line coverage and only partial C1 branch coverage, because if, then and if, then, else statements may be contained all on one line in many languages. C2 - Path coverage measures which of the different possible execution paths through your code were tested. Paths are a combination of linear code execution with alternate branches. Since each new conditional that is encountered gives rise to new path choices, the permutations of possible unique paths gets huge.
  • 107. CamillesCareer@gmail.com 107 Code Coverage C0, C1 and C2 Tools C0- Every automated code coverage tool measures at least Line Coverage. Open source tools rcov for Ruby and Cobertura for Java provide C0 coverage. C1 - Many automated code coverage tools (both open source and commercial) measure Branch coverage. For instance open source tool Emma for Java measures both Line and Branch coverage as does Clover, a commercial Java test tool. C2 – Complete path coverage throughout an entire large application is almost unheard of because of its complexity and cost. I know of only two large software applications (both in Ada) that had 100% path coverage. Path coverage through multiple limited sections of code is much more feasible. All path coverage tools also include line and branch coverage. All are commercial and very expensive.
  • 108. CamillesCareer@gmail.com 108 public class CoverageExample { // This code is logically equivalent to the next example, but because of the // code structure, untested branches are detected by line coverage. public int echo (int x, boolean state1, boolean state2, boolean state3) { if (state1) { x++; } if (state2) { x--; } if (state3) { x = x; } return x; } } // This test provides 100% Line coverage, but it doesn’t provide any branch coverage. public class CoverageTestLineOnly { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @Test public void testReturnInput0FalseFalseFalse() { assertEquals(0, lineBranchPath.echo(0, false, false, false); } } Code Coverage C0 – Line Coverage
  • 109. CamillesCareer@gmail.com 109 Code Coverage C0 – Line Coverage Limitations public class CoverageExample // This code is logically equivalent to the prior example, but because of the // code structure, untested branches are not detected by line coverage. public int echo (int x, boolean state1, boolean state2, boolean state3) { if (state1) { x++; } if (state2) { x--; } if (state3) { x = x; } return x; } } // This test provides 100% Line coverage, but it doesn’t provide any branch coverage. public class CoverageTestLineOnly { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @Test public void allFalse() { assertEquals(0, lineBranchPath.echo(0, false, false, false); } }
  • 110. CamillesCareer@gmail.com 110 Code Coverage C1 – Branch Coverage public class CoverageExample // This code is identical to the prior example. public int echo (int x, boolean state1, boolean state2, boolean state3) { if (state1) { x++; } if (state2) { x--; } if (state3) { x = x; } return x; } } // Adding the new test provide 100% branch coverage, but doesn’t detect the bug. public class CoverageTestLineAndBranch { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @Test public void allFalse() { assertEquals(0, lineBranchPath.echo(0, false, false, false); } @Test public void allTrue() { assertEquals(0, lineBranchPath.echo(0, true, true, true); } }
  • 111. CamillesCareer@gmail.com 111 Code Coverage C2 – Path Coverage Explosion false false false false false true false true false false true true true false false true false true true true false true true true To test all possible paths of N Boolean conditionals is 2 ^ N. So for our example of 3 Booleans that’s 2x2x2 or 8. As shown in the example on the left, it’s not too bad. However, the set of all possible paths grows exponentially, so a combination of 10 Booleans, becomes 2x2x2x2x2x2x2x2x2x2 or 1024 possible paths, an unmanageable number of tests.
  • 112. CamillesCareer@gmail.com 112 Code Coverage C2 – Basis Path Simplification false false false true false false false true false false false true Basis path testing is a simplification of path testing, that significantly lowers the number of needed tests A complete set of basis path sets are the number of boolean decisions +1. It grows linearly instead of exponentially. that’s 3+1 or 4 instead of 8 for our example and 10+1 or 11 instead of 1024. Path 1: Any path will do for the baseline, so pick all trues or all falses for simplification. We are picking false. This is the first path in our basis set. Path 2: To find the next basis path, we flip the first decision (only) in our baseline. Path 3: Next we flip the second decision (only) in our baseline path, the first baseline decision remains fixed with the false outcome. Path 4: Finally, you flip the third decision in your baseline path. Again, the first baseline decision remains fixed with the false outcome.
  • 113. CamillesCareer@gmail.com 113 Code Coverage Line, Branch & Basis Path // JUnit 4 tests with 100% line, 100% branch coverage, and 100% basis path. public class CoverageTestPath { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @After public void runBeforeEveryTest() { lineBranchPath = null ; } @Test // First basis path. public void allFalse() { assertEquals ("All false inputs ", 0, lineBranchPath.echo(0, false, false, false); } @Test // Second basis path. public void trueFalseFalse() { assertEquals ("True-False-False inputs ", 0, lineBranchPath.echo(0, true, false, false); } @Test // Third basis path. public void falseTrueFalse() { assertEquals ("False-True-False inputs ", 0, lineBranchPath.echo(0, false, true, false); } @Test // Fourth basis path. public void falseFalseTrue() { assertEquals ("False-False-True inputs ", 0, lineBranchPath.echo, false, false, true); } }
  • 114. CamillesCareer@gmail.com 114 Cucumber Code Coverage with RCov (today Simplecov would better tool)
  • 115. CamillesCareer@gmail.com 115 RCov and Clover •  RCov, an open source Ruby Gem provides code coverage measurement for Cucumber •  RCov provides C0 – line code coverage •  Clover, a commercial tool provides code coverage measurement for JUnit •  Clover provides C1 – branch code coverage
  • 116. CamillesCareer@gmail.com 116 RCov Cucumber/Ruby Example
  • 117. CamillesCareer@gmail.com 117 RCov Cucumber/Ruby Example
  • 118. CamillesCareer@gmail.com 118 RCov Cucumber/Ruby Example