Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How much testing is enough for software that can condemn a man to death? Traceability in an Agile criminal justice context


Published on

Using tools like TDD and ATDD, Agile provides the means to be confident that your brand new software is well tested-- even for life critical situations such as criminal justice software.  But hold on a minute!  It is a rare mission critical system that is built completely from scratch.  There are always legacy components your team didn't build or doesn't control.  Maybe the previous contractor built it-- but now they are gone and it is your problem.  How can you be certain that everything functions properly in such a situation?  How much testing is enough?  How can you know whether a system has been tested?  These are the questions that standards such as CMMI and PMBOK seek to answer with traceability.
The debate about traceability has been raging for a long time, with passionate advocates on both sides of the argument. Projects following traditional waterfall methods, and projects that conform to PMBOK or CMMI standards often create and maintain a requirements traceability matrix, or RTM, a document that traces “shall” requirements to functional capabilities and testcases. Some Agilists argue that the RTM is rarely consulted in practice, so the significant efforts required to maintain such a document are “waste.” Others point out that agile practices such as TDD provide all the traceability that may be needed. This talk will explore the underlying reasons why traceability may be important and worthwhile in many Federal government contexts, and review exciting new technologies that may provide an “agile answer” to this conundrum.

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here

How much testing is enough for software that can condemn a man to death? Traceability in an Agile criminal justice context

  1. 1. 1   Traceability in an Agile
 Federal Agency Context
   Craeg Strong CTO, Ariel Partners October 21, 2014 Washington, DC   How  Much  Tes,ng  Is  Enough…   When  it  Comes  to  Ma8ers  of     Life  and  Death?    
  2. 2. 2   So&ware  Development  since  1988       Large  Commercial  &  Government  Projects     Turned  Around  Projects  With  Agile     Apache  Ant  Open-­‐Source  Contributor     New  York  &  Washington  DC  Area   CTO,  Ariel  Partners     CSM,  CSP,  CSD,  CSPO,  PSM,     PMI-­‐ACP,  PMP   @ckstrong1   Craeg  Strong   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  3. 3. 3   Agenda   1.  What  Requirements  Traceability  Is   2.  Why  &  When  It  Is  Important   3.  Some  Typical  Agile  PracZces   4.  Why  They  Are  Insufficient   5.  AlternaZve  Approach:  BDD/ATDD  With  SpecFlow   6.  How/When/Who/Why   7.  Solving  The  Problem  With  Agility   8.  How  To  Get  There:  TransiZonal  Strategies   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  4. 4. 4   Context   Examples Use Microsoft technology Equivalents Exist in Other Language Ecosystems (Java, LAMP) q  Air traffic control q  Loss reserve calculation for long-tail insurance q  Threat evaluation for detainee at border crossing q  Nuclear power plant control q  Digitized forensic evidence match at murder trial q  Trading risk circuit breaker for high frequency or options trading q  Resource allocation for disaster relief Heavily  Regulated  Industries   Mission-­‐CriZcal  Systems   Life-­‐CriZcal  Systems   Financial  Systems   Legacy  Systems  Without  Automated  Tests   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  5. 5. 5   What  Is  Requirements  Traceability?   •  Inventory of all requirements •  …traced to features implemented in the software, •  …traced to test cases that cover the functionality •  …with indicator for most recent pass/fail test status Typical Artifact: Requirements Traceability Matrix (RTM) System shall… Module XYZ Test Case 123 ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  6. 6. 6   Why  Is  Requirements  Traceability   Important?   Ø Thousands of complex features Ø …developed over many years Ø …by people who are no longer on the program Ø …few automated tests Ø …a brittle code base How can we know we have tested everything? ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  7. 7. 7   Agile  Approach:     Automated  Unit  Tests   q  One test generally tests one class q  Written by developers q  Intended to be run after every change q  “Classic” unit tests: code only q  xUnit extended family q Database testing q Graphical user interface (GUI) testing q Smoke testing q Performance testing ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  8. 8. 8   Measuring  Unit  Test  Coverage   q Measures how much of the codebase is covered by at least one unit test q Typical target number: 80% ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  9. 9. 9   LimitaZons  of  Unit  TesZng   Difficulty Creating Unit Tests q  Legacy projects may not have good automated test coverage q  Code may not be built for testability Only A Partial Solution q  Non developers cannot understand unit tests q  Coverage of code does not guarantee a feature works as needed Build The Thing Right Build The Right Thing GUI UZliZes   Security  /   Logging   Service   Business  Logic   GUI  Module   Database   Server Persistence   Security  /     Logging   UZliZes   Service   Business  Logic   Feature ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  10. 10. 10   What  about  Test  Driven  Development   (TDD)?   Test Driven Development q  A Design technique q  Tests are written first in order to drive the design q  Additional tests may be added later Why Does It Work? q  There are generally multiple valid ways to design a piece of code q  Writing a test first tends to produce more testable, flexible and modular design More Typical Alternative q  Tests written during/after code q  May require more refactoring to achieve desired results TDD: Good Practice; But Does Not Solve The Problem ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  11. 11. 11   SoluZon:  Behavior-­‐Driven   Development  (BDD)   q Different Process q Different Participants q Different Focus: Build The Right Thing User Story #53 Acceptance Criterion 1 Acceptance Criterion 2 Acceptance Criterion 3 Automated Acceptance Test #53.1 Given a claim has been reported… When the claim is entered… Then the system should return… Also Known as Acceptance Test Driven Development (ATDD) ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  12. 12. 12   Feature  File   BDD  With  SpecFlow   1.  Write   Feature   Step  DefiniZons   2.  Generate   Template   3.   C ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  13. 13. 13   BDD  With  SpecFlow:     Roles  and  Tools   Developer   Tester   Analyst   Product  Owner   Syntax  HighlighZng  Support   Gherkin Plugin Syntax  HighlighZng  Support   Gherkin Plugin PicklesDoc   Document  GeneraZon   CodedUI GUI-­‐Level  AutomaZon   Test  AutomaZon  Base  Fwk      (or  Selenium)      (or  MSTest)   Three  Amigos   Project  Portal/Wiki     Given…   When…   Then…     Given…   When…   Then…     Whiteboard   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  14. 14. 14   BDD  With  SpecFlow:  Process   3.  Tester   Completes   “Happy  Path”   Scenario   Tester     Given…   When…   Then…   5.  Developer     Implements  Code   And  Test  Steps   Developer   SpecFlow  Pair   2.  Create  Dra&   SpecificaZon   During  MeeZng     User  Story  240  -­‐  Acceptance  Criteria   1)  XYZ  Field  should  be  read-­‐only   2)  Total  should  match  999.99       Scenario:  Ensure  XYZ  Field  is   read-­‐only  when…   Scenario:  Ensure  Total  is…   Analyst   1.  Three  Amigos   Discuss  User   Story  or  Bug   Developer   Tester   Analyst   PO   @ignore   Scenario  XYZ            Given…            When…            Then…   4.  Developer  Adds   Dra&  SpecificaZon   To  TFS   Developer   Big?  Risky?   Small?   Fix  Now     Technical   Product  Backlog      User  Story  999   Cleanup  XYZ….   8.  Rework,   Refactoring,  or   Cleanup   IdenZfied?   7.  Review     Completed   Test   Developer   Tester   Analyst   6.  Tester,   Analyst  Add   Addl   Scenarios   Given…   When…   Then…     Given…   When…   Then…   Analyst   Tester   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  15. 15. 15   GeneraZng  DocumentaZon  From   SpecificaZons   q Pickles Report Generator q Features organized In tree structure q More sophisticated commercial options available (SpecFlow+) ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  16. 16. 16   Organizing  Automated  Tests:     By  Type   q Named after class being tested q Test project structure mirrors source code product structure q Named after feature q Test project structure mirrors business domain Unit  Tests   Feature  Tests   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  17. 17. 17   Organizing  Automated  Tests:     Filter  By  Category   Category   ExecuAon  Speed   Dependencies   ExecuAon  Context   @CI   Very  Fast   None:  Code  Only   Developer,  before   every  check-­‐in   @DB   Slower   Database  structure,  may  use   pre-­‐set  test  data   CI  Build   @smoke   Slowest   Full  Running  System   Nightly  Build   @CI   Scenario:  Ensure  abc…     @DB   Scenario:  Ensure  xyz…       @smoke   Scenario:  Ensure  123…   q  Each individual test case is categorized according to its dependencies q  “Run All Tests” automatically runs the correct set of tests depending on execution context ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  18. 18. 18    Organizing  Automated  Tests:     Test  Frameworks   SpecFlow  tests  can  be  wri8en  at  different  levels  –  even  within  the   same  feature  file   Feature  File   Unit! ! CodedUI CodedUI Step  DefiniZons  –  ImplementaZon  AlternaZves   Gherkin   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  19. 19. 19   Organizing  Automated  Tests:   Examples   Acceptance Criteria 1.  Confirm that a trade that would trigger margin call will require secondary authorization protocol 2.  Confirm that a trader is not allowed to enter a quantity of more than 50,000 shares for an options trade 3.  Confirm that an out of the money trade will be flagged User Story #13628 Enforce Trade Limits For Option Trades Possible  Strategy   Why?   CodedUI  Test  using  Moq   Specific  GUI  is  important  and  needs  to  be  tested   NUnit  test,  no  DB   •  CalculaZon  is  most  important  asset  to  test,  GUI  may   change   •  Legacy  system  is  being  rewripen  and  replacement   system  must  support  same  logic   NUnit  Test  with  DB   •  Significant  base  data  setup  required,  mocking  is  not   pracZcal   •  Significant  logic  to  be  tested  in  the  persistence  layer   CodedUI  Test  with  DB   Full-­‐stack  smoke  test  is  desired  (requires  more   maintenance;  there  should  be  relaZvely  few  of  these)   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  20. 20. 20   Extending  BDD  For  Traceability   q Epics  Linked  to  User  Stories  in  TFS   q SpecFlow  Tests  Map  To  User  Stories   User Story 1098 Feature: 1098 Scenario: XYZ Scenario: ABC Epic 1608 User Story 2689 User Story 3104 User Story 7510 Epic 2387 User Story 7513 Feature: 2689 Scenario: XYZ Scenario: ABC Feature: 3104 Scenario: XYZ Scenario: ABC Feature: 7510 Scenario: XYZ Scenario: ABC Feature: 7513 Scenario: XYZ Scenario: ABC Pros   1.  Simple 2.  Easy to track Cons   1.  User story abstraction does not really match feature 2.  Requires maintenance ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  21. 21. 21   Challenges  for  Maintaining  Traceability   Issues 1.  A user story may describe only part of a true business feature 2.  Later user stories may modify, contradict, or replace earlier user stories Mitigation: Let the tests help 1.  A failing SpecFlow test may indicate an older specification needs maintenance 2.  If the earlier SpecFlow test does not fail, consider whether it is really important to update an earlier user story with new information User  Stories  are  designed  to  be  ephemeral  –  only  ac,ve  within  their  Sprint  or  Release   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  22. 22. 22   The  Agile  RTM   q  All system functions expressed as user stories q  All user stories have acceptance criteria q  Each criterion translated to an automated test using structured English (Gherkin) q  Customized report matches epics and user stories to automated acceptance tests q  Test fails unless software is implemented correctly Links  to  Theme   Record  in  TFS   Links  to  Epic   Record  in  TFS   Links  to  User   Story  in  TFS   Links  to  Feature   DocumentaZon   Generated  via  Pickles   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  23. 23. 23   GeneraZng  the  Agile  RTM   TFS   Build   PowerShell  Script   TFS   Themes,  Epics,     User  Stories   Console  Runner   Results  of   Automated  Tests   PicklesDoc   Document  GeneraZon   SpecificaZons   Agile  RTM   Project  Portal  /  Wiki  Combine  &  Transform   XML  Files   Publish   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  24. 24. 24   Building  A  Test  AutomaZon  Capability   BUILD TESTING CAPABILITY q  Technical Frameworks q  NUnit q  SpecFlow q  Moq q  Reporting Capability q  Test Results q  Code Coverage q  Functional Coverage q  Encouraging Adoption q  Examples Everywhere q  Mentoring q  Pair Programming MAKE THE SYSTEM TESTABLE q  Upgrade dot net framework q  Drop support for old operating systems q  Upgrade third party libraries q  Refactor code to isolate dependencies q  Simplify project structure q  Different components may need different strategies Both Pieces Are Required For Success ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  25. 25. 25   TransiZonal  Strategy:   StarZng  with  Manual  Test  Cases   Epic Manual TestCase Manual TestCase Manual TestCaseUser Story Manual TestCase Legacy Req Legacy Req q  Import legacy requirements into TFS q  Convert high-level requirements into epics q  Capture manual test cases in MTM q  Link MTM test cases to their requirements and/or epics q  Capture all new work using user stories q  Link legacy system requirements to epics alongside new user stories Epic Legacy Req Legacy Req Gradually migrate legacy “Shall” requirements to user stories and acceptance criteria Plan ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  26. 26. 26   ConverZng  From  Requirements  to   User  Stories   Why  do  this?   q System  has  long  projected  lifespan   q Planned  re-­‐plarorm,  migraZon,  upgrade,  or   significant  redevelopment     User Story Legacy Req 1  to  1   Legacy Req Acceptance Criterion 1  to  1   User Story Legacy Req 1  to  many   User Story User Story Legacy Req User Storymany  to  1   Legacy Req Legacy Req ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  27. 27. 27   Legacy  System  Replacement   Benefits  of  SpecFlow  for  re-­‐development   q SpecificaZons  confirm  new  system  meets  requirements   q SpecificaZons  represent  up  to  half  of  overall  effort   q SpecificaZons  provide  documentaZon  for  the  new  system   q Highlights  areas  where  new  system  differs  from  old  system   q Significantly  reduces  risk   Legacy System Replacement System Significant   Reuse   Gherkin   SpecificaZons   Step  DefiniZons   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  28. 28. 28   Summary   1.  Traceability  is  important  for  mission  criZcal   and  life  criZcal  so&ware   2.  Automated  test  frameworks  can  provide  the   assurance  we  need   ü Did  we  build  the  right  thing   ü Did  we  build  it  right   ü Are  all  the  required  funcZons  working  correctly   3.  SpecFlow  has  been  proven  effecZve     ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394    
  29. 29. 29   QuesZons?   ©  Copyright  Ariel  Partners  2014                                  *    ((646)  467-­‐7394     We  are  available  for  consulZng  or  Agile  coaching.   Feel  free  to  contact  us!   Ariel  Partners   (646)  467-­‐7394   @arielpartners     Thank  You!