STARWest - Code Coverage is for Testers too - Tutorial

2,339 views

Published on

Projects often set a goal of 80%, 90%, or even 100% code coverage. So, what is code coverage anyway, and why can it mean many different things? How can testers help the team reach its coverage goals? And why should we care about code coverage anyway? From a tester’s perspective, Julie Gardiner explores the variety of code coverage metrics, discusses the advantages and disadvantages of each, and gives examples of how these metrics are used in real projects. Through demonstrations and exercises, learn about a plethora of coverage metrics—statement, branch, decision, branch condition combination, and more. Julie describes the automated tools teams need to calculate and evaluate code coverage. Leave with the confidence to ask about your project’s current coverage levels, the knowledge to set realistic coverage goals, and the ability to guide testers and developers to improve your project’s code coverage.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,339
On SlideShare
0
From Embeds
0
Number of Embeds
902
Actions
Shares
0
Downloads
52
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

STARWest - Code Coverage is for Testers too - Tutorial

  1. 1. Jonathon WrightEmail: jlwright@iee.orgTwitter: @Jonathon_WrightJulie GardinerEmail: julie@qstc.co.ukTwitter: @CheekyTester
  2. 2. Black Box e.g. White Box e.g.• Classification Trees • Statement• Pairwise • Decision• Boundary Value • Branch• etc. Condition Combination • etc. • Defect Based • Heuristics • Fault Attacks • Exploratory
  3. 3. tests 20 “things”our tests have covered 10/20 “things” = 50% “thing” coverage“things” can be: lines of code = statements decision outcomes = componentdecisions level branches integration level modules menu options system level functions
  4. 4. 1 1 # Declare variables.2 2 integer Counter, Quantity Declaration section3 3 float Amount, Price (declares variables)4 45 5 # Program begins here. Comments6 6 MAIN7 78 1 8 Amount = 0.09 2 9 Read Quantity10 1011 11 # Determine total cost.12 3 12 FOR Counter = 1 to Quantity Executable section13 4 13 Read Price (comprises executable14 5 14 Amount = Amount + Price statements)15 6 15 ENDFOR16 16 1 Line numbers17 7 17 Print “Total Cost is “ & Amount18 18 1 Statement numbers19 19 ENDMAIN Labels (optional)
  5. 5. Example control flow chartMake a list of items to sell sequential statementFor each item in the list decision statement (FOR) Calculate cost of postage sequential Write a description of this item statements decision If a photograph is needed statement (IF) Photograph this item sequential Transfer photograph to PC statements Endif end of IF Submit details of item to eBay single statementEndfor end of FOREnd end of program
  6. 6. executable statements
  7. 7. Statement coverage is normally measured by a software tool− a measurement technique− percentage of executable test statements exercised by a test suite = number of statements exercised 1. statement 2. statement total number of statements 3. IF….− example: 4. then 6. else − program has 10 statements 5. … 7. … − tests exercise 8 statements 8. ENDIF 9. statement statement coverage = 8/10 = 80% 10. statement
  8. 8. Example of statement coverage 1 1 read(n) Test Expected Case Input Output 2 IF n > 6 THEN False 3 n=n*2 A 7 14 2 4 ENDIFTrue 5 print n 3 Statement numbers 4 Test Path Statement Case Taken Coverage 5 A 1, 2, 3, 4, 5 100%
  9. 9. False IFTrue
  10. 10. Decision coverage is normally measured by a software tool test= 1. statement 2. statement 3. IF…. 4. then 6. else 5. … 7. … 8. ENDIF 9. statement ½ = 50% 10. statement
  11. 11. Example of decision coverage 1 1 read(n) Test Expected 2 IF n > 6 THEN Case Input Output False 3 n=n*2 A 7 14 2 4 ENDIF B 3 3True 5 print n 3 Statement numbers 4 Test Path Decision Decision Statement Case Taken Outcome Coverage Coverage 5 A 1, 2, 3, 4, 5 True 50% 100% B 1, 2, 4, 5 False 50% 80%
  12. 12. 1 False 2True 3 4 5
  13. 13. ReadRead AIF A > 0 THEN TRUE Print “A positive” A>0 PrintENDIF FALSE 1 ENDIF 2
  14. 14. IF Age < 18 THEN TRUE A<18 no alcohol Print “No alcohol”ELSE FALSE Print “Have an alcoholic drink” alcoholENDIF 2 ENDIF 2
  15. 15. Read stock level for item ReadIF Item in stock THEN Get from stores TRUEELSE In? Get FALSE Drawthe Order from supplier Print “Item back-ordered” diagramENDIF Order in yourPrint “Item processed” notes Print 2 ENDIF 2 Print
  16. 16. Read FALSE ?Read order lineWHILE more items ordered DO TRUE Check availability Check Get from storesENDWHILE GetPrint “Finished processing” ENDWHILE 1 1 Print
  17. 17. Init doPseudo-code:Read QuestionsResult = 0 if r=r+1Right = 0WHILE more Questions end if IF Answer = Correct THEN end Right = Right + 1 while ENDIF resultEND WHILEResult = (Right / Questions)IF Result > 60% THEN if pass Print "pass"ELSE fail Print "fail”ENDIF 2 2 end if
  18. 18. Control flow graph structures Node: IF condition THEN WHILE conditiona single statement or do this do thissequence of sequential ENDIF ENDWHILEstatements (that are notdecisions or branches). FOR condition do this ENDFOR Branch or Edge :a logical link between Statement(s) executedtwo nodes, indicates the IF condition THEN zero or more timesnext statement(s) that do thiscan be executed. ELSE DO do that this Decision Statement ENDIF UNTIL condition or Branching Statement(s) executed Statement one or more times Graph notation as used in Bill Hetzel’s “The Complete Guide to Software Testing”
  19. 19. Producing a control flow graph− identify sequential statements − combine a series into one node − draw one branch coming out from each node− add a node for each decision statement − draw two branches coming out from each decision node− add a node for the end of a decision structure − i.e. where lines join back together− complete graph with an end node − do not leave a branch pointing into space!
  20. 20. Example control flow graphMake a list of items to sell sequential statementFor each item in the list decision statement (FOR) Calculate cost of postage sequential Write a description of this item statements decision If a photograph is needed statement (IF) Photograph this item sequential Transfer photograph to PC statements Endif end of IF Submit details of item to eBay single statementEndfor end of FOREnd end of program
  21. 21. Branch/decision coverage differences Basic blocks Conditional read(a) IF a > 6 THEN branch read(a) TRUE b=a*2 IF a > 6 THEN ENDIF print b FALSE b=a*2 ENDIF branch coverage counts print b all branches decision coverage counts only Unconditional conditional branches branch
  22. 22. read(a)B1 IF a > 6 THEN TRUE Test Input Expected B2 case output FALSE b=a*2 1 7 14 ENDIFB3 print b 2 4 ?Test Path Branch Decisioncase coverage coverage Test case 2 could reveal 1 B1 -> B2 -> B3 67% 50% a fault (‘b’ has not been 2 B1 -> B3 33% 50% defined) Totals: 100% 100%
  23. 23. IF (a < b) AND (c > d) AND (count < 100) THEN Operand values = number of boolean operand values executed total number of boolean operand valuesCondition coverage does not guarantee decision coverage
  24. 24. IF (a < b) AND (c > d) AND (count < 100) THEN= number of boolean operand value combinations executed total number of boolean operand value combinationsMultiple Condition coverage guarantees decision coverage
  25. 25. IF (a < b) AND (c > d) AND (count < 100) THEN = number of boolean operand value (independently affect the decision) total number of boolean operands
  26. 26. = number of executed LCSAJ’s total number of LCSAJ’s
  27. 27. LCSAJs1 Result = 02 From To Target Right = 03 DO While more questions 1 3 74 IF Key = Answer then 1 4 65 Incr Right 1 6 36 END DO 3 3 77 Result = (Right / Questions) x 100 3 4 6 etc.8 IF Result > 50 Then 3 6 39 Print "pass" 6 6 310 ELSE 7 8 1011 Print "fail" 7 10 1212 END 10 12 end 12 12 end
  28. 28. 123412 12 123 ?? ? ? ? ?
  29. 29. 1 2 3 4 5 6 7 8 …. for as many times as it is possible to go round the loop (this can be ? unlimited, i.e. infinite)
  30. 30. Where: All Paths = Path Coverage All Paths L = LCSAJ BCC B = Branch/Decision Coverage L S = Statement Coverage MCDC BCC = Branch Condition B Combination Coverage BC MCDC = Modified Condition Decision Coverage S BC = Branch Condition CoverageSource: BS 7925-2
  31. 31.  structural coverage can measure completeness or incompleteness of a set of specification or experienced based techniques code coverage tools capture the structural coverage achieved by the tests any gaps to achieve 100% are identified very useful for demonstrating compliance to standards especially safety-critical systems
  32. 32. Visual Studio – Products IntelliTrace™ Test Case Management UML Modeling Manual Testing Architecture Explorer Fast Fwd for Manual Test Logical Class Designer Layer Diagram Load Testing Web Testing UI Test Automation Test Impact Analysis Performance Profiling Static Code Analysis Code Coverage Code Metrics Database Change Mgmt. Database Deployment Database Unit Testing Test Data Generation Silverlight Tools Multi-core Development SharePoint Development Cloud Development Web Development Windows Development Generate from Usage Office Development New WPF Editor Customizable IDE
  33. 33. Platform Native CodedUI 3rd Party NotesIE7/8/9+ HTML/AJAXASP.NET MVC 3/4 Fully supported platformWPF 3.5+ Partial solution. Further workWindows Win32 required to completeWindows Forms 2.0+ Best efforts. May work with known issues. No majorFF3 – HTML/AJAX ongoing investmentCitrix/Terminal Services Currently no support but onSharePoint the roadmap for futureSilverlight 3.0/4.0 releasesJava/Java Applets Currently no support and none planned for now.Office Client AppsIE 6 (Legacy)Chrome/Opera/SafariFlash/AIRSAP
  34. 34. generalist specialistmanual some scripting skills strong coding experttesting scripting coding skills develops fully creates scripts some automated to set up lab, coding skills testing create data procedures black box testing white box testing API testing
  35. 35. generalist specialist coded UI test test runner web performance test unit testing load test test case management virtual lab management data diagnostic adapters (video, intelliTrace, action log, event log etc.) team foundation server with reporting (defects, requirements, user stories, code coverage, source control & build information)
  36. 36.  Maintainability Index Cyclomatic Complexity Depth of Inheritance Class Coupling Lines of Code
  37. 37. TDD API testing white box testing BDD component levelblack box testing ATDD Automation integration level Fakes & Stubs UI Automation system level Manual (Assisted)
  38. 38. Add.Book B1 B4 B5 B2 C2 B3 C3 A1 Home C1 Shopping.Cart Submit.Order C5 C4 D2 D1 D3 D4 D5 Search.Book Test Control Variable Confidence Level = High = Medium = LowDynamic Diagnostic Adapters (BPM Assist, IntelliTrace, Amber (Action & Event Logging))
  39. 39. www.microsoft.com/visualstudio/11/ www.specflow.orgwww.microsoft.com/visualstudio/11/en-US/downloads#tfs-groupwww.windows.microsoft.com/en-US/windows-8/download/

×