STARWest - Code Coverage is for Testers too - Tutorial
Upcoming SlideShare
Loading in...5
×
 

STARWest - Code Coverage is for Testers too - Tutorial

on

  • 1,835 views

Projects often set a goal of 80%, 90%, or even 100% code coverage. So, what is code coverage anyway, and why can it mean many different things? How can testers help the team reach its coverage goals? ...

Projects often set a goal of 80%, 90%, or even 100% code coverage. So, what is code coverage anyway, and why can it mean many different things? How can testers help the team reach its coverage goals? And why should we care about code coverage anyway? From a tester’s perspective, Julie Gardiner explores the variety of code coverage metrics, discusses the advantages and disadvantages of each, and gives examples of how these metrics are used in real projects. Through demonstrations and exercises, learn about a plethora of coverage metrics—statement, branch, decision, branch condition combination, and more. Julie describes the automated tools teams need to calculate and evaluate code coverage. Leave with the confidence to ask about your project’s current coverage levels, the knowledge to set realistic coverage goals, and the ability to guide testers and developers to improve your project’s code coverage.

Statistics

Views

Total Views
1,835
Slideshare-icon Views on SlideShare
951
Embed Views
884

Actions

Likes
0
Downloads
18
Comments
0

3 Embeds 884

http://109.228.20.229 869
http://www.linkedin.com 12
http://www.slashdocs.com 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    STARWest - Code Coverage is for Testers too - Tutorial STARWest - Code Coverage is for Testers too - Tutorial Presentation Transcript

    • Jonathon WrightEmail: jlwright@iee.orgTwitter: @Jonathon_WrightJulie GardinerEmail: julie@qstc.co.ukTwitter: @CheekyTester
    • Black Box e.g. White Box e.g.• Classification Trees • Statement• Pairwise • Decision• Boundary Value • Branch• etc. Condition Combination • etc. • Defect Based • Heuristics • Fault Attacks • Exploratory
    • tests 20 “things”our tests have covered 10/20 “things” = 50% “thing” coverage“things” can be: lines of code = statements decision outcomes = componentdecisions level branches integration level modules menu options system level functions
    • 1 1 # Declare variables.2 2 integer Counter, Quantity Declaration section3 3 float Amount, Price (declares variables)4 45 5 # Program begins here. Comments6 6 MAIN7 78 1 8 Amount = 0.09 2 9 Read Quantity10 1011 11 # Determine total cost.12 3 12 FOR Counter = 1 to Quantity Executable section13 4 13 Read Price (comprises executable14 5 14 Amount = Amount + Price statements)15 6 15 ENDFOR16 16 1 Line numbers17 7 17 Print “Total Cost is “ & Amount18 18 1 Statement numbers19 19 ENDMAIN Labels (optional)
    • Example control flow chartMake a list of items to sell sequential statementFor each item in the list decision statement (FOR) Calculate cost of postage sequential Write a description of this item statements decision If a photograph is needed statement (IF) Photograph this item sequential Transfer photograph to PC statements Endif end of IF Submit details of item to eBay single statementEndfor end of FOREnd end of program
    • executable statements
    • Statement coverage is normally measured by a software tool− a measurement technique− percentage of executable test statements exercised by a test suite = number of statements exercised 1. statement 2. statement total number of statements 3. IF….− example: 4. then 6. else − program has 10 statements 5. … 7. … − tests exercise 8 statements 8. ENDIF 9. statement statement coverage = 8/10 = 80% 10. statement
    • Example of statement coverage 1 1 read(n) Test Expected Case Input Output 2 IF n > 6 THEN False 3 n=n*2 A 7 14 2 4 ENDIFTrue 5 print n 3 Statement numbers 4 Test Path Statement Case Taken Coverage 5 A 1, 2, 3, 4, 5 100%
    • False IFTrue
    • Decision coverage is normally measured by a software tool test= 1. statement 2. statement 3. IF…. 4. then 6. else 5. … 7. … 8. ENDIF 9. statement ½ = 50% 10. statement
    • Example of decision coverage 1 1 read(n) Test Expected 2 IF n > 6 THEN Case Input Output False 3 n=n*2 A 7 14 2 4 ENDIF B 3 3True 5 print n 3 Statement numbers 4 Test Path Decision Decision Statement Case Taken Outcome Coverage Coverage 5 A 1, 2, 3, 4, 5 True 50% 100% B 1, 2, 4, 5 False 50% 80%
    • 1 False 2True 3 4 5
    • ReadRead AIF A > 0 THEN TRUE Print “A positive” A>0 PrintENDIF FALSE 1 ENDIF 2
    • IF Age < 18 THEN TRUE A<18 no alcohol Print “No alcohol”ELSE FALSE Print “Have an alcoholic drink” alcoholENDIF 2 ENDIF 2
    • Read stock level for item ReadIF Item in stock THEN Get from stores TRUEELSE In? Get FALSE Drawthe Order from supplier Print “Item back-ordered” diagramENDIF Order in yourPrint “Item processed” notes Print 2 ENDIF 2 Print
    • Read FALSE ?Read order lineWHILE more items ordered DO TRUE Check availability Check Get from storesENDWHILE GetPrint “Finished processing” ENDWHILE 1 1 Print
    • Init doPseudo-code:Read QuestionsResult = 0 if r=r+1Right = 0WHILE more Questions end if IF Answer = Correct THEN end Right = Right + 1 while ENDIF resultEND WHILEResult = (Right / Questions)IF Result > 60% THEN if pass Print "pass"ELSE fail Print "fail”ENDIF 2 2 end if
    • Control flow graph structures Node: IF condition THEN WHILE conditiona single statement or do this do thissequence of sequential ENDIF ENDWHILEstatements (that are notdecisions or branches). FOR condition do this ENDFOR Branch or Edge :a logical link between Statement(s) executedtwo nodes, indicates the IF condition THEN zero or more timesnext statement(s) that do thiscan be executed. ELSE DO do that this Decision Statement ENDIF UNTIL condition or Branching Statement(s) executed Statement one or more times Graph notation as used in Bill Hetzel’s “The Complete Guide to Software Testing”
    • Producing a control flow graph− identify sequential statements − combine a series into one node − draw one branch coming out from each node− add a node for each decision statement − draw two branches coming out from each decision node− add a node for the end of a decision structure − i.e. where lines join back together− complete graph with an end node − do not leave a branch pointing into space!
    • Example control flow graphMake a list of items to sell sequential statementFor each item in the list decision statement (FOR) Calculate cost of postage sequential Write a description of this item statements decision If a photograph is needed statement (IF) Photograph this item sequential Transfer photograph to PC statements Endif end of IF Submit details of item to eBay single statementEndfor end of FOREnd end of program
    • Branch/decision coverage differences Basic blocks Conditional read(a) IF a > 6 THEN branch read(a) TRUE b=a*2 IF a > 6 THEN ENDIF print b FALSE b=a*2 ENDIF branch coverage counts print b all branches decision coverage counts only Unconditional conditional branches branch
    • read(a)B1 IF a > 6 THEN TRUE Test Input Expected B2 case output FALSE b=a*2 1 7 14 ENDIFB3 print b 2 4 ?Test Path Branch Decisioncase coverage coverage Test case 2 could reveal 1 B1 -> B2 -> B3 67% 50% a fault (‘b’ has not been 2 B1 -> B3 33% 50% defined) Totals: 100% 100%
    • IF (a < b) AND (c > d) AND (count < 100) THEN Operand values = number of boolean operand values executed total number of boolean operand valuesCondition coverage does not guarantee decision coverage
    • IF (a < b) AND (c > d) AND (count < 100) THEN= number of boolean operand value combinations executed total number of boolean operand value combinationsMultiple Condition coverage guarantees decision coverage
    • IF (a < b) AND (c > d) AND (count < 100) THEN = number of boolean operand value (independently affect the decision) total number of boolean operands
    • = number of executed LCSAJ’s total number of LCSAJ’s
    • LCSAJs1 Result = 02 From To Target Right = 03 DO While more questions 1 3 74 IF Key = Answer then 1 4 65 Incr Right 1 6 36 END DO 3 3 77 Result = (Right / Questions) x 100 3 4 6 etc.8 IF Result > 50 Then 3 6 39 Print "pass" 6 6 310 ELSE 7 8 1011 Print "fail" 7 10 1212 END 10 12 end 12 12 end
    • 123412 12 123 ?? ? ? ? ?
    • 1 2 3 4 5 6 7 8 …. for as many times as it is possible to go round the loop (this can be ? unlimited, i.e. infinite)
    • Where: All Paths = Path Coverage All Paths L = LCSAJ BCC B = Branch/Decision Coverage L S = Statement Coverage MCDC BCC = Branch Condition B Combination Coverage BC MCDC = Modified Condition Decision Coverage S BC = Branch Condition CoverageSource: BS 7925-2
    •  structural coverage can measure completeness or incompleteness of a set of specification or experienced based techniques code coverage tools capture the structural coverage achieved by the tests any gaps to achieve 100% are identified very useful for demonstrating compliance to standards especially safety-critical systems
    • Visual Studio – Products IntelliTrace™ Test Case Management UML Modeling Manual Testing Architecture Explorer Fast Fwd for Manual Test Logical Class Designer Layer Diagram Load Testing Web Testing UI Test Automation Test Impact Analysis Performance Profiling Static Code Analysis Code Coverage Code Metrics Database Change Mgmt. Database Deployment Database Unit Testing Test Data Generation Silverlight Tools Multi-core Development SharePoint Development Cloud Development Web Development Windows Development Generate from Usage Office Development New WPF Editor Customizable IDE
    • Platform Native CodedUI 3rd Party NotesIE7/8/9+ HTML/AJAXASP.NET MVC 3/4 Fully supported platformWPF 3.5+ Partial solution. Further workWindows Win32 required to completeWindows Forms 2.0+ Best efforts. May work with known issues. No majorFF3 – HTML/AJAX ongoing investmentCitrix/Terminal Services Currently no support but onSharePoint the roadmap for futureSilverlight 3.0/4.0 releasesJava/Java Applets Currently no support and none planned for now.Office Client AppsIE 6 (Legacy)Chrome/Opera/SafariFlash/AIRSAP
    • generalist specialistmanual some scripting skills strong coding experttesting scripting coding skills develops fully creates scripts some automated to set up lab, coding skills testing create data procedures black box testing white box testing API testing
    • generalist specialist coded UI test test runner web performance test unit testing load test test case management virtual lab management data diagnostic adapters (video, intelliTrace, action log, event log etc.) team foundation server with reporting (defects, requirements, user stories, code coverage, source control & build information)
    •  Maintainability Index Cyclomatic Complexity Depth of Inheritance Class Coupling Lines of Code
    • TDD API testing white box testing BDD component levelblack box testing ATDD Automation integration level Fakes & Stubs UI Automation system level Manual (Assisted)
    • Add.Book B1 B4 B5 B2 C2 B3 C3 A1 Home C1 Shopping.Cart Submit.Order C5 C4 D2 D1 D3 D4 D5 Search.Book Test Control Variable Confidence Level = High = Medium = LowDynamic Diagnostic Adapters (BPM Assist, IntelliTrace, Amber (Action & Event Logging))
    • www.microsoft.com/visualstudio/11/ www.specflow.orgwww.microsoft.com/visualstudio/11/en-US/downloads#tfs-groupwww.windows.microsoft.com/en-US/windows-8/download/