SlideShare a Scribd company logo
1 of 26
Jelly Beans
and
Software Verification
Introduction
• Tom Walton
• Software Quality Assurance Specialist
• tom_walton@videotron.ca
Verification
• The processes that are relevant are
– Serial Processes
• Document and Code Inspection
• Unit Test
• Integration Test

– Random Processes
• System Level Verification
Serial Processes
• Pieces are verified one at a time
• Opportunity for 100% Coverage
• Historical data used for
– Estimating total defects
– Severity of remaining defects
Instructions
•
•
•
•

Use the scoop to take a sample
Count the green jelly beans
Record the number of scoops
Record the number of green Jelly beans in
each scoop
• Plot data
– Cumulative Green vs Cumulative Sampled

• Make an estimate of the total number of green
jelly beans present.
Serial Jelly Beans
120
Cum Green
Trend
Actual

100
Cumulative Green

Linear (Cum Green)

y = 2.6442x
R² = 0.9569
80

60

40

20

0
0

5

10

15

20
Trial

25

30

35

40
Code Inspection
Integration Test
Integration Test Defect Trend
350

Cumulative Defects

300
250
200

150
100
74% of Planned Tests
28 of 34 Activities Reporting

50
0
0

500

1000

1500

Cumulative Total Effort (hr)

2000

2500

3000
Using the Data
Code Inspection Performance
30.00

Defect Density (Def/KLOC)

Avg.Def/KLOC
25.00

Act. Def/KLOC

Too Fast

20.00

Too Buggy

15.00

Tend to be large modules

Point 1
Point 2

10.00

Def/KLOC x LOCs/hr = Constant

5.00

0.00
0

100

200

300

400

500

600

700

Inspection Speed (LOCS/hr.)

Measure
Size (LOCS)
Inspection Rate(Locs/Hr)
Defect Rate(Def/Hr)
Defect Density (Def/KLOC)

Average Point 1 Point 2
800
800
78
229
267
0.206
1.43
1.33
3.65
6
5

• t The planning datameasures can establish acceptance criteria. level.
The inspection can be used to be calculated incomponent
acceptance criteria can be applied at the the review.
System Testing Is Random
• For Real Software
– Defects were inserted at random.
– We do not know where the defects are.
– Our tests execute random bits of code.
System Verification
• Verification is to show that the software was
correctly built
– are there any defects?

• By the time system verification starts, the
product is built.
• We should not find any defects – but we do.
• Question is --- How many defects are left?
Instructions
• Use the scoop to take a sample
• Count the green jelly beans and replace them
with purple jelly beans.
• Record the number of scoops
• Record the number of green Jelly beans in each
scoop
• Plot data
– Cumulative Green vs Cumulative Sampled

• Make estimate of the total number of green jelly
beans originally present until 25 samples were
taken.
Replacing Green With Grape
45
40

Cum Green
Trend

Cum Green

35
30
25
20
15
10
5
0
0

5

10

15
Trials

20

25

30
Repeat the Process
• Use the scoop to take a sample
• Count the green jelly and purple beans and replace
them both.
• Record the number of scoops
• Record the number of green and purple jelly beans in
each scoop
• Plot data
– Cumulative Purple Vs Cumulative Sampled
– Cumulative Green vs Cumulative Sampled
– Cumulative Purple and Green vs Cumulative Samples

• Continue until 25 samples were taken.
• Estimate Defects using extrapolation
Have we run out of purple?
50
Cum Green

40

Cum Purple

35
Cum Beans

45

Cum Total

30

Trend

25
20
15
10
5
0
0

5

10

15
Trials

20

25

30
Re-Estimate
• With jelly beans it is difficult to exactly repeat
a trial but with software you can.
• But we know how many purple jelly beans we
put in and how many are left.
• We have three estimating methods available.
Three Estimating Methods
• First Method
– Uses only the data from one set of tests – divide
the effort in half.

• Second Method
– Uses data from both sets of tests
– Problem: needs two test groups both developing
test cases starting from the same requirements

• Third Method
– Extrapolation from one set of tests
Method 1
• Equal test effort finds an equal fraction of the
defects remaining.
• First n trails => X green jelly beans
• Second n trials => Y green jelly beans
• X/N = Y/(N-X)
– where N = original number of green jelly beans.

• Solving N = X2/(X-Y)
Method 2
• First set n trails => replaced X green jelly beans
• Second set m trials => Y green jelly beans and
Z purple jelly beans.
• If Z is a representative fraction
• N = X/Z(Y+Z)
– where N = original number of green jelly beans.
– Have to keep X a secret.
Method 3
• Plot the cumulative defects vs cumulative
effort E (in this case either cumulative trails
or cumulative sampled beans).
• Fit with a curve of the form

• Y = N(1-e-aE)
• Solve iteratively for a best fit of a and N.
Some Added Estimates
• Make an estimate of the number of samples
required to find 99% of the green and purple
jelly beans.
• Plot the fraction of purple jelly beans vs
number of samples.
• What can we conclude from this data?
Equal Effort => Equal Fraction
90
80

Cum Beans

70
60
50
Cum Green

40

Cum Purple

30

Cum Total
Trend

20

Series5

10
0
0

20

40

60
Trials

80

100

120
What Happens When
• The developers are told what defects are
found?
• For two sets of System Test Cases
– The results of the first set of tests have to be kept secret or
the developers will fix them and your tests will no longer
have value.

• For only one set of System Test Cases
• The results of the tests have to be kept secret or the
developers will fix them and your tests will no longer have
value.
Toward Zero Defects
• Give the estimated defect load to the
developers. They have cheaper ways of
finding and fixing defects.
• Only test to get an estimate and give that
estimate to the developers.
• Only retest to re-do the estimate
• System test cases and automation costs a lot
of money and time – don’t throw this away.
POINTS
• Serial
– Produces straight line
– Estimate defect yield from phases with effort
estimate
– Severity distribution
– Estimate repair effort (based on history)
– Used to estimate future projects with same team
– With historical data to estimate total defects
Points
• Random
–
–
–
–
–
–
–

Estimate total defects
Track defect removal progress
Severity distribution
Establish meaningful defect targets
Estimate effort to reach defect targets
Estimate repair effort
Depending on the results – determine a course of action
• Continue testing and fixing
• Send the SW back to the dev.

More Related Content

Viewers also liked

2007 affordable housing development competition
2007 affordable housing development competition2007 affordable housing development competition
2007 affordable housing development competitionKelvin Xuna
 
Brand Choice
Brand Choice Brand Choice
Brand Choice [x+1]
 
Artist Housing business proposal
Artist Housing business proposalArtist Housing business proposal
Artist Housing business proposalKelvin Xuna
 
Juego Simulador de Gimnasio
Juego Simulador de GimnasioJuego Simulador de Gimnasio
Juego Simulador de GimnasioJugar Con Juegos
 
Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015
Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015
Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015John Watton
 
Teaching Argument Using the Documentary
Teaching Argument Using the DocumentaryTeaching Argument Using the Documentary
Teaching Argument Using the DocumentaryMariann Hawken
 
Thought Leadership - Changing the Rules of Engagement
Thought Leadership - Changing the Rules of EngagementThought Leadership - Changing the Rules of Engagement
Thought Leadership - Changing the Rules of EngagementBolaji Okusaga
 
Behavioural Marketing & how to get your customers to love you
Behavioural Marketing & how to get your customers to love youBehavioural Marketing & how to get your customers to love you
Behavioural Marketing & how to get your customers to love youJohn Watton
 
Should The Electoral College Be Abolished
Should The Electoral College Be AbolishedShould The Electoral College Be Abolished
Should The Electoral College Be Abolishedstoklosa
 
Intro to Film: Screenwriting
Intro to Film: ScreenwritingIntro to Film: Screenwriting
Intro to Film: ScreenwritingRob Nyland
 
I don't always test...but when I do I test in production - Gareth Bowles
I don't always test...but when I do I test in production - Gareth BowlesI don't always test...but when I do I test in production - Gareth Bowles
I don't always test...but when I do I test in production - Gareth BowlesQA or the Highway
 
Loomade kohastumused eluks vees
Loomade kohastumused eluks veesLoomade kohastumused eluks vees
Loomade kohastumused eluks veesjelenamahtra
 

Viewers also liked (13)

2007 affordable housing development competition
2007 affordable housing development competition2007 affordable housing development competition
2007 affordable housing development competition
 
Brand Choice
Brand Choice Brand Choice
Brand Choice
 
Artist Housing business proposal
Artist Housing business proposalArtist Housing business proposal
Artist Housing business proposal
 
Juego Simulador de Gimnasio
Juego Simulador de GimnasioJuego Simulador de Gimnasio
Juego Simulador de Gimnasio
 
Eesti jõed
Eesti jõedEesti jõed
Eesti jõed
 
Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015
Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015
Stop Thinking Facts, Start Thinking Emotions - OI15 Conference, 7 May 2015
 
Teaching Argument Using the Documentary
Teaching Argument Using the DocumentaryTeaching Argument Using the Documentary
Teaching Argument Using the Documentary
 
Thought Leadership - Changing the Rules of Engagement
Thought Leadership - Changing the Rules of EngagementThought Leadership - Changing the Rules of Engagement
Thought Leadership - Changing the Rules of Engagement
 
Behavioural Marketing & how to get your customers to love you
Behavioural Marketing & how to get your customers to love youBehavioural Marketing & how to get your customers to love you
Behavioural Marketing & how to get your customers to love you
 
Should The Electoral College Be Abolished
Should The Electoral College Be AbolishedShould The Electoral College Be Abolished
Should The Electoral College Be Abolished
 
Intro to Film: Screenwriting
Intro to Film: ScreenwritingIntro to Film: Screenwriting
Intro to Film: Screenwriting
 
I don't always test...but when I do I test in production - Gareth Bowles
I don't always test...but when I do I test in production - Gareth BowlesI don't always test...but when I do I test in production - Gareth Bowles
I don't always test...but when I do I test in production - Gareth Bowles
 
Loomade kohastumused eluks vees
Loomade kohastumused eluks veesLoomade kohastumused eluks vees
Loomade kohastumused eluks vees
 

Similar to Tom walton - Jelly Beans And Software Verification

Bagley_HNRS_CRM_talk_2015
Bagley_HNRS_CRM_talk_2015Bagley_HNRS_CRM_talk_2015
Bagley_HNRS_CRM_talk_2015Thomas Bagley
 
Bad metric, bad! - Joseph Ours
Bad metric, bad! - Joseph OursBad metric, bad! - Joseph Ours
Bad metric, bad! - Joseph OursQA or the Highway
 
1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...
1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...
1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...Soils FAO-GSP
 
Share_Total_Quality_Management.pptx
Share_Total_Quality_Management.pptxShare_Total_Quality_Management.pptx
Share_Total_Quality_Management.pptxRiyaKaushal21ZO042
 
Method validation terms quality control and assurance
Method validation terms quality control and assurance Method validation terms quality control and assurance
Method validation terms quality control and assurance MuhammadImranMirza2
 
Countries’ presentation on internal quality control: Sri Lanka
Countries’ presentation on internal quality control: Sri LankaCountries’ presentation on internal quality control: Sri Lanka
Countries’ presentation on internal quality control: Sri LankaExternalEvents
 
Countries’ presentation on internal quality control: China 1
Countries’ presentation on internal quality control: China 1Countries’ presentation on internal quality control: China 1
Countries’ presentation on internal quality control: China 1ExternalEvents
 
Dr. Jeff Baxter - Lab Testing Standardization
Dr. Jeff Baxter - Lab Testing StandardizationDr. Jeff Baxter - Lab Testing Standardization
Dr. Jeff Baxter - Lab Testing StandardizationJohn Blue
 
On Farm Trials - BRRG 2015
On Farm Trials  - BRRG 2015On Farm Trials  - BRRG 2015
On Farm Trials - BRRG 2015Manjitinder Deol
 
Control of analytical quality using stable control materials postgrad
Control of analytical quality using stable control materials postgradControl of analytical quality using stable control materials postgrad
Control of analytical quality using stable control materials postgradMedhatEldeeb2
 
Laboratory Internal Quality Control presentation master revision, 2014
Laboratory Internal Quality Control presentation master revision, 2014Laboratory Internal Quality Control presentation master revision, 2014
Laboratory Internal Quality Control presentation master revision, 2014Adel Elazab Elged
 
Statistical Significance Testing in Information Retrieval: An Empirical Analy...
Statistical Significance Testing in Information Retrieval: An Empirical Analy...Statistical Significance Testing in Information Retrieval: An Empirical Analy...
Statistical Significance Testing in Information Retrieval: An Empirical Analy...Julián Urbano
 
6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)
6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)
6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)Bibhuti Prasad Nanda
 
4th SEALNET meeting, item 8: Training on internal quality control - Overview ...
4th SEALNET meeting, item 8: Training on internal quality control - Overview ...4th SEALNET meeting, item 8: Training on internal quality control - Overview ...
4th SEALNET meeting, item 8: Training on internal quality control - Overview ...Soils FAO-GSP
 
Week 11 Model Evalaution Model Evaluation
Week 11 Model Evalaution Model EvaluationWeek 11 Model Evalaution Model Evaluation
Week 11 Model Evalaution Model Evaluationkhairulhuda242
 

Similar to Tom walton - Jelly Beans And Software Verification (20)

Bagley_HNRS_CRM_talk_2015
Bagley_HNRS_CRM_talk_2015Bagley_HNRS_CRM_talk_2015
Bagley_HNRS_CRM_talk_2015
 
Bad metric, bad! - Joseph Ours
Bad metric, bad! - Joseph OursBad metric, bad! - Joseph Ours
Bad metric, bad! - Joseph Ours
 
Bad metric, bad!
Bad metric, bad!Bad metric, bad!
Bad metric, bad!
 
1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...
1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...
1st NENALAB Meeting_Item 32: Perform QA/QC in the lab by Rob De Hayr, GLOSOLA...
 
Share_Total_Quality_Management.pptx
Share_Total_Quality_Management.pptxShare_Total_Quality_Management.pptx
Share_Total_Quality_Management.pptx
 
Method validation terms quality control and assurance
Method validation terms quality control and assurance Method validation terms quality control and assurance
Method validation terms quality control and assurance
 
QA-QC-PSK.pdf
QA-QC-PSK.pdfQA-QC-PSK.pdf
QA-QC-PSK.pdf
 
Countries’ presentation on internal quality control: Sri Lanka
Countries’ presentation on internal quality control: Sri LankaCountries’ presentation on internal quality control: Sri Lanka
Countries’ presentation on internal quality control: Sri Lanka
 
6.2 msa-gauge-r&r
6.2 msa-gauge-r&r6.2 msa-gauge-r&r
6.2 msa-gauge-r&r
 
Countries’ presentation on internal quality control: China 1
Countries’ presentation on internal quality control: China 1Countries’ presentation on internal quality control: China 1
Countries’ presentation on internal quality control: China 1
 
Dr. Jeff Baxter - Lab Testing Standardization
Dr. Jeff Baxter - Lab Testing StandardizationDr. Jeff Baxter - Lab Testing Standardization
Dr. Jeff Baxter - Lab Testing Standardization
 
On Farm Trials - BRRG 2015
On Farm Trials  - BRRG 2015On Farm Trials  - BRRG 2015
On Farm Trials - BRRG 2015
 
Control of analytical quality using stable control materials postgrad
Control of analytical quality using stable control materials postgradControl of analytical quality using stable control materials postgrad
Control of analytical quality using stable control materials postgrad
 
Overview of statistical tests: Data handling and data quality (Part II)
Overview of statistical tests: Data handling and data quality (Part II)Overview of statistical tests: Data handling and data quality (Part II)
Overview of statistical tests: Data handling and data quality (Part II)
 
Quality control of tti
Quality control of ttiQuality control of tti
Quality control of tti
 
Laboratory Internal Quality Control presentation master revision, 2014
Laboratory Internal Quality Control presentation master revision, 2014Laboratory Internal Quality Control presentation master revision, 2014
Laboratory Internal Quality Control presentation master revision, 2014
 
Statistical Significance Testing in Information Retrieval: An Empirical Analy...
Statistical Significance Testing in Information Retrieval: An Empirical Analy...Statistical Significance Testing in Information Retrieval: An Empirical Analy...
Statistical Significance Testing in Information Retrieval: An Empirical Analy...
 
6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)
6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)
6Six sigma-in-measurement-systems-evaluating-the-hidden-factory (2)
 
4th SEALNET meeting, item 8: Training on internal quality control - Overview ...
4th SEALNET meeting, item 8: Training on internal quality control - Overview ...4th SEALNET meeting, item 8: Training on internal quality control - Overview ...
4th SEALNET meeting, item 8: Training on internal quality control - Overview ...
 
Week 11 Model Evalaution Model Evaluation
Week 11 Model Evalaution Model EvaluationWeek 11 Model Evalaution Model Evaluation
Week 11 Model Evalaution Model Evaluation
 

More from QA or the Highway

KrishnaToolComparisionPPT.pdf
KrishnaToolComparisionPPT.pdfKrishnaToolComparisionPPT.pdf
KrishnaToolComparisionPPT.pdfQA or the Highway
 
Ravi Lakkavalli - World Quality Report.pptx
Ravi Lakkavalli - World Quality Report.pptxRavi Lakkavalli - World Quality Report.pptx
Ravi Lakkavalli - World Quality Report.pptxQA or the Highway
 
Caleb Crandall - Testing Between the Buckets.pptx
Caleb Crandall - Testing Between the Buckets.pptxCaleb Crandall - Testing Between the Buckets.pptx
Caleb Crandall - Testing Between the Buckets.pptxQA or the Highway
 
Thomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdfThomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdfQA or the Highway
 
Thomas Haver - Example Mapping.pdf
Thomas Haver - Example Mapping.pdfThomas Haver - Example Mapping.pdf
Thomas Haver - Example Mapping.pdfQA or the Highway
 
Joe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdf
Joe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdfJoe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdf
Joe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdfQA or the Highway
 
Sarah Geisinger - Continious Testing Metrics That Matter.pdf
Sarah Geisinger - Continious Testing Metrics That Matter.pdfSarah Geisinger - Continious Testing Metrics That Matter.pdf
Sarah Geisinger - Continious Testing Metrics That Matter.pdfQA or the Highway
 
Jeff Sing - Quarterly Service Delivery Reviews.pdf
Jeff Sing - Quarterly Service Delivery Reviews.pdfJeff Sing - Quarterly Service Delivery Reviews.pdf
Jeff Sing - Quarterly Service Delivery Reviews.pdfQA or the Highway
 
Leandro Melendez - Chihuahua Load Tests.pdf
Leandro Melendez - Chihuahua Load Tests.pdfLeandro Melendez - Chihuahua Load Tests.pdf
Leandro Melendez - Chihuahua Load Tests.pdfQA or the Highway
 
Rick Clymer - Incident Management.pdf
Rick Clymer - Incident Management.pdfRick Clymer - Incident Management.pdf
Rick Clymer - Incident Management.pdfQA or the Highway
 
Robert Fornal - ChatGPT as a Testing Tool.pptx
Robert Fornal - ChatGPT as a Testing Tool.pptxRobert Fornal - ChatGPT as a Testing Tool.pptx
Robert Fornal - ChatGPT as a Testing Tool.pptxQA or the Highway
 
Federico Toledo - Extra-functional testing.pdf
Federico Toledo - Extra-functional testing.pdfFederico Toledo - Extra-functional testing.pdf
Federico Toledo - Extra-functional testing.pdfQA or the Highway
 
Andrew Knight - Managing the Test Data Nightmare.pptx
Andrew Knight - Managing the Test Data Nightmare.pptxAndrew Knight - Managing the Test Data Nightmare.pptx
Andrew Knight - Managing the Test Data Nightmare.pptxQA or the Highway
 
Melissa Tondi - Automation We_re Doing it Wrong.pdf
Melissa Tondi - Automation We_re Doing it Wrong.pdfMelissa Tondi - Automation We_re Doing it Wrong.pdf
Melissa Tondi - Automation We_re Doing it Wrong.pdfQA or the Highway
 
Jeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdf
Jeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdfJeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdf
Jeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdfQA or the Highway
 
DesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptx
DesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptxDesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptx
DesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptxQA or the Highway
 
Damian Synadinos - Word Smatter.pdf
Damian Synadinos - Word Smatter.pdfDamian Synadinos - Word Smatter.pdf
Damian Synadinos - Word Smatter.pdfQA or the Highway
 
Lee Barnes - What Successful Test Automation is.pdf
Lee Barnes - What Successful Test Automation is.pdfLee Barnes - What Successful Test Automation is.pdf
Lee Barnes - What Successful Test Automation is.pdfQA or the Highway
 
Jordan Powell - API Testing with Cypress.pptx
Jordan Powell - API Testing with Cypress.pptxJordan Powell - API Testing with Cypress.pptx
Jordan Powell - API Testing with Cypress.pptxQA or the Highway
 
Carlos Kidman - Exploring AI Applications in Testing.pptx
Carlos Kidman - Exploring AI Applications in Testing.pptxCarlos Kidman - Exploring AI Applications in Testing.pptx
Carlos Kidman - Exploring AI Applications in Testing.pptxQA or the Highway
 

More from QA or the Highway (20)

KrishnaToolComparisionPPT.pdf
KrishnaToolComparisionPPT.pdfKrishnaToolComparisionPPT.pdf
KrishnaToolComparisionPPT.pdf
 
Ravi Lakkavalli - World Quality Report.pptx
Ravi Lakkavalli - World Quality Report.pptxRavi Lakkavalli - World Quality Report.pptx
Ravi Lakkavalli - World Quality Report.pptx
 
Caleb Crandall - Testing Between the Buckets.pptx
Caleb Crandall - Testing Between the Buckets.pptxCaleb Crandall - Testing Between the Buckets.pptx
Caleb Crandall - Testing Between the Buckets.pptx
 
Thomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdfThomas Haver - Mobile Testing.pdf
Thomas Haver - Mobile Testing.pdf
 
Thomas Haver - Example Mapping.pdf
Thomas Haver - Example Mapping.pdfThomas Haver - Example Mapping.pdf
Thomas Haver - Example Mapping.pdf
 
Joe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdf
Joe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdfJoe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdf
Joe Colantonio - Actionable Automation Awesomeness in Testing Farm.pdf
 
Sarah Geisinger - Continious Testing Metrics That Matter.pdf
Sarah Geisinger - Continious Testing Metrics That Matter.pdfSarah Geisinger - Continious Testing Metrics That Matter.pdf
Sarah Geisinger - Continious Testing Metrics That Matter.pdf
 
Jeff Sing - Quarterly Service Delivery Reviews.pdf
Jeff Sing - Quarterly Service Delivery Reviews.pdfJeff Sing - Quarterly Service Delivery Reviews.pdf
Jeff Sing - Quarterly Service Delivery Reviews.pdf
 
Leandro Melendez - Chihuahua Load Tests.pdf
Leandro Melendez - Chihuahua Load Tests.pdfLeandro Melendez - Chihuahua Load Tests.pdf
Leandro Melendez - Chihuahua Load Tests.pdf
 
Rick Clymer - Incident Management.pdf
Rick Clymer - Incident Management.pdfRick Clymer - Incident Management.pdf
Rick Clymer - Incident Management.pdf
 
Robert Fornal - ChatGPT as a Testing Tool.pptx
Robert Fornal - ChatGPT as a Testing Tool.pptxRobert Fornal - ChatGPT as a Testing Tool.pptx
Robert Fornal - ChatGPT as a Testing Tool.pptx
 
Federico Toledo - Extra-functional testing.pdf
Federico Toledo - Extra-functional testing.pdfFederico Toledo - Extra-functional testing.pdf
Federico Toledo - Extra-functional testing.pdf
 
Andrew Knight - Managing the Test Data Nightmare.pptx
Andrew Knight - Managing the Test Data Nightmare.pptxAndrew Knight - Managing the Test Data Nightmare.pptx
Andrew Knight - Managing the Test Data Nightmare.pptx
 
Melissa Tondi - Automation We_re Doing it Wrong.pdf
Melissa Tondi - Automation We_re Doing it Wrong.pdfMelissa Tondi - Automation We_re Doing it Wrong.pdf
Melissa Tondi - Automation We_re Doing it Wrong.pdf
 
Jeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdf
Jeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdfJeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdf
Jeff Van Fleet and John Townsend - Transition from Testing to Leadership.pdf
 
DesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptx
DesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptxDesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptx
DesiradhaRam Gadde - Testers _ Testing in ChatGPT-AI world.pptx
 
Damian Synadinos - Word Smatter.pdf
Damian Synadinos - Word Smatter.pdfDamian Synadinos - Word Smatter.pdf
Damian Synadinos - Word Smatter.pdf
 
Lee Barnes - What Successful Test Automation is.pdf
Lee Barnes - What Successful Test Automation is.pdfLee Barnes - What Successful Test Automation is.pdf
Lee Barnes - What Successful Test Automation is.pdf
 
Jordan Powell - API Testing with Cypress.pptx
Jordan Powell - API Testing with Cypress.pptxJordan Powell - API Testing with Cypress.pptx
Jordan Powell - API Testing with Cypress.pptx
 
Carlos Kidman - Exploring AI Applications in Testing.pptx
Carlos Kidman - Exploring AI Applications in Testing.pptxCarlos Kidman - Exploring AI Applications in Testing.pptx
Carlos Kidman - Exploring AI Applications in Testing.pptx
 

Recently uploaded

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 

Recently uploaded (20)

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 

Tom walton - Jelly Beans And Software Verification

  • 2. Introduction • Tom Walton • Software Quality Assurance Specialist • tom_walton@videotron.ca
  • 3. Verification • The processes that are relevant are – Serial Processes • Document and Code Inspection • Unit Test • Integration Test – Random Processes • System Level Verification
  • 4. Serial Processes • Pieces are verified one at a time • Opportunity for 100% Coverage • Historical data used for – Estimating total defects – Severity of remaining defects
  • 5. Instructions • • • • Use the scoop to take a sample Count the green jelly beans Record the number of scoops Record the number of green Jelly beans in each scoop • Plot data – Cumulative Green vs Cumulative Sampled • Make an estimate of the total number of green jelly beans present.
  • 6. Serial Jelly Beans 120 Cum Green Trend Actual 100 Cumulative Green Linear (Cum Green) y = 2.6442x R² = 0.9569 80 60 40 20 0 0 5 10 15 20 Trial 25 30 35 40
  • 8. Integration Test Integration Test Defect Trend 350 Cumulative Defects 300 250 200 150 100 74% of Planned Tests 28 of 34 Activities Reporting 50 0 0 500 1000 1500 Cumulative Total Effort (hr) 2000 2500 3000
  • 9. Using the Data Code Inspection Performance 30.00 Defect Density (Def/KLOC) Avg.Def/KLOC 25.00 Act. Def/KLOC Too Fast 20.00 Too Buggy 15.00 Tend to be large modules Point 1 Point 2 10.00 Def/KLOC x LOCs/hr = Constant 5.00 0.00 0 100 200 300 400 500 600 700 Inspection Speed (LOCS/hr.) Measure Size (LOCS) Inspection Rate(Locs/Hr) Defect Rate(Def/Hr) Defect Density (Def/KLOC) Average Point 1 Point 2 800 800 78 229 267 0.206 1.43 1.33 3.65 6 5 • t The planning datameasures can establish acceptance criteria. level. The inspection can be used to be calculated incomponent acceptance criteria can be applied at the the review.
  • 10. System Testing Is Random • For Real Software – Defects were inserted at random. – We do not know where the defects are. – Our tests execute random bits of code.
  • 11. System Verification • Verification is to show that the software was correctly built – are there any defects? • By the time system verification starts, the product is built. • We should not find any defects – but we do. • Question is --- How many defects are left?
  • 12. Instructions • Use the scoop to take a sample • Count the green jelly beans and replace them with purple jelly beans. • Record the number of scoops • Record the number of green Jelly beans in each scoop • Plot data – Cumulative Green vs Cumulative Sampled • Make estimate of the total number of green jelly beans originally present until 25 samples were taken.
  • 13. Replacing Green With Grape 45 40 Cum Green Trend Cum Green 35 30 25 20 15 10 5 0 0 5 10 15 Trials 20 25 30
  • 14. Repeat the Process • Use the scoop to take a sample • Count the green jelly and purple beans and replace them both. • Record the number of scoops • Record the number of green and purple jelly beans in each scoop • Plot data – Cumulative Purple Vs Cumulative Sampled – Cumulative Green vs Cumulative Sampled – Cumulative Purple and Green vs Cumulative Samples • Continue until 25 samples were taken. • Estimate Defects using extrapolation
  • 15. Have we run out of purple? 50 Cum Green 40 Cum Purple 35 Cum Beans 45 Cum Total 30 Trend 25 20 15 10 5 0 0 5 10 15 Trials 20 25 30
  • 16. Re-Estimate • With jelly beans it is difficult to exactly repeat a trial but with software you can. • But we know how many purple jelly beans we put in and how many are left. • We have three estimating methods available.
  • 17. Three Estimating Methods • First Method – Uses only the data from one set of tests – divide the effort in half. • Second Method – Uses data from both sets of tests – Problem: needs two test groups both developing test cases starting from the same requirements • Third Method – Extrapolation from one set of tests
  • 18. Method 1 • Equal test effort finds an equal fraction of the defects remaining. • First n trails => X green jelly beans • Second n trials => Y green jelly beans • X/N = Y/(N-X) – where N = original number of green jelly beans. • Solving N = X2/(X-Y)
  • 19. Method 2 • First set n trails => replaced X green jelly beans • Second set m trials => Y green jelly beans and Z purple jelly beans. • If Z is a representative fraction • N = X/Z(Y+Z) – where N = original number of green jelly beans. – Have to keep X a secret.
  • 20. Method 3 • Plot the cumulative defects vs cumulative effort E (in this case either cumulative trails or cumulative sampled beans). • Fit with a curve of the form • Y = N(1-e-aE) • Solve iteratively for a best fit of a and N.
  • 21. Some Added Estimates • Make an estimate of the number of samples required to find 99% of the green and purple jelly beans. • Plot the fraction of purple jelly beans vs number of samples. • What can we conclude from this data?
  • 22. Equal Effort => Equal Fraction 90 80 Cum Beans 70 60 50 Cum Green 40 Cum Purple 30 Cum Total Trend 20 Series5 10 0 0 20 40 60 Trials 80 100 120
  • 23. What Happens When • The developers are told what defects are found? • For two sets of System Test Cases – The results of the first set of tests have to be kept secret or the developers will fix them and your tests will no longer have value. • For only one set of System Test Cases • The results of the tests have to be kept secret or the developers will fix them and your tests will no longer have value.
  • 24. Toward Zero Defects • Give the estimated defect load to the developers. They have cheaper ways of finding and fixing defects. • Only test to get an estimate and give that estimate to the developers. • Only retest to re-do the estimate • System test cases and automation costs a lot of money and time – don’t throw this away.
  • 25. POINTS • Serial – Produces straight line – Estimate defect yield from phases with effort estimate – Severity distribution – Estimate repair effort (based on history) – Used to estimate future projects with same team – With historical data to estimate total defects
  • 26. Points • Random – – – – – – – Estimate total defects Track defect removal progress Severity distribution Establish meaningful defect targets Estimate effort to reach defect targets Estimate repair effort Depending on the results – determine a course of action • Continue testing and fixing • Send the SW back to the dev.