Your SlideShare is downloading. ×
  • Like
The Tester’s Dashboard: Release Decision Support
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

The Tester’s Dashboard: Release Decision Support

  • 255 views
Published

Industry track paper, November 2, 2010. ISSRE-18, San Jose. Combining reliability, coverage, and an information-theoretic metric provides better feedback for release decisions.

Industry track paper, November 2, 2010. ISSRE-18, San Jose. Combining reliability, coverage, and an information-theoretic metric provides better feedback for release decisions.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
255
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. The Tester’s Dashboard:Release Decision Support Robert V. BinderSystem Verification Associates, LLC rbinder@ieee.org Peter B. Lakey Cognitive Concepts, Inc. peterlakey@sbcglobal.net
  • 2. Overview• Complementary metrics for release decision- support – Model-based testing • Operational profile • Model coverage metrics – Reliability Demonstration Chart – Relative Proximity• Case Study• Observations
  • 3. Release Decision Support
  • 4. Model-based Reliability Estimation• Test suites must be – Proportional to operational profile – Sequentially feasible – Input feasible• Approach – Markov model – Monte Carlo simulation – Post run analytics
  • 5. Model Coverage Metrics • % States Usage Profile ReachedS T • % State- Transitions Observe System Reached Failure Trigger Latent Defect Software System Process Space Observe fault System activated Failure Data Space
  • 6. Reliability Demonstration Chart• Sequential Sampling• Risk- Adjusted• Musa equations http://sourceforge.net/projects/rdc/
  • 7. Relative Proximity• Kullback-Lieber Distance – Information theoretic – Characterizes difference in variation of message population E (expected) and sample A (actual) as “relative entropy” KLD = ∑ 𝐴 𝑖 (𝑙𝑜𝑔2 (𝐴 𝑖 / E 𝑖 ))• Relative Proximity – KLD math doesn’t work unless failures modeled (sum of the actuals must be 1.0) – Assume the target failure rate is aggregate – Allocate failure rate in proportion to each operation
  • 8. Profile Explicit Failure Modes• Assume maximum acceptable failure rate intensity of 1 in 10,000 Operation Mode Standard Explicit Failure Expected Number, Profile Profile 10000 Tests A Pass 0.7 0.6993 6993 B Pass 0.2 0.1998 1998 C Pass 0.1 0.0999 999
  • 9. Profile Explicit Failure Modes Mode Expected Actual KL Distance Actual KL Distance A Pass 6993 7000 10.104 6990 -4.327 Fail 7 0 0.000 10 5.146 B Pass 1998 1990 -11.518 2000 2.887 Fail 2 10 23.219 0 0.000 C Pass 999 980 -27.149 994 -7.195 Fail 1 20 86.439 6 15.510 10000 10000 81.094 10000 12.020• Relative Proximity indicates the difference between actual and observed failure rates• Many possible operation failure rates with better or worse fidelity• RDC based on aggregate FIO, not sensitive to operation variance
  • 10. Case Studies• Stochastic Models• Assumed Failure Rates• Word Processing Application• Ground-Based Midcourse Missile Defense
  • 11. GBMD Test Run, 0-100
  • 12. GBMD Test Run, 1K, 5K
  • 13. GMBD Test Run, 10K
  • 14. GBMD Relative Proximity Trend 1600.00 1484.00 1400.00 1200.00 1000.00 800.00 600.00 418.20 400.00 200.00 67.90 12.00 6.30 0.00 10 100 1000 10000
  • 15. Observations• Model coverage indicates minimal sufficiency – Wouldn’t release without all state-xtn pairs covered – Stochastic can take a long time to do this – Cover with N+ first• RDC assumes “flat” profile – With sequential constraints, may be optimistic – Strength is explicit risk-adjustment• Relative Proximity will indicate when operation- specific Failure Intensity is as expected (or not)
  • 16. Q&A