Your SlideShare is downloading. ×
The Tester’s Dashboard: Release Decision Support
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

The Tester’s Dashboard: Release Decision Support

179

Published on

Industry track, ISSRE-18. San Jose November 2, 2010. …

Industry track, ISSRE-18. San Jose November 2, 2010.
Shows how to the Reliability Demonstration Chart, state transition coverage, and new metric "Relative Proximity" can be used to make better release decisions.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
179
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. The Tester’s Dashboard: Release Decision Support
    Robert V. Binder
    System Verification Associates, LLC
    rbinder@ieee.org
    Peter B. Lakey
    Cognitive Concepts, Inc.
    peterlakey@sbcglobal.net
  • 2. Overview
    Complementary metrics for release decision-support
    Model-based testing
    Operational profile
    Model coverage metrics
    Reliability Demonstration Chart
    Relative Proximity
    Case Study
    Observations
  • 3. Release Decision Support
  • 4.
  • 5. Model-based Reliability Estimation
    Test suites must be
    Proportional to operational profile
    Sequentially feasible
    Input feasible
    Approach
    Markov model
    Monte Carlo simulation
    Post run analytics
  • 6. Model Coverage Metrics
    %States Reached
    % State-Transitions Reached
  • 7. Reliability Demonstration Chart
    Sequential Sampling
    Risk-Adjusted
    Musa equations
    http://sourceforge.net/projects/rdc/
  • 8. Relative Proximity
    Kullback-LieberDistance
    Information theoretic
    Characterizes difference in variation of message population E (expected) and sample A (actual) as “relative entropy”
    KLD = ∑ 𝐴𝑖 (𝑙𝑜𝑔2 (𝐴𝑖 / E𝑖 ))
     
    Relative Proximity
    KLD math doesn’t work unless failures modeled (sum of the actuals must be 1.0)
    Assume the target failure rate is aggregate
    Allocate failure rate in proportion to each operation
  • 9. Profile Explicit Failure Modes
    Assume maximum acceptable failure rate intensity of 1 in 10,000
  • 10. Profile Explicit Failure Modes
    Relative Proximity indicates the difference between actual and observed failure rates
    Many possible operation failure rates with better or worse fidelity
    RDC based on aggregate FIO, not sensitive to operation variance
  • 11. Case Studies
    Stochastic Models
    Assumed Failure Rates
    Word Processing Application
    Ground-Based Midcourse Missile Defense
  • 12. GBMDTest Run, 0-100
  • 13. GBMD Test Run, 1K, 5K
  • 14. GMBD Test Run, 10K
  • 15. GBMD Relative Proximity Trend
  • 16. Observations
    Model coverage indicates minimal sufficiency
    Wouldn’t release without all state-xtn pairs covered
    Stochastic can take a long time to do this
    Cover with N+ first
    RDC assumes “flat” profile
    With sequential constraints, may be optimistic
    Strength is explicit risk-adjustment
    Relative Proximity will indicate when operation-specific Failure Intensity is as expected (or not)
  • 17. Q & A

×