Risk Based Testing
and
Random Testing

Dr. Himanshu Hora
SRMS College of Engineering & Technology
Bareilly (INDIA)
Risk Based Testing
and
Random Testing
• Use of Risk Analysis and Metrics
for Software Testing
• Focus Testing to Save Time...
The Challenges
•
•
•
•

Time Constraints
Resource Constraints
Quality Requirements
Risk Factors:
– New technology
– Lack o...
Risk Analysis and Testing
Test Plan
Test Item Tree

Risk
Strategy

Risk
Identification

Testing,
Inspection etc.

Risk
Ass...
Risk Based Testing - Theory
• The Formula

Re(f) P(f)*C(f)
– Re(f) Risk Exposure of function f
– P(f) - Probability of a f...
Risk Based Testing - Approach
• Plan: Identify Elements to be Tested
– Logical or physical Functions, Modules etc.

• Iden...
Simple Test Metrics
• Test Planning
– Number of test cases per function
– Number of hours testing per function

• Progress...
Risk Based Testing - Metrics
• Identify Areas with “High Risk Exposure”
– Probability and Cost

• All functions/modules sh...
Risk Based Testing - Example
Ranking the functions based on Risk Exposure
The Probability of a Fault
The Cost of a Fault

...
High

Probability

Risk Based Testing
- Reporting

1

2
3
4

TECHNICAL
INTERFACE
RISK

10

1

Low
Low

440

Low Medium

Pr...
Risk Based Testing - Practice
Prior to test execution:
identify critical transactions
1

Test Execution
identifies
“bad” t...
Planning and Progress Tracking

Number of Test
Cases

On-line Test Cases Completed

Planned
Executed

QAed
Date

Started
P...
Progress Indicators - “To be vs. Actual”

“To be
Retested”
vs.
“Actually
Retested”

To Be Fixed

Actually fixed

To Be Res...
Progress Indicators - Hours Used
Number of hours
for finding one
fault and for
fixing one

Hours per Fault for Test and Fi...
“Estimated to Complete”
• ETC for system test based on:

Hours ETC

– Number of hours testing per fault found
– Number of ...
Benefits of Risk Based Testing
• Improved Quality?
– all critical functions tested

• Reduced Time and Money in Testing
– ...
Test Process Work Flow
Risk Identification

Risk Assessment
Basic Test
Data

PD

LD

Case Quality
Standards

Test Exec.
Pr...
Summary
• Risk Based Test Approach
– Focused Testing
• Reduced Resources
• Improved Quality

– Metrics are fundamental

• ...
Random testing
– Start off with a practical look, and some useful
ideas to get you started on the project: random
testing ...
A Little Background
– Generate program inputs at random
– Drawn from some (possibly changing) probability
distribution
“Th...
Random Testing
• Millions of operations and scenarios, automatically
generated
• Run on fast & inexpensive workstations
• ...
The Goals
• Randomize early testing (since it is not possible
to be exhaustive)
– We don’t know where the bugs are

Nomina...
The Goals
• Make use of desktop hardware for early testing – vs.
expensive (sloooow) flight hardware testbeds
– Many fault...
The Goals
• Automate early testing
– Run tests all the time, in the background, while continuing
development efforts

• Au...
Random testing
• Simulated flash hardware layer allows random
fault injection
• Most development/early testing can be done...
Differential Testing
• How can we tell if a test succeeds?
– POSIX standard for file system operations
• IEEE produced, AN...
Differential Testing
• How can we tell if a test succeeds?
– The POSIX standard specifies (mostly) what correct
behavior i...
Random Differential Testing
Choose (POSIX) operation F

Perform F on NVFS

Perform F on Reference
(if applicable)

Compare...
Don’t Use Random Testing for Everything!
• Why not test handing read a null pointer?
– Because (assuming the code is corre...
Principles Used
•
•
•
•

Random testing (with feedback)
Test automation
Hardware simulation & fault injection
Use of a wel...
Synopsis
• Random testing is sometimes a powerful
method and could likely be applied more
broadly in other missions
– Alre...
Thank You

Dr. Himanshu Hora
SRMS College of Engineering & Technology
Bareilly (INDIA)
Upcoming SlideShare
Loading in...5
×

Risk based testing and random testing

507
-1

Published on

Risk based testing and random testing

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
507
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Risk based testing and random testing

  1. 1. Risk Based Testing and Random Testing Dr. Himanshu Hora SRMS College of Engineering & Technology Bareilly (INDIA)
  2. 2. Risk Based Testing and Random Testing • Use of Risk Analysis and Metrics for Software Testing • Focus Testing to Save Time and Money while maintaining quality • How to develop metrics to manage and organise large test projects
  3. 3. The Challenges • • • • Time Constraints Resource Constraints Quality Requirements Risk Factors: – New technology – Lack of knowledge – Lack of experience • Take Control!
  4. 4. Risk Analysis and Testing Test Plan Test Item Tree Risk Strategy Risk Identification Testing, Inspection etc. Risk Assessment Matrix: Cost and Probability Risk Mitigation Test Metrics Risk Reporting Risk Prediction
  5. 5. Risk Based Testing - Theory • The Formula Re(f) P(f)*C(f) – Re(f) Risk Exposure of function f – P(f) - Probability of a fault in function f – C(f) - Cost related to a fault in function f
  6. 6. Risk Based Testing - Approach • Plan: Identify Elements to be Tested – Logical or physical Functions, Modules etc. • Identify Risk Indicators – What is important to predict the probability of faults? • Identify Cost of faults • Identify Critical Elements – I.e. functions, tasks, activities etc. based on Risk Analysis (Indicators and Cost) • Execute: Improve the Test Process and Organization: Schedule and Track
  7. 7. Simple Test Metrics • Test Planning – Number of test cases per function – Number of hours testing per function • Progress Tracking – – – – Number of tests planned, executed and completed Number of faults per function Number of hours used for test and fix Estimated to Complete • Probability of faults - Indicators – – – – New functionality Size Complexity Quality of previous phases and documents • Cost of Faults
  8. 8. Risk Based Testing - Metrics • Identify Areas with “High Risk Exposure” – Probability and Cost • All functions/modules should be tested to a “minimum level” • “Extra Testing” in areas with high risk exposure • Establish Test Plan and Schedule – Monitor Quality • Number of Faults per function and time – Monitor Progress • Number of hours in test and fix -> ETC
  9. 9. Risk Based Testing - Example Ranking the functions based on Risk Exposure The Probability of a Fault The Cost of a Fault C(c) C(s) Re(f) P(f)* 2 Example: Cost Func. Probability New Design Func. Quality Size 5 5 1 Risk Com- Weighted Exp. plexity Averag func. 3 f C( s ) C(c) Avr. Interest Calc. 3 3 3 2 3 3 3 37 111 Close Account 1 3 2 2 2 2 3 31 62 Cust. Profitab. 2 1 1,5 3 3 2 3 41 61,5 Other Probability Factors might include: Function Points, Frequency of Use etc.
  10. 10. High Probability Risk Based Testing - Reporting 1 2 3 4 TECHNICAL INTERFACE RISK 10 1 Low Low 440 Low Medium Probability High 510 439 11 2 2 370 369 5 302 Low Low Medium Medium BUSINESS RISK High High Consequence Consequence High
  11. 11. Risk Based Testing - Practice Prior to test execution: identify critical transactions 1 Test Execution identifies “bad” transactions “Top-20” 2 Extra Testing: - Additional testing by product specialist - Automated regression testing 3
  12. 12. Planning and Progress Tracking Number of Test Cases On-line Test Cases Completed Planned Executed QAed Date Started Planned Actual
  13. 13. Progress Indicators - “To be vs. Actual” “To be Retested” vs. “Actually Retested” To Be Fixed Actually fixed To Be Restested, Actually Retested and Rejected Number of Faults • “To be fixed” vs. “Actually fixed” Number of Faults To Be Fixed and Actually Fixed To be retested Act. retested Rejected
  14. 14. Progress Indicators - Hours Used Number of hours for finding one fault and for fixing one Hours per Fault for Test and Fix Hours per Fault Online Batch Fix Hours per Fault for Test and Fix Hours per Fault Number of hours for finding one fault and for fixing one Test Fix Test Date
  15. 15. “Estimated to Complete” • ETC for system test based on: Hours ETC – Number of hours testing per fault found – Number of hours fixing per fault – Number of faults found Calculated ETC and Actual Hours per function Actual to Complete – Number of at Time t fixes being rejected – Number of remaining tests (functions to be tested) Date Estimated to Complete at Time t
  16. 16. Benefits of Risk Based Testing • Improved Quality? – all critical functions tested • Reduced Time and Money in Testing – effort not wasted on non critical or low risk functions • Improved customer confidence – due to customer involvement and good reporting and progress tracking
  17. 17. Test Process Work Flow Risk Identification Risk Assessment Basic Test Data PD LD Case Quality Standards Test Exec. Procedure Test Case Case Build Procedure Test Completed QC / QA Good Test Exec Risk Mitigation Good/ Bad Bad PTDs Raised Risk Reporting Retest Risk Prediction Problem Mngmnt. Procedure Fix Fix Procedure CR Change Mngmnt. Procedure Regression Test ProAte
  18. 18. Summary • Risk Based Test Approach – Focused Testing • Reduced Resources • Improved Quality – Metrics are fundamental • Process and Organization must support the new strategy – Metrics must support the organization and process
  19. 19. Random testing – Start off with a practical look, and some useful ideas to get you started on the project: random testing for file systems – Then take a deeper look at the notion of feedback and why it is useful: method for testing OO systems from ICSE a couple of years • Then back out to take a look at the general idea of random testing, if time permits
  20. 20. A Little Background – Generate program inputs at random – Drawn from some (possibly changing) probability distribution “Throw darts at the state space, without drawing a bullseye” – May generate the same test (or equivalent tests) many times – Will perform operations no sane human would ever perform
  21. 21. Random Testing • Millions of operations and scenarios, automatically generated • Run on fast & inexpensive workstations • Results checked automatically by a reference oracle • Hardware simulation for fault injection and reset simulation (x 100000) A day (& night) of testing (x 100000) (x 100000) (x 100000)
  22. 22. The Goals • Randomize early testing (since it is not possible to be exhaustive) – We don’t know where the bugs are Nominal Scenario Tests Randomized Testing
  23. 23. The Goals • Make use of desktop hardware for early testing – vs. expensive (sloooow) flight hardware testbeds – Many faults can be exposed without full bit-level hardware simulation
  24. 24. The Goals • Automate early testing – Run tests all the time, in the background, while continuing development efforts • Automate test evaluation – Using reference systems for fault detection and diagnosis – Automated test minimization techniques to speed debugging and increase regression test effectiveness • Automate fault injection – Simulate hardware failures in a controlled test environment
  25. 25. Random testing • Simulated flash hardware layer allows random fault injection • Most development/early testing can be done on workstations • Lots of available compute power – can cover many system behaviors • Will stress software in ways nominal testing will not
  26. 26. Differential Testing • How can we tell if a test succeeds? – POSIX standard for file system operations • IEEE produced, ANSI/ISO recognized standard for file systems • Defines operations and what they should do/return, including nominal and fault behavior File system / POSIX operation Result mkdir (“/eng”, …) mkdir (“/data”, …) creat (“/data/image01”, …) creat (“/eng/fsw/code”, …) mkdir (“/data/telemetry”, …) unlink (“/data/image01”) SUCCESS SUCCESS SUCCESS ENOENT SUCCESS SUCCESS /eng /data image01 /telemetry
  27. 27. Differential Testing • How can we tell if a test succeeds? – The POSIX standard specifies (mostly) what correct behavior is – We have heavily tested implementations of the POSIX standard in every flavor of UNIX, readily available to us – We can use UNIX file systems (ext3fs, tmpfs, etc.) as reference systems to verify the correct behavior of flash – First differential approach (published) was McKeeman’s testing for compilers
  28. 28. Random Differential Testing Choose (POSIX) operation F Perform F on NVFS Perform F on Reference (if applicable) Compare return values Compare error codes Compare file systems Check invariants (inject a fault?)
  29. 29. Don’t Use Random Testing for Everything! • Why not test handing read a null pointer? – Because (assuming the code is correct) it guarantees some portion of test operations will not induce failure – But if the code is incorrect, it’s easier and more efficient to write a single test – The file system state doesn’t have any impact (we hope!) on whether there is a null check for the buffer passed to read • But we have to remember to actually do these non-random fixed tests, or we may miss critical, easy-to-find bugs!
  30. 30. Principles Used • • • • Random testing (with feedback) Test automation Hardware simulation & fault injection Use of a well-tested reference implementation as oracle (differential testing) • Automatic test minimization (delta-debugging) • Design for testability – Assertions – Downward scalability (small model property) – Preference for predictability
  31. 31. Synopsis • Random testing is sometimes a powerful method and could likely be applied more broadly in other missions – Already applied to four file system-related development efforts – Part or all of this approach is applicable to other critical components (esp. with better models to use as references)
  32. 32. Thank You Dr. Himanshu Hora SRMS College of Engineering & Technology Bareilly (INDIA)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×