Your SlideShare is downloading. ×
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Requirements Based Testing (Powerpoint Talk)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Requirements Based Testing (Powerpoint Talk)

944

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
944
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • At first, we started by only verifying properties of the mode logic that we considered safety properties , which is to say, those that could put the aircraft in a hazardous state , but we soon realized that all our functional requirements written as shall statements could be specified as properties over our model. So we went back to our DOORS database and translated each of our shall statement into a CTL property.
  • Role of the FGS in the overall scheme of things
  • Transcript

    • 1. Requirements-Based Testing Dr. Mats P. E. Heimdahl University of Minnesota Software Engineering Center Dr. Steven P. Miller Dr. Michael W. Whalen Advanced Computing Systems Rockwell Collins 400 Collins Road NE, MS 108-206 Cedar Rapids, Iowa 52498 [email_address]
    • 2. Outline of Presentation
      • Motivation
      • Validation Testing
      • Conformance Testing
      • What’s Next
    • 3. How We Develop Software SW High-Level Reqs. Development SW Design Description Dev. (SW Low-Level Reqs. & SW Arch. SW Source Code Dev. SW Integration (Executable Code Production) SW Low-Level Testing SW Integration Testing HW/SW Integration Testing
    • 4. How we Will Develop Software (From V to a Y) SW High-Level Reqs. Development Software Model SW Integration (Executable Code Production) SW Integration Testing HW/SW Integration Testing Can we trust the code generator? How do we know our model is correct? Validation Testing Formal Verification Conformance Testing
    • 5. Outline of Presentation
      • Motivation
      • Validation Testing
      • Conformance Testing
      • What’s Next
    • 6. How we Will Develop Software (From V to a Y) SW High-Level Reqs. Development Software Model SW Integration (Executable Code Production) SW Integration Testing HW/SW Integration Testing How do we know our model is correct?
    • 7. Modeling Process SW High-Level Reqs. Development Software Model SW Integration (Executable Code Production) Desired Model Properties High-Level Requirements Low-Level Requirements
    • 8. Problem—Modeling Frenzy SW High-Level Reqs. Development Software Model SW Integration (Executable Code Production) Desired Model Properties How do we know the model is “right”? How do we test the model? Headfirst into modeling
    • 9. One Solution: Redefine Requirements System Reqs. Development Software Model SW Integration (Executable Code Production) SW Integration Testing HW/SW Integration Testing The model is the requirements Use Engineering Judgment when Testing Software Development Processes (DO - 178B) Software Development Processes (DO - 178B) System Development Processes (ARP4754) System Development Processes (ARP 4754)
    • 10. One Solution: Redefine Requirements System Reqs. Development Software Model SW Integration (Executable Code Production) SW Integration Testing HW/SW Integration Testing The model is the requirements Use Engineering Judgment when Testing Software Development Processes (DO - 178B) Software Development Processes (DO - 178B) System Development Processes (ARP4754) System Development Processes (ARP 4754) My Comment
    • 11. Testing Does not go Away System Reqs. Development Software Model SW Integration (Executable Code Production) SW Integration Testing HW/SW Integration Testing Extensive Testing (MC/DC)
    • 12. It Simply Moves System Reqs. Development Software Model SW Integration (Executable Code Production) SW Integration Testing HW/SW Integration Testing Extensive Testing (MC/DC)
    • 13. Do it Right! SW High-Level Reqs. Development Specification Test – Is the Model Right? Software Model SW Integration (Executable Code Production) Desired Model Properties Analysis (Model Checking, Theorem Proving)
    • 14. How Much to Test? State Coverage Masking MC/DC? Transition Coverage? Decision Coverage? Def-Use Coverage? Something New?? MC/DC Where Do the Tests Come From?
    • 15. Requirements Based Testing SW High-Level Reqs. Development Cover the Properties! Properties are Requirements… Software Model SW Integration (Executable Code Production) Desired Model Properties
    • 16. Properties are Requirements
    • 17. Requirements Based Testing Advantages
      • Objective Measurement of Model Validation Efforts
        • Requirements Coverage in Model-based Development
        • Help Identify Missing Requirements
          • Measure converge of model
      • Basis for Automated Generation of Requirements-based Tests
        • Even If Properties Are Not Used for Verification, They Can Be Used for Test Automation
      How Are Properties “Covered” with Requirements-based Tests?
    • 18. Property Coverage
      • “ If the onside FD cues are off, the onside FD cues shall be displayed when the AP is engaged”
        • G(((!Onside_FD_On & !Is_AP_Engaged) -> X(Is_AP_Engaged -> Onside_FD_On))
      • Property Automata Coverage
        • Cover a Synchronous Observer Representing the Requirement (Property)
      • Structural Property Coverage
        • Demonstrate Structurally “Interesting” Ways in Which the Requirement (Property) Is Met
    • 19. Property Automata Coverage
      • Cover Accepting State Machine As Opposed to Structure of Property
      • Büchi Coverage
        • State Coverage, Transition Coverage, Lasso Coverage…
    • 20. Alternative Machine
      • Different synthesis algorithms give different automata
        • Will affect the test cases required for coverage
    • 21. Structural Property Coverage
      • Define Structural Coverage Criteria for the Property Specification
        • Traditional Condition-based Criteria such as MC/DC Prime Candidates
      • Property Coverage Different than Code Coverage
        • Coverage of Code and Models
          • Evaluate a decision with a specific combination of truth values in the decision
        • Coverage of Properties
          • Run an execution scenario that illustrates a specific way a requirement (temporal property) is satisfied
    • 22. Example
        • G(((!Onside_FD_On & !Is_AP_Engaged) -> X(Is_AP_Engaged -> Onside_FD_On))
      • Demonstrate That Somewhere Along Some Execution Trace Each MC/DC Case Is Met
        • Only the “positive” MC/DC cases
          • The negative cases should have no traces
      • In the Case of G( p )—Globally p Holds—we Need to Find a Test Where
        • in the prefix the requirement p is met
        • we reach a state of the trace where the requirement p holds because of the specific MC/DC case of interest – let us call this case a
        • then the requirement p keeps on holding through the remainder of the trace
      • p U ( a U X(G p ))
      p p a p p p
    • 23. Summary
      • Objective Measurement of Model Validation Efforts
        • Requirements Coverage in Model-based Development
        • Help Identify Missing Requirements
      • Basis for Automated Generation of Requirements-based Tests
        • Even If Properties Are Not Used for Verification, They Can Be Used for Test Automation and Test Measurement
      • Challenges
        • How Are Properties Specified?
          • Combination of Observers and Temporal Properties
        • What Coverage Criteria Are Suitable?
        • How Is Automation Achieved?
        • How Do We Eliminate “Obviously” Bad Tests? Should We?
        • How Do We Generate “Realistic” Test-cases?
        • Rigorous Empirical Studies Badly Needed
    • 24. Outline of Presentation
      • Motivation
      • Validation Testing
      • Conformance Testing
      • What’s Next
    • 25. How we Will Develop Software (From V to a Y) SW High-Level Reqs. Development Software Model SW Integration (Executable Code Production) SW Integration Testing HW/SW Integration Testing Can we trust the code generator?
    • 26. “Correct” Code Generation—How?
      • Provably Correct Compilers
        • Very Hard (and Often Not Convincing)
      • Proof Carrying Code
      • Generate Test Suites From Model
        • Compare Model Behavior With Generated Code
        • Unit Testing Is Now Not Eliminated, but Largely Automated
      Specification/Model Implementation Output Output Specification Based Tests Generate
    • 27. Existing Capabilities
      • Several Commercial and Research Tools for Test-Case Generation
        • TVEC
          • Theorem Proving and Constraint Solving techniques
        • Reactis from Reactive Systems Inc.
          • Random, Heuristic, and Guided Search
        • University of Minnesota
          • Bounded Model Checking
        • NASA Langley
          • Bounded Model Checking/Decision Procedures/Constraint Solving
      • Tools Applicable to Relevant Notations
        • In Our Case Simulink
    • 28. An Initial Experiment
      • Used a Model of the Mode Logic of a Flight Guidance System As a Case Example
      • Fault Seeding
        • Representative Faults
        • Generated 100 Faulty Specifications
      • Generate Test Suites
        • Selection of Common (and Not So Common) Criteria
      • Fault Detection
        • Ran the Test Suites Against the Faulty Specifications
        • Recorded the Total Number of Faults Detected
    • 29. Fault Finding Results Same Effort
    • 30. Model “Cheats” Test Generator FCS Architecture
    • 31. Effect of Test Set Size
    • 32. Summary
      • Automated Generation of Conformance Tests
        • Current Technology Largely Allows This Automation
      • Challenges
        • Development of Suitable Coverage Criteria
        • Effect of Test Set Size on Test Set Effectiveness
        • Effect of Model Structure on Coverage Criteria Effectiveness
        • Traceability of Tests to Constructs Tested
        • Empirical Studies of Great Importance
    • 33. Outline of Presentation
      • Motivation
      • Conformance Testing
      • Validation Testing
      • What’s Next
    • 34. New Challenges for Testing
      • Model Validation – Requirements-based Testing
        • How Do We Best Formalize the Requirements?
        • What Coverage Criteria Are Feasible?
        • Which Coverage Criteria Are Effective (If Any)?
        • How Do We Generate “Realistic” Tests?
        • Will This Be a Practical (Tractable) Solution?
      • Conformance Testing
        • What Coverage Criteria Are Effective?
          • Detecting Faults From Manual Coding
          • Detecting Faults From Code Generation
        • Relationship Between Model Structure and Criteria Effectiveness
        • Traceability From Tests to Model
        • Relationship Between Model Coverage and Code Coverage
          • Optimizations in Code Generator Will Compromise Coverage
    • 35. Discussion
    • 36. Perfection is Not Necessary
      • Tools and Models Only Need To Be Better Than Manual Processes…
        • How Do We Demonstrate This?
          • Empirical Studies Are of Great Importance
      ≥ Missed Faults I Think Many Already Are
    • 37. DO-178B Test Objectives
      • The executable code complies with the high-level requirements.
      • The executable code complies with the specification (low-level requirements).
      • Test coverage of high-level requirements is achieved
      • Test coverage of specification (low-level requirements) is achieved
      • Test coverage of the executable code is achieved
      Requirements-Based Testing Conformance Testing

    ×