Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools

Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools?* Alex Dekhtyar Jane Huffman Hayes Jody Larsen *Funded by NASA

Introduction
What is requirements traceability?
Information retrieval and traceability
What’s different about traceability?
Research questions
Experimental design & validity
Results
Analysis
Future Work
Questions

What is Requirements Traceability?
“ Requirements traceability is the ability to describe and follow the life of a requirement , in both a forward and backward direction , i.e., from its origins, through its development and specification, to its subsequent deployment and use, and through periods of ongoing refinement and iteration in any of these phases.”
Gotel and Finkelstein, 1994.

Why is Traceability Important?
Requirements verification and validation
Impact analysis
Validating contracts and bids
Testing Coverage

Design Document Information Retrieval and Traceability Representation Requirements Document Matching algorithm Analyst

What’s Different about Traceability?
Typical Information Retrieval Problem:
Search Engines: Google, Yahoo
Large and small documents
Document indexes must support arbitrary search queries
High accuracy needed
Requirements Traceability
Queries are known prior to indexing
Documents are all very small
High coverage needed

IR Based Traceability Frameworks
ADAMS Re-Trace
Part of an artifact management suite
Latent Semantic Indexing (LSI) Based
Web Application
Poirot: Tracemaker
Probabilistic Information Retrieval
Web Application
RETRO
Vector Space Model and LSI
Windows Application

RETRO.NET
Candidate Link
Top Link
Feedback
Confirmed Links

Research Questions
What is the optimal upper bound on analyst efficiency using Information Retrieval based tools?
What analyst behaviors produce accurate and complete RTMs most efficiently?

Experimental Design
Subject
A simulated analyst engaging in a specified behavior
Variables
Independent
Analyst Behavior
Dependent
Confirmed True Links
Observed Links
Confirmed Recall
Precision
Selectivity

Measures N - number of low-level requirements; M - number of high-level requirements; Hits - number of confirmed candidate links Strikes - number of false positives Misses - number of missed links Precision = Hits Hits + Strikes Recall = Hits Hits +Misses Selectivity = Hits + Strikes M * N

Global Ordering Without Feedback
All candidate link lists merged
Links processed one at time in descending order
Results in analyst looking at links that have highest global similarity ratings first

Local Ordering Without Feedback
Each candidate link list sorted in descending order
Top link for each requirement element is processed
Results in analyst looking at the links for each requirement by highest similarity rating

Global Ordering With Feedback
All candidate link lists merged
Links processed one at time in descending order
After each link is processed, feedback is executed
Results in analyst looking at links that have highest global similarity ratings first

Local Ordering With Feedback
Each candidate link list sorted in descending order
Top link for each requirement element is processed
After all top links have been processed, feedback is executed
Results in analyst looking at the links for each requirement by highest similarity rating

Dataset
CM-1
http://mdp.ivv.nasa.gov/mdp_glossary.html#CM1
In PROMISE repository
Sanitized dataset for a NASA scientific instrument
Goal is to help analyst avoid examining all 36,556 links
36,556 Total # of retrieved candidate links 70.7% Selectivity 358 Total # of correct links retrieved by basic Vector Space 99% Recall 0.1% Precision # of elements in requirements (high-level) 235 # of elements in design specification (low-level) 220 # of correct links 361

Experimental Design Detail
Two Studies
Impact of Behavior on Analyst Effort when Recall is fixed
Recall set to 89%
Included all four structured behaviors
Impact of Behavior on Precision and Recall when Analyst Effort is fixed
Effort fixed to 1595 observed links
Added two additional behaviors
Global, Random Order
Local, Random Order

Random?
Menzies et al. note that software analysis should “start with random methods because they are so cheap, moving to the more complex methods only when random methods fail.”
Source: Menzies, T., Owen, D., and Richardson, J. The Strangest Thing about Software. IEEE Computer , January 2007, pp. 54 – 60.

Random Methods
Global Filtered Random Selection
All Candidate Link Lists Merged
List Sorted By Similarity
Bottom 75% (Least Relevant) Removed
1595 Links Selected Randomly
Local Filtered Random Selection
All Candidate Link Lists Sorted By Similarity
Bottom 75% (Least Relevant) Removed
7 or 8 Links Selected Randomly For Each Requirement
1595 Links Selected
Executed 1000 times
Median values were used

Experiment Operation
Input:
Requirement and Specification Documents
Desired Analyst Behavior
Operation:
Performs multiple rounds of interaction
Provides link status determination for each observed link
Output:
Candidate RTM
Dependent Variable Data

Experiment Validity
Internal Threats
Accuracy of Requirements Traceability Matrix
Suitability of text for Information Retrieval methods
External Threats
Can Results Based on CM-1 Dataset be Generalized?
Realism of Simulated Analyst Behavior

Study 1 Results, Fixed Recall Observed Links Precision Selectivity Local, Feedback 1595 20% 3% Local, No Feedback 1713 19% 3% Global, Feedback 5399 6% 10% Global, No Feedback 6149 5% 12%

Study 2 Results, Fixed Effort Confirmed Links Precision Recall Local, Feedback 321 20% 89% Local, No Feedback 326 20% 87.5% Global, Feedback 236 15% 65% Global, No Feedback 227 14% 63% Local, Random 65 4% 18% Global, Random 58 3.6% 16%

Study 2 Intermediate Results

Analysis
Knowing when to stop is crucial
Accounts for largest difference between local versus global
Feedback helps
Feedback resulted in a 7% to 13% reduction of effort
Have a system
Unstructured analyst behavior is not efficient

Future Work
User Interface Design
Larger Datasets
Live Analysts

Questions?

Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools

by promise07 on May 20, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 6

Statistics

Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools — Presentation Transcript