Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools?* Alex Dekhtyar Jane Huffman Hay...
Introduction <ul><li>What is requirements traceability? </li></ul><ul><li>Information retrieval and traceability </li></ul...
What is Requirements Traceability? <ul><li>“ Requirements traceability is the ability to describe and  follow the life of ...
Why is Traceability Important? <ul><li>Requirements verification and validation </li></ul><ul><li>Impact analysis </li></u...
Design Document Information Retrieval and Traceability Representation Requirements Document Matching   algorithm Analyst
What’s Different about Traceability? <ul><li>Typical Information Retrieval Problem: </li></ul><ul><ul><li>Search Engines: ...
IR Based Traceability Frameworks <ul><li>ADAMS Re-Trace </li></ul><ul><ul><li>Part of an artifact management suite </li></...
RETRO.NET <ul><li>Candidate Link </li></ul><ul><li>Top Link </li></ul><ul><li>Feedback </li></ul><ul><li>Confirmed Links <...
Research Questions <ul><li>What is the optimal upper bound on analyst efficiency using Information Retrieval based tools? ...
Experimental Design <ul><li>Subject </li></ul><ul><ul><li>A simulated analyst engaging in a specified behavior </li></ul><...
Measures N  - number of low-level requirements; M  - number of high-level requirements; Hits  - number of confirmed candid...
Global Ordering Without Feedback <ul><li>All candidate link lists merged </li></ul><ul><li>Links processed one at time in ...
Local Ordering Without Feedback <ul><li>Each candidate link list sorted in descending order </li></ul><ul><li>Top link for...
Global Ordering With Feedback <ul><li>All candidate link lists merged </li></ul><ul><li>Links processed one at time in des...
Local Ordering With Feedback <ul><li>Each candidate link list sorted in descending order </li></ul><ul><li>Top link for ea...
Dataset <ul><li>CM-1 </li></ul><ul><li>http://mdp.ivv.nasa.gov/mdp_glossary.html#CM1 </li></ul><ul><li>In PROMISE reposito...
Experimental Design Detail <ul><li>Two Studies </li></ul><ul><ul><li>Impact of Behavior on Analyst Effort when Recall is f...
Random? <ul><li>Menzies et al. note that software analysis should “start with random methods because they are so cheap, mo...
Random Methods <ul><li>Global Filtered Random Selection </li></ul><ul><ul><li>All Candidate Link Lists Merged </li></ul></...
Experiment Operation <ul><li>Input: </li></ul><ul><ul><li>Requirement and Specification Documents </li></ul></ul><ul><ul><...
Experiment Validity <ul><li>Internal Threats </li></ul><ul><ul><li>Accuracy of Requirements Traceability Matrix </li></ul>...
Study 1 Results, Fixed Recall Observed Links Precision Selectivity Local, Feedback 1595 20% 3% Local, No Feedback 1713 19%...
Study 2 Results, Fixed Effort Confirmed Links Precision Recall Local, Feedback 321 20% 89% Local, No Feedback 326 20% 87.5...
Study 2 Intermediate Results
Analysis <ul><li>Knowing when to stop is crucial </li></ul><ul><ul><li>Accounts for largest difference between local versu...
Future Work <ul><li>User Interface Design  </li></ul><ul><li>Larger Datasets </li></ul><ul><li>Live Analysts </li></ul>
Questions?
Upcoming SlideShare
Loading in …5
×

Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools

1,485 views

Published on

Alexander Dekhytar, Jane Hayes, and Jody Larson

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,485
On SlideShare
0
From Embeds
0
Number of Embeds
55
Actions
Shares
0
Downloads
68
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools

  1. 1. Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools?* Alex Dekhtyar Jane Huffman Hayes Jody Larsen *Funded by NASA
  2. 2. Introduction <ul><li>What is requirements traceability? </li></ul><ul><li>Information retrieval and traceability </li></ul><ul><li>What’s different about traceability? </li></ul><ul><li>Research questions </li></ul><ul><li>Experimental design & validity </li></ul><ul><li>Results </li></ul><ul><li>Analysis </li></ul><ul><li>Future Work </li></ul><ul><li>Questions </li></ul>
  3. 3. What is Requirements Traceability? <ul><li>“ Requirements traceability is the ability to describe and follow the life of a requirement , in both a forward and backward direction , i.e., from its origins, through its development and specification, to its subsequent deployment and use, and through periods of ongoing refinement and iteration in any of these phases.” </li></ul><ul><li>Gotel and Finkelstein, 1994. </li></ul>
  4. 4. Why is Traceability Important? <ul><li>Requirements verification and validation </li></ul><ul><li>Impact analysis </li></ul><ul><li>Validating contracts and bids </li></ul><ul><li>Testing Coverage </li></ul>
  5. 5. Design Document Information Retrieval and Traceability Representation Requirements Document Matching algorithm Analyst
  6. 6. What’s Different about Traceability? <ul><li>Typical Information Retrieval Problem: </li></ul><ul><ul><li>Search Engines: Google, Yahoo </li></ul></ul><ul><ul><li>Large and small documents </li></ul></ul><ul><ul><li>Document indexes must support arbitrary search queries </li></ul></ul><ul><ul><li>High accuracy needed </li></ul></ul><ul><li>Requirements Traceability </li></ul><ul><ul><li>Queries are known prior to indexing </li></ul></ul><ul><ul><li>Documents are all very small </li></ul></ul><ul><ul><li>High coverage needed </li></ul></ul>
  7. 7. IR Based Traceability Frameworks <ul><li>ADAMS Re-Trace </li></ul><ul><ul><li>Part of an artifact management suite </li></ul></ul><ul><ul><li>Latent Semantic Indexing (LSI) Based </li></ul></ul><ul><ul><li>Web Application </li></ul></ul><ul><li>Poirot: Tracemaker </li></ul><ul><ul><li>Probabilistic Information Retrieval </li></ul></ul><ul><ul><li>Web Application </li></ul></ul><ul><li>RETRO </li></ul><ul><ul><li>Vector Space Model and LSI </li></ul></ul><ul><ul><li>Windows Application </li></ul></ul>
  8. 8. RETRO.NET <ul><li>Candidate Link </li></ul><ul><li>Top Link </li></ul><ul><li>Feedback </li></ul><ul><li>Confirmed Links </li></ul>
  9. 9. Research Questions <ul><li>What is the optimal upper bound on analyst efficiency using Information Retrieval based tools? </li></ul><ul><li>What analyst behaviors produce accurate and complete RTMs most efficiently? </li></ul>
  10. 10. Experimental Design <ul><li>Subject </li></ul><ul><ul><li>A simulated analyst engaging in a specified behavior </li></ul></ul><ul><li>Variables </li></ul><ul><ul><li>Independent </li></ul></ul><ul><ul><ul><li>Analyst Behavior </li></ul></ul></ul><ul><ul><li>Dependent </li></ul></ul><ul><ul><ul><li>Confirmed True Links </li></ul></ul></ul><ul><ul><ul><li>Observed Links </li></ul></ul></ul><ul><ul><ul><li>Confirmed Recall </li></ul></ul></ul><ul><ul><ul><li>Precision </li></ul></ul></ul><ul><ul><ul><li>Selectivity </li></ul></ul></ul>
  11. 11. Measures N - number of low-level requirements; M - number of high-level requirements; Hits - number of confirmed candidate links Strikes - number of false positives Misses - number of missed links Precision = Hits Hits + Strikes Recall = Hits Hits +Misses Selectivity = Hits + Strikes M * N
  12. 12. Global Ordering Without Feedback <ul><li>All candidate link lists merged </li></ul><ul><li>Links processed one at time in descending order </li></ul><ul><li>Results in analyst looking at links that have highest global similarity ratings first </li></ul>
  13. 13. Local Ordering Without Feedback <ul><li>Each candidate link list sorted in descending order </li></ul><ul><li>Top link for each requirement element is processed </li></ul><ul><li>Results in analyst looking at the links for each requirement by highest similarity rating </li></ul>
  14. 14. Global Ordering With Feedback <ul><li>All candidate link lists merged </li></ul><ul><li>Links processed one at time in descending order </li></ul><ul><li>After each link is processed, feedback is executed </li></ul><ul><li>Results in analyst looking at links that have highest global similarity ratings first </li></ul>
  15. 15. Local Ordering With Feedback <ul><li>Each candidate link list sorted in descending order </li></ul><ul><li>Top link for each requirement element is processed </li></ul><ul><li>After all top links have been processed, feedback is executed </li></ul><ul><li>Results in analyst looking at the links for each requirement by highest similarity rating </li></ul>
  16. 16. Dataset <ul><li>CM-1 </li></ul><ul><li>http://mdp.ivv.nasa.gov/mdp_glossary.html#CM1 </li></ul><ul><li>In PROMISE repository </li></ul><ul><li>Sanitized dataset for a NASA scientific instrument </li></ul><ul><li>Goal is to help analyst avoid examining all 36,556 links </li></ul>36,556 Total # of retrieved candidate links 70.7% Selectivity 358 Total # of correct links retrieved by basic Vector Space 99% Recall 0.1% Precision # of elements in requirements (high-level) 235 # of elements in design specification (low-level) 220 # of correct links 361
  17. 17. Experimental Design Detail <ul><li>Two Studies </li></ul><ul><ul><li>Impact of Behavior on Analyst Effort when Recall is fixed </li></ul></ul><ul><ul><ul><li>Recall set to 89% </li></ul></ul></ul><ul><ul><ul><li>Included all four structured behaviors </li></ul></ul></ul><ul><ul><li>Impact of Behavior on Precision and Recall when Analyst Effort is fixed </li></ul></ul><ul><ul><ul><li>Effort fixed to 1595 observed links </li></ul></ul></ul><ul><ul><ul><li>Added two additional behaviors </li></ul></ul></ul><ul><ul><ul><ul><li>Global, Random Order </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Local, Random Order </li></ul></ul></ul></ul>
  18. 18. Random? <ul><li>Menzies et al. note that software analysis should “start with random methods because they are so cheap, moving to the more complex methods only when random methods fail.” </li></ul><ul><li>Source: Menzies, T., Owen, D., and Richardson, J. The Strangest Thing about Software. IEEE Computer , January 2007, pp. 54 – 60. </li></ul>
  19. 19. Random Methods <ul><li>Global Filtered Random Selection </li></ul><ul><ul><li>All Candidate Link Lists Merged </li></ul></ul><ul><ul><li>List Sorted By Similarity </li></ul></ul><ul><ul><li>Bottom 75% (Least Relevant) Removed </li></ul></ul><ul><ul><li>1595 Links Selected Randomly </li></ul></ul><ul><li>Local Filtered Random Selection </li></ul><ul><ul><li>All Candidate Link Lists Sorted By Similarity </li></ul></ul><ul><ul><li>Bottom 75% (Least Relevant) Removed </li></ul></ul><ul><ul><li>7 or 8 Links Selected Randomly For Each Requirement </li></ul></ul><ul><ul><li>1595 Links Selected </li></ul></ul><ul><li>Executed 1000 times </li></ul><ul><li>Median values were used </li></ul>
  20. 20. Experiment Operation <ul><li>Input: </li></ul><ul><ul><li>Requirement and Specification Documents </li></ul></ul><ul><ul><li>Desired Analyst Behavior </li></ul></ul><ul><li>Operation: </li></ul><ul><ul><li>Performs multiple rounds of interaction </li></ul></ul><ul><ul><li>Provides link status determination for each observed link </li></ul></ul><ul><li>Output: </li></ul><ul><ul><li>Candidate RTM </li></ul></ul><ul><ul><li>Dependent Variable Data </li></ul></ul>
  21. 21. Experiment Validity <ul><li>Internal Threats </li></ul><ul><ul><li>Accuracy of Requirements Traceability Matrix </li></ul></ul><ul><ul><li>Suitability of text for Information Retrieval methods </li></ul></ul><ul><li>External Threats </li></ul><ul><ul><li>Can Results Based on CM-1 Dataset be Generalized? </li></ul></ul><ul><ul><li>Realism of Simulated Analyst Behavior </li></ul></ul>
  22. 22. Study 1 Results, Fixed Recall Observed Links Precision Selectivity Local, Feedback 1595 20% 3% Local, No Feedback 1713 19% 3% Global, Feedback 5399 6% 10% Global, No Feedback 6149 5% 12%
  23. 23. Study 2 Results, Fixed Effort Confirmed Links Precision Recall Local, Feedback 321 20% 89% Local, No Feedback 326 20% 87.5% Global, Feedback 236 15% 65% Global, No Feedback 227 14% 63% Local, Random 65 4% 18% Global, Random 58 3.6% 16%
  24. 24. Study 2 Intermediate Results
  25. 25. Analysis <ul><li>Knowing when to stop is crucial </li></ul><ul><ul><li>Accounts for largest difference between local versus global </li></ul></ul><ul><li>Feedback helps </li></ul><ul><ul><li>Feedback resulted in a 7% to 13% reduction of effort </li></ul></ul><ul><li>Have a system </li></ul><ul><ul><li>Unstructured analyst behavior is not efficient </li></ul></ul>
  26. 26. Future Work <ul><li>User Interface Design </li></ul><ul><li>Larger Datasets </li></ul><ul><li>Live Analysts </li></ul>
  27. 27. Questions?

×