Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Mining Application in a Software Project Management Process


Published on

  • Be the first to comment

Data Mining Application in a Software Project Management Process

  1. 1. Data Mining Application in a Software Project Management Process Richi Nayak School of Information Systems, QUT, Brisbane Tian Qiu EDS Credit Services, Adelaide, Australia
  2. 2. Agenda <ul><li>Motivation </li></ul><ul><li>Data Pre-processing </li></ul><ul><li>Data Modelling </li></ul><ul><li>Analysis of Results </li></ul><ul><li>Conclusion </li></ul>
  3. 3. Motivation <ul><li>A software engineering project leader manages a project with several issues involved such as project planning and scheduling, code implementation, testing and release. </li></ul><ul><li>It is difficult to precisely estimate the project duration in advance. </li></ul><ul><li>However, he can utilise the data Mining methods and make the accurate estimation by learning from the information gained by previous projects. </li></ul><ul><li>He can also eliminate several potential problems when a pattern appears in his current project similar to the one that have caused problems in previous projects. </li></ul><ul><li>Data mining techniques have been successfully applied to various areas such as marketing, medical, and financial. </li></ul><ul><li>However, few of them can be currently seen in software engineering domain. </li></ul>
  4. 4. Data Acquisition <ul><li>Every year, more than 50 software projects are carried out in MASC, more than ten thousands lines of code are created, and thousands pages of documents are released to various customers. </li></ul><ul><li>In order to control the problems appearing during a project life cycle and to improve the working efficiency, MASC has set up a series of processes such as Software Configuration Management, Software Risk Management, Software Project Metric Report and Software Problem Report Management. </li></ul><ul><li>The Software Problem Report (PR) management data is chosen as the focus of this research. </li></ul><ul><li>A software bug-tracking system, GNATS ( A T racking S ystem by GN U), is set up on MASC Intranet to collect and maintain all PRs raised from every department and individual within MASC. </li></ul><ul><ul><li>Currently the GNATS system stores more than 40,000 of problem reports. </li></ul></ul>
  5. 5. Structure of a PR
  6. 6. Data Pre-processing: Defining Goals (1) <ul><li>Finding a precise estimation figures on bug fixing or estimation work at the early stage of a project </li></ul><ul><ul><li>There is few or even no tool exists that can be used for both estimation and project problem reasoning stage. </li></ul></ul><ul><ul><li>Project team members can only give estimations based on their own experience from previous projects. </li></ul></ul><ul><ul><li>If the current project is not within their familiar topics, the accuracy of the estimation becomes worse. </li></ul></ul><ul><ul><li>A PR (problem report) fixing work also becomes more tedious when the responsible person can not estimate the time to fix the problem. </li></ul></ul><ul><ul><li>It will bring great cost savings and accurate progress control to the development team and to the organization. </li></ul></ul>
  7. 7. Data Pre-processing: Defining Goals (2) <ul><li>Improving control over the PR fixing and deduction of the project cycle time </li></ul><ul><ul><li>Currently the GNATS system has no actual database management system implemented. </li></ul></ul><ul><ul><li>This limits the potential benefits to software engineers who can obtain valuable information if the existing PR data is being analysed. </li></ul></ul><ul><ul><li>This is useful especially when a programmer is struggling with a bug while a resolution may already hide behind the knowledge that can be derived from the previous similar problems. </li></ul></ul>
  8. 8. Data Pre-processing: Field Selection (1) <ul><li>Whenever a PR is raised, a project leader will have to find answers </li></ul><ul><ul><li>How severe the problem is (customer impact)? </li></ul></ul><ul><ul><li>What is the impact of the problem on project schedule (Cost & Team priority)? and </li></ul></ul><ul><ul><li>What type of the problem it is (a Software bug or a design flaw)? </li></ul></ul><ul><ul><li>How long it will take to fix? </li></ul></ul><ul><ul><li>How many people are available? </li></ul></ul><ul><li>Attributes Severity, Priority, Class, Arrival-Date, Closed-Date, Responsible, and Synopsis are considered for mining. </li></ul><ul><li>Several fields such as Confidential, Submitter-ID, Environment, Fix, Release Note, Audit Trail, the associated project name and PR number are not considered during mining. </li></ul><ul><li>All PRs with a closed value in their State field are chosen. </li></ul>
  9. 9. Data Pre-processing: Field Selection (2) <ul><li>The attribute ‘ class ’ is chosen as the target attribute in order to find out any valuable knowledge among the type of a problem and the rest of the PR attributes. </li></ul><ul><ul><li>Association or characteristics rules can reveal the relationship between the fix effort (in time) and the PR class. </li></ul></ul><ul><ul><li>A project leader can analyse the fix effort versus the human resources available, and put it in the schedule and resource plan. </li></ul></ul><ul><li>The Synopsis field is pure text briefly describing what the problem is in the associated project. </li></ul><ul><ul><li>Text mining is used to analyse this data. </li></ul></ul>
  10. 10. <ul><ul><li>PR_ID|Category|Severity|Priority|Class|Arrival-Date|Close-Date|Synopsis </li></ul></ul><ul><li>17358| bambam| serious| high| sw-bug | 20:50 May 25 CST 1999 | 11:35 Mar 24 CST 1999 | STI STR register not being reset at POR </li></ul><ul><li>17436 |bambam| serious| high| support| 18:10 Mar 30 CST 1999| 12:00 May 24 CST 1999| sequence_reg varable in the RDR_CHL task is not defined </li></ul><ul><li>580 | bingarra| serious| low| doc-bug| 10:10 May 31 1996| | In URDRT2 of design doc, the word 'last' should be 'first‘ </li></ul><ul><li>6205 | gali| serious| medium| SW-bug | 14:30 Nov 5 1997| 13:14 Dec 1 1997| grouping of options in dialog box </li></ul>Data examples from the original PR data set Data Pre-processing: Cleaning Use of Different Terminology over the time Start-time > close-time Missing Value No Time Zone Value
  11. 11. Data Pre-processing: Data Transformation <ul><li>Attributes Arrival-Date, Closed-Date and Responsible are used to calculate the time spent to fix a PR. </li></ul><ul><ul><li>assuming that the derived attribute is total time spent to fix a problem if there is only one person involved . </li></ul></ul><ul><li>This attribute is discretized with cutting points be one day (1), half week (3 days), one week (7), two weeks (14), one month (30) and one quarter (90 days), half year (180 days) and more than one year (360 days). </li></ul><ul><ul><li>So that the mining results are not very highly depended on the exact human resource involved but gives an approximate estimate, allowing a minor change in human resource. </li></ul></ul>
  12. 12. Data Modelling and Mining <ul><li>Prediction modelling on the time consuming patterns of the PR data to make estimation. </li></ul><ul><ul><li>C5 </li></ul></ul><ul><ul><li>CBA, </li></ul></ul><ul><li>Link analysis to discover association among the contents/values of the variables being selected. </li></ul><ul><ul><li>CBA </li></ul></ul><ul><li>Text Mining to analyse Synopsis field. </li></ul><ul><ul><li>TextAnalyst http:// </li></ul></ul>
  13. 13. Classification and Association Rule Mining <ul><li>Initial Data set: 40,00 PRs </li></ul><ul><li>Pre-processed Data set: 10,765 PRs </li></ul><ul><ul><li>Covering more than 120 projects from 1996 to 2000 </li></ul></ul><ul><li>Different number of data sets are used in training </li></ul>
  14. 14. Data Mining result example in CBA
  15. 15. CBA Results: All PRs coming from one project 0.09 47.49 1.04 47.059 11 MSAD 0.10 47.56 1.01 45.18 9 MSMD 0.08 52.94 1.00 46.16 15 SSAD 0.07 47.56 1.01 45.180 10 SSMD Testing time cost (seconds) Prediction Inaccuracy on testing data (%) Training time cost (seconds) Prediction Inaccuracy on training data (%) Number of rules 9541 PR Number in the Testing set 1224 PR Number in the Training set
  16. 16. CBA Results: large size training set * large size of 5381 PRs from all software projects 1.2 58.91 0.45 58.45 12 MSAD 1.0 58.25 0.44 59.10 21 MSMD 1.3 58.09 0.44 57.39 18 SSAD 1.1 51.95 0.41 57.04 41 SSMD Testing time cost (seconds) Prediction Inaccuracy on testing data (%) Training time cost (seconds) Prediction Inaccuracy on training data (%) Number of rules 5384 PR Number in the Testing set 5381 PR Number in the Training set
  17. 17. CBA Results: equally distributed target value <ul><li>342 PRs with change-request value of ‘ Class ’ from all software projects </li></ul><ul><li>1000 PRs with sw-bug, doc-bug and support values. </li></ul>1.6 46.9 1.6 46.5 15 MSAD 1.9 45.1 1.6 46.5 15 MSMD 2.1 43.8 1.6 43.6 15 SSAD 2.0 43.5 2.2 43.51 20 SSMD Testing time cost (seconds) Prediction Inaccuracy on testing data (%) Training time cost (seconds) Prediction Inaccuracy on training data (%) Number of rules 7423 PR Number in the Testing set 3342 PR Number in the Training set
  18. 18. CBA Results: Cross-validation   25.365 48.87 CV-MSAD   28.944 45.02 CV-MSMD   25.345 50.5 CV-SSAD   25.00 46.05 CV-SSMD   Mining time (seconds) Prediction Inaccuracy (%) 10765 PR Number in the data set  
  19. 19. CBA Results: Rules <ul><li>If severity= non-critical and Time-to-fix = 3 to 30 days and priority= medium </li></ul><ul><li>Then class = doc-bug . Confidence = 76.6%, Support = 2.7% </li></ul><ul><li>If severity= critical and Time-to-fix = less than 3 days and </li></ul><ul><li>priority = high </li></ul><ul><li>Then class = sw-bug . Confidence = 75.2%, Support = 2.3% </li></ul><ul><li>Conclusion : All software related bugs can be fixed within 3 days with above 75% confidence if they have high priority and are in critical condition. </li></ul><ul><li>It may take 3 months to fix the problem if the corresponding priority and severity are graded as medium and serious. </li></ul><ul><li>There is no rule that has confidence value larger than 80%, however they do describe some characters of the PR fixing patterns. Therefore they are useful for software project management in estimation bug fixing related time issues. </li></ul>
  20. 20. CBA Results: Summary <ul><li>In general, around 46% p rediction mismatch in training data set </li></ul><ul><ul><li>the lowest is 43.51%, the highest is above 59.10%. </li></ul></ul><ul><li>Above 51% p rediction mismatch is in testing data set </li></ul><ul><ul><li>the lowest is 43.51%, the highest is 58.25%. </li></ul></ul><ul><li>Attempt to improve the accurate prediction in the way of equal-distributed target-value samples does not lead much change. </li></ul><ul><li>The error rates from using multiple supports are higher and the number of extracted rules is lower than those from using single support mining engine. </li></ul><ul><li>The error rates from Multiple Support and Single Support mining engine on Automatic Discretized value are usually higher than the manually discretization. </li></ul>
  21. 21. C5 results: Original Data Set 41.1 37.7 5.6 Process Time (seconds) 44.1 39.4 40.3 Error Rate of trees (%) 121.9 N/A. 141 Size of Created tree 43.9 41.3 41.5 Error Rate of rules (%) 57.7 N/A. 51 Number of Created rules CROS_VAL BOOSTING NORMAL 10765 TOTAL NUMBER OF PR RECORDS
  22. 22. C5 results: equally distributed target value 1.1 0.4 0.2 Process Time (seconds) 43.1 42.6 42.5 Error Rate of trees (%) 17.4 N/A 21 Size of Created tree 42.8 42.6 42.6 Error Rate of rules (%) 12.4 N/A 11 Number of Created rules CROS_VAL BOOSTING NORMAL 5681 (3342 +2339) TOTAL NUMBER OF PR RECORDS
  23. 23. C5 Results: Rules <ul><li>When a PR is in low priority and the time spent is around half a day (0.5 day) Then the rule has a high probability (87.5% Confidence) to classify a bug to be a document related bug . </li></ul><ul><li>When a PR is in medium priority with non-critical severity and the time spent is around 1.1 day Then the rule has 84.6% Confidence to classify a bug to be a document related bug . </li></ul><ul><li>When a PR is in low priority and the time spent for fixing is around 1 week </li></ul><ul><li>Then the rule has 83.3% Confidence to classify a bug to be a software bug . </li></ul><ul><li>In general, around 42% error rate in rules from the training set </li></ul><ul><ul><li>the lowest is 40.3%, the highest is 43.9%. </li></ul></ul><ul><li>Similar error rate value is for the generated trees in testing data set </li></ul><ul><ul><li>the lowest is 39.4%, the highest is 44.1%. </li></ul></ul><ul><li>Both of the rates are better than CBA results. </li></ul><ul><li>The time efficiency of C5 is also better than CBA. </li></ul>
  24. 24. Text Mining <ul><li>The analysis of the text together with the rules obtained from classification and association can more accurately predict the time and cost of fixing PRs. </li></ul><ul><li>automatically summarise the pure text data and extract some valuable rules. </li></ul><ul><li>TextAnalyst automatically creates the semantic network based on the structure, vocabulary and volume of the analysed text, without any predefined rules. </li></ul>
  25. 27. Existing problems <ul><li>The error rates of testing data sets in both CBA and C5 are higher than expected. </li></ul><ul><ul><li>non-uniform value distribution (57, 25, 15, 3%) </li></ul></ul><ul><ul><li>indicating some amount of noise is still existent in data. </li></ul></ul><ul><li>The relationship between PRs and human resources within a particular project plays a great impact. </li></ul><ul><li>The time needed to fix a bug is different for each project depending upon the actual human resources available. </li></ul><ul><li>Only the attribute ‘ Responsible’ is used to indicate the human resource available. </li></ul><ul><li>Truly, the relationship with the human resources available for past projects whose data was analysed is needed to use time patterns to help project leaders to predict time consummations more accurately. </li></ul><ul><li>The use of additional data source ‘Change Request data set’ that records all customer request process data may rectify this problem. </li></ul>
  26. 28. Conclusion <ul><li>use of data mining techniques on a set of data collected from the software engineering process under a real software business environment. </li></ul><ul><li>useful rules are inferred on the time patterns of the PR fixing and the relationship between the content and the type of a PR in the form of association rules, classification rules or semantic trees. </li></ul><ul><li>the scale of this data mining task is limited. </li></ul><ul><li>Future work: apply data mining to different phases of software development such as software quality data, etc. </li></ul>
  27. 29. Thank You!! Questions?