Information Needs for Software Development Analytics

1,456 views

Published on

Talk from ICSE 2012

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Information Needs for Software Development Analytics

  1. 1. Ray Buse Tom Zimmermann
  2. 2. Ray Buse • Information Needs for Software Development Analytics6/20/2012 22
  3. 3. Ray Buse • Information Needs for Software Development Analytics6/20/2012 33
  4. 4. Analytics Measurements Measurements Metrics What? How much? Exploratory Analysis Quantitative Analysis Analytics Why? Qualitative Analysis Experiments What if? Insights Goal InsightsRay Buse • Information Needs for Software Development Analytics6/20/2012 55
  5. 5. Where is Software Eng. Today? Few Insights Many InsightsRay Buse • Information Needs for Software Development Analytics 6
  6. 6. Dependencies Code Clones Telemetry VelocityChange Type Readability Architecture Expertise Data RichOwnership Churn Failure Documentation Models Test Defect Complexity Coverage Density
  7. 7. Data is plentifulResearcher Project Manager
  8. 8. Acting on data is hard Readability is 4.8 So what?Researcher Complexity is 9.6 … Researcher Project Manager
  9. 9. Acting on data is hard Has to learn about Knows how to analysis to understand analyze data and act on resultsResearcher Makes assumptions Knows project about project very well Researcher Project Manager
  10. 10. Analytics ToolsDraws on Meets specificpowerful analyses information needs Researcher Product Manager
  11. 11. Existing ToolsAnalytics: Focus on data collection and developerawareness rather than information needsRay Buse • Information Needs for Software Development Analytics 12
  12. 12. But what information is needed?Ray Buse • Information Needs for Software Development Analytics6/20/2012 1313
  13. 13. But what information is needed?… and how should it be presented?Ray Buse • Information Needs for Software Development Analytics6/20/2012 1414
  14. 14. This TalkWhat is Software Analytics? Guidelines Survey at Microsoft Ray Buse • Information Needs for Software Development Analytics 15
  15. 15. This Talk Guidelines Survey at MicrosoftRay Buse • Information Needs for Software Development Analytics6/20/2012 1616
  16. 16. Survey of Microsoft Engineers 110 responses. 53 developers and 57 managers. 30 25 20 15 Developers 10 Managers 5 0 Entertainment Microsoft Online Server & Tools Windows and & Devices Business Services Division Windows Live Division Division Division DivisionRay Buse • Information Needs for Software Development Analytics6/20/2012 1717
  17. 17. Survey of Microsoft Engineers 110 responses. 53 developers and 57 managers. 30 25 20 15 Developers 10 Managers 5 0 Entertainment Microsoft Online Server & Tools Windows and & Devices Business Services Division Windows Live Division Division Division DivisionRay Buse • Information Needs for Software Development Analytics6/20/2012 1818
  18. 18. What factors most influence your decisions?Ray Buse • Information Needs for Software Development Analytics6/20/2012 1919
  19. 19. Important Decision FactorsRay Buse • Information Needs for Software Development Analytics6/20/2012 2020
  20. 20. Important Decision FactorsManagers relymore on data Developers rely on experienceRay Buse • Information Needs for Software Development Analytics6/20/2012 2121
  21. 21. What questions are important or difficult to answer?Ray Buse • Information Needs for Software Development Analytics6/20/2012 2222
  22. 22. Ray Buse • Information Needs for Software Development Analytics6/20/2012 2323
  23. 23. Ray Buse • Information Needs for Software Development Analytics6/20/2012 2424
  24. 24. Ray Buse • Information Needs for Software Development Analytics6/20/2012 2525
  25. 25. Which artifacts are important to measure?Ray Buse • Information Needs for Software Development Analytics6/20/2012 2626
  26. 26. Important Artifacts to ReportRay Buse • Information Needs for Software Development Analytics6/20/2012 2727
  27. 27. Important Artifacts to Report Important to tie datato deliverable artifacts Developers more concerned with code artifactsRay Buse • Information Needs for Software Development Analytics6/20/2012 2828
  28. 28. What indicators do you currently use? What would you like to use?Ray Buse • Information Needs for Software Development Analytics6/20/2012 2929
  29. 29. Fraction of Developers Fraction of Developers that currently use this that WOULD use this Indicator Fraction of Managers Fraction of Managers that WOULD use this that currently use thisRay Buse • Information Needs for Software Development Analytics6/20/2012 3030
  30. 30. Ray Buse • Information Needs for Software Development Analytics6/20/2012 3131
  31. 31. Desired AND Highly Available Desired AND Highly AvailableRay Buse • Information Needs for Software Development Analytics6/20/2012 3232
  32. 32. Desired but NOT Highly AvailableRay Buse • Information Needs for Software Development Analytics6/20/2012 3333
  33. 33. All useful to at least half of respondentsRay Buse • Information Needs for Software Development Analytics6/20/2012 3434
  34. 34. What decisions could analytics help with?Ray Buse • Information Needs for Software Development Analytics6/20/2012 3535
  35. 35. Targeting Testing“Targeting testing of a product needs informationon the code that changed from build to buildand [the] result of bug fixes so we could moreeasily map out what features and what othercode requires re-examination.”Ray Buse • Information Needs for Software Development Analytics6/20/2012 3636
  36. 36. Targeting Refactoring“The number of bug reports for a certain featurearea helps us decide whether that feature area ismature for a refactoring.”“Telemetry allows us to prioritize investment forcode cleanup and bug fixing in a way that hassubstantial customer impact.”Ray Buse • Information Needs for Software Development Analytics6/20/2012 3737
  37. 37. Understanding Customers“Analytics helps us understand how a user isusing our product. Are they performing tasks weexpect? Performing tasks we didn’t anticipate?We can determine effectiveness of features, aswell.”Ray Buse • Information Needs for Software Development Analytics6/20/2012 3838
  38. 38. Targeting Inspection“If I had decent metrics for software readability,dependency changes, component level changes,etc., it could help drive decisions on what levelsof code review we would force. I’d also like tomake comparisons between the results of theanalytics and the actual number of issues foundduring code review (in order to tune theanalytics).”Ray Buse • Information Needs for Software Development Analytics6/20/2012 3939
  39. 39. Analytics ToolsRay Buse • Information Needs for Software Development Analytics6/20/2012 4040
  40. 40. Recommendations • Be easy to use. • Be fast and produce concise or summary output.Ray Buse • Information Needs for Software Development Analytics 41
  41. 41. Recommendations • Measure many artifacts using many indicators. • Map indicators to features and dates to milestones.Ray Buse • Information Needs for Software Development Analytics 42
  42. 42. Recommendations • Be current and interactive. • Focus on characterizing the past and present over predicting the future. • Focus on information relevant to the target audience.Ray Buse • Information Needs for Software Development Analytics 43
  43. 43. Tool Spectrum Past Present Future Exploration Trends Alerts Forecasting Analysis Summarization Overlays GoalsExperimentation Modeling Benchmarking SimulationRay Buse • Information Needs for Software Development Analytics6/20/2012 4444
  44. 44. Tool Spectrum Past Present Future Exploration Trends Alerts Forecasting Analysis Summarization Overlays GoalsExperimentation Modeling Benchmarking SimulationRay Buse • Information Needs for Software Development Analytics6/20/2012 4545
  45. 45. Tool Spectrum Past Present Future Exploration Trends Alerts Forecasting Analysis Summarization Overlays GoalsExperimentation Modeling Benchmarking SimulationRay Buse • Information Needs for Software Development Analytics6/20/2012 4646
  46. 46. Tool Spectrum Past Present Future Exploration Trends Alerts Forecasting Analysis Summarization Overlays GoalsExperimentation Modeling Benchmarking SimulationRay Buse • Information Needs for Software Development Analytics6/20/2012 4747
  47. 47. Make Data Actionable and Accessible Data collection Data quality User needs Education Privacy User experience
  48. 48. Make Data Actionable and Accessible Data collection Data quality User needs Education Privacy User experience
  49. 49. Make Data Actionable and Accessible Data collection Data quality User needs Education Privacy User experience
  50. 50. Make Data Actionable and Accessible Data collection Data quality User needs Education Privacy User experience
  51. 51. Make Data Actionable and Accessible Data collection Data quality User needs Education Privacy User experience
  52. 52. Make Data Actionable and Accessible Data collection Data quality User needs Education Privacy User experience
  53. 53. Make Data Actionable and Accessible Data collection Data quality User needs Education Privacy User experience
  54. 54. Thank you!Ray Busehttp://arrestedcomputing.comTom Zimmermannhttp://thomas-zimmermann.com
  55. 55. Ray Buse • Information Needs for Software Development Analytics6/20/2012 5656

×