9. Acting on data is hard
Readability
is 4.8 So what?
Researcher
Complexity
is 9.6
…
Researcher Project Manager
10. Acting on data is hard
Has to learn about
Knows how to analysis to understand
analyze data and act on results
Researcher Makes
assumptions Knows project
about project very well
Researcher Project Manager
11. Analytics Tools
Draws on
Meets specific
powerful analyses
information needs
Researcher Product Manager
12. Existing Tools
Analytics: Focus on data collection and developer
awareness rather than information needs
Ray Buse • Information Needs for Software Development Analytics 12
13. But what information is needed?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1313
14. But what information is needed?
… and how should it be presented?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1414
15. This Talk
What is Software Analytics? Guidelines
Survey at Microsoft
Ray Buse • Information Needs for Software Development Analytics 15
16. This Talk
Guidelines
Survey at Microsoft
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1616
17. Survey of Microsoft Engineers
110 responses. 53 developers and 57 managers.
30
25
20
15
Developers
10 Managers
5
0
Entertainment Microsoft Online Server & Tools Windows and
& Devices Business Services Division Windows Live
Division Division Division Division
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1717
18. Survey of Microsoft Engineers
110 responses. 53 developers and 57 managers.
30
25
20
15
Developers
10 Managers
5
0
Entertainment Microsoft Online Server & Tools Windows and
& Devices Business Services Division Windows Live
Division Division Division Division
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1818
19. What factors most influence your
decisions?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1919
21. Important Decision Factors
Managers rely
more on data
Developers rely
on experience
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2121
22. What questions are important or
difficult to answer?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2222
23. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2323
24. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2424
25. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2525
26. Which artifacts are
important to measure?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2626
27. Important Artifacts to Report
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2727
28. Important Artifacts to Report
Important to tie data
to deliverable artifacts
Developers more concerned
with code artifacts
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2828
29. What indicators do you currently use?
What would you like to use?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2929
30. Fraction of Developers Fraction of Developers
that currently use this that WOULD use this
Indicator
Fraction of Managers Fraction of Managers
that WOULD use this
that currently use this
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3030
31. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3131
32. Desired AND Highly Available
Desired AND
Highly Available
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3232
33. Desired but NOT
Highly Available
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3333
34. All useful to at
least half of
respondents
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3434
35. What decisions could
analytics help with?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3535
36. Targeting Testing
“Targeting testing of a product needs information
on the code that changed from build to build
and [the] result of bug fixes so we could more
easily map out what features and what other
code requires re-examination.”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3636
37. Targeting Refactoring
“The number of bug reports for a certain feature
area helps us decide whether that feature area is
mature for a refactoring.”
“Telemetry allows us to prioritize investment for
code cleanup and bug fixing in a way that has
substantial customer impact.”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3737
38. Understanding Customers
“Analytics helps us understand how a user is
using our product. Are they performing tasks we
expect? Performing tasks we didn’t anticipate?
We can determine effectiveness of features, as
well.”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3838
39. Targeting Inspection
“If I had decent metrics for software readability,
dependency changes, component level changes,
etc., it could help drive decisions on what levels
of code review we would force. I’d also like to
make comparisons between the results of the
analytics and the actual number of issues found
during code review (in order to tune the
analytics).”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3939
41. Recommendations
• Be easy to use.
• Be fast and produce concise or
summary output.
Ray Buse • Information Needs for Software Development Analytics 41
42. Recommendations
• Measure many artifacts using
many indicators.
• Map indicators to features and
dates to milestones.
Ray Buse • Information Needs for Software Development Analytics 42
43. Recommendations
• Be current and interactive.
• Focus on characterizing the past
and present over predicting the
future.
• Focus on information relevant to
the target audience.
Ray Buse • Information Needs for Software Development Analytics 43
44. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4444
45. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4545
46. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4646
47. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4747
48. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience
49. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience
50. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience
51. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience
52. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience
53. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience
54. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience
