This document summarizes the results of a survey conducted at Microsoft to understand the information needs for software development analytics. Some key findings include: - Managers rely more on data when making decisions, while developers rely more on their experience. - Respondents said the most important questions to answer relate to quality, dependencies, and defects. - Code-related artifacts like code complexity and test coverage were seen as most important to measure. - Commonly used and desired indicators included bugs, code churn, complexity, and test coverage. - Analytics could help with decisions around targeting testing, refactoring, and resource allocation.

1. Ray Buse
Tom Zimmermann

2. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 22

3. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 33

5. Analytics
Measurements
Measurements
Metrics What?
How much?
Exploratory Analysis
Quantitative Analysis
Analytics Why?
Qualitative Analysis
Experiments
What if?
Insights Goal
Insights
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 55

6. Where is Software Eng. Today?
Few Insights Many Insights
Ray Buse • Information Needs for Software Development Analytics 6

7. Dependencies Code Clones Telemetry
Velocity
Change Type
Readability
Architecture
Expertise
Data Rich
Ownership
Churn
Failure Documentation
Models
Test
Defect
Complexity Coverage
Density

8. Data is plentiful
Researcher Project Manager

9. Acting on data is hard
Readability
is 4.8 So what?
Researcher
Complexity
is 9.6
…
Researcher Project Manager

10. Acting on data is hard
Has to learn about
Knows how to analysis to understand
analyze data and act on results
Researcher Makes
assumptions Knows project
about project very well
Researcher Project Manager

11. Analytics Tools
Draws on
Meets specific
powerful analyses
information needs
Researcher Product Manager

12. Existing Tools
Analytics: Focus on data collection and developer
awareness rather than information needs
Ray Buse • Information Needs for Software Development Analytics 12

13. But what information is needed?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1313

14. But what information is needed?
… and how should it be presented?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1414

15. This Talk
What is Software Analytics? Guidelines
Survey at Microsoft
Ray Buse • Information Needs for Software Development Analytics 15

16. This Talk
Guidelines
Survey at Microsoft
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1616

17. Survey of Microsoft Engineers
110 responses. 53 developers and 57 managers.
30
25
20
15
Developers
10 Managers
5
0
Entertainment Microsoft Online Server & Tools Windows and
& Devices Business Services Division Windows Live
Division Division Division Division
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1717

18. Survey of Microsoft Engineers
110 responses. 53 developers and 57 managers.
30
25
20
15
Developers
10 Managers
5
0
Entertainment Microsoft Online Server & Tools Windows and
& Devices Business Services Division Windows Live
Division Division Division Division
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1818

19. What factors most influence your
decisions?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 1919

20. Important Decision Factors
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2020

21. Important Decision Factors
Managers rely
more on data
Developers rely
on experience
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2121

22. What questions are important or
difficult to answer?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2222

23. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2323

24. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2424

25. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2525

26. Which artifacts are
important to measure?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2626

27. Important Artifacts to Report
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2727

28. Important Artifacts to Report
Important to tie data
to deliverable artifacts
Developers more concerned
with code artifacts
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2828

29. What indicators do you currently use?
What would you like to use?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 2929

30. Fraction of Developers Fraction of Developers
that currently use this that WOULD use this
Indicator
Fraction of Managers Fraction of Managers
that WOULD use this
that currently use this
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3030

31. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3131

32. Desired AND Highly Available
Desired AND
Highly Available
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3232

33. Desired but NOT
Highly Available
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3333

34. All useful to at
least half of
respondents
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3434

35. What decisions could
analytics help with?
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3535

36. Targeting Testing
“Targeting testing of a product needs information
on the code that changed from build to build
and [the] result of bug fixes so we could more
easily map out what features and what other
code requires re-examination.”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3636

37. Targeting Refactoring
“The number of bug reports for a certain feature
area helps us decide whether that feature area is
mature for a refactoring.”
“Telemetry allows us to prioritize investment for
code cleanup and bug fixing in a way that has
substantial customer impact.”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3737

38. Understanding Customers
“Analytics helps us understand how a user is
using our product. Are they performing tasks we
expect? Performing tasks we didn’t anticipate?
We can determine effectiveness of features, as
well.”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3838

39. Targeting Inspection
“If I had decent metrics for software readability,
dependency changes, component level changes,
etc., it could help drive decisions on what levels
of code review we would force. I’d also like to
make comparisons between the results of the
analytics and the actual number of issues found
during code review (in order to tune the
analytics).”
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 3939

40. Analytics Tools
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4040

41. Recommendations
• Be easy to use.
• Be fast and produce concise or
summary output.
Ray Buse • Information Needs for Software Development Analytics 41

42. Recommendations
• Measure many artifacts using
many indicators.
• Map indicators to features and
dates to milestones.
Ray Buse • Information Needs for Software Development Analytics 42

43. Recommendations
• Be current and interactive.
• Focus on characterizing the past
and present over predicting the
future.
• Focus on information relevant to
the target audience.
Ray Buse • Information Needs for Software Development Analytics 43

44. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4444

45. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4545

46. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4646

47. Tool Spectrum
Past Present Future
Exploration Trends Alerts Forecasting
Analysis Summarization Overlays Goals
Experimentation Modeling Benchmarking Simulation
Ray Buse • Information Needs for Software Development Analytics
6/20/2012 4747

48. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience

49. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience

50. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience

51. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience

52. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience

53. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience

54. Make Data Actionable and Accessible
Data collection Data quality
User needs Education
Privacy User experience

55. Thank you!
Ray Buse
http://arrestedcomputing.com
Tom Zimmermann
http://thomas-zimmermann.com

56. Ray Buse • Information Needs for Software Development Analytics
6/20/2012 5656