Why should we take the time to measure our pain and start working together as an industry?

1. Gradle Summit
2016

2. How to Measure
Your #GradleDX
Janelle Klein

3. , Developer, Consultant, CTO @
Specialized in Statistical Process Control (SPC)
and Supply Chain Optimization from Lean Manufacturing (data geek)
Continuous Delivery infrastructure, automation strategy & technical mentorship
Janelle Klein
Who Am I?
How to Measure the PAIN
in Software Development
Janelle Klein
Author of “Idea Flow”
leanpub.com/ideaflow
Yay!
Tour Speaker
Founder of
newiron.com

4. What is This Talk About?

5. #OpenDX
An Open Standard for Measuring PAIN
Universal definition of “Better”
(Specification for Data Collection)
Developer
Experience

6. #GradleDX is a Pilot Project
#OpenDX
within
#GradleDX

7. Feedback Loop
Community
Pain Sensor
Tools
Pilot:
Supplier Partnership
#GradleDX Community Project
Plugin
Plugin
Plugin
Plugin

8. Why does partnership matter
in a software supply-chain?

9. My Code
is AWESOME!
100% Code Coverage
Low Cyclomatic
Complexity
Usability
Problems
This is
PAINFUL!
Depend
On API
WTF?!
WTF?!
WTF?!
WTF?!
WTF?! WTF?!

10. My Code
is AWESOME!
100% Code Coverage
Low Cyclomatic
Complexity
This is
PAINFUL!
Depend
On API
WTF?!
WTF?!
WTF?!
WTF?!
WTF?! WTF?!
We can both be looking at the exact same code, and
have a totally different experience.

11. 100% Code Coverage
Low Cyclomatic
Complexity
Usability
Problems
Depend
On API
WTF?!
WTF?!
WTF?!
WTF?!
WTF?! WTF?!
Here’s what
caused me PAIN
Painful Experience
Oh…
I can fix that!

12. 100% Code Coverage
Low Cyclomatic
Complexity
Depends
On API
WTF?!
WTF?!
WTF?!
WTF?!
WTF?! WTF?!
Oh…
I can fix that!

13. 100% Code Coverage
Low Cyclomatic
Complexity
Depend
On API
Oh…
I can fix that!
No More Pain
This
is AWESOME!

14. 100% Code Coverage
Low Cyclomatic
Complexity
Depend
On API
Yay!
It’s AWESOME!
No More Pain
This
is AWESOME!
Visibility Facilitates the Conversation

15. Application
3rd Party
Dependencies
Local
Components
Usability
Problems
90% of our Software is HERE

16. Application
3rd Party
Dependencies
Local
Components
Usability
Problems
90% of our Software is HERE
This is
PAINFUL!

17. Bad Software has become the new Environment Pollution
Cost
&
Risk
Human
Limitations
We are HERE:
Difficulty of Work
What’s Causing our Pain?
Cumulative Usability Problems in the Software Supply Chain

18. Alert!
Alert!
Our Application
PRESSURE
for better quality parts
PAIN
What if we started measuring PAIN
across the software supply chain?
This is
PAINFUL!

19. Everything is
AWESOME!
Our Application
PRESSURE
for better quality parts
What if we started measuring PAIN
across the software supply chain?

20. Feedback Loop
Community
Pain Sensor
Tools
#GradleDX: Pilot for Supply-Chain Feedback Loop
Plugin
Plugin
Plugin
Plugin
Pilot:
Supplier Partnership

21. Why Should YOU Care?
source: Domics - “What’s a Shrug?”

22. Every few years we rewrite our software…
Start%
Over%
Unmaintainable%
So0ware%
Across the Industry

23. We Start with the Best of Intentions
High Quality Code
Low Technical Debt
Automated Deployment
Good Code Coverage

24. Then This Happens!

25. PAIN
The Classic Story of Project Failure
Problems get deferred
Builds start breaking
Releases get chaotic
Productivity slows to a crawl
Developers begging for time
It’s never enough
Project Meltdown

26. Our Solution…

27. Until we can’t take it anymore.

28. On to greener pastures…

29. Has anyone struggled with the
challenge of business pressure?
Has anyone ever tried to fix this problem
within their organization?
Has anyone ever succeeded?

30. This is the biggest cause of FAILURE
in our industry:

31. Because PAIN Doesn’t Show Up
on a Balance Sheet
If we don’t MEASURE it,
Why?
they don’t SEE it.

32. “Better”
“Better”
What if you could get managers and developers
all pulling the same direction in your company?
Managers
Developers

33. “Better”
“Better”
What if we could solve this problem
in ALL of our companies?
Managers
Developers

34. Two Options:
Option 1
Stay
the Course
Here’s the Catch:
In order for this to work, we have to start
working together as a community.
Option 2
Build
Happiness
or
Let’s Do This

35. LEARN YOUR WAY TO AWESOME.
Free to Join Industry Peer Mentorship Network
1. Make the PAIN visible
2. Breakdown the causes into patterns
3. Help each other to learn & improve

36. We are working together to INVENT solutions for:
Community Analytics Platform
Process Control,
Supply-Chain Optimization &
Throughput Accounting
for
Software Organizations

37. #OpenDX
An Open Standard for Measuring PAIN
Based on:

38. The Goal
Eliyahu M. Goldratt
“Throughput Accounting”
System Productivity =
Rate of Revenue
Rate of Investment
Theory of Constraints
Optimizing Flow in a Supply Chain System

39. “Throughput Accounting”
System Productivity =
Rate of Revenue
Rate of Investment
Throughput Accounting measures how efficiently
we turn effort into money.
Our PAIN Matters

40. How Do We Measure PAIN?

41. Great Team
Disciplined with Best Practices
Constantly Working on Improvements+
Project FAILURE
About 8 Years Ago…

42. ApplicationTests
Our Test Suite

43. ApplicationTests
Our Test Suite
“What are we supposed to do?!”

44. The Retrospective
“What are we supposed to do?!”
Our biggest problem
“Well, if we don’t understand
a problem, we should
collect data.”

45. The Retrospective
Our biggest problem
“What data would help us
understand the problem?”
“What are we supposed to do?!”

46. Technical Debt Mistakes
I thought the main obstacle was
Technical Debt

47. ?
Most of our mistakes were in the
most well-written parts of the code.
Mistakes

48. We made significantly more mistakes
in code that we didn’t write ourselves.
Lower
Familiarity
More
Mistakes=
There had to be more to the story...

49. Complex(
So*ware(
PAIN
This is what I knew...
What made development feel painful?

50. Unexpected
Behavior
Problem
Resolved
Tracking Painful Interaction with the Code (Friction)
Troubleshooting
Progress
5 hours and 18 minutes of troubleshooting...
PAINFUL

51. The amount of PAIN was caused by…
Likeliness(of((
Unexpected(
Behavior(
Cost(to(Troubleshoot(and(Repair(
High(Frequency(
Low(Impact(
Low(Frequency(
Low(Impact(
Low(Frequency(
High(Impact(
PAIN(

52. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
Most of the pain was caused by human factors.
What causes PAIN?

53. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
Most of the pain was caused by human factors.
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm

54. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Most of the pain was caused by human factors.

55. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
PAIN is a consequence of how we interact with the code.
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm

56. PAIN occurs during the process of
understanding and extending the software
Complex(
So*ware(
PAIN
Not the Code.
Optimize “Idea Flow”

57. We deleted everything…
The automation wasn’t
solving the problem!

58. What are the specific risks in this release?
How can we mitigate the risks?
Handful of
manual tests
Custom
Tools
+

59. Time-Bound Releases
RISK-Bound Releases

60. Release
Size
(weeks)
Time
Defect Rates
RISK-Bound Releases
Continuous Delivery
is about learning how to
effectively manage risk

61. My team spent tons of time working on
improvements that didn’t make much difference.
We had tons of automation, but the
automation didn’t catch our bugs.

62. My team spent tons of time working on
improvements that didn’t make much difference.
We had well-modularized code,
but it was still extremely time-consuming to troubleshoot defects.

63. The hard part isn’t solving the problems
it’s identifying the right problems to solve.
“What are the specific problems
that are causing the team’s pain?”

64. Complex(
So*ware(
PAIN
Measure Friction in “Idea Flow”
OpenDX measures the “friction” that occurs
during a developer’s experience

65. The Rhythm of “Idea Flow”
Write a little code.
Work out the kinks.
Write a little code.
Work out the kinks.
Write a little code.
Work out the kinks.

66. ConflictConfirm
Rework'
Learn'
Validate(
Modify'
Progress Loop Conflict Loop
Troubleshoot'
Write a little code. Work out the kinks.
The Rhythm of “Idea Flow”

67. On Intelligence
Jeff Hawkins

68. Confirmed
Expectations
Our brain works like a giant prediction engine…

69. Violated
Expectations
Our brain works like a giant prediction engine…

70. ConflictConfirm
Rework'
Learn'
Validate(
Modify'
Progress Loop Conflict Loop
Troubleshoot'
The Rhythm of “Idea Flow”

71. “Friction” in Idea Flow
ConflictConfirm
Rework'
Learn'
Validate(
Modify'
Progress Loop Conflict Loop
Troubleshoot'

72. “Idea Flow Map”
“Friction”

73. Idea Flow Mapping Tools
(Open Source, Supported GA ~July 2016)
github.com/ideaflow/tools

74. Typical Idea Flow Maps
Single Problem
Multi-Problem

75. We can read Visual Indicators in Idea Flow Maps
Le#$Atrium$
Le#$Ventricle$
Right$Ventricle$
Right$Atrium$
What’s$causing$this$pa7ern?$
Similar to how an EKG helps doctors diagnose heart problems...

76. ...Idea Flow Maps help developers diagnose software problems.
Problem-Solving
Machine
We can read Visual Indicators in Idea Flow Maps

77. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Based on:
Grandon Gill and Richard Hicks models of Task Complexity
and John Boyd’s OODA Loop

78. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Choose a general strategy

79. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Understand the system

80. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Code & work out the kinks

81. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Back to the drawing board

82. The symptoms we experience depend on where the
disruptions are in the problem-solving process

83. Trial and Error
The Friction is Here
Troubleshooting
Progress
Learning
Rework

84. Assumption Risk (Rework)
Likelihood)of))
making)a))
Bad)Assump4on)
Cost)to)Correct)Decisions)
High)Uncertainty)
Low)Delay)
Low)Uncertainty)
Low)Delay)
Low)Uncertainty)
High)Delay)
PAIN)

85. Struggling with Understanding the Code
The Friction is Here
Troubleshooting
Progress
Learning
Rework

86. Familiarity Risk (Learning)
Likelihood)of))
working)with)
Unfamiliar)
Code)
Cost)to)Learn)
High)Frequency)
Easy)to)Learn)
Low)Frequency)
Easy)to)Learn)
Low)Frequency)
Hard)to)Learn)
PAIN)

87. Struggling with Feedback
The Friction is Here
Troubleshooting
Progress
Learning
Rework

88. Quality Risk (Troubleshooting)
Likelihood)of))
Unexpected)
Behavior)
Cost)to)Troubleshoot)and)Repair)
High)Frequency)
Low)Impact)
Low)Frequency)
Low)Impact)
Low)Frequency)
High)Impact)
PAIN)

89. ”Friction” is the time spent on:
x
Troubleshooting
x
Learning
x
Rework
Quality Risk Familiarity Risk Assumption Risk

90. “What caused the pain
in this case?”
Categorize the Problems with #HashTags
#ReportingEngine
#Hibernate
#MergeHell

91. 1. Problem A
2. Problem B
3. Problem C
1000 hours/month
Add up the Pain by Category
What’s the biggest problem to solve?

92. Case studies.

93. Troubleshooting
Progress
Learning
7:070:00
0:00 19:52
12 year old project after all original developers left.
Case Study: Huge Mess with Great Team
70-90% of dev capacity on “friction”

94. The Team’s Improvement Focus:
Increasing unit test coverage by 5%
Case Study: Huge Mess with Great Team
1. Test Data Generation
2. Merging Problems
3. Repairing Tests
1000 hours/month
The Biggest Problem:
~700 hours/month generating test data

95. 18 months after a Micro-Services/Continuous Delivery rewrite.
Troubleshooting
Progress
Learning
40-60% of dev capacity on “friction”
0:00 28:15
12:230:00
Case Study: From Monolith to Microservices

96. The Architecture Looked Good on Paper
Team A Team B Team C
Complexity Moved Here

97. We Don’t Have TIME To Fix It!

98. The Cost of Escalating Risk
0%
100%
Release 1 Release 2 Release 3
Troubleshooting
Progress
Learning
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
(extrapolated from samples)

99. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Figure out what to do
Learning is front-loaded
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

100. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Rush Before the Deadline
Validation is Deferred
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

101. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Pain Builds
Baseline friction keeps rising
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

102. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Chaos Reigns
Unpredictable work stops
fitting in the timebox
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

103. Over 1000 Hours/Month in Downtime!
They tried to solve their design problems
with more test automation.
25 developers down for 2 days
(It didn’t work)

104. Lots of Friction in BRAND NEW code:
27:15
“WHY?”

105. Application
3rd Party
Dependencies
Local
Components
Usability
Problems
90% of our Software is HERE
This is
PAINFUL!

106. “The Idea Flow Factory”
(supply chain model)
Optimize the Rate of Idea Flow
Across ANY Software Supply Chain

107. Build Test Deploy
Typical manufacturing metaphor mapping:
Customer
The Problem: Focus on short-term effects

108. Software Supply Chain as the Factory Floor
Customer

109. Software Supply Chain as the Factory Floor
Customer
Dev = Operator “flows” the components

110. Architecture Changes Create Bottlenecks

111. Customer
Lack of Familiarity Creates Bottlenecks
WTF?!

112. Customer:
Big Kahuna
Customers:
Seg 2
Seg 3
Customers:
Seg 2
Seg 4
Seg 5
Connect Maintenance Effort to Revenue
through Software Dependencies

113. “Big Kahuna”
Effort
Revenue
“Throughput Accounting” Metrics
Cost of Support vs Revenue per Customer Segment (%)

114. Effort
Revenue
“Throughput Accounting” Metrics
Cost of Support vs Revenue per Customer Segment (%)

115. Breakdown of Development Capacity
If you were an investor:
wouldn’t YOU want to reduce friction?
“Throughput Accounting” Metrics

116. With #OpenDX, we can apply the entire suite of
Risk Management Tools from Lean Manufacturing
Statistical Process Control (SPC)
Optimal Friction
Upper Control Limit
X"
“Out of Control”
20min
0m
Average Pain per Incident
Target
Control
Limit
Supply Chain Optimization
Throughput Accounting

117. We have the power to solve this problem:
If we choose to do it.

118. Application
3rd Party
Dependencies
Local
Components
Usability
Problems
This is
PAINFUL!
We have the power to solve this problem:

119. Two Options:
Option 1
Stay
the Course
Option 2
Build
Happiness
or
Let’s Do This
Miko Matsumura: “Build Happiness” is a
philosophy that happiness must be built.

120. Miko: “That’s not a happy elephant!”

121. Miko: “That’s not a happy elephant!”

122. Focus on the PAIN & Build Happiness

123. Let’s Talk About It.
openmastery.slack.com
#OpenDX #GradleDX

124. Janelle Klein
openmastery.org @janellekz
Check out openmastery.org for details.
Read my Book.
Think About It.
FREE with
MembershipBuy It
How to Measure the PAIN
in Software Development
Janelle Klein

125. Learn more at www.gradle.org