What if we could measure the indirect costs of pain building up on a software project? What if we could measure the loss of productivity, the escalating costs and risks, and could steer our projects with a data-driven feedback loop? By measuring the friction in “Idea Flow”, the flow of ideas between the developer and the software, we can create a data-driven feedback loop for learning what works. Rather than making decisions based on anecdote and gut feel, we can start driving our improvement decisions with real data. Data-Driven Software Mastery is about learning and improving faster than ever. Find out how you can: • Identify the biggest causes of productivity loss on your software project. • Avoid spending tons of time solving the wrong problems • Collaborate with other industry professionals in the art of data-driven software mastery Idea Flow gives us a universal language for describing our experience, so we can share the patterns and principles of what works. With a feedback loop, we can run real experiments! Idea Flow turns the development community into a scientific community.

1. Janelle Klein
openmastery.org @janellekz
Tour Speaker
Data-Driven
Software Mastery
newiron.com

2. , Developer, Consultant, CTO @New Iron
Specialized in Statistical Process Control (SPC)
and Supply Chain Optimization from Lean Manufacturing (I <3 cool data)
Continuous Delivery infrastructure, automation strategy & technical mentorship
Janelle Klein
Who Am I?
I’m also a hobbyist Cognitive Scientist
How to Measure the PAIN
in Software Development
Janelle Klein
Author of “Idea Flow”
leanpub.com/ideaflow

3. LEARN YOUR WAY TO AWESOME.
Community of Practice for
Data-Driven Software Mastery
Why Are We Here?

4. Why Should You Care?
Problem #1: Organizational Dysfunction

5. Why Should You Care?
Problem #2: Solving the Wrong Problem

6. Why Should You Care?
This talk is about a
REALISTIC STRATEGY
to solve these problems.

7. I’m going to show you:
POINT A
(I’ve been working on this for 5 years)

8. I Need Your Help to Get to:
POINT AWESOME
Point A Point AWESOME?
Iterate!

9. LEARN YOUR WAY TO AWESOME.
Why Are We Here?
Learning Takes Work.

10. Let’s Make the PAIN Visible!
#OpenDX
Our Campaign:

11. This isn’t About Me.
This is About ALL OF US.

12. Five Years Ago I Read This Book:
How to Build a Learning Organization

13. Five Disciplines of a Learning Organization
Personal Mastery
Mental Models
Shared Vision
Team Learning
Systems Thinking
These disciplines were emergent practice on our team.

14. Five Disciplines of a Learning Organization
Personal Mastery
Mental Models
Shared Vision
Team Learning
Systems Thinking
These disciplines are emergent practice in
software development

15. What’s the Gap?
What was different about our team?

16. My Day Job is Technical Assessment
GeorgeMe
What does “better quality” mean to you?

17. What does “better” really mean?

18. What does “better” really mean?
Where does my
“better” come from?

19. “Better”
Fifth Discipline: “Shared Vision” = “Shared Better”
The Key to Team Learning:
“Better”

20. Five Disciplines of a Learning Organization
Personal Mastery
Mental Models
Team Learning
Systems Thinking
This book is about the art of group problem-solving.
Shared LearningShared Better

21. Five Years to Put “Better” into Words
How to Measure the PAIN
in Software Development
Janelle Klein
This is the my story.

22. We were trying to do
all the “right” things.
About 8 Years Ago…

23. We were building a
factory automation system...

24. We shipped to production...

25. We shipped to production... (again)

26. We couldn’t reproduce the problem.

27. We shipped to production... (AGAIN)
The rollback failed!

28. Project FAILURE

29. The Retrospective
“What are we going to do?!”
Our biggest problem
“Well, we know we’ve got
a quality problem right?”

30. The Retrospective
“What are we going to do?!”
Our biggest problem
“The problem is we don’t have
enough test automation!”

31. So the Test Automation Began…
Our regression testing
took 3-4 weeks…
Let’s automate the tests!

32. 20%
Percent Automated:
So the Test Automation Began…

33. 50%
Maintenance
started getting PAINFUL…
Let’s refactor the tests!
Percent Automated:
So the Test Automation Began…

34. 80%
It was still really PAINFUL…
“Well, at least our regression
cycle is faster, right?”
Our regression cycle still took 3-4 weeks!
Percent Automated:
So the Test Automation Began…

35. ApplicationTests
Our Test Suite

36. ApplicationTests
Our Test Suite
Any ideas?

37. The Retrospective
“What are we supposed to do?!”
Our biggest problem
“Well, if we don’t understand
a problem, we should
collect data.”

38. The Retrospective
Our biggest problem
“What data would help us
understand the problem?”
“What are we supposed to do?!”

39. Technical Debt Mistakes
I thought the main obstacle was
Technical Debt

40. ?
Most of our mistakes were in the
most well-written parts of the code.
Mistakes

41. We made significantly more mistakes
in code that we didn’t write ourselves.
Lower
Familiarity
More
Mistakes=
There had to be more to the story...

42. Complex(
So*ware(
PAIN
This is what I knew...
What made development feel painful?

43. Unexpected
Behavior
Problem
Resolved
Tracking Painful Interaction with the Code (Friction)
Troubleshooting
Progress
5 hours and 18 minutes of troubleshooting...
PAINFUL

44. The amount of PAIN was caused by…
Likeliness(of((
Unexpected(
Behavior(
Cost(to(Troubleshoot(and(Repair(
High(Frequency(
Low(Impact(
Low(Frequency(
Low(Impact(
Low(Frequency(
High(Impact(
PAIN(

45. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
Most of the pain was caused by human factors.
What causes PAIN?

46. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
Most of the pain was caused by human factors.
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm

47. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Most of the pain was caused by human factors.

48. What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
PAIN is a consequence of how we interact with the code.
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm

49. PAIN occurs during the process of
understanding and extending the software
Complex(
So*ware(
PAIN
Not the Code.
Optimize “Idea Flow”

50. Analyzing the Types of Mistakes
#1 Cause of Mistakes:
Misunderstandings of how the system worked

51. ApplicationTests
The Bugs were in Our Conceptual Models
Fix: We built a Data Dictionary

52. The “Coat of Armor” Principle
Software
Coat of Armor

53. “Alarm Pain”
Six hours of troubleshooting and repairing tests
when nothing is broken.

54. The Waxy Coating Principle
Optimize the signal to noise ratio.
Software
(before)
Software
(after)
Tests are like a waxy coating poured over the code.

55. We deleted everything…
The automation wasn’t
solving the problem!

56. What are the specific risks in this release?
How can we mitigate the risks?
Handful of
manual tests
Custom
Tools
+

57. Release
Size
(weeks)
Time
Defect Rates
We Could See the Effects…
Effective Risk Management Looks Like This

58. My team spent tons of time working on
improvements that didn’t make much difference.
We had tons of automation, but the
automation didn’t catch our bugs.

59. My team spent tons of time working on
improvements that didn’t make much difference.
We had well-modularized code,
but it was still extremely time-consuming to troubleshoot defects.

60. The hard part isn’t solving the problems
it’s identifying the right problems to solve.
“What are the specific problems
that are causing the team’s pain?”

61. “Better”
Fifth Discipline: “Shared Vision” = “Shared Better”
The Key to Team Learning:
“Better”

62. “Better”
following best practices
(solution-focused) (problem-focused)
“Better”
solving the problems
What does “better” really mean?
How do we identify the right problems to solve?

63. Cost & Risk are a Function of Increased Difficulty
Cost
&
Risk
Difficulty of Work
The Difficulty of Doing Our Jobs
Human
Limitations

64. Likelihood)of))
Unexpected)
Behavior)
Cost)to)Troubleshoot)and)Repair)
High)Frequency)
Low)Impact)
Low)Frequency)
Low)Impact)
Low)Frequency)
High)Impact)
PAIN)
“Better” = Reduce the Risk
Reduce Likeliness
=
Mistake Proofing
Reduce Recovery Cost = Mistake Tolerance

65. What Exactly does Idea Flow Measure?

66. Complex(
So*ware(
PAIN
Not the Code.
Optimize “Idea Flow”

67. FRICTION is a Function of Increased Difficulty
Cost
&
Risk
Difficulty of Work
The Difficulty of Doing Our Jobs
Human
Limitations

68. The Rhythm of “Idea Flow”
Write a little code.
Work out the kinks.
Write a little code.
Work out the kinks.
Write a little code.
Work out the kinks.

69. ConflictConfirm
Rework'
Learn'
Validate(
Modify'
Progress Loop Conflict Loop
Troubleshoot'
Write a little code. Work out the kinks.
The Rhythm of “Idea Flow”

70. On Intelligence
Jeff Hawkins

71. Confirmed
Expectations
Our brain works like a giant prediction engine…

72. Conflicting
Expectations
Our brain works like a giant prediction engine…

73. ConflictConfirm
Rework'
Learn'
Validate(
Modify'
Progress Loop Conflict Loop
Troubleshoot'
The Rhythm of “Idea Flow”

74. “Friction” in Idea Flow
ConflictConfirm
Rework'
Learn'
Validate(
Modify'
Progress Loop Conflict Loop
Troubleshoot'

75. “Idea Flow Map”
“Friction”
A “conflict” starts with an unexpected observation and
ends when understanding is resolved.

76. Idea Flow Mapping Tools
(Open Source, Supported GA ~June 2016)
github.com/ideaflow/tools

77. Typical Idea Flow Maps
Single Problem
Multi-Problem

78. Reading Visual Indicators in Idea Flow Maps
Le#$Atrium$
Le#$Ventricle$
Right$Ventricle$
Right$Atrium$
What’s$causing$this$pa7ern?$
Similar to how an EKG helps doctors diagnose heart problems...

79. ...Idea Flow Maps help developers diagnose software problems.
Problem-Solving
Machine
Reading Visual Indicators in Idea Flow Maps

80. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)

81. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Choose a general strategy

82. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Understand the system

83. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Code & work out the kinks

84. = Solution Strategy
The “Heart” of Software Development
(the problem-solving machine)
Back to the drawing board

85. The symptoms we experience depend on where the
disruptions are in the problem-solving process

86. Trial and Error
The Friction is Here
Troubleshooting
Progress
Learning
Rework

87. Assumption Risk (Rework)
Likelihood)of))
making)a))
Bad)Assump4on)
Cost)to)Correct)Decisions)
High)Uncertainty)
Low)Delay)
Low)Uncertainty)
Low)Delay)
Low)Uncertainty)
High)Delay)
PAIN)

88. Struggling with Understanding the Code
The Friction is Here
Troubleshooting
Progress
Learning
Rework

89. Familiarity Risk (Learning)
Likelihood)of))
working)with)
Unfamiliar)
Code)
Cost)to)Learn)
High)Frequency)
Easy)to)Learn)
Low)Frequency)
Easy)to)Learn)
Low)Frequency)
Hard)to)Learn)
PAIN)

90. Struggling with Feedback
The Friction is Here
Troubleshooting
Progress
Learning
Rework

91. Quality Risk (Troubleshooting)
Likelihood)of))
Unexpected)
Behavior)
Cost)to)Troubleshoot)and)Repair)
High)Frequency)
Low)Impact)
Low)Frequency)
Low)Impact)
Low)Frequency)
High)Impact)
PAIN)

92. measures the time spent on:
Idea Flow
x
Troubleshooting
x
Learning
x
Rework
Quality Risk Familiarity Risk Assumption Risk

93. What percentage of time do you
think your team spends on friction?

94. Troubleshooting
Progress
Learning
7:070:00
0:00 19:52
12 year old project after all original developers left.
Case Study: Huge Mess with Great Team
70-90% of dev capacity on “friction”

95. The Team’s Improvement Focus:
Increasing unit test coverage by 5%
Case Study: Huge Mess with Great Team
1. Test Data Generation
2. Merging Problems
3. Repairing Tests
1000 hours/month
The Biggest Problem:
~700 hours/month generating test data

96. The Retrospective
“What’s the biggest opportunity for improvement?”
Our biggest problem

97. “What’s the biggest opportunity for improvement?”
“The awful email
template engine code!”
Our biggest problem
The Retrospective

98. “What’s the biggest opportunity for improvement?”
“Fill in missing
unit tests!”
Our biggest problem
The Retrospective

99. “What’s the biggest opportunity for improvement?”
“I know how to improve
database performance!”
Our biggest problem
The Retrospective

100. “What’s the biggest opportunity for improvement?”
“Let’s improve maintainability
of our test framework!”
Our biggest problem
The Retrospective

101. “What’s the biggest opportunity for improvement?”
Just because a problem comes to mind,
doesn’t mean it’s an important problem to solve.
Our biggest problem
The Retrospective

102. “What’s the biggest opportunity for improvement?”
Our biggest problem
What do I feel the
most intensely about?
Daniel Kahneman
Thinking Fast and Slow
The Retrospective

103. “What’s the biggest opportunity for improvement?”
“The awful email
template engine code!”
Recency Bias
Our biggest problem
The Retrospective

104. “What’s the biggest opportunity for improvement?”
Guilt Bias
“Fill in missing
unit tests!”
Our biggest problem
The Retrospective

105. “What’s the biggest opportunity for improvement?”
“I know how to improve
database performance!”
Known Solution Bias
Our biggest problem
The Retrospective

106. “What’s the biggest opportunity for improvement?”
Sunk Cost Bias
“Let’s improve maintainability
of our test framework!”
Our biggest problem
The Retrospective

107. Relying on Gut Feel:

108. 18 months after a Micro-Services/Continuous Delivery rewrite.
Troubleshooting
Progress
Learning
40-60% of dev capacity on “friction”
0:00 28:15
12:230:00
Case Study: From Monolith to Microservices

109. The Architecture Looked Good on Paper
Team A Team B Team C
Complexity Moved Here

110. We Don’t Have TIME To Fix It!

111. PAIN
The Classic Story of Project Failure
Problems get deferred
Builds start breaking
Releases get chaotic
Productivity slows to a crawl
Developers begging for time
It’s never enough
Project Meltdown

112. The Cost of Escalating Risk
0%
100%
Release 1 Release 2 Release 3
Troubleshooting
Progress
Learning
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
(extrapolated from samples)

113. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Figure out what to do
Learning is front-loaded
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

114. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Rush Before the Deadline
Validation is Deferred
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

115. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Pain Builds
Baseline friction keeps rising
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

116. 0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Chaos Reigns
Unpredictable work stops
fitting in the timebox
Troubleshooting
Progress
Learning
The Cost of Escalating Risk

117. This is the Biggest Cause of FAILURE
In Our Industry:
Have you struggled with these problems in your organization?

118. “Better”
“Better”
Fifth Discipline: “Shared Vision” = “Shared Better”
Developer
Developer
Manager Manager
The Key to Organizational Learning:

119. Lessons Learned

120. How can we reduce the impact of
bad architecture assumptions?
(Assumption Risk)
The cost of bad architecture decisions
in the microservices world
is EXTREMELY HIGH.

121. If you’ve got beautiful code
Because you pushed all the PAIN
to your clients…
Your code SUCKS.
How do we know what the
client’s experience will be like?

122. Lots of Friction in BRAND NEW code:
27:15
“WHY?”

123. Application
3rd Party
Dependencies
Local
Components
Usability problems cascade through dependencies
Usability
Problems
This is Painful!

124. Application
3rd Party
Dependencies
Local
Components
Usability
Problems
This is Painful!
90% of our Software is HERE

125. “The Idea Flow Factory”
(supply chain model)
Optimize the Rate of Idea Flow
Across ANY Software Supply Chain

126. What if we learned our way to “better”, together?
Company
Company
Company
Company
Company
Company Company

127. LEARN YOUR WAY TO AWESOME.

128. Start with Why
Simon Sinek

129. Why - Your Purpose
How - Your Process
What - Your Solution
Humans Align Around Shared Purpose

130. Let’s Make the PAIN Visible!
#OpenDX

131. Why?
#OpenDX

132. Why?
Because This Really SUCKS.

133. Why?
And So Does This.

134. #OpenDX
Measure the PAIN that you CARE About

135. LEARN YOUR WAY TO AWESOME.
We BELIEVE That:
Even though we may not have all the answers,
what matters is that we know how to learn.

136. How Do We Learn Our Way There?
Point A Point AWESOME?
Iterate!

137. Because we’re part of a community
with other smart people…
We BELIEVE That:
We will get better, faster, by working together.

138. #OpenDX
How?

139. Community Analytics Platform
(Long-Term Roadmap)

140. Community Problem #1:
Solve This Pain:

141. Team Visibility Platform
IFM Data
Collection
Team Visibility
Raw
Data
Timeline
Abstraction
1. Record an Idea Flow Map
2. Review/Annotate an Idea Flow Map
3. Identify the Biggest Pains
Project
Abstraction
#Private
Taxonomy

142. Community Analytics Platform
Anonymizer Anonymizer Anonymizer Anonymizer
#Private
Taxonomy
Team
Visibility
#Private
Taxonomy
Team
Visibility
#Private
Taxonomy
Team
Visibility
#Private
Taxonomy
Team
Visibility
Community Analytics
#Shared
Taxonomy

143. Community Analytics Platform
Community Analytics
Shared Taxonomy
Tiger
Bear
Requirement: Analyze Patterns & Codify Lessons Learned
Community
Focus
Community Analytics
#Shared
Taxonomy

144. Community Problem #2:
Solve This Pain:

145. Team
Server
Team
Server
“Idea Flow Factory”
Supply Chain Cost Model
Supply Chain Optimization
Team
Server
“Customer Factory”
Revenue Model
(Ash Maurya)
Throughput Accounting
(Effort vs Revenue)
Organizational Mastery Platform

146. Community Problem #3:
Solve This Pain with Lack of Skills:
Company
Nope.Nope.Nope.Nope.Nope.
Nope.Nope.Nope.Nope.
Nope.Nope.
Nope.Nope.Nope.
Nope.Nope.Nope.Nope.Nope.Nope.Nope.Nope.
Nope.
Yay!
We need a
Sr Developer!

147. Open Certification Platform
(Crowd-Sourced Peer Mentorship)
Decision Assessment
Archetype Examples
& Decision-Making Tests
Mentorship Training
M1
M2
M3
C1
C2
C3
Crowd-Sourced
Peer Video Assessment
#Shared
Taxonomy

148. Community Problem #4:
Solve This Pain with Upstream Usability:

149. #OpenDX
Bring visibility up to the highest levels of our
organizations and across the industry

150. The “Open Source” Roadmap
Point A Point AWESOME?
Iterate!

151. Get together over lunch and share lessons learned.
Leaders Circle
Developers Circle

152. Open Mastery Circle Meetings
Circle Leader
Circle Member
Focus: What’s the problem to solve?
What: Ask questions about the facts
Why: Breakdown the causes
How: Strategies to reduce the pain
Codify: Lessons Learned into #Patterns.
Observation
Questions
The Explicit Mastery Loop

153. The Explicit Mastery Loop
Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
1. What2. Why3. How
F ocus!
Output: Pain Signal
Improve Quality of Decisions

154. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
1. What2. Why3. How
F ocus!
Output: Pain Signal
Target - The direction of “better”
Target: Optimize the Rate of Idea Flow
The Explicit Mastery Loop

155. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
1. What2. Why3. How
F ocus!
Output: Pain Signal
Input - The constraints that limit our short-term choices…
The Explicit Mastery Loop

156. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
1. What2. Why3. How
F ocus!
Output: Pain Signal
Output - The pain signal we’re trying to improve
The Explicit Mastery Loop

157. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
2. What3. Why4. How
F ocus!
Output: Pain Signal
1. !
1. Focus on the biggest pain
F ocus!
The Explicit Mastery Loop

158. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
2. What3. Why4. How
F ocus!
Output: Pain Signal
1. !
2. What - Identify the specific symptoms
2. What
The Explicit Mastery Loop

159. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
2. What3. Why4. How
F ocus!
Output: Pain Signal
1. !
3. Why - Breakdown cause and effect
3. Why
The Explicit Mastery Loop

160. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
2. What3. Why4. How
F ocus!
Output: Pain Signal
1. !
4. How
4. How - Run Experiments to learn what works
The Explicit Mastery Loop

161. Input:
Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
2. What3. Why4. How
F ocus!
Output: Pain Signal
1. !
5. Codify Lessons Learned and Modify Behavior
The Explicit Mastery Loop

162. #OpenDX
What?

163. Companies
Community
Groups
HQ in
Austin
Open Mastery
Austin
WE ARE a “Pay it Forward”
Industry Peer Mentorship Network

164. LEARN YOUR WAY TO AWESOME.

165. Five Years Ago I Read This Book:
How to Build a Learning Organization

166. —>
2.0
Visibility Changes Everything
How to Measure the PAIN
in Software Development
Janelle Klein
Visibility
+
Lean Manufacturing

167. —>+
Everyday Science
(Lean Startup)
We get a design template for how to build a learning organization.
Visibility Changes Everything
2.0

168. Focus: Quality of Execution Decisions
Focus Traction
$$$ $$
Focus: Quality of Investment Decisions
Risk
Translator
(interface)
Management
(Coordination)
Engineering
(Execution)
Open Mastery Learning Framework
Idea Flow Pain Sensor

169. is a Learning Organization

170. Visibility Starts a Chain Reaction
#OpenDX

171. Next Steps:
openmastery.slack.com