Frame, Build, Measure, Learn: Responding to Change

  • 493 views
Uploaded on

The key to running effective experiments is to first frame the experiment thoughtfully. This takes time, but the investment pays off many-fold. This presentation will guide you through the key steps …

The key to running effective experiments is to first frame the experiment thoughtfully. This takes time, but the investment pays off many-fold. This presentation will guide you through the key steps of framing, which, when done properly, reward your brain–and your organization–with learning.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
493
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
0
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Frame, Build, Measure, Learn:
 Responding to Change" Zach Nies – Chief Technologist! www.rallydev.com @zachnies! ©2014!
  • 2. Zach Nies Chief Technologist Rally Software Faculty University of Colorado at Boulder @zachnies www.rallydev.com @zachnies! ©2014!
  • 3. The biggest risk you face
 is changing" www.rallydev.com @zachnies! ©2014!
  • 4. Risk:
 failing to respond to market shifts" www.rallydev.com @zachnies! ©2014!
  • 5. ANOTHER RISK…? www.rallydev.com @zachnies! ©2014!
  • 6. An example…" www.rallydev.com @zachnies! ©2014!
  • 7. Local consumer data company" www.rallydev.com @zachnies! ©2014!
  • 8. 90% We have 80% market share" 80% 70% Market Share 60% 50% 40% 30% 20% 10% 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 9. 90% We need
 a mobile app" 80% 70% Market Share 60% 50% 40% 30% 20% 10% 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 10. 90% Mobile app specification begins" 80% 70% Market Share 60% 50% 40% 30% 20% 10% 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 11. 90% 80% 70% Market Share 60% 50% 40% 30% 20% 10% Startup competitor ships 1.0" 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 12. 90% 80% 70% Market Share 60% 50% 40% 30% 20% 10% Startup competitor ships 1.1" 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 13. 90% App
 defined
 
 Dev
 begins" 80% 70% Market Share 60% 50% 40% 30% 20% 10% 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 14. 90% 80% 70% Market Share 60% 50% 40% 30% 20% 10% Startup competitor ships 1.2" 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 15. 90% Mobile app ships" 80% 70% Market Share 60% 50% 40% 30% 20% 10% 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 16. 90% 80% 70% Market Share 60% 50% 40% 30% 20% 10% We forever lost half our market share in one year" 0% Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Desktop www.rallydev.com @zachnies! Q1 2013 Q2 2013 Q3 2013 Q4 2013 Q1 2014 Mobile ©2014!
  • 17. Gut wrenching" www.rallydev.com @zachnies! ©2014!
  • 18. What’s
 going on here?" www.rallydev.com @zachnies! ©2014!
  • 19. Time needed
 to change
 market expectations" www.rallydev.com @zachnies! ©2014!
  • 20. Some examples…" www.rallydev.com @zachnies! ©2014!
  • 21. 1984! www.rallydev.com @zachnies! ©2014!
  • 22. 2006! www.rallydev.com @zachnies! ©2014!
  • 23. About 20 years! www.rallydev.com @zachnies! ©2014!
  • 24. 2007! www.rallydev.com @zachnies! ©2014!
  • 25. 2010! www.rallydev.com @zachnies! ©2014!
  • 26. 3 years! www.rallydev.com @zachnies! ©2014!
  • 27. 2010! www.rallydev.com @zachnies! ©2014!
  • 28. 2012! www.rallydev.com @zachnies! ©2014!
  • 29. 2 years! www.rallydev.com @zachnies! ©2014!
  • 30. 1979! www.rallydev.com @zachnies! ©2014!
  • 31. 1989! www.rallydev.com @zachnies! ©2014!
  • 32. 10 years! www.rallydev.com @zachnies! ©2014!
  • 33. 5 years! www.rallydev.com @zachnies! ©2014!
  • 34. 2.5 years! www.rallydev.com @zachnies! ©2014!
  • 35. 1.5 years! www.rallydev.com @zachnies! ©2014!
  • 36. www.rallydev.com @zachnies! ©2014! http://hbr.org/2013/03/big-bang-disruption/
  • 37. Huge market shifts can happen
 in a year" www.rallydev.com @zachnies! ©2014!
  • 38. Are you
 Agile enough
 to balance" www.rallydev.com @zachnies! ©2014!
  • 39. operating the business
 and
 responding to change?" www.rallydev.com @zachnies! ©2014!
  • 40. Typical approach" www.rallydev.com @zachnies! ©2014!
  • 41. Create a plan to predict the future" www.rallydev.com @zachnies! ©2014! http://www.flickr.com/photos/isaacmao/380257543
  • 42. Market research" Product definition" Product requirements" Detailed plan" Alpha release" Beta release" Final release" FIRST CONTACT WITH CUSTOMERS" www.rallydev.com @zachnies! ©2014!
  • 43. Market research" Project definition" Product requirements" Detailed plan" Alpha release" Beta release" Final release" FIRST CONTACT WITH CUSTOMERS" www.rallydev.com @zachnies! ©2014!
  • 44. Market research" Project definition" Project requirements" Detailed plan" Alpha release" Beta release" Final release" FIRST CONTACT WITH CUSTOMERS" www.rallydev.com @zachnies! ©2014!
  • 45. Market research" Project definition" Project requirements" Detailed plan" Alpha release" Beta release" Final release" FIRST CONTACT WITH CUSTOMERS" www.rallydev.com @zachnies! ©2014!
  • 46. Market research" Project definition" Project requirements" Detailed plan" Development" " Final release" FIRST CONTACT WITH CUSTOMERS" www.rallydev.com @zachnies! ©2014!
  • 47. Market research" Project definition" Project requirements" Detailed plan" Development" User acceptance testing" " FIRST CONTACT WITH CUSTOMERS" www.rallydev.com @zachnies! ©2014!
  • 48. Market research" Project definition" Project requirements" Detailed plan" Development" User acceptance testing" Final release" FIRST CONTACT WITH CUSTOMERS" www.rallydev.com @zachnies! ©2014!
  • 49. HOW MANY GUESSES?" www.rallydev.com @zachnies! ©2014!
  • 50. FIRST CONTACT WITH USERS" www.rallydev.com @zachnies! ©2014!
  • 51. www.rallydev.com @zachnies! ©2014!
  • 52. www.rallydev.com @zachnies! ©2014!
  • 53. www.rallydev.com @zachnies! ©2014! http://www.flickr.com/photos/oregondot/4132135156
  • 54. Obscure
 the uncertainty
 with details of a false precision" www.rallydev.com @zachnies! ©2014!
  • 55. Embrace, reveal and mitigate
 the uncertainty" www.rallydev.com @zachnies! ©2014!
  • 56. Why does this
 feel unnatural?" www.rallydev.com @zachnies! ©2014!
  • 57. www.rallydev.com @zachnies! ©2014!
  • 58. “the greatest prosperity can exist only when that individual has reached his highest state of efficiency; that is, when he is turning out his largest daily output.”" www.rallydev.com @zachnies! ©2014!
  • 59. www.rallydev.com @zachnies! ©2014!
  • 60. www.rallydev.com @zachnies! ©2014!
  • 61. www.rallydev.com @zachnies! ©2014!
  • 62. Assumes:
 Certainty is high" 
 Goal: 
 Maximize individual efficiency" www.rallydev.com @zachnies! ©2014!
  • 63. 100 years later" www.rallydev.com @zachnies! ©2014!
  • 64. Status Quo
 Budgets" " Comprehensive
 Analysis" " Detailed
 Plans" " Waterfall
 Execution" www.rallydev.com @zachnies! ©2014!
  • 65. Works when:" " Certainty is high" www.rallydev.com @zachnies! ©2014!
  • 66. Is your business getting
 more certain?" www.rallydev.com @zachnies! ©2014!
  • 67. Agile Execution
 
 Lean Thinking" www.rallydev.com @zachnies! ©2014!
  • 68. Balance
 execution and exploration" www.rallydev.com @zachnies! ©2014!
  • 69. Assumes:
 Uncertainty exists" " Goal:
 Maximize value delivery" www.rallydev.com @zachnies! ©2014!
  • 70. Adaptive
 Steering" " Disciplined
 Exploration" " Empirical
 Learning" " Agile
 Execution" www.rallydev.com @zachnies! ©2014!
  • 71. Adaptive
 Steering" " Disciplined
 Exploration" " Empirical
 Learning" " Agile
 Execution" www.rallydev.com @zachnies! ©2014!
  • 72. Turn Uncertainty" Into Knowledge" www.rallydev.com @zachnies! ©2014!
  • 73. www.rallydev.com @zachnies! ©2014! http://www.flickr.com/photos/memekiller/4564410767
  • 74. www.rallydev.com @zachnies! ©2014!
  • 75. Business" " " " Initiative" www.rallydev.com @zachnies! ©2014!
  • 76. www.rallydev.com @zachnies! ©2014!
  • 77. www.rallydev.com @zachnies! ©2014!
  • 78. Assumptions Canvas Name: Key Assumption Reminders will help gift givers Dependent Assumptions Key Assumption Suggestions will motivate givers Dependent Assumptions We know when We can give giving events reasonable will happen suggestions Text messages are acceptable reminders We can send timely text messages We can get receiver preference data Date: Key Assumption Falsified Assumptions We can make money through partner referrals Dependent Assumptions Partners want referral leads We can get Other Assumptions Other Assumptions Facebook adds Recipients will drive want gifts on traffic time A mobile first enough parter product inventory strategy Our leads will buy gifts I-1 www.rallydev.com @zachnies! ©2014!
  • 79. Key assumptions" Knowledge" www.rallydev.com @zachnies! ©2014!
  • 80. Run experiments" www.rallydev.com @zachnies! ©2014!
  • 81. Frame" www.rallydev.com @zachnies! ©2014!
  • 82. www.rallydev.com @zachnies! ©2014!
  • 83. www.rallydev.com @zachnies! ©2014!
  • 84. www.rallydev.com @zachnies! ©2014!
  • 85. www.rallydev.com @zachnies! ©2014!
  • 86. www.rallydev.com @zachnies! ©2014!
  • 87. www.rallydev.com @zachnies! ©2014!
  • 88. www.rallydev.com @zachnies! ©2014!
  • 89. www.rallydev.com @zachnies! ©2014!
  • 90. www.rallydev.com @zachnies! ©2014!
  • 91. www.rallydev.com @zachnies! ©2014!
  • 92. www.rallydev.com @zachnies! ©2014!
  • 93. Build" www.rallydev.com @zachnies! ©2014!
  • 94. Scrum Machine Daily Scrum Product Backlog Refinement Sprint Planning Sprint 1-2 Weeks Burndown Chart Product Backlog www.rallydev.com Sprint Backlog @zachnies! Sprint Demo & Review Sprint Retrospective Product Increment ©2014!
  • 95. Measure" www.rallydev.com @zachnies! ©2014!
  • 96. Collect" www.rallydev.com @zachnies! ©2014! http://www.flickr.com/photos/mrpetersononline/4977025601
  • 97. Analyze" www.rallydev.com @zachnies! ©2014! http://www.flickr.com/photos/mrpetersononline/4992505907
  • 98. Organize" www.rallydev.com @zachnies! ©2014! http://www.flickr.com/photos/mrpetersononline/4992505641
  • 99. Learn" www.rallydev.com @zachnies! ©2014!
  • 100. www.rallydev.com @zachnies! ©2014!
  • 101. www.rallydev.com @zachnies! ©2014!
  • 102. www.rallydev.com @zachnies! ©2014!
  • 103. www.rallydev.com @zachnies! ©2014!
  • 104. 25% www.rallydev.com @zachnies! ©2014!
  • 105. 25% www.rallydev.com @zachnies! 55% ©2014!
  • 106. Empirical Learning" www.rallydev.com @zachnies! ©2014!
  • 107. www.rallydev.com @zachnies! ©2014!
  • 108. Advice
 from a master…" www.rallydev.com @zachnies! ©2014!
  • 109. www.rallydev.com @zachnies! http://www.richard-feynman.net/gallery.htm ©2014!
  • 110. www.rallydev.com @zachnies! ©2014!
  • 111. One last thing to keep in mind" www.rallydev.com @zachnies! ©2014!
  • 112. Technology adoption is a human phenomenon " www.rallydev.com @zachnies! ©2014!
  • 113. An example…" www.rallydev.com @zachnies! ©2014!
  • 114. www.rallydev.com @zachnies! ©2014!
  • 115. Need an Agile and human centered approach! www.rallydev.com @zachnies! ©2014!
  • 116. Balance execution and exploration" www.rallydev.com @zachnies! ©2014!
  • 117. LEARN MORE… rallydev.com/els rallydev.com/frame www.rallydev.com @zachnies! ©2014!