Your SlideShare is downloading. ×
'Metrics That Matter': Gabrielle Benefield @ Colombo Agile Con 2014
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

'Metrics That Matter': Gabrielle Benefield @ Colombo Agile Con 2014

393
views

Published on

Evolve Beyond Managing Director Gabrielle Benefield on 'Metrics That Matter: Outcomes Over Outputs', which discusses the pitfalls of traditional metrics and how they are not fit for purpose, then …

Evolve Beyond Managing Director Gabrielle Benefield on 'Metrics That Matter: Outcomes Over Outputs', which discusses the pitfalls of traditional metrics and how they are not fit for purpose, then provides an alternative approach that teams are adopting worldwide using the Mobius framework

Published in: Software

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
393
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
24
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Metrics that matter
  • 2. Waterfall Agile Mobius Ac#vity  based  tracking     Output  based  tracking     Outcome  based  tracking     What do we measure? Planning Development Design Testing Maintain Launch
  • 3. OPTION A Improve network Increase # checkout conversions Baseline:15% Target:30% Current:18% 5 Op#on   Value   Risk   Cost   Impact   Op#on   A   ?   M   M   ?   Op#on   B   ?   H   H   ?   Op#on   C   ?   L   L   ?   People start the buying process but don’t complete it HYPOTHESIS Improving the page load speed will help conversions Increase checkout conversions Improve page load speed DATA 85% of users abandon checkout How to create outcomes case study OPTION B Build a local datacenter DATA 86% of users abandon checkout when page loads times = > 4 seconds CREATE OPTIONS Generate ideas on how to fix the page load speed INSIGHT The further users are from the Datacenter, the slower the page loads OPTION C Compress page elements IMPLEMENT Image compression EXPERIMENT Test one server with 0.002% split test RESEARCH Run network diagnostics with a trace route NEXT IMPROVEMENTS What else can we do to improve the conversion rate? IMPLEMENT: Add instrumentation and exit survey to pages OPTION A Fix the network OPTION Find out why people are leaving the service
  • 4. Tom Gilb’s ‘Ilities’ ( Reliability ( Availability ------ ( Maintainability ( ( Integrity ------------ ( Threat ( ( Security ( ( ( ( Connectability ( ( Flexibility------------ ( Tailorability ---( Extendibility Quality ------------ ( Adaptability ----- ( ( Interchangeability ( ( ( ( Upgradeability ----- ( Installability ( ( Portability ( ( Improvability ( ( ( ( Entry level experience ( ( Training attribute ( Usability --------- ( Handling ability ( Likeability ( Demonstratability ( Financial saving Resource saving ---( Time saving ( Effort saving ( Equipment saving ( Throughput Workload capacity ( Response time ( Storage capacity Quality outcomes
  • 5. How contracts measure success     PLAN       OUTPUT       ACTIVITIES       OVERARCHING   GOAL   CONTRACT  CONTRACT   Measurement  of  success  
  • 6. Why there is friction     ACTUAL   OUTCOMES      ACTIONS       PLANS       OVERARCHING   GOAL   KNOWLEDGE  GAP   The  difference  between   what  we  would  like  to  know   and  what  we  actually  know   EFFECTS  GAP   The  difference  between  what  we   expect  our  ac#ons  to  achieve  and   what  they  actually  achieve   ALIGNMENT  GAP   The  difference  between   what  we  want  people  to  do   and  what  they  actually  do   Source:  ‘The  Art  of  Ac#on’  by  Stephen  Bungay  
  • 7. Medium  to  large  projects  experience  25-­‐35%  requirements  change.     Capers  Jones  on  So,ware  Development     Our  customers  say  that  if  it  weren’t  for  the  contract  making  it  so  expensive  and  #me-­‐consuming,   they  would  have  had  more  changes,  and  could  have  built  beSer  products.     Locking in scope, locks out opportunities
  • 8. FRAMEWORK  AGREEMENT   Warran#es   IP   Constraints  ($,  Time,  Regulatory)   Statement  Of   Target  Outcome   (SOTO)   $   The contracts have to be Agile
  • 9. Snowden’s  Cynefin  model   Cause  and  effect   rela#ons  repeatable,   perceivable  and   predictable   Cause  and  effect  are   only  coherent  in   retrospect  and  do   not  repeat     Cause  and  effect   separated  over  #me   and  space   CHAOS   SIMPLE   (Known)   COMPLEX   (Unknowable)     COMPLICATED   (Knowable)   No  cause  and  effect   rela#onships   perceivable  
  • 10. Benefits of Agile
  • 11. Questions?
  • 12. Nearly 50 countries use Systematic’s solutions  Systematic’s offices   Manage  complex  projects  to  succes  using  CMMI,  Lean  and  Scrum   hSp://www.systema#c.com     Public Sector Intelligence & National Security Healthcare Defence
  • 13. Standard Project Status Report Managing normal projects is a prerequisite to manage large projects CMMI and Lean provides a solid foundation for both Status   Analysis  of  measures   Risk  top  3   Ac#ons   Risk  Measure   12  std  KPI’s  
  • 14. Example: Flow of story implementation and fix-time after failed build Productivity is optimized: •  Flow of work - maximized •  Fix-time of failed builds - minimal •  Sprint test and release - minimal à Variance reduced Measures  inspired  from  Lean Fix time for failed builds Flow of story implementation
  • 15. Questions?
  • 16. You get what you measure “Perhaps what you measure is what you get. More likely, what you measure is all you get. What you don’t (or can’t) measure is lost.” H Thomas Johnson (2006)
  • 17. Friction “Fric7on  manifests  itself  when  human  beings  with  independent  wills  try   to  achieve  a  collec7ve  purpose  in  a  fast-­‐changing,  complex   environment  where  the  future  is  fundamentally  unpredictable.”     ‘The  Art  of  Ac#on’  by  Stephen  Bungay,  2011   •  Fric#on:   Ø  An  accumula#on  of  innumerable  peSy  circumstances     e.g.  uncertain#es,  errors,  accidents,  technical  difficul#es,  the  unforeseen     Ø  …  which  impact  on  decisions,  morale  and  ac#ons   Ø  …  thereby  preven#ng  you  from  achieving  your  goal    
  • 18. Causes of friction Human  finitude   •  Services  are  performed  by  people:   Ø  People  are  imperfect     Ø  People  are  intrinsically  unpredictable     Ø  People  have  different  interpreta#ons   •  The  environment  is  complex  and  chao#c:   Ø  The  environment  is  non-­‐linear  and  fundamentally  unpredictable   Ø  External  factors  contribute  towards  a  change  in  the  environment   Ø  Even  if  the  environment  is  stable,  some  of  the  effects  of  our  ac#ons  will  be   unintended.      
  • 19. Pay with a one step pay button Step   Search for book Step   Decrease time to pay TO   Decrease time to find a book TO   Find one book and get it shipped Goal   Setup one step pay option Goal   Call and pay Step   Enter payment details Step   Save payment details Step   Our  team  tries  to  get  feedback  and  hit  the  outcomes  with  the  least   amount  of  effort.  Goal  mapping  helps  us  set  a  goal  and  find  the   simplest  path  to  reach  it.     22  
  • 20. Outcomes Op#ons   In  Progress Done 23   Decrease  page   load   #me   Setup local datacenter Option   Cache page elements Option   Decrease  #me   to     find  products   Redesign search results Option   Clickstream analysis Research     Datacenter  split   test  Australia   Research     Test  caching   one  page   Research     Split  test   redesign   Research     Trace  route   analysis   Research   Teams  track  outcomes  and  op#ons  visually  during  delivery.  
  • 21. How  much  progress     did  we  make?   Decrease  #me  to  find  products   Baseline:   7  minutes   Target:   <  3  minutes   Current:   5  minutes   Decrease  page  load  #me   Baseline:   12  seconds   Target:   <  4  seconds     Current  #me:  9  seconds   We  saved  2   minutes   We  reduced   page  load   speed  by  3   seconds   24  
  • 22. We  are  constantly  adap#ng   our  model  based  on   feedback.     When  we  see  the  value   diminish,  we  can  choose   other  op#ons  or  outcomes   to  explore   25   x Point  of  diminishing  returns   Time   Outcomes   (value)  
  • 23. 26     Value  Type     Value  Driver     Scale  -­‐  What  to  Measure     Scale  Defini#ons     Meter  -­‐  How  to   Measure   Performance   Reliability   Mean  #me  between  failure   over  #me  period   Time  Period:  current   month,  trailing  12  months   Monitoring  system   Performance   Responsiveness   Elapsed  #me  between  user   reques#ng  and  receiving   informa#on  for  percen#le  of   transac#ons   Time:  seconds  ;  Percen#le:   90th  ;  Transac#ons:   <Define  specific   transac#ons>   Performance  tes#ng  system   Profitability   Increase  Revenue   Total  revenue  over  #me  period  Time  Period:  current   month,  trailing  12  months   Financial  system   Delivery   Improve  Customer     Sa#sfac#on   Average  scores  pertaining  to   survey  ques#ons  for  current   customers…   Survey  Ques#ons:  <define   specific  ques#ons>  or  Net   Promoter  score   Survey  system   Quality   Reduce  Incident  Rate   Rate  of  new  incidents  over   #me  period   Time  Period:  current   month,  trailing  12  months   Report  from  issue  tracking   system   Example  
  • 24. A comparison of the activities PRODUCT   BACKLOG   Waterfall  Project   Agile  Project   REQUIREMENTS   Discrete  modules  of  somware     Integra#on  late     in  the  process   Each  sprint/itera#on  builds  on  the  earlier  sprints/itera#ons  
  • 25. Outcomes  are  the  desired  results  that  deliver  value.       E.g.  Increase  checkout  conversion  rates  by  19%       What do you want to improve? ©  Gabrielle  Benefield   28  
  • 26. Create  target  outcomes   Set  a  baseline     e.g.  Currently  only  11%  of  people   complete  checkout   Set  a  target     e.g.  30%  of  people  complete   checkout   Baseline:  11%   Target:  30%   Give  it  a  name  in  verb  +  noun  phrase  form   Increase  checkout  conversion  rates     29  
  • 27. Set  target  outcomes  for  the  ‘ili#es’   30   Scalability Must scale to 3 million users. Availability System must have a minimum of 99.9 uptime Usability Must be easy to use and fast to learn. Interoperability Must integrate with all customer systems.
  • 28. 31   Pay with a one step pay button Step   Cache page elements Step   Search for book Step   Decrease time to pay TO   Decrease time to load pages TO   Decrease time to find a book TO   Find one book and get it shipped Goal   Setup one step pay option Goal   Call and pay Step   Read details Step   Create  a  Goal  map   Start  with  the  Goal  and  work  back  to  find  the  least  amount  of  steps  needed  to  reach   the  goal   Enter payment details Step   Save payment details Step   Decrease the page load times Goal  Setup 1 server and split test Step  
  • 29. Which  op#ons  should  we  do  first?     We  can  guess,  or  we  can  test  with  an  experiment   ? 32   Decrease  page     load  #me   Setup local datacenter Option   Test caching elements on the payment page Experiment   Cache page elements Option   Setup one server test with 1% of traffic Experiment  
  • 30. Create  experiments   ? 33   Take  a  target  outcome   Design experiments to test ideas or de-risk your work Experiment  
  • 31. Outcomes Options In Progress Done Visually  track  outcomes     and  op#ons  during  delivery   ©  The007model.com   34   Decrease  page  load   #me   Setup local datacenter Option   Cache page elements Option   Decrease  #me  to     find  products   Redesign search results Option   Clickstream analysis Research     Datacenter  split   test  Australia   Research     Test  caching  one   page   Research     Split  test   redesign   Research     Trace  route   analysis   Research  
  • 32. How  much  progress     did  we  make?   Decrease  #me  to  find  products   Baseline:   7  minutes   Target:   <  3  minutes   Current:   5  minutes   Reduce  customer  complaints   Baseline:   250  calls  per  day   Target:   <  150  calls  per  day   Current:   190  calls  per  day   We  saved   2  minutes   We   reduced   calls  by  60   per  day   ©  The007model.com   35   Redesign checkout process Option  
  • 33. Snowden’s  cynefin  model   Cause  and  effect   rela#ons  repeatable,   perceivable  and   predictable   Cause  and  effect  are   only  coherent  in   retrospect  and  do   not  repeat     Cause  and  effect   separated  over  #me   and  space   CHAOS   SIMPLE   (Known)   COMPLEX   (Unknowable)     COMPLICATED   (Knowable)   No  cause  and  effect   rela#onships   perceivable  
  • 34. Outcomes  are  the  desired  results  that  deliver  value.       E.g.  Increase  checkout  conversion  rates  by  19%       What do you want to improve? ©  Gabrielle  Benefield   37  
  • 35. We  are  constantly  adap#ng   our  model  based  on   feedback.     When  we  see  the  value   diminish,  we  can  choose   other  op#ons  or  outcomes   to  explore   38   x Point  of  diminishing  returns   Time   Outcomes   (value)  
  • 36. READY and competence match When work allocated to sprint is READY, flow and stability is achieved 0,00 20,00 40,00 60,00 80,00 100,00 120,00 140,00 160,00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 Actual effort Ready NOT Ready 0,00% 10,00% 20,00% 30,00% 40,00% 50,00% 60,00% 70,00% 80,00% 90,00% 100,00% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 Flow Ready NOT  Ready Flow Actual effort spent Projects measures how ”READY” a sprint plan is Measures of key competencies  needed  in  project   Meeting READY criteria creates stability
  • 37. Cost of change Time Low High Plan Develop Design Test Launch The  point  where  we  have  the  least   informa#on,  the  contract  is  signed   locking  in  that  ignorance  for  the   whole  project.   When  the  customer  sees  what   has  been  built,  when  tes#ng   begins.       When  the  cost  of  change  and   fixing  defects  is  highest.