Measuring Success and Business
                                   Value: Metrics and Analysis
                            ...
#e2conf @Ted_Hopton @@Donnalc300




 Who We Are
                                      Ted Hopton
                        ...
#e2conf @Ted_Hopton @@Donnalc300




 United Business Media

 Who is UBM?
        – ~ 6000 employees world-wide across hu...
#e2conf @Ted_Hopton @@Donnalc300




 Objectives

              Our Community’s Objectives Are Simply Stated




© Copyrig...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics
       Minimal Engagement Level: Logins and Contributors




  ...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics

      Consuming Content: Page Views




      Per Member Views...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics
                      Active Members, Consumers and Contributor...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics
             Page Views by Area




© Copyright 2010 2.0 Adopti...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics
               Page Views by Area, cont.




© Copyright 2010 2...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics
       Page Views Per Employee




       Comparable view acros...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics




© Copyright 2010 2.0 Adoption Council, MITRE Corporation, U...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics




© Copyright 2010 2.0 Adoption Council, MITRE Corporation, U...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics

                Where In the Community Is Activity Taking Plac...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics

                      Who Is Creating and Viewing Content?



...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics
            Active Members by Level of Activity




    Minimal...
#e2conf @Ted_Hopton @@Donnalc300




 Quantitative Metrics
            Active Members by Level of Activity




    Minimal...
#e2conf @Ted_Hopton @@Donnalc300




 Objectives

              But, Back to Our Objectives…




© Copyright 2010 2.0 Adop...
#e2conf @Ted_Hopton @@Donnalc300




 Qualitative Metrics: Survey
    Annual User Survey
          – How often they used ...
#e2conf @Ted_Hopton @@Donnalc300




 Qualitative Metrics: Survey
 List of positive outcomes
        – Strongly Agree
   ...
#e2conf @Ted_Hopton @@Donnalc300




 Qualitative Metrics: Survey
  Why Don’t You Use It More?
        – Strongly Agree
 ...
#e2conf @Ted_Hopton @@Donnalc300




 Qualitative Metrics: Survey

 The Bottom Line Question
        – How likely are you...
#e2conf @Ted_Hopton @@Donnalc300




 Qualitative Metrics: Wins

 Success Stories: “Wins”
        – Invite people to writ...
#e2conf @Ted_Hopton @@Donnalc300




 Lessons Learned

 Metrics will (and should) evolve
        – Technology evolves new...
#e2conf @Ted_Hopton @@Donnalc300




 Lessons Learned

 There are no generally accepted benchmarks for
  Enterprise 2.0 c...
#e2conf @Ted_Hopton @@Donnalc300




 The MITRE Corporation
   MITRE is a private, independent, not-for-profit organizati...
#e2conf @Ted_Hopton @@Donnalc300



  Use Case 1: Improve MITRE’s Research
  Program Selection Process
    Deploy an inno...
#e2conf @Ted_Hopton @@Donnalc300




   Idea Market




                                                                  ...
#e2conf @Ted_Hopton @@Donnalc300




  First Year Numbers (as of 20 May 2009)
   2842 of 7278 (39%) MITRE employees click...
#e2conf @Ted_Hopton @@Donnalc300



  MIP User Satisfaction
  Summary: 2008 vs. 2009
     Respondents reported considerab...
#e2conf @Ted_Hopton @@Donnalc300



 Use Case 2: Social Bookmarking
 (with Laurie Damianos)


     Hypothesized social bo...
#e2conf @Ted_Hopton @@Donnalc300




      Social Bookmarking
                                                            ...
#e2conf @Ted_Hopton @@Donnalc300




onomi Pilot Usage
(from 2007/2008)




          Average information provider has 56
...
#e2conf @Ted_Hopton @@Donnalc300




© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd....
#e2conf @Ted_Hopton @@Donnalc300


   MITRE-Babson SNA Study
   (Bill Donaldson, Donna Cuomo, Sal Parise, Bala Iyer)


   ...
#e2conf @Ted_Hopton @@Donnalc300




  Measuring Business Value
    Understand and explicitly document your goals for dep...
#e2conf @Ted_Hopton @@Donnalc300




                   Engage. Evangelize. Empower.
                                     ...
Upcoming SlideShare
Loading in …5
×

Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value

2,749 views

Published on

The 2.0 Adoption CouncilEnterprise 2.0 Black Belt Workshop: Measuring Success and Business Value by Ted Hopton & Donna Cuomo @ Enterprise 2.0 Conference Boston, June 2010

  • Be the first to comment

Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value

  1. 1. Measuring Success and Business Value: Metrics and Analysis Ted Hopton, United Business Media Donna Cuomo, The MITRE Corporation © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
  2. 2. #e2conf @Ted_Hopton @@Donnalc300 Who We Are Ted Hopton Wiki Community Manager United Business Media www.adventuresinsocialmedia.org @Ted_Hopton Donna Cuomo, PhD Chief Information Architect The MITRE Corporation dcuomo@mitre.org @Donnalc300 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 2
  3. 3. #e2conf @Ted_Hopton @@Donnalc300 United Business Media  Who is UBM? – ~ 6000 employees world-wide across hundreds of offices plus telecommuters – B2B media:  “We serve specialist business communities with tradeshows and other live ‘in person’ events; data, marketing and information products; print products; and targeting, distribution and monitoring services.” – Organizationally stove-piped  More than a dozen divisions serving distinct markets  Each has a CEO and operates largely independently – Commonalities across divisions include  Exhibitions/conferences (BTW, we produce this event, E2.0!)  Print publications  Digital/Online Services – Launched internal Jive SBS community Sept. 2008 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 3
  4. 4. #e2conf @Ted_Hopton @@Donnalc300 Objectives Our Community’s Objectives Are Simply Stated © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 4
  5. 5. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Minimal Engagement Level: Logins and Contributors Contributors = Created content of some kind © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 5
  6. 6. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Consuming Content: Page Views Per Member Views: Comparable over time as community grows © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 6
  7. 7. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members, Consumers and Contributors © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 7
  8. 8. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Page Views by Area © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 8
  9. 9. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Page Views by Area, cont. © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 9
  10. 10. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Page Views Per Employee Comparable view across divisions of different size © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 10
  11. 11. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 11
  12. 12. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 12
  13. 13. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Where In the Community Is Activity Taking Place? © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 13
  14. 14. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Who Is Creating and Viewing Content? Learn what the active members are doing so you can share success stories and ideas © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 14
  15. 15. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members by Level of Activity Minimal = <6 activities Light = 6 - 20 activities Moderate = 21 - 50 activities Heavy = 51- 200 activities Super = >200 activities © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 15
  16. 16. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members by Level of Activity Minimal = <6 activities Light = 6 - 20 activities Moderate = 21 - 50 activities Heavy = 51- 200 activities Super = >200 activities © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 16
  17. 17. #e2conf @Ted_Hopton @@Donnalc300 Objectives But, Back to Our Objectives… © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 17
  18. 18. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  Annual User Survey – How often they used it in specific ways  Several times/week  Weekly  Monthly  Occasionally  Never © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 18
  19. 19. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  List of positive outcomes – Strongly Agree – Somewhat Agree – Somewhat Disagree – Strongly Disagree © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 19
  20. 20. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  Why Don’t You Use It More? – Strongly Agree – Somewhat Agree – Somewhat Disagree – Strongly Disagree  Blunt, negative statements – Ask for it! – Let them tell you how they feel  OK, this is going to hurt a little – It’s a benchmark – Listen, learn, then improve © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 20
  21. 21. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  The Bottom Line Question – How likely are you to recommend using it to a colleague? – Scale of 0-10, with zero least likely and 10 most likely  Net Promoter Score© (NPS) – Scores of 9-10 = Promoters (7-8 = Passive Positives) – Scores of 0-6 = Detractors – NPS = % Promoters - % Detractors  Positive NPS means more people promoting than detracting  Negative NPS means the opposite   A Truly Comparable Metric – Even very different communities can compare NPS © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 21
  22. 22. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Wins  Success Stories: “Wins” – Invite people to write up and submit wins  Small, medium and large  All kinds: •cost-saving, revenue •“Soft” wins: collaboration, communication, efficiency, innovation… – Celebrate Wins – Share them as examples for others © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 22
  23. 23. #e2conf @Ted_Hopton @@Donnalc300 Lessons Learned  Metrics will (and should) evolve – Technology evolves new measurement capabilities – Your skill with the technology will evolve better at pulling out data you need – Your understanding of your community & objectives will evolve with experience – Your community will evolve it’s alive, it will change, it will surprise you!  Numbers + Surveys + Stories – Collect all three – Look at the whole picture © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 23
  24. 24. #e2conf @Ted_Hopton @@Donnalc300 Lessons Learned  There are no generally accepted benchmarks for Enterprise 2.0 communities – No one knows what results you *should* see for your community – Your Culture + Your Objectives = A Unique Community  Benchmark Against Yourself – Aim for progress and improvement over time – Compare different parts of your community with each other © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 24
  25. 25. #e2conf @Ted_Hopton @@Donnalc300 The MITRE Corporation  MITRE is a private, independent, not-for-profit organization, chartered to work in the public interest  Founded in 1958 to provide engineering and technical services to Bedford, Mass. the U.S. Air Force  Currently manages 4 Federally Funded Research and Development Centers McLean, Va. – Department of Defense 7,000 employees worldwide – Federal Aviation Administration – Internal Revenue Service/Department of Veterans Affairs – Department of Homeland Security  Supports a broad and diverse set of sponsors within the U.S. government, as well as internationally © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 25 25
  26. 26. #e2conf @Ted_Hopton @@Donnalc300 Use Case 1: Improve MITRE’s Research Program Selection Process  Deploy an innovation management tool as the host environment for the research competition  Enable codification of the research competition process  Collect all ideas in one place and record all participation (eliminate early “weeding out” of ideas)  Better support “teaming” of proposers from across the corporation with similar ideas (improve collaboration)  Encourage much broader participation in both proposing ideas and commenting on ideas  Improve user experience and satisfaction of participation (visibility, feedback, standard process, idea targeting)  Use an externally-hosted application (cloud service) © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 26
  27. 27. #e2conf @Ted_Hopton @@Donnalc300 Idea Market FY10 Research Strategic Plan, powered by Sharepoint Wiki Idea Market, powered by Spigit as a cloud service Face-to-face “elevator proposals” © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 27 27
  28. 28. #e2conf @Ted_Hopton @@Donnalc300 First Year Numbers (as of 20 May 2009)  2842 of 7278 (39%) MITRE employees clicked into Idea Market – 1445 Read-only Visitors – 597 Idea Owners (non-stakeholders, submitted ideas, commented, voted) – 575 Voters (only voted on ideas) – 192 Commenting Users (non-stakeholders, non-idea owners who provided written content) – 33 Members are Active Stakeholders (MIP Leadership who submitted an idea, review, comment, reply or vote)  840 Ideas submitted  750 Comments (threads) initiated  969 Replies to Comments  463 Reviews submitted  643 MIP Tech Support inquiries handle  5564 Votes cast Achieved transparency, breadth of participation © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 28
  29. 29. #e2conf @Ted_Hopton @@Donnalc300 MIP User Satisfaction Summary: 2008 vs. 2009  Respondents reported considerable engagement with the process  Clear improvement in perceptions that the process is understandable and predictable  Clear improvement in perceptions of fairness/consistency in proposal evaluation  Still only a (sizable) minority of respondents who feel that they understand or have received adequate rationale for funding decisions  More participants (41%) agreed that the process improved their competition experience than disagreed (27%) Used survey technique to assess user experience © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 29
  30. 30. #e2conf @Ted_Hopton @@Donnalc300 Use Case 2: Social Bookmarking (with Laurie Damianos)  Hypothesized social bookmarking would improve resource sharing, leveraging others research across teams and corporation – Some use of external tools like delicious already occurring  Could feed expertise finding  Could replace current “knowledge zones” – subject-based websites maintained by corporate stewards (consider corporate goals, not just end user goals)  Increase participation in knowledge sharing  Subscription to topic areas  More granular information management technique via tagging  Ease of ‘re-finding’ information or highlighting resources (recommended, lesson learned, etc) © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 30
  31. 31. #e2conf @Ted_Hopton @@Donnalc300 Social Bookmarking RSS Tips Popular topics Corporately stewarded Bookmarked collections resources urc © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 31
  32. 32. #e2conf @Ted_Hopton @@Donnalc300 onomi Pilot Usage (from 2007/2008) Average information provider has 56 bookmarks tagged with 5.4 terms. 154K tags 12000 page visits/mos 18K unique 27K bookmarks tags 17% internal bookmarks Supports need for internal social bookmarking service onomi bookmarks by tag my bookmarks by tag other bookmarks 83% external bookmarks add bookmark edit bookmark my bookmarks by tag We have no other way to share external add bookmark by pop-up resources browse users my tag collection delete bookmark bookmark page (URL) most popular activity: viewing search other bookmarks by tag help other people’s bookmarks poponomi onomi tag collection add bookmark via bookmarklet tag filters 14% of onomi’s visitors are delicious top 15 quick lookup (user, tag) edit my tag contributors other tag collections © Copyright 2010 2.0 Adoption 0% Council, MITRE Corporation, United Business 30% Ltd. All rights reserved. 5% 10% 15% 20% 25% Media, 32
  33. 33. #e2conf @Ted_Hopton @@Donnalc300 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 33
  34. 34. #e2conf @Ted_Hopton @@Donnalc300 MITRE-Babson SNA Study (Bill Donaldson, Donna Cuomo, Sal Parise, Bala Iyer) Size: tenure Study Findings • Its not just frequency of use that is important with these tools, but rather, who you are connecting to (uniqueness) • Brokerage, in both social networks and technology- mediated networks, has a positive impact on personal innovativeness • Both of the technology networks (ListServ and social bookmarking/ tagging) provide unique, significant value Center # Color • The social network & the 1 Lavender technology networks 2 Blue 5 Red complement each other 19 LightMagenta 20 Sepia 48 Melon © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 34
  35. 35. #e2conf @Ted_Hopton @@Donnalc300 Measuring Business Value  Understand and explicitly document your goals for deploying a particular social tool. What benefits do you expect to achieve?  Think about what evaluation methodologies and objective metrics would help you understand if these desired behaviors are occurring  End users are not your only user group – Research analysts, commenters/reviewers, expertise finding, recommending, ….  Increasing the number of unique connections in a person’s social network has value  Cutter study notes it isn’t always possible to measure business value of a social tool from a prototype – Need critical mass, long tail effect of many one-to-one benefits, takes 2-3 years for behavior changes to occur, etc  Use realistic benchmarks of participation to assess the adoption success of your tool – Social tool use and patterns not always comparable to business tool use © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 35
  36. 36. #e2conf @Ted_Hopton @@Donnalc300 Engage. Evangelize. Empower. www.20adoptioncouncil.com © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 36

×