Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value

  • 3,155 views
Uploaded on

The 2.0 Adoption CouncilEnterprise 2.0 Black Belt Workshop: Measuring Success and Business Value by Ted Hopton & Donna Cuomo @ Enterprise 2.0 Conference Boston, June 2010

The 2.0 Adoption CouncilEnterprise 2.0 Black Belt Workshop: Measuring Success and Business Value by Ted Hopton & Donna Cuomo @ Enterprise 2.0 Conference Boston, June 2010

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,155
On Slideshare
3,151
From Embeds
4
Number of Embeds
1

Actions

Shares
Downloads
77
Comments
0
Likes
5

Embeds 4

http://www.slideshare.net 4

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Measuring Success and Business Value: Metrics and Analysis Ted Hopton, United Business Media Donna Cuomo, The MITRE Corporation © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
  • 2. #e2conf @Ted_Hopton @@Donnalc300 Who We Are Ted Hopton Wiki Community Manager United Business Media www.adventuresinsocialmedia.org @Ted_Hopton Donna Cuomo, PhD Chief Information Architect The MITRE Corporation dcuomo@mitre.org @Donnalc300 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 2
  • 3. #e2conf @Ted_Hopton @@Donnalc300 United Business Media  Who is UBM? – ~ 6000 employees world-wide across hundreds of offices plus telecommuters – B2B media:  “We serve specialist business communities with tradeshows and other live ‘in person’ events; data, marketing and information products; print products; and targeting, distribution and monitoring services.” – Organizationally stove-piped  More than a dozen divisions serving distinct markets  Each has a CEO and operates largely independently – Commonalities across divisions include  Exhibitions/conferences (BTW, we produce this event, E2.0!)  Print publications  Digital/Online Services – Launched internal Jive SBS community Sept. 2008 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 3
  • 4. #e2conf @Ted_Hopton @@Donnalc300 Objectives Our Community’s Objectives Are Simply Stated © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 4
  • 5. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Minimal Engagement Level: Logins and Contributors Contributors = Created content of some kind © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 5
  • 6. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Consuming Content: Page Views Per Member Views: Comparable over time as community grows © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 6
  • 7. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members, Consumers and Contributors © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 7
  • 8. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Page Views by Area © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 8
  • 9. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Page Views by Area, cont. © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 9
  • 10. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Page Views Per Employee Comparable view across divisions of different size © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 10
  • 11. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 11
  • 12. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 12
  • 13. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Where In the Community Is Activity Taking Place? © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 13
  • 14. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Who Is Creating and Viewing Content? Learn what the active members are doing so you can share success stories and ideas © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 14
  • 15. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members by Level of Activity Minimal = <6 activities Light = 6 - 20 activities Moderate = 21 - 50 activities Heavy = 51- 200 activities Super = >200 activities © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 15
  • 16. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members by Level of Activity Minimal = <6 activities Light = 6 - 20 activities Moderate = 21 - 50 activities Heavy = 51- 200 activities Super = >200 activities © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 16
  • 17. #e2conf @Ted_Hopton @@Donnalc300 Objectives But, Back to Our Objectives… © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 17
  • 18. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  Annual User Survey – How often they used it in specific ways  Several times/week  Weekly  Monthly  Occasionally  Never © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 18
  • 19. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  List of positive outcomes – Strongly Agree – Somewhat Agree – Somewhat Disagree – Strongly Disagree © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 19
  • 20. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  Why Don’t You Use It More? – Strongly Agree – Somewhat Agree – Somewhat Disagree – Strongly Disagree  Blunt, negative statements – Ask for it! – Let them tell you how they feel  OK, this is going to hurt a little – It’s a benchmark – Listen, learn, then improve © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 20
  • 21. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  The Bottom Line Question – How likely are you to recommend using it to a colleague? – Scale of 0-10, with zero least likely and 10 most likely  Net Promoter Score© (NPS) – Scores of 9-10 = Promoters (7-8 = Passive Positives) – Scores of 0-6 = Detractors – NPS = % Promoters - % Detractors  Positive NPS means more people promoting than detracting  Negative NPS means the opposite   A Truly Comparable Metric – Even very different communities can compare NPS © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 21
  • 22. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Wins  Success Stories: “Wins” – Invite people to write up and submit wins  Small, medium and large  All kinds: •cost-saving, revenue •“Soft” wins: collaboration, communication, efficiency, innovation… – Celebrate Wins – Share them as examples for others © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 22
  • 23. #e2conf @Ted_Hopton @@Donnalc300 Lessons Learned  Metrics will (and should) evolve – Technology evolves new measurement capabilities – Your skill with the technology will evolve better at pulling out data you need – Your understanding of your community & objectives will evolve with experience – Your community will evolve it’s alive, it will change, it will surprise you!  Numbers + Surveys + Stories – Collect all three – Look at the whole picture © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 23
  • 24. #e2conf @Ted_Hopton @@Donnalc300 Lessons Learned  There are no generally accepted benchmarks for Enterprise 2.0 communities – No one knows what results you *should* see for your community – Your Culture + Your Objectives = A Unique Community  Benchmark Against Yourself – Aim for progress and improvement over time – Compare different parts of your community with each other © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 24
  • 25. #e2conf @Ted_Hopton @@Donnalc300 The MITRE Corporation  MITRE is a private, independent, not-for-profit organization, chartered to work in the public interest  Founded in 1958 to provide engineering and technical services to Bedford, Mass. the U.S. Air Force  Currently manages 4 Federally Funded Research and Development Centers McLean, Va. – Department of Defense 7,000 employees worldwide – Federal Aviation Administration – Internal Revenue Service/Department of Veterans Affairs – Department of Homeland Security  Supports a broad and diverse set of sponsors within the U.S. government, as well as internationally © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 25 25
  • 26. #e2conf @Ted_Hopton @@Donnalc300 Use Case 1: Improve MITRE’s Research Program Selection Process  Deploy an innovation management tool as the host environment for the research competition  Enable codification of the research competition process  Collect all ideas in one place and record all participation (eliminate early “weeding out” of ideas)  Better support “teaming” of proposers from across the corporation with similar ideas (improve collaboration)  Encourage much broader participation in both proposing ideas and commenting on ideas  Improve user experience and satisfaction of participation (visibility, feedback, standard process, idea targeting)  Use an externally-hosted application (cloud service) © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 26
  • 27. #e2conf @Ted_Hopton @@Donnalc300 Idea Market FY10 Research Strategic Plan, powered by Sharepoint Wiki Idea Market, powered by Spigit as a cloud service Face-to-face “elevator proposals” © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 27 27
  • 28. #e2conf @Ted_Hopton @@Donnalc300 First Year Numbers (as of 20 May 2009)  2842 of 7278 (39%) MITRE employees clicked into Idea Market – 1445 Read-only Visitors – 597 Idea Owners (non-stakeholders, submitted ideas, commented, voted) – 575 Voters (only voted on ideas) – 192 Commenting Users (non-stakeholders, non-idea owners who provided written content) – 33 Members are Active Stakeholders (MIP Leadership who submitted an idea, review, comment, reply or vote)  840 Ideas submitted  750 Comments (threads) initiated  969 Replies to Comments  463 Reviews submitted  643 MIP Tech Support inquiries handle  5564 Votes cast Achieved transparency, breadth of participation © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 28
  • 29. #e2conf @Ted_Hopton @@Donnalc300 MIP User Satisfaction Summary: 2008 vs. 2009  Respondents reported considerable engagement with the process  Clear improvement in perceptions that the process is understandable and predictable  Clear improvement in perceptions of fairness/consistency in proposal evaluation  Still only a (sizable) minority of respondents who feel that they understand or have received adequate rationale for funding decisions  More participants (41%) agreed that the process improved their competition experience than disagreed (27%) Used survey technique to assess user experience © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 29
  • 30. #e2conf @Ted_Hopton @@Donnalc300 Use Case 2: Social Bookmarking (with Laurie Damianos)  Hypothesized social bookmarking would improve resource sharing, leveraging others research across teams and corporation – Some use of external tools like delicious already occurring  Could feed expertise finding  Could replace current “knowledge zones” – subject-based websites maintained by corporate stewards (consider corporate goals, not just end user goals)  Increase participation in knowledge sharing  Subscription to topic areas  More granular information management technique via tagging  Ease of ‘re-finding’ information or highlighting resources (recommended, lesson learned, etc) © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 30
  • 31. #e2conf @Ted_Hopton @@Donnalc300 Social Bookmarking RSS Tips Popular topics Corporately stewarded Bookmarked collections resources urc © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 31
  • 32. #e2conf @Ted_Hopton @@Donnalc300 onomi Pilot Usage (from 2007/2008) Average information provider has 56 bookmarks tagged with 5.4 terms. 154K tags 12000 page visits/mos 18K unique 27K bookmarks tags 17% internal bookmarks Supports need for internal social bookmarking service onomi bookmarks by tag my bookmarks by tag other bookmarks 83% external bookmarks add bookmark edit bookmark my bookmarks by tag We have no other way to share external add bookmark by pop-up resources browse users my tag collection delete bookmark bookmark page (URL) most popular activity: viewing search other bookmarks by tag help other people’s bookmarks poponomi onomi tag collection add bookmark via bookmarklet tag filters 14% of onomi’s visitors are delicious top 15 quick lookup (user, tag) edit my tag contributors other tag collections © Copyright 2010 2.0 Adoption 0% Council, MITRE Corporation, United Business 30% Ltd. All rights reserved. 5% 10% 15% 20% 25% Media, 32
  • 33. #e2conf @Ted_Hopton @@Donnalc300 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 33
  • 34. #e2conf @Ted_Hopton @@Donnalc300 MITRE-Babson SNA Study (Bill Donaldson, Donna Cuomo, Sal Parise, Bala Iyer) Size: tenure Study Findings • Its not just frequency of use that is important with these tools, but rather, who you are connecting to (uniqueness) • Brokerage, in both social networks and technology- mediated networks, has a positive impact on personal innovativeness • Both of the technology networks (ListServ and social bookmarking/ tagging) provide unique, significant value Center # Color • The social network & the 1 Lavender technology networks 2 Blue 5 Red complement each other 19 LightMagenta 20 Sepia 48 Melon © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 34
  • 35. #e2conf @Ted_Hopton @@Donnalc300 Measuring Business Value  Understand and explicitly document your goals for deploying a particular social tool. What benefits do you expect to achieve?  Think about what evaluation methodologies and objective metrics would help you understand if these desired behaviors are occurring  End users are not your only user group – Research analysts, commenters/reviewers, expertise finding, recommending, ….  Increasing the number of unique connections in a person’s social network has value  Cutter study notes it isn’t always possible to measure business value of a social tool from a prototype – Need critical mass, long tail effect of many one-to-one benefits, takes 2-3 years for behavior changes to occur, etc  Use realistic benchmarks of participation to assess the adoption success of your tool – Social tool use and patterns not always comparable to business tool use © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 35
  • 36. #e2conf @Ted_Hopton @@Donnalc300 Engage. Evangelize. Empower. www.20adoptioncouncil.com © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 36