Online Communities: Metrics and Reporting 2009


Published on

A research project I conducted while running the Online Community Research Network at Forum One. Interesting to look back at state of the art 4 years ago and also interesting to see what issues we / the platform providers are still struggling with.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Online Communities: Metrics and Reporting 2009

  1. 1.   1  Online Communities: Metrics and Reporting Current Practice Research / August 2009 RESEARCH COORDINATION: FORUM ONE NETWORKS SAN FRANCISCO / WASHINGTON D.C.
  2. 2.  RESEARCH SPONSORS HiveLive HiveLives on-demand software enables companies to rapidly deploy custom-fit enterprise communities built around people, their relationships, and their shared knowledge.   Lithium   Lithium Technologies provides market-leading solutions for deploying and managing successful customer communities, including proven forums, blogs, and ideas. Online Community Research Network A research-based network of online community professionals   working to understand key community building issues and best practices.Online Communities: Metrics and Reporting 2009 2
  3. 3.  EXECUTIVE SUMMARYThe Online Communities: Metrics and Reporting research study was initiated in late July 2009, andran until the second week of August 2009. The intention of the study was to get a broad look at whatonline community metrics organizations are tracking, how organizations determine and report on theongoing value of their online community initiatives, and the reporting and metrics tools that helpcompanies in their efforts.We received approximately 175 responses to our survey. Participants represent a healthy swath of thetypes of organizations participating in online community culture. Participating industry categoriesinclude: software companies, hardware companies, consumer goods firms, non-profit organizations,independent consultants and media companies, among others.A sample of the 175 organizations that participated include (with their permission):Autodesk, BusinessWeek, Electronic Arts, EMC, Environmental Defense Fund, Google, Great Schools,IBM, Juniper Networks, Microsoft, Quest Software, Sony Computer Entertainment, Sun Microsystems.Viacom, and VMware.Overall, we were very encouraged by the level and quality of response to the research questions.Several key issues pertaining to online community and social media metrics surfaced during this report,including:  In general, organizations need to do a more thorough job of defining their business objectives for online community engagement, assessing ways to measure progress towards these objectives, reaching beyond their native platform metrics capabilities, and finding ways to measure the more qualitative components of community member engagement.  The Role of the Community Manager is increasingly important to developing and refining business process, and measuring performance in these new “social spaces.”  There is a growing need for community metric standards that are platform and vendor- independent.Online Communities: Metrics and Reporting 2009 3
  4. 4.  TABLE OF CONTENTSKey Insights: ...........................................................................................................................................501 Organization Type..............................................................................................................................802 Platform ..............................................................................................................................................903 Community Age ...............................................................................................................................1004 Total Employees ..............................................................................................................................1105 Annual Budget .................................................................................................................................1206 Clarity of Expectations ....................................................................................................................1307 Purpose of Community ...................................................................................................................1408 Alignment Metrics and Business Goals ........................................................................................2009 Responsibility ..................................................................................................................................2110 Determining What to Measure ........................................................................................................2211 Technology for Tracking .................................................................................................................2312 Metrics Currently Being Tracked....................................................................................................2513 Determining ROI...............................................................................................................................3014 Top 3 Key Performance Indicators.................................................................................................3415 Satisfaction ......................................................................................................................................3916 Additional Metrics............................................................................................................................4117 Additional Advice.............................................................................................................................45Final Comments ....................................................................................................................................51Online Communities: Metrics and Reporting 2009 4
  5. 5.  Key Insights:20% of the respondents have a custom-built community platform. The three most common 9primary community platforms are:  Web: Facebook, Twitter, Blog  Clearspace Forums – Social Business Software  Lithium – Social CRMOver a third of the participants belong to an online community that is younger than one year 10(38%), while over a fifth of the respondents belong to a community that has existed for morethan 5 years (21%).Over half of the respondents indicated that their company employs less than 500 employees 11(62%), the majority of those companies employ less than 50 employees (39%).Over half of the respondents (51%) indicated that their online community platform annual 12budget is less than $50,000 and (10%) indicated it is over $500,000.Just under half of the respondents (45%) felt that their management team has loosely 13defined the expectations regarding ROI and (38%) indicated that expectations areclearly defined.The three most common primary purposes of online communities are: 14  Promote Services / Products  Collaborate / Gather Information  Specialty Group NetworkAlmost half of the respondents communicated that their community’s metrics goals were 20created in support of the organization’s business goals (47%).(31%) indicated that they had been created independently but were helping to redefineexisting business goalsThe three most common roles that are responsible for setting metrics and reporting goals are: 21  Senior Management  Community Manager  Marketing Team61% of the participants said that they determine what is important to measure and report by 22working from a strategy based on business goals and finding solutions to help measure whatis needed.Online Communities: Metrics and Reporting 2009 5
  6. 6.  73% of respondents indicated that they either used the technology that their platform 23provided for tracking and reporting or developed in-house measurement solutions.Google Analytics (40%) and Omniture (26%) were the two most common supportingtechnology tools that online communities use for tracking and reporting.The top 3 items that online communities currently measure for tracking and reporting are the 25same for non-profit and profit related organizations, and are as follows:  Unique Visitors  New Member Registrations  Page ViewsThe top 3 items that online communities don’t measure, but want to are:  Member Satisfaction  Influencer / Evangelism  Member Life CycleThe five most mentioned “other” metrics that respondents tracked within their online 30communities are:  Time On Site  Posts by Type  Member Demographics  User Types  User EngagementAlmost a third of respondents (34%) indicated that User Activity / Engagement is the most 31important item for their online community to track to determine ROI. This was considered thenumber one item for both profit and non-profit organizationsThe top 3 key performance indicators in the reports respondents sent to management are: 35  User Activity / Engagement  Membership Count [New Registrations, Active]  Number of Posts / CommentsOver half of the participants felt that reports are either just satisfactory or that they need 40improvement.The most common metrics that stakeholders would like to see, that are not currently reported, 42are:  Activity Levels / Engagement  ROI: Revenue / Sales (Up sells / Renewals)  User Conversion Metrics (Social Network Analysis)Online Communities: Metrics and Reporting 2009 6
  7. 7.  The three most common pieces of advice that respondents wanted to offer to peers regarding 46metrics and reporting are:  Measure Metrics that Supports Community Goals  Define Key Business Objectives First  Monitor Quality of Interactions, not NumbersOnline Communities: Metrics and Reporting 2009 7
  8. 8.  01 Organization TypeDerived from Question 5: In which industry or market segment does your company dobusiness?SUMMARY:Respondents represent a healthy sampling of the types of industries that participate in onlinecommunity culture:  18% (31) work in Software  14% (24) work for a Non-Profit  11% (18) work in Online Community/ Social Media Destination  9% (16) work in Media Company  6% (10) work in Online Retail  6% (10) work for an Agency  6% (10) work as a Consultant  6% (10) work in Education  4% (7) work for a Technology Manufacturer  3% (6) work for a Community Platform Provider  2% (4) work for the Government  25% (25) Other“Other” organizations include; Research Organizations, Retail Organizations and FinancialInstitutions among others.Online Communities: Metrics and Reporting 2009 8
  9. 9.  02 PlatformFrom Question 6: What is your PRIMARY community platform? (Vendor, Product andVersion, please)SUMMARY:20% (30) of the respondents said that their company had a custom built community platform. A largepercentage of other respondents, 16% (24), indicated that they used web-based social media platformssuch as Facebook and Twitter as their community platform.Other popular community platforms / software are as follows:  9% (14) Clearspace Forums  8% (12) Lithium  6% (9) Telligent Enterprise  5% (8) Sharepoint  5% (8) Drupal  5% (7) vBulletin  5% (7) NingOnline Communities: Metrics and Reporting 2009 9
  10. 10.  03 Community AgeFrom Question 7: How long has your online community existed?SUMMARY:  13% (22) 0 – 3 Months  7% (12) 4 – 6 Months  18% (31) 7 – 12 Months  10% (17) 13 – 18 Months  10% (18) 19 – 24 Months  13% (23) 3 Years  5% (9) 4 Years  3% (6) 5 Years  21% (36) More than 5 YearsTAKEAWAY:More than one third of respondents, 38% (65), have a community that is younger than one year, while21% (36) have a community that is at least 5 years old.Online Communities: Metrics and Reporting 2009 10
  11. 11.  04 Total EmployeesFrom Question 8: How many people does your company employ in total?SUMMARY:  39% (68) Less than 50  23% (40) 50 – 499  8% (15) 500 – 999  13% (23) 1,000 – 4,999  4% (7) 5,000 – 9,999  13% (22) 10,000 or moreTAKEAWAY:Over half of the respondents, 62% (108), indicated that their company employs less than 500 persons,while 39% (68) of those belong to a company that employs less than 50.  Online Communities: Metrics and Reporting 2009 11
  12. 12.  05 Annual BudgetFrom Question 9: What is the annual budget for your online community platform andservices?SUMMARY:  33% (40) Less than $50,000  18% (22) $10,000 – 49,000  11% (14) $50,000 – 99,000  28% (34) $100,000 – 500,000  6% (8) $500,000 – 999,000  2% (3) $1 million - $4,999,000  2% (2) Over 5 millionTAKEAWAY:Of the 123 respondents that selected an annual budget, over half (51%), indicated that their onlinecommunity platform annual budget is less than $50,000 and (10%) indicated it is over $500,000.52 respondents chose not to disclose their annual community platform budget.Online Communities: Metrics and Reporting 2009 12
  13. 13.  06 Clarity of ExpectationsFrom Question 10: How clearly has your management team defined their expectationsabout value and return on investment (ROI) for your community?SUMMARY:  10% (18) Very clearly defined value and targets  28% (48) Important milestones and benchmarks are defined  45% (78) Value is loosely defined  7% (12) Community team decides without input from management  10% (17) We do not have value and ROITAKEAWAY:Most participants felt that their management team had some definition of expectations about ROI,although just under half, (45%), expressed that the value was loosely defined. As noted in latersections, the concept of value is made up of quantitative and qualitative measures, and is often not ahard dollar measurement of return. It is encouraging to see that most participants have at least a looseagreement about community value and ROI. Lack of agreement between management and staff oncommunity value is one of the biggest risks in the sustained development and support of communityprograms.Online Communities: Metrics and Reporting 2009 13
  14. 14.  07 Purpose of CommunityFrom Question 11: What is the primary purpose of your community?SUMMARY:Most of the respondents indicated the primary purpose of their online community was to share andgather information, support clients / customers, or to promote their company’s services / products. Thefollowing list encompasses the top five primary purposes for online communities:  24% (38) Promote Services / Products  24% (37) Collaborate / Gather Information  20% (32) Specialty Group Network  17% (26) Share Information  15% (24) Customer Support / SatisfactionTAKEAWAY:24% (38) of the respondents felt that a primary purpose of their online community was to promoteservices / products, including: driving leads, building brand advocacy and product evangelism, andencouraging product / service sales and renewals. 7% (15) of the respondents ranked productawareness and support as another related primary purpose of some communities.Other respondents indicated that the primary purpose of their community was to collaborate / gatherinformation, 24% (37), or simply to share information, 17% (26). Some respondents used their onlinecommunity for team members, development or consumers to share and gather information, includingsharing information on brands, ideas, issues and trends.Some participant’s online communities had the primary purpose of bringing people together who belongto specialty groups, 20% (32), (minority groups, mothers, game players, travel enthusiasts) or thosethat support a common cause 2% (5) (spiritual, cultural, global concerns).Customer support / satisfaction, 15% (24), was a common primary purpose of some onlinecommunities.Providing a platform for clients to network, share best practices and provide product support to oneother ranked at 8% (16).Online Communities: Metrics and Reporting 2009 14
  15. 15.   WRITE-IN RESPONSES:Below are just a few of the responses from participants that describe the primary purpose of their onlinecommunity:“Promote Service / Product” related responses include: Drive leads and some market research. To breed product advocates and evangelists by offering direct contact, support, and conversation with the company. We run several communities on behalf of our clients – all share one primary purpose, which is to turn loyal fans into brand advocates. Developer community outreach and evangelizing. Generating buzz about our work, moving the one-time visitor to become online advocate for our org. Increase engagement among Enterprise customers – increase renewals, up-sell, and cross-sell.Online Communities: Metrics and Reporting 2009 15
  16. 16.   To communicate and promote product updates. Generate revenue and retain customers. Provide community based support and to build brand advocacy. Build the brand and spread our work externally. Drive conversion. Build visibility & generate dialogue.“Collaborate / Gather Information” related responses include: Collaborate on a new way to measure and manage sales and marketing by measuring and managing the relationship tied to profit. Knowledge Management, Quality Management, Collaboration. Knowledge sharing, peer-to-peer support, collaboration on content in wiki, learn from bloggers. To assist developers with the platform, and to promote collaborative development. Discuss emerging trends, technologies and business issues. Collaboration and open dialogue between team members. Team and subject-matter collaboration. To gather ongoing in-depth consumer insights. Sharing & gathering news and information on our focus areas, tracking the conversation about our brand and issues. Internal collaboration community for our Staff and Contractors. Share findings and best practices, ask questions, have conversations about industry trends. Share information between employees, surface new business ideas, replace old corporate intranet. Online Communities: Metrics and Reporting 2009 16
  17. 17.  “Specialty Group Network” related responses include: To network with others who focus their time and energy on sustainable businesses (renewable energy, green building, organic food, etc). To provide a means for our members and subscribers to receive advice, support and feedback from each other relative to their online and offline dating and romantic relationships. Support the growth and development of knowledge focused networks of professionals and others interested in all the fields related to science, engineering and technology in order to boost our membership. Educational community. Create collaborate space for international health researchers in the fields of mental health and education. An online community of teachers to bridge their isolation and develop 21st century skills, improve practice, and increase professional engagement. “Share Information” related responses include: In places with poor connectivity, we connect health workers with each other and to relevant clinical and public health information using technology. Engage members to share experiences, which will lead to better choices by other members. Provide members with information they need to be successful in their work. Knowledge sharing and practice development in local government. Educate, entertain, and interact with women and people interested in marketing to women. Enable peers to leverage one another and to share best practices. “Customer Support / Satisfaction” related responses include: To build customer loyalty by providing our existing customers with a direct communication channel to our development teams to ask questions, take part in our betas, and have an influence on future product direction.Online Communities: Metrics and Reporting 2009 17
  18. 18.   Our primary goal is facilitate the awareness and understanding of our clients and customers. To increase our authority in our subject area and give our clients confidence in our ability and our knowledge. The platform allows brands to include the customer voice in the decision making process. 360 degree Client Satisfaction. Internally for increased collaboration and communication regarding products, troubleshooting, engineering, etc. Our forum is for customer support, tutorials, etc for increased customer care. “Platform for Clients to Network” related responses include: Enable customer-to-customer discussions regarding issues and solutions. Increase open communication amongst global customers in the marine industry. Enabling customers to help each other, reducing load on support and increasing sense of community. Peer support, customer co-collaboration, and generally, help customers get more engaged and use the solution better. Allow customers to share best practices and network. Give customers a place to talk with other customers... connect, ask questions, share best practices. Provide an environment for product users to communicate with other and help on technical problems. “Product Awareness / Support” related responses include: Product support, brand awareness. Help customers complete and file their income tax returns; and help customers use their Quicken products. Moving toward discussions on personal finance. Provide an environment for product users to communicate with other and help on technical problems.Online Communities: Metrics and Reporting 2009 18
  19. 19.   To increase product utilization. To provide a place for our customers and partners to learn about and discuss our product, ask questions and get answers.   Online Communities: Metrics and Reporting 2009 19
  20. 20.   08 Alignment Metrics and Business GoalsFrom Question 12: Were your community’s metrics created in support of yourorganization’s broader business goals or were they created independent of a corporatebusiness?SUMMARY:  47% (81) Created to support existing business goals  31% (53) Created independently but helping redefine existing business goals  5% (9) Not in support of existing business goals at all  7% (12) No corporate business goals yet, metrics goals come first  10% (18) No corporate business goals or metrics goals yetTAKEAWAY:Almost half of the respondents, 47% (81), indicated that their community’s metrics goals were createdin support of the organization’s business goals. 31% (53) indicated that they had been createdindependently but were helping to redefine existing business goals. It is worth highlighting the relativelyhigh percentage of respondents, 31% (53), whose community programs are having a transformativeeffect on the organization.Online Communities: Metrics and Reporting 2009 20
  21. 21.  09 ResponsibilityFrom Question 13: Who is responsible for setting metrics and reporting goals? Checkall that apply.SUMMARY:  22% (88) Senior Management  20% (80) Community Manager  16% (63) Marketing Team  14% (54) CEO / President  9% (36) Product Team  5% (21) Development Team  5% (18) Research / Data  2% (7) Sales  1% (4) System / Network administrator  6% (24) Other“Other” employees responsible for setting metrics and reporting goals include: Program Manager,COO, Business Analyst, Social Media Strategist, User Groups, Product Marketing Team, Web Team,Board of Directors, and Internal Communications.TAKEAWAY:Respondents stated that Senior Management was primarily responsible for setting community metricsand reporting goals, followed closely by Community Managers and Marketing teams. This data speaksto the increasingly important role the Community Manager (and team) play in the organization.Online Communities: Metrics and Reporting 2009 21
  22. 22.  10 Determining What to MeasureFrom Question 14: How does your organization determine what is important to measureand report?SUMMARY:  20% (34) We stick with what the platform can provide  61% (100) We work from a strategy based on business goals and find solution to help us measure what we need  19% (31) We try to measure everything, will develop more of a strategy laterTAKEAWAY:Respondents are primarily shaping metrics strategies based on business goals, (61%), even if theirplatform doesn’t support gathering or tracking the desired metrics. Platform metrics are generallyspeaking, not comprehensive or extensible enough to create a meaningful dashboard to see overallcommunity health, get an accurate visualization of the community’s social graph, or to understand theongoing insight created by the sentiments of the community population. The risk in relying only on datathat a platform can provide, as (20%) of the respondents indicated, is that the data sets aren’tcomprehensive or contextual to the organizations’ needs. Measuring everything, as (19%) ofrespondents indicated, would likely overwhelm the community team and stakeholders, and is unlikely toyield meaningful performance data or insight without some rigor in the analysis.Online Communities: Metrics and Reporting 2009 22
  23. 23.  11 Technology for TrackingFrom Question 15: What type of technology do you use for tracking and reporting?Please provide detail of the solution.SUMMARY:  42% (69) We use what the platform provides  31% (51) We have developed in-house measurement solutions  19% (31) We use outside vendors for measurement  8% (14) We have purchased softwareOnline Communities: Metrics and Reporting 2009 23
  24. 24.  NON PLATFORM TECHNOLOGY:The most common technologies that respondents used for tracking and reporting in their onlinecommunities (apart from reporting available from their software platform) are as follows:  40% (28) Google Analytics  26% (18) Omniture  4% (3) Social Media Tools: Twitter, Facebook  3% (2) Business Objects  3% (2) SQL Databases  24% (17) Other toolsTAKEAWAY:The most common supporting technology tools that respondents use for tracking and reporting in theironline communities are Google Analytics (40%) and Omniture (26%). Some respondents use GoogleAnalytics and Omniture exclusively, while others use them in addition to in-house solutions, outsidevendors, or in addition to the reporting available from their software platform.       Online Communities: Metrics and Reporting 2009 24
  25. 25.  12 Metrics Currently Being TrackedFrom Question 16: What do you currently measure?SUMMARY:The top five items that online communities measure for tracking and reporting are as follows:  152 Responses - Unique Visitors  150 Responses - New Member Registrations  143 Responses - Page Views  126 Responses - Visitors  116 Responses - Message PostsThe top five items that online communities don’t measure, but want to, are as follows:  90 Responses - Member Satisfaction  90 Responses - Influencer / Evangelism  84 Responses - Member Life Cycle  83 Responses - Member Loyalty  73 Responses - Referrals to CommunityTAKEAWAY:The top five items that online communities currently measure for tracking and reporting are the samefor both profit and non-profit organizations and include Unique Visitors, New Member Registrations,Page Views, Visitors and Message Posts.Non-profit organizations concentrate on measuring Podcasts & Video Links and MemberSatisfaction, more often than other organizations. Commercial organizations place more attention onmeasuring Retention / Attrition, Member Loyalty, Member Blog Posts and Conversion than non-profit organizations.As organization’s community strategies mature, the trend to primarily report on basic web metrics (pageviews, registrations) will be replaced by metrics that speak to the health of the community, the strengthof members’ networks, the quality and type of member participation, and more robust measurements ofmember engagement. The data suggest that we are on the cusp of the evolution from “basic”community metrics to more robust and contextual reporting.CURRENTLY MEASURING – PROFIT AND NON PROFITNote: Graph is detailed on the next page. A large .PDF of the graph can be found at: Communities: Metrics and Reporting 2009 25
  26. 26.   TOP 20:Online Communities: Metrics and Reporting 2009 26
  28. 28.  TOP 20: CURRENTLY MEASURING – FOR PROFITTOP 20: CURRENTLY MEASURING - NON PROFITOnline Communities: Metrics and Reporting 2009 28
  29. 29.  TOP 20: WANT TO MEASURE – PROFIT AND NON PROFITTOP 20: DON’T MEASURE – PROFIT AND NON PROFITOnline Communities: Metrics and Reporting 2009 29
  30. 30.  13 Determining ROIFrom Question 18: Of the items you track, what do you consider the most important fordetermining ROI?SUMMARY:  34% (49) User Activity / Engagement  13 Number of Page Views / Clicks  11 General Participation / Engagement  10 Number of Unique Visits  9 Number of Site Visits  7 Number of Active Users / Members  3 Number of Threads  3 Number of Returning Visitors  2 Amount of Time Users Spend On Site  10% (15) Sales Revenue – Renewals, Up Sells, Cross Sells…  9% (13) Number of Evangelism / Referrals  8% (12) Number of Posts / Comments  8% (11) New Member Registrations  7% (10) Member Satisfaction / Loyalty  6% (8) Number of Conversions  5% (7) Member-to-Member support  5% (7) Retention  4% (6) Reduction of Technical Support Calls  3% (5) Number of Downloads  1% (2) Fundraising GeneratedTAKEAWAY:Over a third, (34%), of respondents indicated that User Activity / Engagement was the most importantitem for their online community to track to determine ROI, this included Number of Page Views /Clicks, General Participation and Number of Unique Visits.Other participants felt that Sales Revenue, (10%), including renewals, up sells and cross sells, as wellas Number of Conversions, (6%), and New Member Registrations, (8%), were the most importantitems to track.Member-to-Member support, (5%), and the resulting Reduction of Technical Support Calls, (4%)were considered the most important factors for determining ROI for other participants.User Activity / Engagement, (34%), was the number one item to track for both profit and non-profitorganizations, although within this category, profit based organizations had a significantly higherpercentage response rate for tracking Number of Page Views, whereas non-profit organizations had asignificantly higher percentage response rate for tracking Number of Active Users / Members.Another common response from profit based organizations was related to tracking Sales Revenue,while non-profit organizations had a higher percentage response rate for tracking New MemberRegistrations and Member Satisfaction / Loyalty.Online Communities: Metrics and Reporting 2009 30
  31. 31.  USER ACTIVITY / ENGAGEMENT 34% (BROKEN OUT):  Online Communities: Metrics and Reporting 2009 31
  33. 33.  USER ACTIVITY / ENGAGEMENT – PROFITUSER ACTIVITY / ENGAGEMENT – NON-PROFITOnline Communities: Metrics and Reporting 2009 33
  34. 34.  14 Top 3 Key Performance IndicatorsFrom Question 19: What are the 3 most important community key performanceindicators in the reports you send to management?SUMMARY:  32% (74) User Activity / Engagement  21% (49) Membership Count (New Registrations, Active)  18% (42) Number of Posts / Comments  5% (12) Member Satisfaction / Loyalty  4% (10) Number of Questions Answered  4% (10) Sales Revenue - Up Sell, Cross Sell, Renewals  4% (8) Leeds / Referrals Generated  3% (6) Number of Downloads  3% (6) Number of Influencers / Evangelists  2% (5) Visitor Retention  2% (5) Number of Conversions  1% (2) Donations Received  1% (2) Visitor Geographic DispersalTAKEAWAY:Almost a third (32%) of respondents indicated that User Activity / Engagement is one of the mostimportant key performance indicators in the reports that they sent to their management. The followingthree key performance indicators were the most commonly reported in the User Activity /Engagement category:  33 – Number of Page Views / Clicks  22 – Number of Site Visits  19 – Number of Unique VisitsThe other two key performance indicators that many respondents input into management reports areMembership Count, (21%), (including new membership and total membership count) and the Numberof Posts / Comments, (18%), received on their site.Both profit and non-profit organizations list User Activity / Engagement as the number one item totrack. Within this category the specific key performance indicators were dispersed similarly, with theprofit based organizations having a slightly higher percentage ratio on key performances such as theNumber of Threads reported and General Participation. Non-profit organizations, on the other hand,have a slightly higher percentage ratio on reporting metrics such as Number of Returning Visitorsand the Number of Site Visits.Another common response from profit based organizations was related to reporting key performancessuch as Sales Revenue and the Number of Conversions, while non-profit organizations had a higherpercentage response rate for reporting the amount of Donations Received.Online Communities: Metrics and Reporting 2009 34
  35. 35.  USER ACTIVITY / ENGAGEMENT 34% (BROKEN OUT):Online Communities: Metrics and Reporting 2009 35
  36. 36.  USER ACTIVITY / ENGAGEMENT – PROFITUSER ACTIVITY / ENGAGEMENT – NON-PROFITOnline Communities: Metrics and Reporting 2009 36
  37. 37.  USER ACTIVITY / ENGAGEMENT – PROFITUSER ACTIVITY / ENGAGEMENT – NON PROFITOnline Communities: Metrics and Reporting 2009 37
  38. 38.  PROFIT & NON-PROFIT COMPARISON CHARTOnline Communities: Metrics and Reporting 2009 38
  39. 39.  15 SatisfactionFrom Question 20: How satisfied are executive stakeholders with current metricsreports?SUMMARY:  22% (37) Current reports are excellent  27% (45) Reports are just satisfactory  29% (47) Reports need improvement  9% (15) Reports are extremely poor  13% (21) We don’t currently track and report metricsTAKEAWAY:Over half of the participants, (58%), felt that reports are either just satisfactory or that they needimprovement. 13% of the respondents indicated that they do not currently track and report metrics,although several commented that while they do not track metrics for management purposes, they trackmetrics for their own use. It is a relatively straight line from unclear management expectations (Section06) through poor alignment of management and community goals (Section 08) to get to low stakeholdersatisfaction with the metrics reports being produced.Online Communities: Metrics and Reporting 2009 39
  40. 40.  WRITE-IN RESPONSES:Below are key write-in responses from participants, outlining the level of satisfaction that executivestakeholders have with their current metrics reports listed below:User responses include: Unfortunately the executive stakeholders see curves of large numbers all up and to the right, and have been satisfied. Only now are they beginning to ask what the real business value is. It is really the team that is responsible for trying to unify and coordinate cross-divisional strategy that is frustrated by a lack of focus on KPIs. We have many measurements but no real analysis of the data. I am working with one system that will measure a number of factors regularly that I and others can analyze. They dont really know what they need, they rely on us to tell them. We always wish we could glean more understanding from the activity and content, but find number crunching takes us away from that goal. Show greater correlation to sales and ROI than traditional media channels. We have received excellent data from member surveys. We get feedback periodically, but our reports are used mainly to confirm were on track – management isnt really that engaged. Metric reports are NEVER good enough. Stakeholders want real-time, dynamic metrics delivered at will. Our systems cannot provide that at this point. The value we see on our level is excellent, executive stakeholders are accustomed to seeing other data (traditional marketing values which more clearly demonstrate ROI than community engagement) which makes it more difficult for that value to be relayed properly in a way that makes sense for executive stakeholders. We have not yet reported anything other than # members and # discussions. The community is very new. We do not send metrics reports to executives, but we do track them for our own internal use.Online Communities: Metrics and Reporting 2009 40
  41. 41.  16 Additional MetricsFrom Question 21: What metrics would stakeholders like to see that are not currentlyreported?SUMMARY:  31% (18) Activity Levels / Engagement  17% (10) ROI: Revenue / Sales (Up sells / Renewals)  16% (9) User Conversion Metrics (Social Network Analysis)  12% (7) Content Evaluation / Tracking  7% (4) Members directed from Social Media Tools  7% (4) Qualitative Engagement Measuring  5% (3) Retention  5% (3) ROI: Reduces CostsTAKEAWAY:Almost a third (31%) of the participants indicated that their stakeholders would like to see metrics thatare related to Activity Levels and User Engagement. The engagement category consisted ofresponses such as the number of active members, number of site visits, the number of pageviews, time spent on site and content views. In addition to these behavioral metrics, a properengagement metric would also include attitudinal data that illustrates member satisfaction, affinity, andlikelihood to recommend.Return on Investment was another metric that stakeholders wanted to see in their reports, includingSales, renewals and up sells, (17%), and how the online community affects Reduction in costs,(5%).Another common response from participants was the need for stakeholders to see data related to UserConversions, (16%), in their reports, including the number of conversions and how, when, and wherethe conversions are taking place.Other metrics that stakeholders felt that they would like to start seeing in their reports, is ContentEvaluation, (12%), which includes quality and usefulness of the content posted and a record of keydistributors.A small percentage of participants indicated that they would like to see a record of Members Directedfrom Social Media Tools, (7%), Qualitative Engagement Measuring, (7%), and Retention, (5%).Online Communities: Metrics and Reporting 2009 41
  42. 42.      WRITE-IN RESPONSES:Below are key write-in responses from participants outlining the metrics that stakeholders would like tosee that are not currently being reported:“Activity Levels / Engagement” related responses include: Active members, customer journeys. Life cycle of the user. And the seasonal visiting, as we face a problem in identifying our growth, we have seasonal visitors, so we need to compare our growth with a similar season not just the previous month. Unfortunately they are not yet aware of the power of the community, thus they have not asked for so many.... But, account managers and consultants want to see the activity of their customers. We would like to see a Time Spent on Site metric. Page views/member vs. non-member would be very useful. Number of visits before initiating registration would be useful. Specific content views. We track this, but our content is not tagged and it is difficult to organize forOnline Communities: Metrics and Reporting 2009 42
  43. 43.   reporting. Time spent in the community (average per user, min, max, etc.).“ROI Revenue / Sales” related responses include: Attrition, cross-sell / up-sell, member lifecycle (all the real ROI stats!). What percentage of the users are product owners? What level of purchase have they made? Theyd like to see more hard evidence of how the community reduces costs or increases revenues. As we all know, communities tend to have a indirect impact on those metrics, but we need more case studies and metrics that can help us communicate the value of communities to our exec chain. Correlation with sales - renewals and up-sells. ROI to include collateral benefits, i.e., a question may be answered and clearly benefit the original poster, but subsequent viewers will also benefit, and we can only infer that impact.“User Conversion Metrics” related responses include: Social graph related to conversion and retention. Where the conversations are taking place - i.e. Social Network Analysis. Lead conversion. More accurate monthly conversion rates, customer lifecycle. Comprehensive social media tracking.“Content Evaluation / Tracking” related responses include: What are the characteristics of social media content that lead to new customers. Objective "quality" of content produced. Overall how useful is the content to our consumers. Analytics on how information based on relations flow, who are the information distributors? Who areOnline Communities: Metrics and Reporting 2009 43
  44. 44.   the bottlenecks?“Other” related responses include: The number of members from other social media tools such as Twitter, Facebook, MySpace, YouTube, Flickr which go through to the website and donate money. Pipeline of customers filtering from loyalty program (highest level) to company portal (mid-level) to blog (lowest level) Sentiment of comments - text analytics/mining Top ideas (from idea generation tool) - summarized by themes automatically (text mining) Customer satisfaction rating at individual level based on survey answers, comments, ideas etc. Need more bench marking to make the current data meaningful. More info on the members themselves. Reach, quality of conversation, fuzzy things like that. Click through rate from Facebook stream, engagement rate for items that appear in Facebook stream.Online Communities: Metrics and Reporting 2009 44
  45. 45.  17 Additional AdviceFrom Question 22: What additional advice do you have for peers about metrics andreporting?SUMMARY:  20% (13) Measure Metrics that Supports Community Goals  20% (13) Define Key Business Objectives First  11% (7) Monitor Quality of Interactions, not Numbers  6% (4) Analyze Metrics to Find the Story In the Numbers  5% (3) Follow Standards  5% (3) Ensure Metrics Supports Qualitative Data  5% (3) Measure Everything, from the Beginning  5% (3) Dont Track Metrics Just For Management Reporting  4% (3) Dont Measure Everything  4% (3) Measuring Metrics is Crucial to Business Success  3% (2) Use Platform that can Report ROI Stats to Execs  3% (2) Share Experiences  3% (2) Measure Metrics against Past Historical Periods  3% (2) Change Goals as Necessary  3% (2) Measure Metrics Reflecting What Membership WantsTAKEAWAY:The two most common pieces of advice were related to online community business goals andobjectives. The first was to Define Key Business Object First, (20%), thus preventing disjointedstrategies and additional development work later on, and the second was to Measure Metrics thatSupported Community Goals, (20%), and business objectives that are actionable.Other pieces of advice were related to the type of metrics being tracked. Some respondents gave theadvice to Monitor Quality of Interactions, not just Numbers and Not to Track Metrics Just ForManagement Reporting, instead learn to analyze content and to Report on User Interaction andSatisfaction, instead of just reporting on a bunch of numbers. Some respondents suggested that bothqualitative (content) and quantitative (data) data should be tracked, but that Quantitative Data shouldSupport Qualitative Data.Some respondents gave conflicting advice on the importance of metrics and reporting, someparticipants gave advice to Measure Everything from the Beginning, whereas others gave theadvice, Don’t Measure Everything.There were many other pieces advice that respondents offered, such as the importance of FollowingStandards, Using Platforms that can Report ROI Statistics to Management, SharingExperiences, Measuring Metrics against Historical Periods, Being Flexible and Changing Goalsas Necessary, and Measuring Metrics Reflecting What the Membership Wants.Online Communities: Metrics and Reporting 2009 45
  46. 46.  RESPONDENT RESPONSES:Below are just a few of the responses from participants, outlining any additional advice that participantshad for peers regarding metrics and reporting.“Measure Metrics that Supports Community Goals” related responses include: Focus on metrics that are actionable and relevant to key business objectives. Dont have too many metrics, use metrics that get to the core of your business goals - choose metrics that allow you to gauge your progress towards specific goals (business & member goals) - include both qualitative and quantitative goals - dont be afraid to get rid of goals and add others... wash, rinse, repeat. Make sure your metrics correspond to your purpose for the community, and also that your purpose corresponds with your community members purpose for being there. Dont track it if it wont change your decisions. Make sure that your data has significance to an overall business plan. It makes no sense to collectOnline Communities: Metrics and Reporting 2009 46
  47. 47.   nice numbers if those counting the beans only care about unique visitors or time spent on site etc… Dont get lost in metrics that dont affect your business outcomes. Metrics are important but more relevant are the actions based on the metrics So much information is available you need to think through what really matters and watch it closely. Be sure to link community metrics to the business metrics. If the business metric is to increase renewals, you should track renewal metrics. Dont get bogged down in metrics that dont relate to the ultimate business goal.“Define Key Business Objectives First” related responses include: We did not successfully defined the key business goals first, and then created community metrics that measure our drive to those goals. This has resulted in a me too strategy where many internal stakeholder groups are wanting to build a community site for their customers but have no clear answer to the reason why?’ What business goal are you trying to serve? How do you measure success? Holding the line early and defining these up front will save a lot of long-term rework and will prevent fragmented and disjointed strategies. Get involved early in development so metrics code gets inserted. Inserting this code after the fact adds months to completion think ahead about the metrics and reporting infrastructure early, so that queries and reporting can be done efficiently without additional development work. Balance metrics with top quality surveys related to the business. Neither gives the full picture. The worst: vanity metrics that dont indicate the health of the business, and lazy surveys that dont generate insight into the customers experience. Invest in BI tools, only after you know what data you want to measure. Determine what you want to measure first, then build the system with appropriate web statistical tools. Make sure the outcomes of your community are crystal-clear and defined first ... then, put your resources (i.e. $$$ and time) against driving those outcomes. Your metrics are really just indicators of the performance of those resources. Everything else is unnecessary. Think about how to track/measure it before you build it. Create a baseline as you get started with any major community project and use reports to show results proactively. This way you set the parameters that you can manage or create transparencyOnline Communities: Metrics and Reporting 2009 47
  48. 48.   on what is feasible and what is important to your company. Determine what your broader goals are for what youre trying to achieve with your online community, then determine what to measure, how to measure it, and then how to analyze it. A community cant be successful until you determine how youre measuring success.“Monitor Quality of Interactions, not Numbers” related responses include: Make sure to push back against management desire for defining success by # of registrations and page views. A better measurement of success is to show increases in the number of users who regularly participate. Pay attention to quality of interactions, not necessarily numbers of them. While it is important to show metrics our execs, as a community manager it is critical that your metrics are helping you understand what the membership wants and needs are. Remember, communities exist to benefit the membership. If the membership sees no value in the community, the community will cease to exist. Learn to analyze the content, rather than a bunch of numbers, that often (usually?) dont mean what theyre supposed to represent. For example, time spent can indicate engagement; or, it can indicate the visitor got a phone call while on the site. Numbers never tell the whole story. The best metric is the pulse of the community. Find measures that report interaction and satisfaction, and dont benchmark quantitative measures against other communities.“Analyze Metrics to Find the Story in the Numbers” related responses include: Metrics only go so far, listening to and understanding why certain trends may occur are for more important. For example, we may see a surge in login activity during large updates, this is due to our services being offline with the exception of web services (website/forums) should we see a rise in a period where normally there shouldnt be it is my responsibility to assess and investigate why this is. Metrics should always be supplementary to help support qualitative feedback, not the other way around. Metrics alone tell you very little. You need to find someone who can both find the story in the numbers AND be able to tell that story to others. Thats a rare person. Whilst I see the importance of metrics for some stakeholders, it is important to have a dedicated community manager or team to read between the lines of figures and ascertain the immeasurable. What sort of stories do those figures tell, or belie? Are they related to content trends that cant beOnline Communities: Metrics and Reporting 2009 48
  49. 49.   repeated – e.g. members complaining about performance issues.“Other” related responses include: You cant properly run or grow your online business without proper metrics and reporting. If you dont know who your customers are, what they buy, when and why they buy - you wont stay in business long. MEASURE EVERYTHING from day 1! Its the best investment in your community, organization, or business. Dont obsess too much over them - if the community is buzzing and happy, the good stuff will rise to the surface... Many general website statistics are nice, but do your best to get a platform that can report important ROI stats...makes the growing idea of Community and Enterprise 2.0 much more palatable to management. Its crucial if you want to have a successful community. If you arent actively looking at your metrics and using the analysis and results to improve your website, youll die on the vine. Monitor and measure everything, even if managers are not asking for a particular metric. The likelihood is that they will ask for it at some point in the future. It would be very nice to have the right resources and tools to do proper metrics. Why not come together and start sharing our experiences? Separate out "indicators" like page views, time on site, number of comments per post from true "performance" metrics that relate to sales growth or cost savings. Ensure your performance metrics tie into an ROI model, even if your management isnt currently asking for this - community longevity depends on ROI. Standardization is needed. I prefer doing two versions of metrics, one much more general and top-line that people can scan and read once a month, but also a much more in depth one for the web/Public Affairs teams to delve into. There needs to be a balance between quantity (metrics) and quality (of content, feedback, etc.). Collect anecdotes, thank you’s, and other feedback received through your collaboration tools to help complement your metrics and better represent the value of your communities. Gather everything (as long as it is not too time-consuming), but look for opportunities for comparison and metrics that prompt action/response.Online Communities: Metrics and Reporting 2009 49
  50. 50.   There is no standard when it comes to reporting systems as to what should be measured and reported. In fact, the same metric reported by two different sources may be calculated quite differently. An industry standard needs to be created that 1) debunks "hits" as being valuable 2) solidifies exactly what constitutes a unique visitor and over what standard time period that is measured 3) standardizes the methods of determining each metric.Online Communities: Metrics and Reporting 2009 50
  51. 51.  Final CommentsThis research was conducted with sponsorship support from HiveLive and Lithium Technologies.We would like to thank all of the participants who spent time giving us their frank and candidanswers and feedback. Thank you. We really appreciate your interest and effort!We hope that you continue to find value in the qualitative feedback that we are presenting in thisformat. Please let me know if you have any suggestions that would help improve our process.Forum One will continue to explore the emerging issues around online community and socialmedia as the primary purpose and agenda of the Online Community Research Network.http://www.onlinecommunityresearch.comBest regards,Bill JohnstonChief Community OfficerForum One Networksbjohnston@forumone.comAbout Forum One Networks & The Online Community Research NetworkForum One Networks connects those who connect othersonline. We provide events, research, networking and strategic consulting for online communityand social media professionals.We firmly believe that the best source of information about online communities and social mediaare those who play in this space daily. All of our activities are based on this premise.The Online Community Research Network (OCRN) isa collaborative effort of online community professionals to better understand the principalchallenges of building and managing online communities. The Research Network coordinatesthe collection, analysis and dissemination of useful information among online communityprofessionals.For further information, or for sponsorship / partnership opportunities, please contact BillJohnston bjohnston@forumone.comOnline Communities: Metrics and Reporting 2009 51