1. Measuring Digital Communitiesin Results-Driven EnvironmentsNelly Leonidis | @nellleo | MarketingManaging online communities is a dream job for those of us who enjoy connecting people, technologies and businesses. Community managers have soft skillsthat help them connect with the members they’re looking after, and have analytical and strategic skills that allow them to connect a business’s goals to thecommunity.But measuring and reporting on the health and performance of digital communities isn’t a straight forward one-formula approach. I’ve observed (and fell into) afew traps when it comes to reporting (as accurately as one can). This presentation will go over the problems and the solutions that have worked.
2. Why is Indiana Jones here? Because I’m going to use him as my presentation’s tour guide.I believe community managers are adventurers by profession. We go to unchartered territories (hello, subReddits), create tribes, join existing cultures, learn tospeak new languages, share memories with people we meet along the way, guide crowds to safety, occasionally come out with some scars, but mostly - everycommunity is its own adventure. So using Indiana: totally legit in this case.
3. The snake pit of vanity metricsHello and welcome to the snake pit of Vanity Metrics. Gross, slimy, and hard to get out of without getting bit (usually by a “shiny-metrics”-obsessed seniorexec).Vanity metrics = chasing the “likes”, followers, web visits, YouTube views and all the numbers that make you feel good about your work at the end of the day.But could mean absolutely nothing for your business.If you have 10,000 comments on your post, but not one of those comments generated a single cent of revenue for your business, then what’s the point of thecommunity? At the end of the day, you have a stated goal behind creating the community: increase revenue, generate brand impressions, public outreach,decrease costs, retain loyalty, and so on.
4. Trap #1: The snake pit of vanity metricsLooking at the number of followers/visits/”likes” and reporting it asan achieved resultSolutionCreate a comprehensive measurement framework around yourbusiness goals for the community, including:• direct,• indirect, and• complementary measuresIf you report a vanity metric as a result, you’re misleading those you report to. This is an easy trap to fall into, and rather hard to get out of. Once managers getused to the big figures, they want more - with or without context.Your job will be to condition them to see the actions behind the numbers. An increase in the number of followers is great, but you should focus on what haschanged, been delivered or goal met.What works? A comprehensive measurement framework that takes into account 3 measures: direct, indirect, and complementary.Direct = measured changes in your call of action for the community. For example, the use of community-specific coupons for purchase, or the number of phonecalls that come from the community (using a special # to help tracking, for example).Indirect = the interactions between your community pertaining to your call of action. Are more people talking about your coupon? Are they discussing yourcustomer service through the telephone? What are they saying or communicating that can explain the change in your direct measures? Take that into accountand provide snippets of information in your report.Complementary = the external circumstances that influence your direct and indirect measures. For example, a news story that talks about one of youringredients perhaps. A conversation in the community follows, and sales increase/decrease. These are factor you can’t help, but you need to note and report onbecause they are likely to have a very direct impact on both direct and indirect measures.Your report can help explain the “why” behind the vanity metrics by framing the results using your observations of direct, indirect and complementary measures.Always tie them back to your community’s goal(s).
5. The contracting walls of community life cyclesIf your reports aren’t addressing the life cycles of your community, brand, or individual members, then you’re quickly working towards the demise of yourcommunity (and your job, possibly).Life cycles here refers to three different cycles that dictate how your community should be performing and at what point.Community life cycle: Did it just launch? Are you at the maturity level? Have you hit post-maturity segmentation?Brand life cycle: Is the product new? That means you’re in brand-recognition mode. Is the market saturated? Then you’re likely focusing on retention and loyalty.Members’ life cycle: Are they students graduating? Is this a parent-heavy community? Seasonal purchases?A good way of framing the obstacle is to pretend that you’re the community manager for a car-manufacturing community. You’re dealing with three cycles all atonce, and every one of these will have an effect on what you measure, when you measure it and what you need to take into account to explain and analyze yourdata.If your report makes your community look like it’s underperforming because your goals aren’t reflecting the life cycle(s) you’re in right now, it will feel like thecontracting walls are coming to squish you and your members out of that space!
6. Trap #2: The contracting walls of community lifecyclesFailing to take into account natural community life-cycles whenmeasuring and reportingSolutionSet realistic expectations for your community’s performance basedon:• your audience• industry trends• platform specificsIn order to avoid analyzing and reporting data that doesn’t accurately reflect the life cycle of your community, set realistic expectations for your community’performance.Do this by understanding your audience, the industry and the platform specifics on which you run your community, and place the information within the contextof the life cycles your community is likely going through.Setting realistic expectations with respect to your audience means understanding their behaviour online, their tendencies, what particular life cycle they aregoing through and their networking habits. For example, a community for high school teachers will see a decline in participation during peak teaching periods inthe day, and a slump during the summer time. Reporting on the goals achieved for the duration of measurement needs to note these shifts in the community.Additionally, understanding the next course of the life cycle will help the community manager set realistic goals of growth (or maintenance) and move towardsachieving them at a pace most suited to their particular community.Industry trends give the community manager markers that will help analyze the collected data for the measurement period and forecast (realistically) thegrowth/return potential of the community. Your can use information you collect in house about your industry, or rely on products like Gartner Reports for yourmarket.Finally, a community that runs on a developed platform (a Facebook page, for example), has a flatter learning curve for members. This means engagement andreach can happen much quicker. It also allows for you to target a large group of people at once, while channeling your energy on keeping them interested in yourbrand and community. But a community built on a specific platform (for example, a moderated community through your website), will have a steeper learningcurve for joining members as they need to get used to your community’s functionalities. This means they’ll spend more time in the beginning learning thanengaging, so your reports need to account for that extended period of learning.
7. The giant rolling boulder of externalitiesExternal factors that you have absolutely no control over will be affecting the performance of your community.These factors could be economic (growth, recession), industry-related (shift away from books to e-readers, for example), publicity (an unflattering news articleabout one of your product’s ingredients, for example), and so on.
8. Trap #3: The giant rolling boulder of externalitiesTreating your community as if it exists in a vacuumSolutionLook from the inside out:• frequent snapshots of news in your industry• peer/competitor activity monitoring (not comparing!)• regular briefings with all teamsThe best way to deal with something you cannot control is to try and see what you can do before it rolls onto your and your community.Environmental scans for the win!Study your industry thoroughly; learn about the shifts (small or large) that are affecting you, your peers, and competitors. Keep an eye on the news; what’shappening in your community, what are the key issues affecting buying/purchasing/service sectors/not-for-profits and so on. The more you know about what’scoming, the more time you’ll have to adjust your community management strategy and provide context to your data findings.Watch what your peers and competitors are doing to deal with the very same issues you cannot control (economy, industry, etc). Monitor, but do not compare.Each community’s performance is shaped by many variables that cannot be blindly applied across the board, so comparing your performance to anyone else’s isunfair. Monitoring however, will give you an idea of what seems to be working or not working in response to these externalities.Share your concerns and insights with other members of your organization, and learn from their insights. For example, a decrease in consumer spending has aneffect on your entire company’s performance. What are your marketing or operations teams doing to help the business stay profitable?
9. The quicksand of analysis paralysisA very common problem with data-driven communities? Spending too much time collecting, analyzing and reporting on data - with long intervals of no action.Whether or not you’re the one doing the data-collecting and analysis, you’re likely checking on insights regularly and trying to see how your community isperforming in terms of your business goals. And between looking at your direct results, capturing indirect measures, keeping an eye out for industry shifts,noting life cycles, and so on - when do you put into action the recommendations following your report?
10. Trap #4: The quicksand of analysis paralysisWe’re good at getting data and analyzing it. Now what?SolutionPractice agile analysis:• keep reporting cycles short• always be testing• tweak, tweak, tweakRealizing that your digital community is a living breathing organism that acts, reacts, and overreacts to events unfolding every second online helps you escapeform the quicksand of constant measuring, analyzing and reporting then acting too late on information.Create a more adaptive pattern of deploying tweaks to your community management strategy. For example, check your data on Thursdays and note what hasworked and what hasn’t over the last week. Note actions/behaviour that can be considered as a hit/miss. On the following Mondays, introduce adjustments toyour strategy based on your metrics and what you’ve observed. Replicate the hits, and avoid what hasn’t worked. Seize on opportunities you spot immediately,because it means getting results in shorter periods of time and keeping at pace with your community.You should also introduce a hypothesis that explains why some calls for actions are met and some aren’t, then change some elements in your communitymanagement that could prove or disprove your theory to help you earn what works (or doesn’t) in a shorter period of time. For example, you can move the timeof posting from 10 a.m. to 1 p.m. to see if it makes a difference in the number of comments you get (or purchases, re-shares, etc). You can also experiment withtone, or content.Continuously adapting your community management strategy to move faster on timely insights and capitalize on continuous “wins” will save you time andresources in the long run, because you are not looking to introduce sweeping changes to your strategy or spend more time analyzing quarterly results.
11. The myopia of relying on passive dataIndiana Jones gives his dad the opportunity to shoot down the enemy, and Jones Sr. ends up shooting their only means of escape down.I’m always surprised by the number of businesses that do not tap into their most valuable resource to learn more about their performance and report moreaccurately.
12. Trap #5: The myopia of relying on passive dataForgetting to ask your community members what they wantSolutionTap into your best data source - the community:• surveys (polls, short answers)• focus groups• “wants and needs” analysis• listening programsPassive data in this sense means data that you’re collecting by-the-by. For example, exit pages through your web analytics dashboard.You don’t really know why people are leaving that particular page, but you can make theories: they dislike the content, the title is misleading, their smartphonebatteries ran out, power outage in your particular area one night, etc.The easiest most effective thing to do would be to ask the community about your observations: “Hi everyone, what to you think of this page? Feedback is alwaysappreciated!”.The insights you get from your very own community are worth more than everything else you could passively collect, because you’re getting first hand accountsof what’s working and what doesn’t. This makes you understand your community’s behaviour better, and you’ll know why your calls for action are met (or not),and how big is the gap between your business goals and what your members thing they’re contributing to your community.Use surveys: quick polls do well, and can be done frequently. Longer surveys (no more than 5 questions) on less frequent basis also work well.Introduce focus groups to your community: ask for a number of volunteers and interact with them about a particular idea, product, and so on. This is also aneffective way to turn your new members into regulars through direct dialogue, and move them to brand champions based on the new interpersonal relation youhave with them.Through a “wants and needs” analysis, you’ll be able to understand what your community members think they’re getting our of your community, and how italigns (or not at all) with your business goals. This could be the making or breaking of your strategy; if there is a big gap between what members are doing andwhat you’d like them to do, this is an excellent way to start bridging the gap and introducing new content/features/conversations/offerings that will help youdeliver on your goals.Listening programs refer to the way you interact with your members based on what they say. For example, if someone comments on your post, how quickly doyou respond? How thoroughly? How honestly? If you have the ability to get in on the conversations (steer them or just join), you’ll see more members adding theinsights that your business could benefit from.Thing about the worst thing that would happen if you began tapping into your community directly for insights. Nothing. You would receive nothing, and the riskto you is nothing.The best thing that would happen? New insights, business ideas, improved analysis, better reports, more realistic goals for your growth, and in the end a happiercommunity manager!
13. With respect to measuring digital communities:“X never, ever marks the spot.”It’s very important to know that while these traps are common measurement and reporting traps that I’ve come across (personally and in conversation withother community managers) over the last few years, there is no fixed formula to effectively measure a community’s performance.Measuring and reporting on your digital community - as you know - is a combination of different measures and insights - all of which apply only to yourcommunity at a particular period of time.There is no “holy grail” of measuring a digital community, but there are some ways to avoid providing unhelpful or widely inaccurate reports on the health ofyour community.
14. Bonus: lessons learned in the trenches1. Educate: horizontally and vertically2. Clear presentation of findings = 90% of the battle3. Always, ALWAYS, know the business value of your community4. Share ALL the things with ALL the people!Here are a few lessons that will save you time, blood, sweat and tears:1. Educate those around you about what you do, your community, how to understand the results of your work, and the effect it has on business. Educatevertically: upwards to your managers - so they stop bugging you for vanity metrics, and downwards to those doing the research for you - so they see the biggerpicture and how it all helps you manage a better community online. Take the time to educate your peers as well. You’re all working towards the same corporategoals, why not provide them with a little more insight? You can do this by holding lunch and learn sessions (for example, to share cool trends emerging in yourindustry that have been validated through your community, or just to explain how you engage with people online and what they’re saying about your brand).You can also write case studies about what works for your brand in the communities, what doesn’t, how do you overcome the difficulties, and what are thelessons learned that can be of benefit to your organization.2. Present your reports in a clear and jargon-free manner. Use metaphors, and use examples. Have an executive summary that explains what you’ve tried to dofor the time period measured, what you observed, highlights, and recommendations. You can then provide more in-depth visuals on quantifiable growth, includescreenshots of conversations and interesting information shared, share the lessons learned, and list the recommendations for the upcoming period. Add anappendix with your metrics dashboard data if you wish, industry scans, associated reports, or details on your survey or focus group results. This way, you haveconsumable information for any type of reader regardless of their background and how much or little they know about digital communities and measurement.The executive summary helps your managers look through the report quickly and decide how much or little detail they want to go into. The appendix makes thedata-happy people ... happy.3. Never walk into a meeting without having a few “media lines” ready about your work and the results it recently achieved. As a community manager, your job isalready misunderstood by most of the people you’ll meet. “So you sit on Facebook all day long and chat” is one of the most frequent reactions we hear afterintroducing ourselves as community managers. The best way to prove your (and your community’s) worth immediately? Have shareable insights ready. Forexample, “In the last 6 months, our community was responsible for 15% of the growth of our brand awareness, which generated 2% sales in the past quarter”.No one is going to ask you to justify your work or your community’s existence after that. And you get to defend your members to any people that doubt theiroverall contribution to your business’ success or growth.4. No point keeping useful information an insights from your own company. Share your reports with interested groups, add useful information to your emailsthat could be of help to product developers or public relations staff, and so on. In my experience, I’ve seen many fantastic ideas about content come from theleast expected sources: people who weren’t at all part of my division or discipline. Bring your community’s insights to your organization, and help themunderstand how their work impacts those you mean to do business with.
15. tl;drTrap #1:The snake pit of vanity metricsSolution:Tap into your community; ask and listenSolution:Comprehensive measurement frameworkSolution:Set realistic expectationsSolution:Use insights from environmental scansSolution:Practice agile analysisTrap #2:The contracting walls of community life cyclesTrap #3:The giant rolling boulder of externalitiesTrap #4:The quicksand of analysis paralysisTrap #5:The myopia of relying on passive dataA quick recap slide of the 5 traps and solutions to help you recollect the last few minutes.
16. Images from:www.indianajones.com / LucasFilm Ltd.Source of images.