Community Metrics at Novell


Published on

Published in: Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Community Metrics at Novell

  1. 1. Community Metrics The Novell Approach Lee Romero 2009 October
  2. 2. Contents <ul><li>Communities at Novell 3 </li></ul><ul><li>Setting up the Discussion 6 </li></ul><ul><li>Membership Metrics 10 </li></ul><ul><li>Activity Metrics 17 </li></ul><ul><li>Tying into Performance Mgmt 22 </li></ul><ul><li>Some Advanced Metrics 25 </li></ul><ul><li>Additional Metrics 28 </li></ul><ul><li>Final Words 32 </li></ul>
  3. 3. Communities at Novell
  4. 4. Community Metrics at Novell A brief overview of the program, part 1 <ul><li>Novell started a CoP program in 2002/2003 time frame </li></ul><ul><li>The program was primarily focused around the “solutions” and products Novell has marketed and sold </li></ul><ul><li>The overall program responsibility fell to the Enterprise KM group: </li></ul><ul><ul><ul><ul><li>KM group established the goals, methodologies and processes </li></ul></ul></ul></ul><ul><ul><ul><ul><li>KM group also identified a set of standard infrastructure tools and supported those </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>No new significant development / implementation was supported </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>This was primarily a case of identifying what was in place and providing an a la carte menu for communities to work with </li></ul></ul></ul></ul></ul>
  5. 5. Community Metrics at Novell A brief overview of the program, part 2 <ul><li>The CoP program identified levels of communities that largely guided investment and formal support for individual communities </li></ul><ul><li>There were approximately a dozen “top level” communities over time </li></ul><ul><li>There were also dozens of informal communities supported to a limited extent </li></ul><ul><li>The a la carte menu of tools included (among other things): </li></ul><ul><ul><ul><ul><li>Intranet web sites </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Mailing lists </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Enterprise Wiki </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Team spaces (implemented in LiveLink initially) </li></ul></ul></ul></ul>
  6. 6. Setting up my discussion
  7. 7. What is a community member? It all started with a simple question <ul><li>The rest of this presentation will focus on a set of metrics we developed at Novell that were manageable, scalable and, I think, provided useful insights </li></ul><ul><li>To start with, though, early on in our program, we were faced with a simple question from our community leaders: </li></ul><ul><li>“ How large is my community?” </li></ul><ul><li>Which quickly turned into: </li></ul><ul><li>“ What is a community member?” </li></ul>
  8. 8. An Answer A definition of community membership <ul><li>Because of our development limitations, we were faced with trying to track membership using either our existing tools or having some type of manual support </li></ul><ul><li>In looking at our tools, we found that our mailing list infrastructure provided a reasonably good solution: </li></ul><ul><ul><ul><ul><li>It already had (list) membership management functions </li></ul></ul></ul></ul><ul><ul><ul><ul><li>People could subscribe / unsubscribe on their own </li></ul></ul></ul></ul><ul><ul><ul><ul><li>We already knew that the lists needed to be aligned to communities </li></ul></ul></ul></ul><ul><li>With that insight and no other realistic option, we decided on a definition: </li></ul><ul><ul><ul><ul><li>You are a member of a community if you are subscribed to any mailing list associated with the community </li></ul></ul></ul></ul>
  9. 9. An Implementation How we tracked membership <ul><li>Our mailing list server provided a simple XML format for members for each list </li></ul><ul><ul><ul><ul><li>We implemented a mechanism to sync this member list data from the XML format into a SQL database </li></ul></ul></ul></ul><ul><ul><ul><ul><li>This, combined with some integration with other databases already available, provided a wide variety of membership reporting </li></ul></ul></ul></ul><ul><ul><li>We later added on a mechanism that captured an “event” for each post to a list in the same SQL database, which led to a variety of activity based reporting </li></ul></ul>
  10. 10. Membership Metrics
  11. 11. Basic Metrics <ul><li>With the basic structure in place, we found we could provide data to answer many questions: </li></ul><ul><ul><ul><ul><li>How big is any given mailing list? </li></ul></ul></ul></ul><ul><ul><ul><ul><li>How big is a community? </li></ul></ul></ul></ul><ul><ul><ul><ul><li>How much change is there in community population over time? </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Various slicing and dicing by different demographics </li></ul></ul></ul></ul><ul><li>The following slides provide examples of many of these </li></ul>
  12. 12. Basic Metrics Community size <ul><li>Simple to query current size </li></ul><ul><ul><ul><ul><li>By tracking “join” and “departure” events, it was also straightforward to provide growth over time </li></ul></ul></ul></ul><ul><li>We could also measure the total overall population of the “community program” </li></ul>
  13. 13. Basic Metrics Community penetration <ul><li>Knowing the employee population size (total or by demographics), we could provide measures of “penetration” of the community program as a whole and by community against different groupings </li></ul>
  14. 14. Basic Metrics Demographics <ul><li>By combining community membership with HR information, we could provide break downs of membership (both overall program and individual community) on a variety of dimensions </li></ul>
  15. 15. Basic Metrics Demographics, continued
  16. 16. Basic Metrics Other questions <ul><li>Some other (not necessarily actionable) questions we could easily answer included: </li></ul><ul><ul><ul><ul><li>How many communities is an employee a member of on average? </li></ul></ul></ul></ul><ul><ul><ul><ul><li>How many communities is a community member a member of on average? </li></ul></ul></ul></ul>
  17. 17. Activity Metrics
  18. 18. Activity Metrics <ul><li>Once we implemented a means to capture an “event” for each post in each mailing list, we were able to understand community activity (in this one tool) very easily </li></ul><ul><li>We also were also able to answer (admittedly, simplistically) another key question we had been asked: </li></ul><ul><ul><ul><ul><li>“ How many *active* members do I have in my community?” </li></ul></ul></ul></ul><ul><li>We answered that with: </li></ul><ul><ul><ul><ul><li>A member is active if they have posted at least one post during the time period of interest. </li></ul></ul></ul></ul>
  19. 19. Activity Metrics <ul><li>Some examples of specific metrics we were able to track </li></ul>
  20. 20. Activity Metrics <ul><li>We could also gain other potentially useful insights </li></ul><ul><li>Identifying most active members </li></ul><ul><ul><ul><ul><li>Potentially useful for identifying SMEs or core team members (or even community leaders) </li></ul></ul></ul></ul><ul><li>Percent (and number and even individual identification) of lurkers in communities </li></ul><ul><ul><ul><ul><li>Useful within a community to know who is there but not “active” </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Useful across communities to know when a community has a significant different rate of “lurkers” compared to other communities </li></ul></ul></ul></ul>
  21. 21. Activity Metrics Networks within communities <ul><li>Using the activity data to connect people who corresponded, it was even possible to do data mining to get a sense of the network within a community </li></ul>
  22. 22. Performance Management
  23. 23. Performance Management <ul><li>Working with the HR group, we eventually were able to community involvement with our performance management program </li></ul><ul><li>This was achieved by having community involvement embedded in the “employee self service” and “manager self service” portals </li></ul>
  24. 24. Performance Management <ul><li>The intent was not that a specific goal was desirable but that this provided a way to initiate conversations between a manager and an employee about involvement, encouraging: </li></ul><ul><ul><ul><ul><li>Some thought about which communities were valuable for an employee </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Some reflection on level of involvement and activity </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Development of employees over time </li></ul></ul></ul></ul>
  25. 25. Advanced Metrics
  26. 26. Advanced Metrics Some experiments into compound metrics <ul><li>Another simple question prompted some digging into other uses of the data we had: </li></ul><ul><ul><ul><ul><li>“ Why do we need to provide navigation to community sites [on the intranet], anyway? They don’t get any traffic at all?” </li></ul></ul></ul></ul><ul><li>The resulting analysis attempts to draw a comparison between “web site visits” and community membership / activity </li></ul><ul><ul><ul><ul><li>I came to call this measurement “knowledge flow” </li></ul></ul></ul></ul><ul><li>You can find a detailed description at: </li></ul><ul><ul><ul><ul><ul><li> </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>The eventual formula I worked out is: </li></ul></ul></ul></ul></ul>K c = 2 * P c * A c - P c * A c 2 M c
  27. 27. Advanced Metrics Knowledge Flow examples Animations (in four dimensions!) available at:
  28. 28. Additional Metrics
  29. 29. Additional Metrics <ul><li>On top of these metrics based on a mailing list implementation, the Novell CoP program also used a number of additional metrics </li></ul><ul><li>These included: </li></ul><ul><ul><ul><ul><li>Web site usage </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Intellectual asset production </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Specifically called out white papers as well </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>“ Anecdotes” from community members </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Knowledge share events </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>And attendance at same </li></ul></ul></ul></ul></ul><ul><li>For each of these, we provided quarter-to-quarter changes as well </li></ul>
  30. 30. Additional Metrics Wiki metrics <ul><li>Another area we investigated but did not lock in on was use / edits of the corporate wiki </li></ul><ul><li>Novell had (has) a corporate implementation of MediaWiki (in place since about 2003) </li></ul><ul><li>We explored some metrics related to the Wiki for communities: </li></ul><ul><ul><ul><ul><li>Establishing a standard Category for each community to use </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Using that Category assignment, we could: </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Track usage over time for pages in the category (it was integrated with Omniture) </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Track edits (“contributions”) over time to pages in that category </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Potentially identify anyone who edits such a page to be a community member </li></ul></ul></ul></ul></ul>
  31. 31. Final Words
  32. 32. In Summary <ul><li>Beyond all of the details here, a key takeaway is this: </li></ul><ul><ul><ul><ul><li>With a very simple definition (of community membership) and a pretty simple technical approach to implement that definition, Novell was able to gain a wide variety of insights </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Look at the tools you have, make some definitions (even if you know they are not perfect) and start tracking! </li></ul></ul></ul></ul><ul><li>Also, when you are thinking about what metrics you want to track, make sure all your metrics are actionable </li></ul><ul><ul><ul><ul><li>How can your metrics be turned into actions to improve your communities? </li></ul></ul></ul></ul>
  33. 33. Read more <ul><li>If you would like to read more about the work behind this material, there are extensive write-ups available at: </li></ul><ul><li> </li></ul><ul><li>Specifically, look under “Collaboration / Communities” on the “Posts by Topic” page </li></ul>