Your SlideShare is downloading. ×

Community Metrics at Novell


Published on

Published in: Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Community Metrics The Novell Approach Lee Romero 2009 October
  • 2. Contents
    • Communities at Novell 3
    • Setting up the Discussion 6
    • Membership Metrics 10
    • Activity Metrics 17
    • Tying into Performance Mgmt 22
    • Some Advanced Metrics 25
    • Additional Metrics 28
    • Final Words 32
  • 3. Communities at Novell
  • 4. Community Metrics at Novell A brief overview of the program, part 1
    • Novell started a CoP program in 2002/2003 time frame
    • The program was primarily focused around the “solutions” and products Novell has marketed and sold
    • The overall program responsibility fell to the Enterprise KM group:
          • KM group established the goals, methodologies and processes
          • KM group also identified a set of standard infrastructure tools and supported those
            • No new significant development / implementation was supported
            • This was primarily a case of identifying what was in place and providing an a la carte menu for communities to work with
  • 5. Community Metrics at Novell A brief overview of the program, part 2
    • The CoP program identified levels of communities that largely guided investment and formal support for individual communities
    • There were approximately a dozen “top level” communities over time
    • There were also dozens of informal communities supported to a limited extent
    • The a la carte menu of tools included (among other things):
          • Intranet web sites
          • Mailing lists
          • Enterprise Wiki
          • Team spaces (implemented in LiveLink initially)
  • 6. Setting up my discussion
  • 7. What is a community member? It all started with a simple question
    • The rest of this presentation will focus on a set of metrics we developed at Novell that were manageable, scalable and, I think, provided useful insights
    • To start with, though, early on in our program, we were faced with a simple question from our community leaders:
    • “ How large is my community?”
    • Which quickly turned into:
    • “ What is a community member?”
  • 8. An Answer A definition of community membership
    • Because of our development limitations, we were faced with trying to track membership using either our existing tools or having some type of manual support
    • In looking at our tools, we found that our mailing list infrastructure provided a reasonably good solution:
          • It already had (list) membership management functions
          • People could subscribe / unsubscribe on their own
          • We already knew that the lists needed to be aligned to communities
    • With that insight and no other realistic option, we decided on a definition:
          • You are a member of a community if you are subscribed to any mailing list associated with the community
  • 9. An Implementation How we tracked membership
    • Our mailing list server provided a simple XML format for members for each list
          • We implemented a mechanism to sync this member list data from the XML format into a SQL database
          • This, combined with some integration with other databases already available, provided a wide variety of membership reporting
      • We later added on a mechanism that captured an “event” for each post to a list in the same SQL database, which led to a variety of activity based reporting
  • 10. Membership Metrics
  • 11. Basic Metrics
    • With the basic structure in place, we found we could provide data to answer many questions:
          • How big is any given mailing list?
          • How big is a community?
          • How much change is there in community population over time?
          • Various slicing and dicing by different demographics
    • The following slides provide examples of many of these
  • 12. Basic Metrics Community size
    • Simple to query current size
          • By tracking “join” and “departure” events, it was also straightforward to provide growth over time
    • We could also measure the total overall population of the “community program”
  • 13. Basic Metrics Community penetration
    • Knowing the employee population size (total or by demographics), we could provide measures of “penetration” of the community program as a whole and by community against different groupings
  • 14. Basic Metrics Demographics
    • By combining community membership with HR information, we could provide break downs of membership (both overall program and individual community) on a variety of dimensions
  • 15. Basic Metrics Demographics, continued
  • 16. Basic Metrics Other questions
    • Some other (not necessarily actionable) questions we could easily answer included:
          • How many communities is an employee a member of on average?
          • How many communities is a community member a member of on average?
  • 17. Activity Metrics
  • 18. Activity Metrics
    • Once we implemented a means to capture an “event” for each post in each mailing list, we were able to understand community activity (in this one tool) very easily
    • We also were also able to answer (admittedly, simplistically) another key question we had been asked:
          • “ How many *active* members do I have in my community?”
    • We answered that with:
          • A member is active if they have posted at least one post during the time period of interest.
  • 19. Activity Metrics
    • Some examples of specific metrics we were able to track
  • 20. Activity Metrics
    • We could also gain other potentially useful insights
    • Identifying most active members
          • Potentially useful for identifying SMEs or core team members (or even community leaders)
    • Percent (and number and even individual identification) of lurkers in communities
          • Useful within a community to know who is there but not “active”
          • Useful across communities to know when a community has a significant different rate of “lurkers” compared to other communities
  • 21. Activity Metrics Networks within communities
    • Using the activity data to connect people who corresponded, it was even possible to do data mining to get a sense of the network within a community
  • 22. Performance Management
  • 23. Performance Management
    • Working with the HR group, we eventually were able to community involvement with our performance management program
    • This was achieved by having community involvement embedded in the “employee self service” and “manager self service” portals
  • 24. Performance Management
    • The intent was not that a specific goal was desirable but that this provided a way to initiate conversations between a manager and an employee about involvement, encouraging:
          • Some thought about which communities were valuable for an employee
          • Some reflection on level of involvement and activity
          • Development of employees over time
  • 25. Advanced Metrics
  • 26. Advanced Metrics Some experiments into compound metrics
    • Another simple question prompted some digging into other uses of the data we had:
          • “ Why do we need to provide navigation to community sites [on the intranet], anyway? They don’t get any traffic at all?”
    • The resulting analysis attempts to draw a comparison between “web site visits” and community membership / activity
          • I came to call this measurement “knowledge flow”
    • You can find a detailed description at:
            • The eventual formula I worked out is:
    K c = 2 * P c * A c - P c * A c 2 M c
  • 27. Advanced Metrics Knowledge Flow examples Animations (in four dimensions!) available at:
  • 28. Additional Metrics
  • 29. Additional Metrics
    • On top of these metrics based on a mailing list implementation, the Novell CoP program also used a number of additional metrics
    • These included:
          • Web site usage
          • Intellectual asset production
            • Specifically called out white papers as well
          • “ Anecdotes” from community members
          • Knowledge share events
            • And attendance at same
    • For each of these, we provided quarter-to-quarter changes as well
  • 30. Additional Metrics Wiki metrics
    • Another area we investigated but did not lock in on was use / edits of the corporate wiki
    • Novell had (has) a corporate implementation of MediaWiki (in place since about 2003)
    • We explored some metrics related to the Wiki for communities:
          • Establishing a standard Category for each community to use
          • Using that Category assignment, we could:
            • Track usage over time for pages in the category (it was integrated with Omniture)
            • Track edits (“contributions”) over time to pages in that category
            • Potentially identify anyone who edits such a page to be a community member
  • 31. Final Words
  • 32. In Summary
    • Beyond all of the details here, a key takeaway is this:
          • With a very simple definition (of community membership) and a pretty simple technical approach to implement that definition, Novell was able to gain a wide variety of insights
          • Look at the tools you have, make some definitions (even if you know they are not perfect) and start tracking!
    • Also, when you are thinking about what metrics you want to track, make sure all your metrics are actionable
          • How can your metrics be turned into actions to improve your communities?
  • 33. Read more
    • If you would like to read more about the work behind this material, there are extensive write-ups available at:
    • Specifically, look under “Collaboration / Communities” on the “Posts by Topic” page