Return On Contribution (ROC):
A Metric for Enterprise Social Software
Michael Muller, Jill Freyne*, Casey Dugan,
David R Millen, & Jennifer Thom-Santelli
IBM Research & IBM Center for Social Software
Cambridge MA USA
*Jill Freyne is now at Tasmanian ICT Center, CSIRO, Australia
1

Agenda
• How to measure the benefits of social software in
organizations?
• Return On Contribution (ROC)
• Applying ROC to
– Enterprise social software applications
– Types of resources in social software applications
– Points of articulation in social software applications
– Individual users (with a few notes on privacy)
• Conclusion and Next Steps
2

Benefits of Social Software
• Informal arguments are known
– Better knowledge-sharing
– Better personal effectiveness
– Improved ability to manage one’s reputation
– In organizations, better satisfaction and retention
• There are few strong studies to support those claims
• Management’s desire: Return On Investment
– Has been shown for niche social software applications
• Customer-support operations
• Customer communities
– Has been shown for advertising opportunities
• Social networking sites
3

Return On … What?
• Return On Investment
– ROI = Benefit / Cost € / € (unitless economic ratio)
– We hope for ROI >> 1.0
• Social software benefits – and even costs – are
difficult to measure
– Is the purpose of social software to increase productivity?
– How do you calculate ROI of a telephone? an IM product?
– How do you calculate ROI of a relationship?
4

Return On Contribution (ROC)
• A social ratio
– ROI = € / €
– ROI = Benefit / Cost
– ROC = Beneficiaries / Contributors (unitless social ratio)
= Consumers / Producers
• Return On Contribution
– A measure of social effectiveness – do more people benefit
(or consume) than contribute (or produce)?
– Rational Choice theory (Pirolli, 2007)
• Over time, people’s work-oriented decisions are beneficial to them
• Measure those choices and summarize them as a metric
5

ROC for Two Enterprise Services
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewa
– Overall usage – Overall usagell”
– Common goods – Common goods
• Bookmarks • Photos
• Tags • Lists (HiveFives)
• Events
• (Person-summaries) • (Person-summaries)
6

ROC for Two Enterprise Services
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewall”
– Overall usage – Overall usage
– Common goods – Common goods
• Bookmarks • Photos
• Tags • Lists (HiveFives)
• Events
• (Person-summaries) • (Person-summaries)
7

ROC for Two Enterprise Services
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewall”
– Overall usage – Overall usage
– Common goods – Common goods
• Bookmarks • Photos
• Tags • Lists (HiveFives)
• Events
• (Person-summaries) • (Person-summaries)
8

ROC for Two Enterprise Services
• Dogear • Beehive
9

ROC for Two Enterprise Services
• Dogear • Beehive
10

ROC for Types of Contributions
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewall”
– Overall usage – Overall usage
– Common goods – Common goods
• Bookmarks • Photos
• Tags • Lists (HiveFives)
• Events
• (Person-summaries) • (Person-summaries)
11

ROC for Types of Contributions
• Dogear • Beehive
– “Social bookmarking – “Social networking
Monthly Social-Networking ROC for three media types
behind the firewall” 20
18
behind the firewall”
Photo
List
– Overall usage 16
14
– Overall usage
Event
12
– Common goods – Common goods
ROCC
10
Data range of Figure 1
8
• Bookmarks 6
• Photos
• Tags 4
2
• Lists (HiveFives)
0
Jun-07
•
Jul-07
Events
Aug-07 Sep-07 Oct-07 Nov-07 Dec-07 Jan-08
Date
• (Person-summaries) • (Person-summaries)
12

ROC for Specific Contributions
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewall”
– Overall usage – Overall usage
– Common goods – Common goods
• Bookmarks • Photos
• Tags – specific tags • Lists (HiveFives)
• Events
• (Person-summaries) • (Person-summaries)
13

ROC for Specific Contributions
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewall”
– Overall usage – Overall usage
– Common goods – Common goods
• Bookmarks • Photos
• Tags – specific tags Tagging• for audiences (Thom-Santelli
Lists (HiveFives)
et al., 2008)
• Events
• Publishers
• (Person-summaries) • (Person-summaries)
• Evangelists
14

ROC for Specific Contributions
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewall”
– Overall usage – Overall usage
– Common goods – Common goods
• Bookmarks • Photos
• Tags – specific tags • Lists (HiveFives)
– Publisher: podcast tag “Tag-City” • Events
Tag ROC = 7.41 readers/contributor – Personal ROC = 63.00
• (Person-summaries)
– Evangelist: tag “web2.0”
Tag ROC = 1.95 readers/contributor – Personal ROC = 1245.00
• (Person-summaries)
15

ROC for Specific Contributions
• Dogear • Beehive
– “Social bookmarking – “Social networking
behind the firewall” behind the firewall”
– Overall usage – Overall usage
– Common goods – Common goods
• Bookmarks • Photos
• Tags – specific tags • Lists (HiveFives)
– Publisher: podcast tag “Tag-City” • Events
Tag ROC = 7.41 readers/contributor – Personal ROC = 63.00
• (Person-summaries)
– Evangelist: tag “web2.0”
Tag ROC = 1.95 readers/contributor – Personal ROC = 1245.00
• (Person-summaries)
16

ROC for Other Social Applications
Service Beneficiaries Contributors ROC C
Dogear 10896 4213 2.59
Beehive 21453 8397 2.55
Wiki server 238838 36377 6.57
Discussion server 150000 23000 6.52
Person-tagging 20973 3102 6.76
File-sharing 68762 11276 6.19
17

Implications for Design or Potential Use
• Track the development of organizational value of an application
over time
– Does it increase? Does it stabilize?
• Compare the organizational value of different types of
contributions over time
• Compare the organizational value of specific contribution
instances
• Assist the development of individual contributors, especially in
assigned roles such as “evangelist” or “publisher”, by providing
private views of her/his personal ROC
• Monitor, on an anonymous basis, the development of social
capital through aggregate, summary ROC measures across all
beneficiaries and contributors
18

Unanswered Questions about ROC
• Are there characteristic “signature” ROC values for
different types of applications?
• How to determine “stabilization” of ROC over time?
• What should the “target” ROC be for a discussion
forum?
– ROC >> 6.0 for some applications looked very nice
– However, ROC=1.0 suggests full democratic participation
– When are different values of ROC desirable?
• What should the “target” ROC be for a type of object,
or a particular object (e.g., a tag)?
• Can ROC help to show the value of “lurkers”? When is
it permissible (under privacy rules) to study “lurking”?
19

Summary of Contributions
• Lurkers as non-public participants (Nonnecke and
Preece, 2001) and as altruists (Takahashi et al., 2003)
– Employees in some jobs are “paid to lurk”
– Lurkers’ “consumption” of shared objects is a test of the
organizational value of those objects
• ROC provides a “social value” metric for
– Social software applications (Dogear, Beehive)
– Types of contributions (Photos, Lists, Events in Beehive)
– Specific contributions (Tags in Dogear)
– (where permitted) specific contributors (Taggers in Dogear)
• ROC can help organizations and researchers to assess
and study the value of social media
20

Thank you!
Slides may be found on slideshare.net
michael_muller@us.ibm.com
21

ROC for Two Enterprise Services
• Dogear • Beehive
22

ROC for Two Enterprise Services
• Dogear • Beehive
Two ways to think about benefit
• All users (including contributors)
ROC C = AllUsers/Contributors
• Lurkers only
ROC L = Lurkers/Contributors
23

ROC C and ROC L
• Dogear • Beehive
Two ways to think about benefit
• All users (including contributors)
ROC C = AllUsers/Contributors
• Lurkers only
ROC L = Lurkers/Contributors
24

ROC for Two Enterprise Services
• Dogear • Beehive
25