Methods and metrics for measuring the success of Enterprise Social Software


Published on


Abstracht: With Enterprise Social Software (ESS) being increasingly used for internal communication and collaboration, an important question is how their success can be measured. In this paper we contribute to answering this question by investigating which methods and metrics organizations are currently applying for measuring ESS success. To do so, we have conducted 26 interviews with persons responsible for the deployment and use of ESS in their company. We found that measuring ESS success in the studied companies mainly focuses on the analysis of usage. We attribute this to the fact that ESS is primarily characterized by an abundance of user-generated content that can be easily analyzed. At the same time, companies still face difficulties in the evaluation of the business value of the platforms. We attribute this to some characteristics of ESS that we will discuss in this paper. Our study is supposed to provide a comprehensive overview of the methods and metrics applied in practice to measure ESS success.

Published in: Business, Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Methods and metrics for measuring the success of Enterprise Social Software

  1. 1. www.kooperationssysteme.deMethods and metrics formeasuring the success of ESSWhat we can learn from practiceand vice versaChristian Herzog21st European Conference on Information SystemsJune, 2013 Utrecht
  2. 2.• Companies use Enterprise Social Software (ESS) for their internalcommunication and collaboration• The positive impact has been recognized but the measurable effects are stillnot obvious and easy to prove• IT executives are under pressure to justify the investments and todemonstrate the benefits of ESS by appropriate methods and meaningfulindicatorsMotivationUserStakeholderInterested in: Seeing thebenefits of the platform fortheir own work (What is infor me?)2Christian Herzog„What kinds of methods and metrics are currently being used to measurethe success of Enterprise Social Software?”Platform ownersInterested in: Improving theplatform and its usage(What can we do to improvecollaboration?)ManagementInterested in: Justification forinvestment(Is it worth to sponsor?)The impact of different stakeholders should be considered during success measurement and thechoice of success metrics.
  3. 3. www.kooperationssysteme.deWhat do we know?
  4. 4. www.kooperationssysteme.deSuccess measurement models4Christian Herzog• Differences of ESS to other IS (create own content, connect with other users;mostly used to support unstructured tasks; no clearly defined objective) not possible to adopt IS success models without modification• Studies on IS success are missing detailed success measurement methodsand proposals for the collection of these metrics• ESS success models provide more specific methods for measuring the success• ESS studies frequently focus on measuring the usage ( user-generatedcontent)• Weblog success model (Du and Wagner,2006)• Return On Contribution (ROC) (Muller etal. 2009)• Measuring the value of social software(Cooper et al. 2010)• Impact of organizational social web siteusage on individual and teamperformance (Raeth et al., 2011)• Theory-Based Measuring Model (Lehnerand Haas 2011)Enterprise Social Software• Technology Acceptance Model (Davis,1989)• IS Success Model (DeLone & McLean,1992)• The construct space for IS effectiveness(Grover et al., 1996)• Unified Theory of Acceptance and Use ofTechnology (Venkatesh et al., 2003)• IS-Impact Measurement Model (Gable etal., 2008)Information Systems
  5. 5. www.kooperationssysteme.deOur interviewstudy…
  6. 6. www.kooperationssysteme.deInterview StudyMethodologyInterviews (n=26)with “Enterprise 2.0”companies (n=24)6Topics / ObjectivesWhat kind of methods do you use inorder to measure the ESS success?Christian Herzog• Qualitative approach by semi-structured interviews; interview guide with 32questions  exploratory assessment• Companies from German-speaking area (<10 till >400.000 employees);all have practical experience in the application of ESS• Conducted by telephone, one was held face to face (Dez. 11– Jun. 12); 35-50 min;recorded, transcripted and coded• Discussion in two focus groups (10 and 3 participants) with experts from practiceand research ( reduce subjectivity, discuss interim results)Do you measure the success of yourESS?What metrics are examined in thesuccess measurement?Are there any barriers or limitations?
  7. 7. 7ResultsA practice-oriented set of methods for measuring ESSsuccessChristian Herzog• Focus of ESS success measurement can change with the maturity of the systemi25: “In first place we really wanted to know if there is activity on it [the platform]. But once if there is regularactivity on it, then you are suddenly interested in how it can bring economic benefits.[…] interests also move the more mature a system is.”
  8. 8. 8ResultsESS success measurement is applied in practice…i22: “So when you measure the success of the tools, it means I take my staff and users seriously. Im interestedin what you are really interested in, what is working, what is not working, what do you actually want andwhat should be different.”i18: “How much do I invest in the measurement? The problem is simple: To have something ... to arguesomething waterproof, I need to invest in the measurement so much that I know the profit is almostgone.”• No definition of success or success measurement was given to the interviewees Different views of success and its evaluationi13: “…what is difficult in this issue, is first per se the definition: What is success? So right now we are stillbased on the criteria for success, which we can measure.“• Different opinions on the point of when success measuring beginsi11:“Yes surely you can quantitatively measure the usage, but I think what matters at the end of the day,especially in Enterprise 2.0 projects - is what has changed qualitatively.”Christian Herzog
  9. 9.• 20 of 24 of organizations collectusage statisticsi08: “Someone who uses the tool intensively willproduce so much more benefit and also recognizemore benefit than someone who does not use it. Sothose are certainly self-reinforcing cycles”• Objectives influence whether ratherqualitative or quantitative methodsare usedi25: “It depends on the target horizon […]. Usingquantitative methods on a relatively new field ofresearch is very difficult. Suitable for this are morequalitative evaluations.”9ResultsUsageChristian HerzogMethod MetricsContent andusageanalysisadjusted ideas; intensity ofcollaboration; degree of cross-linkingDatabasequeries /Log fileanalysisno. of: blog-posts, communitiy spaces,authors, attachments, visits, edits, log-ins, user, new users, messages per day,unique visitors, comments, blogfollowers, average comments on a blogor discussion forum, unique users andhits per time period, sessions, wikipages per day, posts, readers of a post,praises per post;development of the use; posts withmost readers; average time per userper visit; session timeUserInterviewsuser requirements for the plattform;usage behavior; use case validation /user satisfaction with the processes ortools; review of the toolUsersurveysuser satisfaction with the plattform;usage types of the tool; frequency ofuse; applicability of the tools;knowledge of the users about thepossibilities of the tools; satisfactionwith the availibility through the newtools; self-assesment of the affinity inusing new tools; usability benefit
  10. 10. 10ResultsBusiness ValueChristian HerzogMethod MetricsCalculation ofcostssavings of travel costs; savings ofhardware costsImputedassumptions /estimatesopportunity proceeds of projects;ROI; cost-benefit ratioMeasurementof emailtrafficemail frequency; no. of emails;correlation of email frequency andblog posts from one personOutputmeasurementno. of implemented ideas; no. ofawardsProcess andusage analysistime spent for reading, writing andansweringUserInterviewsROI of use casesUser surveyseffort for working with the tools;individual business value; ROI; savedmoney; generating new revenue;new customer acquisition; percievedbenefits for the organization;business value• Difficult and complex; often based onestimates or subjective perceptionsi16: “[…] In ROI yes. So, this is very difficult for us.There is talk of reducing travel costs, speed ofinnovation, time to market, cost to market, etc.Unfortunately we cannot yet.• Malleability makes the measurementdifficulti25: “The experience is that over time you will findwhole new use cases that were not planned andwhich nevertheless generate a benefit.”• Precise definitions of use cases,application scenarios or specificobjectives are neededi25: “[…] the more it goes toward financial metrics,the more accurate you have to know thetargets.“
  11. 11. www.kooperationssysteme.deSo what?
  12. 12. www.kooperationssysteme.deDiscussion12Christian HerzogUsage• Usage is of central importance for thesuccess measurement of ESS as it is drivenby user-generated content• Almost all companies collect usage statistics simplicity of collecting usage data viaanalysis tools at low costs meets the need for quantifiable successmetrics of an IT manager participation in an ESS platform isvoluntary: just happens if users acceptplatform and find benefits for their ownworkBUT• Usage statistics alone have a limitedinformative value regarding the company’ssuccess and must be considered in contextBusiness Value• Subjective estimates and imputedassumptions for collecting monetarymetrics  makes it difficult to obtain validdata• With increasing maturity the focus shiftsmore to business value• ESS is mostly used to support unstructuredtasks and can be characterized asmalleable end-user software with noclearly a priori defined usage scenarios• Email traffic is seen as an indicator ofinformation processing workloadBUT moving to ESS does not simultaneouslybring a reduction of labor costs• Number of different metrics which are notconsidered by practitioners
  13. 13. 13Conclusions• ESS success measurement is applied in practice by a variety of differentmethods and metrics• Different views on ESS success: The dimensions usage and businessvalue are distinguished and supplemented by a set of methods• Usage is an important indicator for ESS success ( characteristics of ESS)but usage statistics require additional interpretations• Measuring the monetary or economic output is not easy due to the lackof detailed objectives and missing definitions of processes in specificuse cases as well as due to the malleability• Further research in terms of a quantitative study and in order to developthe results towards comprehensive guidelinesChristian Herzog
  14. 14. www.kooperationssysteme.deThank you very much!?QuestionsPlease don´t hesitate to contact me ifyou have any further questionsafterwards:christian.herzog@unibw.deChristian HerzogAlexander RichterMelanie SteinhüserUwe HoppeMichael KochCooperation Support Center MunichInstitute for software technologyBundeswehr University MunichInstitute for Information Managementand Information SystemsUniversity of Osnabrück14