SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metrics Game


Published on

Measurement is not just about looking for a bottom-line result to justify investments. It’s also a tool to provide feedback about where the organization is along the road to successfully leveraging investments in SharePoint and the business outcomes it provides. At every stage in the development of your solution, metrics provide a valuable means for focusing attention on desired behaviors and results. This presentation showcases a practical and realistic framework for SharePoint metrics based on real world examples and successes.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metrics Game

  1. 1. SharePoint Moneyball – The Art ofWinning the SharePoint Metrics GameSusan Hanley – President, Susan Hanley LLC
  2. 2. 2
  3. 3. 3
  4. 4. 4
  5. 5. 5Why Measure? – The Four “F” WordsFeedbackFundingFollow-onFocus
  6. 6. Measurement throughout the life-cycleBeforeMake thebusiness caseDuringProvide a targetMake tradeoffsTune theimplementationprocessAfterDevelopbenchmarksDevelop lessonslearned6
  7. 7. 7Make your case for the solution SharePoint enables
  8. 8. 8Measurement Process3. Who are themetricsSTAKEHOLDERS?6. What do themetrics TELL us aboutwhat we need toCHANGE?1. What are theBUSINESSOBJECTIVES?2. How should thesolution beDESIGNED to meetthe objectives?5. How can weCOLLECT the metrics?4. What are theMETRICS and howshould we PRESENTthem?AiddecisionmakingModifytheprocessor toolModifythemeasures
  9. 9. 1. What are the BUSINESS OBJECTIVES?Without a critical business initiative …… career limiting move9
  10. 10. 10Be the main event
  11. 11. It’s easy to go for the “motherhood” objectives …More innovative products and servicesMore effective marketingBetter access to knowledgeLower cost of doing business – reduction intravel and other operational costsHigher revenuesImproved employee, customer, and partnersatisfaction11
  12. 12. 12It’s better to Get SMART!Measurable (quantifiable, comparable)Achievable (feasible, actionable)Realistic (consider resources)Time-bound (deadline driven)Specific (concrete and well-defined)
  13. 13. 13SMART objective for a proposal libraryReduce the average amount of time it takesto produce complex proposals by 10% in thenext yearSpecificMeasurableTime-boundAchievableRealisticReduce the average amount of time it takesto produce complex proposals by 10% inthe next yearReduce the average amount of time it takesto produce complex proposals by 10% in thenext year
  14. 14. 142. How should the solution be DESIGNED to meetthese objectives?SiteArchitectureTechnicalInfrastructureFeaturesCustomization SecurityGovernance Roles and ResponsibilitiesTraining and Communications
  15. 15. 15Your business case ispersonal
  16. 16. 3. Who are the metrics STAKEHOLDERS?They’re at all levels -especially in the middleThey care about differentthings16
  17. 17. 17For each key stakeholder, ask …What counts?What keeps you up atnight?What do you alreadyuse?What do I need to tellyou?
  18. 18. 184. What are the METRICS and how should wePRESENT them?Identify the type• Quantitative• QualitativeConsider the life-cycleEstablish a baselineGain commitmentabout targetsDecide the bestway tocommunicate
  19. 19. Good metrics come in multiple types … plan on bothQuantitativePerformance between pointsSpot trendsQualitativeProvide contextUsed when numbers aren’t easy(storytelling)Used at early project stages (futurescenarios)Richer (“serious anecdotes”)19
  20. 20. Return on InvestmentBenefit > CostBe careful: whoever controls the spreadsheetand the assumptions can make an ROI that canjustify anything.Resources:Total Economic Impact™ of Microsoft SharePoint Server 2010(Forrester) for platform investment, less helpful for individual solutionsCloud vs. On Premise Calculator (Andrew McAfee and GoogleAnalytics) (for small to medium businesses)20
  21. 21. 21ROI is only PART of the storyGood metrics are:Related to outcomeRelevant to stakeholdersCollected at low costBalanced
  22. 22. 22Consider two types of quantitative metricsBUSINESS METRICSSYSTEM METRICS
  23. 23. Sample Business MetricsHours per week to execute a processNumber of Proposals/Contracts per yearNumber of “[My Organization]-All” emailsNumber of email attachmentsCall center or support call deflectionAverage application training costsCost savings to retire a legacy application23
  24. 24. 24Business Metrics Example: “Support Call Deflection” +BusinessGoal• Increase member satisfaction with HQ byproviding better access to helpful resources anytime/any place• Reduce the number of support calls to be ableto serve increasing numbers of members withsame staff• Reduce cost of “authorship” by HQ due to―crowd-sourced‖ contentApproach• Number of member-generated resources• Number of member-generated resources with > x―likes‖• Number of posts and comments in forums• Number of downloads of HQ and member contentSystem Metrics• Membership survey to seek out specific re-use cases, membershipsatisfaction• Number of support request calls to HQ• Reduction in simple requestsBusiness Metrics
  25. 25. 25Business Metrics Example: Process ImprovementApproachBusinessGoal• Allocate limited SharePoint Resources forProcess Improvement Projects• SMART Objective Example – Reduce theamount of time for a task by x% in 90 daysx x xT = Time on task (in minutes)E = Number of employees performing that taskN = Number of times per year task isperformedS = Average employee loaded salary cost perminute
  26. 26. 26Qualitative Metrics – the stories that drive it homeKeep it realIn the storyteller’swordsSerious Anecdotes
  27. 27. Serious Anecdote | ConsultingI joined the organization on March 16 without previousexperience. After one week of training, I joined a project team.After one day of training on the project, I was assigned a task tolearn a particular technology that was new to everyone on theteam. I was given a bunch of books and told that I had threedays to learn how to create a project using this technology.In my first week of training, I learned about the company’sintranet where people described their expertise. I sent an emailto four people I found with a search for that technology askingfor their help. One of them sent me a link to a documentcontaining exactly what I needed.27Instead of three days, my task wascompleted in 4 hours.
  28. 28. 28Serious Anecdote | Pharma – The NeedA scientist withThrombotic & JointDiseases in Germanybegan a project toisolate and culturemacrophages andneeded some help. Meanwhile, two scientists inthe US had deep experiencein protocols for this area.
  29. 29. 29Serious Anecdote | Pharma – The ResultBenefit: The German scientist wasable to leverage existing internalexpertise and, in the process, reducehis research effort by four weeks.Both scientists quicklyresponded with assistance.One helped him with culturingprotocols and the otherhelped him with informationon magnetic cell sorting.The German scientistconsulted the expertisedirectory to find that expertiseexisted within the companyand contacted the two USscientists he found in hissearch.
  30. 30. 30How do you spell success? Have a Baseline andTarget
  31. 31. 31Presenting MetricsBalanced ScorecardDashboard“Report Card”
  32. 32. 32Balanced Scorecard DimensionsCapabilitiesBusiness ValueHealth
  33. 33. 33Balanced Scorecard Example | Expertise LocationBusinessValue:Health:Capabilities& Culture:Metric Target Pilot Outcome# searches/user/week .25 .58Usefulness rating 3.5 out of 5 3.6 out of 5% of users who say “Don’t take itaway”66% 83%Usability/friendliness rating 3.5 out of 5 4.1 out of 5# Anecdotes (repeat metric) 10 serious 22% of participants attending training 75% 85%# of Anecdotes 10 serious 10 serious + 12 transactionalMinimum $ value/anecdote $X $2XEstimated time saved X months X + 3 months
  34. 34. 5. How can we collect the metrics?Try not to over-achieve – balance counting with“doing”Automate where possibleGet creative when it comes to qualitative metricsAskSurveyUsability TestingActive ListeningSeekSend out a ―journalist‖TrackClassify by typeKeep storyteller value metrics – what was the benefit to you?34
  35. 35. 35Example Survey QuestionsIf given the choice, would you KEEP it?How does this COMPARE?How EASY was it to …?
  36. 36. 6. What do the metrics tell us about how we need tochange?Are we doing the right thing?What areas are most successful?What areas should we be promoting?In which areas should we be investing?Which initiatives should we discontinue?36
  37. 37. 37Keep in mindMetrics alone won’t make your program successfulYou need someonewhose job it is tomonitor themYou need someone who isaccountable for makingchanges based on analysisIt’s as important to have a plan for acting on metricsas it is to have a plan for collecting them!
  38. 38. 38Call to ActionDevelop a plan to capturequantitative and qualitative metrics.Make sure metricsare part ofsomeone’s job.Identify baseline measures – and gain commitment ontargets – before you start!!Develop a library or list tocapture and categorizequalitative metrics.Develop an approach toproduce and promotemetrics.
  39. 39. About Me• Governance• User Adoption• Metrics• Information Architecture• Knowledge Management• Portals• Collaboration Solutions• President, Susan Hanley LLC• Led national Portals, Management Collaboration,and Content practice for Dell• Director of Knowledge Management at AmericanManagement Systemssusanhanleysue@susanhanley.com39August 2013!
  40. 40. Susan HanleySusan Hanley LLCsue@susanhanley.com301 469 0770 (o)301 442 0127 (m)@susanhanleywww.susanhanley.com
  41. 41. Thank you for your attention!This presentation will be available on the TorontoSharePoint Summit web site a few days after the event.
  42. 42. Please rate this session!Fill out the survey and get a chance to win a Surface
  43. 43. Extras43
  44. 44. White PaperFor a white paper that explains the concepts inthis presentation in more detail – with lots moreexamples, please go to
  45. 45. 45Examples of Intranet System MeasuresKey Measure Objective MetricWhich features of theintranet are mostimportant?Knowing which pages are most used can help to prioritize which pages should beimproved or developed.You can also see which business units are the biggest intranet users and whichbusiness unit’s content is used the most.Page Hits―Dwell‖ Time (Time onPage/Site)Which features are notbeing used?If certain pages have low usage numbers, it is an indication that either the page isnot very popular—and therefore should be a lower priority to develop—or thatpeople are just not aware of its existence (which might be a communications or"promotion" problem).Page HitsDocument DownloadsIs the site navigationeffective?A high number of hits on a page that is not easily accessible from the main pageindicate that the popular page should be moved up in the hierarchy.Search results with no hits present opportunities to both promote content and addbest bets.Page Hits on pages deepin the hierarchyWhich team sitesshould be archived ordeleted?Sites that have not been accessed in the past 12 months might be candidates forarchival or deletion if the content is no longer useful.Page HitsWhat are the peak/lowusage times?Monitoring usage trends helps identify patterns or problems and potentially alertsthe Exchange Business Owner and Portal Administrator of potential user orperformance issues – ideally, before they become a problem.Usage by timeHow is usagetrending?Trending reports are available for a limited period of time within SharePoint 2010.Third-party tools are required to do multi-year detailed trend analysis.Number of users andnumber of unique usersover time
  46. 46. 46Sample System Metrics (“out of the box” SharePoint2010)Metric ObjectiveNumber of UniqueUsers (month tomonth)• Provides a proxy for adoption, which is a loose proxy for value.Most ViewedPages/Sites• Provides a proxy for the most valuable content.• Sites not being used help identify content that might either need to bepromoted or deleted.Top Queries(search)• Identifies ―trending ― content.• Top queries can also provide insights about what content should bepromoted to the home page.Failed Queries / NoResults Queries• Identifies candidates for best bets or synonyms and identifiesemerging business terms or concepts.Best BetSuggestion Report• Helps the business owner improve user outcomes by identifying URLsas most likely results for search queries.Best Bet Usage • Helps identify which best bets are adding value and as an input todetermine new best bets or best bets that need updating.
  47. 47. Additional Useful System Metrics (third-party for 2010)Metric ObjectiveMost Viewed Documents Provides a proxy for the most valuable content.Document Contribution/EditingAnalysisProvides a way to measure sustained adoption from theperspective of employee engagement.Team Site Summary InformationTotal Number of Team SitesViewed in Past 30 daysModified in Past 30 daysSites with no access in past 12monthsTrend of the number of team sitescreatedProvides a way to understand which sites are activelybeing used to monitor the health of the collaborativeteam sites.Can be used to identify which sites are no longer beingused and might be able to be deleted or archived.Provides a proxy for whether or not team sites areadding value.My Site Summary and TrendsTotal number of My SitesViewed in past 30 daysModified in past 30 daysAverage sizeIdentifies adoption of people-to-people collaborationfeatures.Proxy for employee engagement.47
  48. 48. 48System Metrics in SharePoint 2013 Online
  49. 49. 49One sample metric for each stage (more in WhitePaper)Life-cycleStageExample QuantitativeMetricExample QualitativeMetricSourcesPlanning • Time to perform currentprocess• "Day in the life" futurestories• Work measurementstudies• Interviews of keystakeholdersStart up • N/A • Immediate term ―day inthe life‖ stories• Employee surveysPilotConclusion• Same metrics you usedfor baselines• Usage anecdotes –specific examples frompilot• Follow up workmeasurement studies• Surveys and follow upinterviewsOngoing • Additional metrics relevantto the business problemavailable with newprocess• Usage anecdotes with a―serious‖ punch line thatyou collect and catalogueon an ongoing basis• New solution systemmetrics• Employee surveys andfollow up interviews
  50. 50. Objectives Critical Success Factors Source Sample MetricsGain frequent andsustained adoption ofsolution• High volume of needs that can’t be metthrough existing channels• Positive impact on existing workload orwork processes• System metrics• User Surveys• # of searches per week• # of average users per week• # unique users per week• # of “hits” on key pages/sites• “Usefulness rating” from user surveys• % of users who say “don’t take it away” at theend of the pilotProvide reliable,easy-to-usetechnology that canbe incorporated intowork processes• Solution user-friendliness and intuitiveness• Solution reliability• Integration of the solution with workprocesses and existing tools• System metrics• User Surveys• Direct measurement• “Usefulness rating” from user surveys• # of searches per week• # of average users per week• # unique users per week• # of “hits” on key pages/sites• # Help Desk calls/weekEnsure usersunderstandobjectives and how toleverage the solution• User training• Effective help resources• Persistent, clear communications• Active, sustained management support• Incorporation of collaboration intoperformance objectives and evaluations• System metrics• User Surveys• Direct measurement• % of users trained• % of pilot milestones achieved• # of communications events/activitiesDemonstrate clearvalue with respect tothe business strategy• Tangible, quantifiable examples ofreductions in process cycle time• “Serious” Anecdotescollected via surveys• Estimates and/or directmeasurement of cycletime• # of anecdotes• $ value of anecdotes• Cycle time improvement (in hours)BusinessValueSolutionHealthCapabilitiesBalancedScorecard Framework50
  51. 51. 51Survey Example: Post-Pilot UsabilityUsability Question MetricIf presented the choice, do you want to keep thesolution?―Don’t Take itAway‖Dont take it away Take it awayUsability/friendliness - how does the usability of thissolution compare to other solutions you use on aregular basis?―UserFriendlinessRating‖Much easier touseEasier to useAbout the sameHarder to useMuch harder to useHow easy and intuitive was the solution to use foreach of the following [specific task]?―IntuitivenessRating‖Very easyEasyModerateDifficultVery Difficult
  52. 52. Other ResourcesHow to Measure Anything: Finding the Value of Intangibles in Business byDouglas HubbardJakob Nielen’s Alert Box - Current Issues in Web Usability: the Value of Social Business ROI: Myths, Facts, and PotentiallyHigh Returns: Software for Business Performance: The missing link in social software:Measureable business process performance: Lifecycle Management Solution with Project Server 2010: