1. IBM Total Information ExperienceDefining and Evaluating Success:Metrics and Metrics Frameworksfor Information ArchitectsAndrea L. AmesInformation Experience Strategist/Architect/DesignerAlyson RileyTIE Content StrategistSTC Summit07 May 2013© IBM Corporation 2013. All Rights Reserved
2. IBM Total Information ExperienceAbout AndreaTechnical communicator since 1983Areas of expertiseInformation experience design: Content strategy,information architecture, and interaction designfor content display and delivery, within productsand interactive information delivery systemsArchitecture, design, and development of embedded assistance(content within or near the product user interface)Information and product usability, from analysis through validationUser-centered process for information development andinformation experience designIBM Senior Technical Staff Member on corporate Total InformationExperience team in IBM CIO’s officeUniversity of CA Extension certificate coordinator and instructorSTC Fellow, past president (2004-05), former member ofBoard of Directors (1998-2006), and Intercom columnist (with AlysonRiley) of The Strategic IAACM Distinguished Engineer 2
3. IBM Total Information Experience3About AlysonTechnical communicator since 1995Areas of expertiseContent strategyContent metrics—the business value of contentInformation architecture (my first love!)Interaction design for content delivery vehicles,and interactive contentInformation and product usability, from analysisthrough validationUser-centered processes for content strategyand scenario-driven information architectureIBM Senior Content Strategist on corporateTotal Information Experience team in the IBMCIO OfficeMember of STC, and Intercom columnist (withAndrea Ames) of The Strategic IA
4. IBM Total Information ExperienceAgendaTelling the right story through metricsTelling the right story to the right audienceThe importance of a closed-loop metrics processEvaluation frameworksBuilding an evaluation frameworkQuestions
5. IBM Total Information ExperienceJared Spool says it SO well…Metric: Something you can measureThe time a user spends on a Web siteWhether a particular Web page has a high“bounce rate” (indicates it’s often the lastpage users visit)How much revenue a Web site makesHow many times the letter E is used on a Web site home pageAll of these are measurable…BUT…being measurabledoesn’t make those measures informative or useful
6. IBM Total Information ExperienceWe care, because…An information architect is (among other things)a story-teller:Define the right visionTell a compelling, true story that inspires people to buy inWhat makes a story true? Facts—things you can prove.What makes a story compelling? It speaks to what matters most.What matters most? Depends on your audience. Duh, right?Prove the value of information architecture and content with metricsValue is in the eye of the beholder.Who’s your “beholder?” Understand who your beholders actually are—that is, the real decision-makers and influencers in your world. (Manageyour stakeholders!)Use metrics that target actual decision-makers.Your actual decision-makers are probably business people—executives,managers, and others who hold the purse-strings.Figure out what your audience values—their metrics for success.
7. IBM Total Information ExperienceProblem: Metrics have gotten a bad rapNumbers can be hard for word peopleThe right numbers are hard for everyoneGetting metrics to work for you requires a significant shift in thinkingSolution: Rethink metricsMetrics are another form of audience analysis (who cares about what?)Metrics are another form of usability testing (what works for whom?)Motivation for change: Metrics are a powerful tool for getting what you want(and making sure you want the right things)Metrics transform opinion into factMetrics remove emotion from analysisStrategize with metrics: Use metrics at every project phaseBeginning: identify opportunity, prove the strategy is rightMiddle: show incremental progress, course-correctEnd: prove value and earn investment for the future
8. IBM Total Information ExperienceBe sure you’re talking to the right audienceAudience 1: Business peopleUnless you can make a direct connection between your IA metrics and themetrics that drive business, you are telling the wrong story for thisaudience.You need this audience! The business community funds us. We have tosell our vision to them, with a metrics story that resonates with them.We must learn to speak “business”—that is, prove the value of content andthe information experience using metrics that matter to business.Audience 2: Content peopleTypically many kinds of content people will help implement an informationarchitecture—your work may span departments and business units.Content people tend to reflect the values of their leadership and businessunit in which they’re located.This means that even kindred spirits—other content people—can havewidely different goals and metrics.Your job is to define common ground by speaking to what matters most tothis audience, too.
9. IBM Total Information ExperienceSelling information architecture to a business audienceThe metrics that we use to build effective contentstrategies don’t resonate with most executives,managers, and finance people. Sometimes we “talk to ourselves”—that is, usemetrics that resonate with content people, not theactual people we need to support our strategy. “Page hits” resonate with us. “Sales leads”resonates with business.You cannot directly connect things like page hitsand bounce rates to core business metrics.You need an informational professional’s intuitionto know how content supports business metrics—most business people don’t have that intuition.The business audience funds us. We have to sellour vision and prove our value to them, with ametrics story that speaks to what they care aboutmost.Examplebusiness metrics:Revenue streamsSales leadsCost per leadCustomer satisfactionCustomer loyaltyReturn on investment (ROI)Time to valueMarket shareMindshare
10. IBM Total Information ExperienceSelling content strategy to a content audienceAnalyze each team that contributes to your information architectureIn what business unit are they located?Who are their executives, sponsors, and stakeholders?Who “grades them” on their performance?Who funds them?What matters to them?How do they measure their progress or results?What are they doing well (both in your analysis and theirs)?Where can they improve (both in your analysis and theirs)?Identify areas of similarity and differenceWhere do their goals align with yours? build bridges!Where do their goals conflict with yours? build business cases!Use metrics to craft a story that:Shows problems and opportunities that the content team cares aboutMaps in key areas to their goals for contentDiverges from their current goals in ways that would increase their value to sponsorsand stakeholders
11. IBM Total Information ExperienceCritical: A closed-loop processClosed loop: end up at the beginning!Start with metrics—use at project outset to:Identify problems and opportunitiesDefine the visionProve that the vision is rightContinue with metrics—use during implementation to:Measure the success of your progress in small incrementsStay on-target through implementationDetermine when it’s time to course-correct (before change gets expensive)Keep your sponsors and stakeholders engaged throughout the long haulEnsure that you remain connected to the broader goals and metrics of thesurrounding businessEnsure that you stay responsive and adapt to changeEnd with metrics—use at project conclusion to:Prove the business value of information architectureProve the business value of your work—enhance your credibility and careerEncourage future investment in information architecture
12. IBM Total Information ExperienceHow do you close the loop?Measure what you’ve said you’ll measure, of course…Build a framework, aka, scorecard, to evaluate howyou’re doingBefore you begin: Identify and begin managing stakeholders!1. Identify what to measure and collect, and determine types *2. Normalize “results” to “scores” *3. Categorize and weight metrics *4. Create summaries *5. Validate the framework *6. Launch! **Stakeholder-management point
13. IBM Total Information ExperienceBefore you begin: Identify and begin managing stakeholdersYour best political asset—your stakeholders!A rigorous stakeholder management process will help you takerigorous advantage of this key assetThink through the ways that your stakeholders can help you—start by identifying and analyzing:Their status relative to your project—advocate, supporter, neutral,critic, blockerTheir top interests and hot issuesTheir key performance indicators (KPIs) and metricsThe level of support you desire from themThe role on your project that you desire for themThe actions that you want them to take (and their priority)The messages that you need to craft for them to enable theoutcome you wantThe actions and communication that you need to make happenwith each stakeholder to achieve your desired outcomeKeep your stakeholder management plan current“Stakeholdermanagement iscritical to thesuccess of everyproject in everyorganization … Byengaging the rightpeople in the rightway in your project,you can make a bigdifference to itssuccess...and to your career.”—Rachel ThompsonSource and freestakeholdermanagementworksheet here:Thompson, Rachel.StakeholderManagement:PlanningStakeholderCommunication.MindTools. Web.12 April 2013.http://bit.ly/8UnUdj
14. IBM Total Information Experience1. Determine what to measure by considering:– Who are your stakeholders?– What is important to them?– What are your goals?– What will best help you tell your story to get yourstakeholders aligned around your goals?1. Determine type1Identify what to measure, and determine types
15. IBM Total Information ExperienceMetrics for an audience of business people (stakeholders)Use the research you didduring the today-state analysisphaseTarget the key decisions-makers—those who hold thepurse-stringsIdentify what the key businessdecision-makers care aboutUse language that resonateswith that business audienceRemember: unless you can tiea particular goal or result to ameasurement that thestakeholder cares about, thatresult ultimately doesn’t matterStakeholder Example metricsMarketingExecutive ROI Cost per lead Campaign performance Conversion metricsSalesExecutive Viable leads Sales growth Product performanceSupportExecutive Call volume Call length Customer satisfaction Ticket deflectionDevelopmentExecutive Development costs Market share Lines of code Compliance Quality and test results
16. IBM Total Information ExperienceMetrics for an audience of content peopleMap thepeople whodelivercontent to themetrics theycare aboutRememberthat eachcontent teamhas their owndecision-makers who:Approve theirgoalsDeterminetheir fundingDeterminetheir futuresStakeholder ExamplemetricsExample associatedcontent teamsExamplecontent metricsMarketingExecutive ROI Cost per lead Campaignperformance Conversionmetrics Web team Social team Event team Web traffic Click-throughs Likes and shares Conversions Collateral distributed Cost per unit producedSalesExecutive Viable leads Sales growth Productperformance Sales enablement Education & training Beta programs Proofs of Concept(PoCs) to sale Number of classes Beta programparticipants Cost per unit producedSupportExecutive Call volume Call length Customer sat. Ticketdeflection Web support team Call center team Amount of webinformation produced Number of calls reduced Time of calls reduced Cost per unit producedDevelopmentExecutive Dev cost Market share Lines of code Compliance Quality and test Productdocumentation team Developers whopublish whitepapersand case studies Product communityforums and wikis Lines of text, number ofpages, etc. Cost per unit produced Web traffic Number of forumparticipants
17. IBM Total Information ExperienceMetrics-based goalsSo what are the goals for your information architecture? Express those goals inthe form of business metrics and content metrics. Some examples:Business metrics Sample IA metrics Sample content goalsPurchase decisions(revenue) Reach—visits, etc. Engagement—referrals,etc.Contribute to revenue streamthrough referrals fromtechnical content that becomesales leads.Product quality(customer loyalty) Reach—visits, etc. Engagement—referrals,etc.Contribute to product qualitythrough by simplifying theamount of content in the userexperience.Customer satisfaction(ROI) Web traffic Direct feedback Ratings Shares (social)Create high value content thatspeeds customer time tosuccess.Perceptions ofcompany (mindshare) Sentiment—nature ofsocial dialogue, etc. Direct feedbackCreate high quality, highlyusable content delivered in anelegant informationexperience.
18. IBM Total Information ExperienceWhat types of metrics are you measuring?QuantitativeDescribe the what, or how many of the whatCan be measured with numbers—absolutely, mathematicallyExamples: See the last three charts QualitativeDescribe intangibles, like the whyNon-numericalExamples: Sentiment, how important something is, how much therespondents like something, how likely they are to recommendCompliance itemsQuantitative—you can count themBut you’re only counting “1”More important is the perceived value of that “1”Examples: Models applied, templates used, processes followed
19. IBM Total Information ExperienceA simple example: RunningQuantitativeTimeDistanceHeartrateComplianceDopingQualitativeHow I feel at start of runHow I feel at end of runHow the weather conditionswere
20. IBM Total Information Experience1. Evaluate qualitative metrics – are scales the same2. Evaluate quantitative metrics – how good is good3. Evaluate compliance items – how important is it4. Select a scale• Numerical• Good/bad• High/med/low (e.g., priority)• Crawl/walk/run Make room for an “innovation” or “exemplary” level—the“top” of your scale should be attainable by everyone!Consider:Do teams have a way to be successful from wherethey are? (e.g., can they reasonably get to crawl?)Will you be able to measure incremental improvement2Normalize results to scores
21. IBM Total Information ExperienceQualitative metrics: Are the result scales thesame?How I feel at start of runHorrible OK Good Great!How I feel at end of runHorrible OK Good Great!What the weather conditions were*Freezing Cold Cool Warm/humid Hot/humid* More accurate to measure temperature and humidity as quantitative metrics
22. IBM Total Information ExperienceQualitative metrics: Better!How I feel at start of runHorrible OK Good Great!How I feel at end of runHorrible OK Good Great!What the weather conditions were*Horrible OK Good Great!(Hot/humid or freezing  cool/dry)* More accurate to measure temperature and humidity as quantitative metrics
23. IBM Total Information ExperienceQuantitative metrics: How good is “good?”TimeBeginner: 15 minute mileIntermediate: 11 minute mileAthlete: 8 minute mileDistance: % of goal25-50%50-75%75-100%HeartrateBelow fat-burning zoneFat burning zoneAerobic zone
24. IBM Total Information ExperienceQuantitative metrics: NormalizedTimeHorrible (implied)  More than 15-minute mileOK  Beginner: 15 minute mileGood  Intermediate: 11 minute mileGreat  Athlete: 8 minute mileDistance: % of goalHorrible (implied)  Less than 25%OK  25-50%Good  50-75%Great  75-100%Heart rateHorrible (implied)  No heart rate OK  Below fat burning zoneGood  Fat burning zoneGreat  Aerobic zone
25. IBM Total Information Experience1. Determine what to measure by considering:– Who are your stakeholders?– What is important to them?– What are your goals?– What will best help you tell your story to get yourstakeholders aligned around your goals?1. Use your IA brain; this is a classic IA problem. 3Categorize and weight scores
26. IBM Total Information ExperienceA possible categorization schemeThink about how youwant to tell your story formost impactWhat is your clients’product-use life cycle?Other possible schemesBy function (to makespecific business cases tothem)By “standard” (tounderstand specificcompliance issues)A generalized view ofIBM’s product-use lifecycle
27. IBM Total Information Experience1. Determine what summaries you need to tell theright story at a glance• For management• For the larger organization• For the team being measured1. Be careful not to create an unwieldy mess 4Create a summaries
28. IBM Total Information ExperienceThis could be an entire session by itself!
29. IBM Total Information Experience1. Find as many diverse “products” as possible tomeasure.2. Gut check with several “independent” IAs• Individual metrics• Categories• The whole1. Compare results across products forconsistency2. Compare results to uncover gaps: Is somethingnote being measured that should be?3. If possible, find other frameworks to compare toyour approach—not necessarily an IAframework…maybe a user experience orsupport framework5Validate framework
30. IBM Total Information Experience1. Determine how the framework will be used• As a self-evaluation tool?• In an objective, outside evaluation?• Do users need to be “certified” to use it? Orevaluators to evaluate with it?1. Communicate!2. Educate!3. Consult?6Launch!
31. IBM Total Information ExperienceQuestions?
32. IBM Total Information ExperienceReferencesJared Spool: KPIs are metrics, but not all metrics are KPIs from UIEBrainsparks blog, Oct 5, 2012: http://bit.ly/VFYvF2Bhapkar, Neil. 8 KPIs Your Content Marketing MeasurementsShould Include. Content Marketing Institute. Web. 12 April 2013.http://bit.ly/Wnb7CyKlipfolio. The KPI Dashboard—Evolved. Web. 12 April 2013.http://bit.ly/LhzeL9Muldoon, Pamela. 4 metrics every content marketer needs tomeasure: Interview with Jay Baer. Content Marketing Institute.Web. 12 April 2013. http://bit.ly/X8IvMJThompson, Rachel. Stakeholder management: Planningstakeholder communication. MindTools. Web. 12 April 2013.http://bit.ly/8UnUdj
33. IBM Total Information ExperienceBackupIBM Confidential© IBM Corporation 2013. All Rights Reserved
34. IBM Total Information ExperienceProving the business value of content—IBM exampleShamelessad:Watch for theMay issue ofSTC’s Intercommagazine for anew article thatwe wrote onproving thebusiness valueof content.At IBM, we’re learning to tell a better story for a business audienceWe conducted a survey from 2010-2012 with clients and prospectiveclients about the value of content—here’s the hot-off-the-press data:
35. IBM Total Information ExperienceA map of content metrics to stakeholder metricsTie your IA metrics to the metrics that matter most to your stakeholdersso you can tell a story that inspires the outcomes you want.This means researching how content influences the metrics that aremost important to the specific people you need for success.Start your research with these hints:How does content speed usersuccess and time-to-value?direct link to customer valueHow does content drivepurchase decisions?direct link to the revenue streamHow does content impactproduct quality?direct link to customer loyaltyHow does content influencecustomer satisfaction?direct link to ROIHow does content shape clients’perceptions of yourcompany?direct link to mindshare
36. IBM Total Information ExperienceEvaluation frameworks: Heuristics and heuristic evaluationsHeuristicsRules of thumbNo one single “rule” or set of “rules”Heuristic evaluationsInspection methodPerformed by “experts”Based on heuristicsBoth: Based on principles derived from observing andanalyzing user actions and intentionsThese principles enable development of your ownevaluation framework, based on your own user data