Improving Findability in the Enterprise
Upcoming SlideShare
Loading in...5

Improving Findability in the Enterprise



A presentation delivered at the 2013 APQC KM Conference in May 2013. Authored by Lee Romero and Stan Garfield

A presentation delivered at the 2013 APQC KM Conference in May 2013. Authored by Lee Romero and Stan Garfield



Total Views
Views on SlideShare
Embed Views



4 Embeds 25 18 5 1 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Improving Findability in the Enterprise Improving Findability in the Enterprise Presentation Transcript

  • Deloitte Touche Tohmatsu LimitedAPQC KM ConferenceMay 3, 2013Houston, TXImproving Findabilityin the Enterprise
  • © 2013 Deloitte Global Services LimitedAuthor1Lee Romero, Global Consulting Knowledge Management• 4 years with Deloitte• Leads SharePoint initiatives for Global Knowledge Services• Portal program manager for ConsultingAs used in this presentation, Deloitte means Deloitte Touche Tohmatsu Limited and its member firms
  • © 2013 Deloitte Global Services LimitedIntroduction2•Many organizations are implementing an enterprise search solution andenterprise social networking•They also hear frequently from users that “search doesn’t work”•This session will:o Provide insights about how enterprise social networking tools canimprove findabilityo Cover specific case studies of how social search is becomingmore prevalent and how enterprise social networking tools can beused to specifically improve the findability of content within anenterprise search solutiono Provide detailed insights on actionable analytics which canimprove search resultso Reveal the challenges you will face in understanding what yourusers are looking for and how you can work around thato Suggest how information architecture can be used to improvefindability
  • © 2013 Deloitte Global Services LimitedOverview3•Enterprise search can make large collections of content available tomore users and, in doing so, it makes it more and more difficult toensure that users can find what theyre looking for•Enterprise social networking tools bring more and more users togethervirtually and allow for simpler many-to-many conversations•At Deloitte, we have implemented an enterprise search solution•Our search covers over 1,000,000 separate targets, and that number isgrowing all of the time, both through organic growth of existingcollections and addition of whole new collections to the search•We have also implemented an enterprise social networking tool thatenables users across Deloitte to interact and post questions
  • Social search
  • © 2013 Deloitte Global Services LimitedSocial search5How our enterprise social networking tool has had the effect of improvingfindability of content by enabling our users to perform a social search –searching by asking questions of othersThis can be applied to the enterprise but in general here I will be talkingabout another kind of impact of social context to findabilityFrom Patrick Lambe’s forward to Heather Hedden’s “The Accidental Taxonomist”:This problem [of organization and retrieval of digital content collections] has largely beenaddressed on the internet through increasingly sophisticated search. Search can satisfythis need on the internet because it can exploit two unique features of the internet:1. Super availability of information means that useful content can be found for almostany purpose2. Social clues about useful content such as links and references can lift content thathas been recognized as useful to the top of search results
  • © 2013 Deloitte Global Services LimitedShare a link. “Here is a link to the latest Forrester Wavereport on social networking.”Ask a question. “Has anyone encountered this problembefore, and if so, how was it solved?”Find a resource. “Looking for a specialist in retirementbenefits to help win a bid in Calgary.”Answer a post. “Here are links to three relevant examples inthe project database.”Recognize a colleague. “Thanks to John Doe for hosting anexcellent planning session today.”Inform about your activities. “Will be in the Philadelphia officetoday; does anyone wish to meet?”Suggest an idea. “Local office TV screens should display theglobal conversation stream.”Recommend uses for Enterprise Social Networks
  • © 2013 Deloitte Global Services LimitedWays to do a social search7•Post in the all company stream•Post in a topic-specific group•Post in the knowledge help desk group
  • © 2013 Deloitte Global Services LimitedAll company8
  • © 2013 Deloitte Global Services LimitedTopic-specific group9
  • © 2013 Deloitte Global Services LimitedKnowledge help desk group10
  • © 2013 Deloitte Global Services LimitedExamples of queries and reaction to responses11•Any links to thought leadership articles on change management, talentand human capital in general?o Amazing! Thanks.•I am looking for an expert in the field of Enterprise Social Networks orWeb 2.0 technologies in a Knowledge Management context in general.o I already scheduled two additional interviews - thanks!•I am working on a proposal. Do you know of any cases where we havebeen responsible for coordinating interviews with the media and/oridentifying the proper spokesperson to speak with the media?o Thank you very much!
  • Crowdsourcing
  • © 2013 Deloitte Global Services LimitedCrowdsourcing13How the enterprise search program has used the enterprise socialnetworking tool to crowdsource targeted improvements in search resultsfor users for important and yet underperforming searches
  • © 2013 Deloitte Global Services LimitedCrowdsourcing improvements14A key process in improving any search solution is to be able to identify“problematic” search terms and then work to improve results for thoseterms.The next section on Analytics will discuss different ways to identify these“problematic” search terms for further analysis.Here, I will describe how we’ve been using our enterprise socialnetworking tool to help improve results.
  • © 2013 Deloitte Global Services LimitedIdentifying desirable targets15We have adopted a process to post on aregular basis (“Search term of the week”) witha specific search term.We ask for our users to collectively identifyand “vote” on specific resources that theyconsider the “best” for that term.The most commonly voted resources aredeemed as the best.This also obviously feeds into the manual“best bets.”
  • © 2013 Deloitte Global Services LimitedConfirming desirable targets16A similar process we’ve used as analternative is to pre-identify the resourcesand ask users to vote on those (or possiblysuggest alternatives).We adopted this as an alternative whensome terms elicited very little feedbackfrom users.We had a knowledge manager do someresearch on these terms to identifypotential targets (any existing best betswere also considered).
  • © 2013 Deloitte Global Services LimitedCrowdsourcing improvements – next steps17The remainder of the process, once “best” resources are identified forspecific terms is part of our standard process:• Use the resources as benchmarks for assessing relevancy of search(their position in the results)• Focus content optimization efforts on those resources (ensuring theyare indexed, they are well-titled, well-tagged, etc.)• Consider changes in weightings within the search engine to improvethe position of these results for the specified search terms• Work to actively move these resources up in the results (generally)
  • © 2013 Deloitte Global Services LimitedCrowdsourcing a strategy18Another specific way we used ourenterprise social networking tool wasduring a re-evaluation of our searchstrategy.We elicited input from our end users aswell as encouraging discussions aroundthe strategy.This approach engendered a significantconversation – some of which is shownhere.Many of the items and comments wereeventually incorporated into our newstrategy for 2015.
  • Analytics
  • © 2013 Deloitte Global Services LimitedAnalytics20A review of specific analytics that can provide useful insights on youruser behavior and support improvements to your enterprise searchsolutionYour search engine may provide some of theseYou may need to integrate your web analytics tool with your search toobtain some of theseAnd some may require you directly use the log file generated from yoursearch engine
  • © 2013 Deloitte Global Services LimitedBasic analytics21“Basic” analytics are ones that typically would be provided directly byyour search engine or analytics tool. They include:• Total searches• Top N searches• “Super search terms”• Not found searches / error searches / failed searches• Total distinct search terms
  • © 2013 Deloitte Global Services LimitedSome advanced analytics22These “advanced” analytics are typically ones that will require you to pulldata from multiple sources, or to process the data to obtain more in-depth understanding• Percent of visits to your portal / website that include a search• Pages of results viewed per search visit• Pages of results viewed for search terms• Words per search term• Categories of search terms
  • © 2013 Deloitte Global Services LimitedUnderstanding results usage23Another important view of your search usage is to understand the usageof the results. Some metrics to consider here include:• Most common result clicked for your top searches• Most commonly clicked on result across all searches• Percent of traffic for specific resources that is driven from search• Percent of results clicked on from “best bets”• Another view of “failed” searches• Percent of results clicked by page (1st vs. 2nd, etc.)
  • © 2013 Deloitte Global Services LimitedWhat kind of search do you support?24To effect meaningful changes in your search solution, I recommend youalso identify the power curve of your search logThis curve shows the percent of all searches performed against thepercent of distinct search terms usedThe steeper the curve, the moreconcentrated your searchesThe shallower the curve, themore exploratory your users areThis chart shows two searchsolutions I work with* I’ll have more on this topic in the next section
  • © 2013 Deloitte Global Services LimitedKey performance indicators25The metrics described here are not necessarily good as key performanceindicators for your search solution. They can be useful to manage yoursearch solution but will not mean much to others.The best key performance indicator would go above and beyond yoursearch solution – it would be to determine the percent of your users whoget to the information they need, regardless of whether they use search,navigation or both.That can be very hard to measure, but there are some indicators that youcan use from your search analytics that can give you directionalunderstanding.• Percent of visits to your portal that use a search• Average pages of results a user looks at in a search visit• Percent of searches performed where at least one result is clicked
  • The Challenge of your SearchLog
  • © 2013 Deloitte Global Services LimitedUnderstanding your search logFor enterprise search solutions1, the “80-20” rule is not trueThe language variability is very high in a couple of ways (covered inthe next few slides)Yet having a good understanding of the language, frequency andcommonality in your search log is critical to being able to makesustainable improvements to your search1 This does not seem to apply equally to e-commerce solutions27
  • © 2013 Deloitte Global Services LimitedSome facts about search termsThere’s an anecdote that goes something like, “80% of your searches arefrom 20% of your search terms”• Equivalently, some will say that you can make significant impact bypaying attention to a few of your most common terms (you can, but inlimited ways)Fact: in enterprise search solutions the curve can be much shallower:In the second case, it takes 13% of terms to cover 50% of searches, andthat is over 7000 distinct terms in a typical month!This chart shows theinverted power curve fortwo different solutionsI’m currently workingwith28
  • © 2013 Deloitte Global Services LimitedSome facts about search terms: part 2Another myth: a large percent of searches repeat over and over againFact: on enterprise search solutions, there is surprisingly littlecommonality month-to-monthOver a recent six month period, which saw a total of ~289K distinctsearch terms, only 11% of terms occurred in more than 1 month!# of months # terms % of searches1 257665 89.2%2 17994 6.2%3 5790 2.0%4 2900 1.0%5 2019 0.7%6 2340 0.8%29
  • © 2013 Deloitte Global Services LimitedSome facts about search terms: part 3Another myth: a good percentage of your search terms will repeat insequential periodsFact: There is much more churn even month-to-month than you mightexpect – in the period covered below, only about 13% of termsrepeated from one month to the next (covering about 36% ofsearches)30
  • © 2013 Deloitte Global Services LimitedWhat to do with your search log?The summary of the previous slides:• It is hard to understand a decent percentage of terms within a giventime period (month)!• If you could do that, the problem during the next time period isn’t thatmuch easier!The next sections describe a couple of research projects I’ve beenworking on to tackle these issues31
  • Information architecture
  • © 2013 Deloitte Global Services LimitedEngage a findability expert33•Assess your current information architecture and environment•Make concrete recommendations for improving findability•Possible expertsAndrew Hinton Peter Morville Lou Rosenfeld
  • © 2013 Deloitte Global Services LimitedMap topics to targets34•Compile a list of all important topics to the organization•Map these to the best existing sources and defineo Best betso Next best linkso Synonyms (“see also”)•Add too Searcho Navigationo Indexo Other alternatives•For topics lacking a current targeto Create a stub pageo Assign a page ownero Flesh out the page
  • © 2013 Deloitte Global Services LimitedDevelop an information architecture35•Include all instances in your enterprise ofo Intranetso Knowledge assetso Collaboration tools•Design effective navigation to reacho Siteso Pageso Contento Discussionso Profiles
  • © 2013 Deloitte Global Services LimitedProvide multiple alternatives for users to find content36•Search•Navigation•Chat•Voice•Email•Social search•Indexes•Guides•Top 10 lists•Most popular lists•Multiple user categories•User assigned tagging (social bookmarking)•Facets•Filters
  • © 2013 Deloitte Global Services LimitedDevelop a user-centered design37•Include all instances in your enterprise ofo Siteso Pageso Contento Discussionso Profiles•Specify required information, such aso Date last modifiedo Contact nameo “Like” buttono Tagging•Use this to provide a consistent user experience
  • © 2013 Deloitte Global Services LimitedKeep content current38•Develop a process too Reviewo Refresho Remove•For allo Siteso Pageso Contento Discussionso Profiles•Automate detection ofo Broken linkso Orphan siteso Sites not recently updated
  • © 2013 Deloitte Global Services LimitedEnable user participation39•Evolve from static intranet pages to dynamic social interaction•Allow users to help manage the information ecosystemo Social bookmarkingo Taggingo “Liking”o Commentingo Recommending content for reuse or removal•Integrateo Intranetso Knowledge assetso Collaboration tools
  • Understanding yourusers’ informationneeds
  • © 2013 Deloitte Global Services LimitedCategorizing your users’ language•Given the challenges previously laid out, using the search log tounderstand user needs seems very challenging•Beyond the first several dozen terms, it is hard to understand what usersare looking for‒And those several dozen terms cover a vanishingly small percentage of allsearches!•However, it would be very useful to understand your users’ informationneeds if we could somehow understand the entirety of the search log•How do we handle this? Categorize the search terms!41
  • © 2013 Deloitte Global Services LimitedCategorizing your users’ language, p2•So we need to categorize search terms to really be able to understandour users’ information needs.•To do this, we face two challenges1. What categorization scheme should we use?2. How do we apply categorization in a repeatable, scalable and manageableway?•For the first challenge, I would recommend you use your taxonomy•The second challenge is a bit more difficult but is addressed later in thisdeck42
  • © 2013 Deloitte Global Services LimitedCategories to use•Proposal: Start with your own taxonomy and its vocabularies as thecategories into which search terms are grouped•Some searches will not fit into any of these categories, so you cananticipate the need to add further categories•This exercise actually provides an excellent measurement tool for yourtaxonomy‒You can quantitatively assess the percent of your users’ language that isclassifiable with your taxonomy‒A number you may wish to drive up over time43
  • © 2013 Deloitte Global Services LimitedAutomating categorization•Now we turn to the hairier challenge – how can we automaticallycategorize search terms?•To describe the problem, we have:1. A set of categories, which may be hierarchically related (most taxonomies are)2. A set of search terms, as entered by users, that need to be assigned to thosecategoriesCategoryCategoryCategory.........CategoryCategoryCategoryCategorySearch TermSearch TermSearch TermSearch TermSearch Term...??44
  • © 2013 Deloitte Global Services LimitedAutomating categorization, p2•The proposed solution is based on a couple of concepts:1. You can think of this categorization problem as search!2. You are taking each search term and searching in an index from which theset of results is the set of categories•Question: What is the “body” of what you are searching?•Answer: Previously-categorized search terms!•Using this approach, you can consider the set of previously-categorizedsearch terms as a corpus against which to search‒You can apply all of the same heuristics to this search as any search:•Word matching (not string matching)•Stemming•Relevancy (word ordering, proximity, # of matches, etc.)45
  • © 2013 Deloitte Global Services LimitedAutomating categorization, p3•Here’s a depiction of this solutionCategoryCategoryCategory.........CategoryCategoryCategoryCategorySearch TermSearch TermSearch TermSearch TermSearch Term...PreviouslycategorizedtermsPreviouslycategorizedtermsPreviouslycategorizedterms46
  • © 2013 Deloitte Global Services LimitedAutomating categorization, p4: Bootstrapping•This approach depends on matching to previously-categorized terms‒Every time you categorize a new search term, you expand the set ofcategorized terms, enabling more matches in the future•Bootstrapping: You can take the names of the categories (the terms inyour taxonomy) as the first set of “categorized search terms”‒This allows you to start with no search terms having been categorized at all‒You run a first round of matching against the categories to find first-levelmatches‒Take those that seem like “good” matches and pull those into the set ofcategorized search terms for a second iteration, etc.‒Using this in initial testing resulted in 10% of distinct terms from a month beingassociated with at least one category•Another aspect: Any manual categorization of common search terms willadd to the success of categorization47
  • © 2013 Deloitte Global Services LimitedAutomating categorization, p5: IterativeCategoryCategoryCategory.........CategoryCategoryCategoryCategorySearch TermSearch TermSearch TermSearch TermSearch Term...PreviouslycategorizedtermsPreviouslycategorizedtermsPreviouslycategorizedtermsNew categorizationsNew categorizationsNew categorizations
  • © 2013 Deloitte Global Services LimitedAutomating categorization, p5: Iterative•This approach also needs to be applied iteratively‒You start with a set of categorized search terms and a new set of(uncategorized) search terms‒You then apply this matching to the uncategorized search terms, getting a setof newly-categorized search terms (with some measure of probability of“correctness” of the match, i.e., relevancy)‒You pull in the newly-categorized search terms and run the matching processagain‒Each time, as you expand the set of categorized search terms (from aprevious match), you increase the possibility of more matches (in subsequentmatches)49
  • © 2013 Deloitte Global Services LimitedAutomating categorization, p6: Iterative•It will be beneficial to have a human review the set of matches for eachiteration and determine if they are accurate enough‒The measurement of relevancy is intended to do this but would likely only bepartially successful•Over time, using this process, you build up a larger and larger set ofcategorized search terms‒This makes it more likely in future iterations that more terms will becategorizable50
  • © 2013 Deloitte Global Services LimitedAutomating categorization, p7: No matches•There will always be search terms that do not get matched.‒This may be because the terminology used does not match‒This may be because there are no categories in the global taxonomy thatwould be useful for categorization•The first issue would require a human to recognize the association(thus, categorizing the term and then enabling matches on future uses ofthat term)•The second issue would require adding in new categories (not part ofthe global taxonomy)‒And then categorizing the term into the newly-added category(ies)51
  • © 2013 Deloitte Global Services LimitedSummary•With this approach, we can take a set of search terms at any time andcategorize them (partially) automatically‒Over time, the accuracy of the matching will improve through human review-and-approval of matches•We then are able to relate these information needs to a variety of otherpieces of data:‒Volume of content available to users – significant mismatches can highlightneed for new content‒Rating of content in these categories – can highlight that a particular area ofinterest has content but it isn’t quality content‒Usage of content in these categories – could highlight navigational issues(e.g., when a category is much more highly represented in search than indownloads)•This does not require directly working with end-users and is scalable52
  • © 2013 Deloitte Global Services LimitedAdditional benefits: Measuring your taxonomy•As mentioned earlier, part of the challenge will be that there will beterms that do not match the starting categories (i.e., the globaltaxonomy)•This actually highlights some valuable insight obtainable from this:‒We can identify gaps in our taxonomy (terms requiring new categories)‒We can identify areas of our taxonomy where we have many search termsassociated with a taxonomy term and consider if we need to either add orsplit search terms in order to better match our users’ real language‒We can identify areas of the taxonomy that are of little use in terms of thelanguage used by our users53
  • © 2013 Deloitte Global Services LimitedAdditional benefits: What are key words for your users?•Word counts – independent of term usage,what are the most common individualwords?•Word networks – we can understandthe inter-relationships betweenindividual words (which pairsoccur commonly together,which words occur commonlyfor a given word)These are not as much about information needs as about understanding thelanguage users use (so this insight can help shape categorization)Word Distinct Terms Searchesmanagement 3128 8283sap 1931 3873strategy 1414 3728business 1558 3599it 1343 2992process 1515 2920data 1264 2899project 1249 2823model 1296 2791plan 987 217054
  • Lessons learned
  • © 2013 Deloitte Global Services LimitedLessons learned56Tryalternatives• Ask differentgroups• Try differentquestions• Offer the answerand ask peopleto poke holes initRegularlysearch• Put yourself inyour users’shoes• See the realresults• Work to improvekey searches –invest in this!Use outsiders• Users• ExpertsYour search log• Work onunderstandingthe languageyour users areusing• Go beyond the“short head”
  • © 2013 Deloitte Global Services LimitedMore resources57•For more about search and enterprise search, I recommend•“Search Patterns: Design for Discovery” by Peter Morville andJeffery Callender•“Enterprise Search” by Martin White•“Search Analytics for your Site” by Louis Rosenfeld•For more information about information architecture, I recommend•“Information Architecture for the World Wide Web: Designing Large-Scale Web Sites” by Peter Morville and Louis Rosenfeld•Some web sites or blogs of potential interest• – edited by Avi Rappoport• - my own writings on this topic (and many othersaround content and collaboration)
  • © 2013 Deloitte Global Services LimitedDeloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network ofmember firms, each of which is a legally separate and independent entity. Please see about for a detailed descriptionof the legal structure of Deloitte Touche Tohmatsu Limited and its member firms.Deloitte provides audit, tax, consulting, and financial advisory services to public and private clients spanning multiple industries. With aglobally connected network of member firms in more than 150 countries, Deloitte brings world-class capabilities and high-quality serviceto clients, delivering the insights they need to address their most complex business challenges. Deloitte’s approximately 182,000professionals are committed to becoming the standard of excellence.This presentation contains general information only, and none of Deloitte Touche Tohmatsu Limited, its member firms, or their relatedentities (collectively, the “Deloitte Network”) is, by means of this presentation, rendering professional advice or services. Before makingany decision or taking any action that may affect your finances or your business, you should consult a qualified professional adviser. Noentity in the Deloitte Network shall be responsible for any loss whatsoever sustained by any person who relies on this presentation.