Deloitte's search solution indexes approximately 130,000 documents and sees 1.7 million searches per year. To manage the solution, Deloitte dedicates about half an FTE to search analytics and configuration. The document recommends five practices for actively managing enterprise search: robust governance; analyzing and acting on metrics; routine maintenance; evaluating user perception; and communicating progress.

1. © 2011 Deloitte Global Services Limited
Search quality in real world
Search functionality + Active management
= Customer satisfaction
Lee Romero and Ankit Desai
November 2011

2. © 2011 Deloitte Global Services Limited
Deloitte’s search environment *
A brief review of our search solution:
To manage this solution, we resource approximately 1 FTE toward search:
• About 50% of one search analyst who:
• Gathers and reviews search usage data and recommends actions
• Collects metrics for leadership reporting
• Performs routine configuration activities like best bets, etc.
• Smaller percentages of Ankit and Lee’s time
2
Documents indexed ~130,000
Total searches ~ 1.7M searches/year
Total visits to search ~ 425,000/year
* As used in this document, “Deloitte” means Deloitte Consulting LLP, a subsidiary of Deloitte LLP. Please see
www.deloitte.com/us/about for a detailed description of the legal structure of Deloitte LLP and its subsidiaries.

3. © 2011 Deloitte Global Services Limited
Why active management for search?
“There is nothing permanent except change.”
- Heraclitus of Ephesus (c.535 BC - 475 BC)
Enterprise search is no different. After search is launched, many factors
could change:
• User information needs
• User language
• Organizational structure
• Vendor products
• Functionality requirements
• New classes of users may be added
• New data could be indexed
• And, the list goes on …
All of the above necessitates continuous maintenance of the search
platform.
3

4. © 2011 Deloitte Global Services Limited
Active management of search
4
The following five leading practices have worked for us:
1. Robust governance
2. Analyzing and acting on search metrics
3. Routine search maintenance
4. Evaluating end-user perception – search survey and usability
5. Communicating “wins” and “issues”

5. © 2011 Deloitte Global Services Limited
#1
Robust
governance

6. © 2011 Deloitte Global Services Limited
Robust governance
Recommendation: Provide appropriate governance and actively maintain
that governance.
Deloitte search governance and process documentation covers:
• Management of best-bet terms and targets
• Selection of keywords for content items
• Management of synonyms/thesaurus
• Ways to improve relevancy on an ongoing basis
We recommend reviewing your governance documentation periodically so
it reflects your users and business needs at all times.
6

7. © 2011 Deloitte Global Services Limited
#2
Analyzing and
acting on
search metrics

8. © 2011 Deloitte Global Services Limited
Metrics we track
Recommendation: Ensure you are capturing metrics, regularly reviewing
key measures, and acting on those measures.
Search metrics show what users are looking for in the system.
We track several metrics on a monthly and quarterly basis, such as:
8
Total searches Saved search usage
Visits that include Search Fallout rate
Searches/visitor Zero result searches
Searches/search visit Overall and individual facet usage
Distinct best-bet terms and targets Best-bet usage

9. © 2011 Deloitte Global Services Limited
Act on search metrics
There are many ways to act on search metrics:
• Look for trends on a periodic basis. If a particular metric changes
significantly for more than two time periods, strive to understand the
deeper reasons behind those changes.
• Study the impact of these metrics across other aspects of your
information access strategy. We look at potential navigation,
taxonomy, and content changes among other things.
• Look for any potential UI changes that you can make to search
interface based on these metrics.
9

10. © 2011 Deloitte Global Services Limited
#3
Routine
search
maintenance

11. © 2011 Deloitte Global Services Limited
Routine search maintenance
Recommendation: Define some standard maintenance tasks and plan to
spend time executing against those tasks on a regular basis.
We perform regular maintenance in a number of areas, including:
• Actively configuring new best bets for content items
• Actively managing keywords for content items
• Review weights of the content sources and adjusting ranking of those
sources (this is engine-dependent)
• Reviewing priority of individual targets and adjusting accordingly
• Identifying zero-return searches and analyzing them
• Evaluating relevancy of highly used search terms (a.k.a. super search
terms) and improving relevancy
• Reviewing highly used content items and making them more accessible
11

12. © 2011 Deloitte Global Services Limited
#4
Evaluating end-
user perception

13. © 2011 Deloitte Global Services Limited
Search surveys
Recommendation: Spend time interacting with your users asking for their
feedback.
We survey our end users annually. Doing so:
• Allows us to identify trends on a yearly basis, understand what users really
want, and understand what they think of search.
• Helps us plan search improvements for the year.
Our search survey covers:
• What do users like most about search? Which features, what content?
• What do users not like about search? And why?
• How do users rate the search UI?
• How do users rate search relevancy?
13

14. © 2011 Deloitte Global Services Limited
Search usability
Usability testing of the search implementation is a good tool to evaluate the
search user interface and to understand how users use search in “real life.”
Recently, we conducted a usability study with a wide range of participants
on a one-on-one basis.
The tester was given a set of real-life scenarios by
evaluator to execute on the search interface.
Observer noted the tester’s response and
recorded key findings.
Based on this study, we made a number of changes to the UI and to the
functionality and content.
14

15. © 2011 Deloitte Global Services Limited
#5
Communicating
your “wins” and
“issues”

16. © 2011 Deloitte Global Services Limited
Communication is the key
Recommendation: Establish an effective communication plan for your
search program.
Through all of the recommendations previously mentioned, we regularly
make improvements to our search solution, but without communication
they will go largely unnoticed.
We recommend that you communicate your wins in improving search and
the issues you uncover.
Communication is key because it:
• Helps raise visibility within the firm
• Helps gain support from other teams within the firm, when that is
required to improve search
• Garners executive support
16

17. © 2011 Deloitte Global Services Limited
Conclusion

18. © 2011 Deloitte Global Services Limited
Conclusion – Key takeaways
• Improving search should be part of your information access strategy not
just another operational activity
• Have a dedicated resource or a team to deal with search
• It all starts with the governance – make it as robust as possible, and
update it when necessary
• Define a set of metrics that you want track on a regular basis. Identify
trends and understand deeper reasons, when required.
• Identify a set of search activities that you will perform on a regular basis
• Understand end-user perception using search surveys and search
usability studies
• And, most importantly, communicate your “wins” as well as “issues”
18

19. © 2011 Deloitte Global Services Limited
As used in this communication, ‘Deloitte’ means Deloitte Touche Tohmatsu Limited and its member firms.
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms,
each of which is a legally separate and independent entity. Please see www.deloitte.com/about for a detailed description of the legal structure of
Deloitte Touche Tohmatsu Limited and its member firms.
Deloitte provides audit, tax, consulting, and financial advisory services to public and private clients spanning multiple industries. With a globally
connected network of member firms in more than 150 countries, Deloitte brings world-class capabilities and high-quality service to clients,
delivering the insights they need to address their most complex business challenges. Deloitte's approximately 182,000 professionals are committed
to becoming the standard of excellence.
This publication contains general information only, and none of Deloitte Touche Tohmatsu Limited, its member firms, or their related entities
(collectively, the “Deloitte Network”) is, by means of this publication, rendering professional advice or services. Before making any decision or
taking any action that may affect your finances or your business, you should consult a qualified professional adviser. No entity in the Deloitte
Network shall be responsible for any loss whatsoever sustained by any person who relies on this publication.