1. Search Quality Management
Agnes Molnar
Enterprise Search Consultant, Founder
Search Explained

2. Introduction – Agnes Molnar
 8 years MVP (SharePoint Server & Office 365)
 Enterprise Search, Information Architecture
 Community Leader
 Enterprise Search Consultant, Founder
 Worldwide Consultancy, Guidance, Trainings,
Workshops
 Web: http://SearchExplained.com
 Yammer: http://yammer.com/SearchExplained
 Twitter: @molnaragnes / @SearchExplained
 E-mail: agnes.molnar@SearchExplained.com

3. Agenda
 Enterprise Search – Real World Challenges
 What is Enterprise Search?
 Quality of Enterprise Search
 Action Plan

4. Enterprise Search – Real World
Challenges

5. Search is Easy!?

6. Find is the REAL Challenge!

7. Search = Users + Content + Context
Content
UsersContext

8. Common Requirements
 ‘I want Google-like Search in my Enterprise’
 ‘I want it just work.’
 ‘We have 666 million documents. I want to be able
to find everything.’

9. Formal Requirements
Business
goals
Scenarios &
use cases
Application
Lifecycle
Systems to
integrate
User
Experience
Metrics
Milestones Resources Experts

10. Enterprise Search – IT Project?
 Content evolves
 Users evolve
 Context evolves
 Search has to evolve, too!
 Add/Modify content source
 Add/Modify/Remove New metadata
 Modify/Improve UI
 New Search-Driven Applications
 Obsolete content
 …
Content
UsersContext

11. Enterprise Search – Business
Process!
Planning
ImplementationAnalysis

13. What Makes a Good
Enterprise Search?

14. Search Quality
 Quality of Results
 Relevance of Results
 Quality of User Experience
 Supportability and Maintainability
Quality of Information Architecture!!!

15. Quality of Results
 Various Content Sources
 Various Types of Content
 Structured and Unstructured Data
 Security is critical
 Metadata is essential
 User tagging
 Auto-tagging
 Advanced features vary system by system

16. Relevance of Results
 The order of results
 Enterprise SEO = Information Architecture!
 Ranking – METADATA based!
Custom Ranking Model On-Prem only
Query Rules and XRANK On-Prem + Cloud

17. Supportability
 Well-defined processes
 Well-defined roles
 Experts involved
 Consistent Management of
 Content Sources
 Search Metadata
 Search Configuration (Result Sources, Query Rules, …)
 User Interface (Results, Refiners, Hover Panel, custom
components, …)

18. Enterprise Search – Roles
Team Members
Internal External Liaison
Business Sponsors
Stakeholders
Project Management
IT
Search Admins
Developers
…
Consultants
Service Providers
Product Vendors
Developers
Act as Internal
Have Wide Search
Experience
and Knowledge
Understands the
Business
Well Organized
Liaise between Internal +
External
Quality Management
“Active Insurance”

19. How we Know Search is Good
(Enough)?
Users don’t complain
Users keep using it

20. Search Metrics
Number of
Searches
Number of
successful
Searches
Number of
Searches with
Zero results
Results with no
click
Most popular
Queries
Results clicked
the most
Who’s
performing the
most searches
Correlated
searches
Speed /
latency
Total distinct
search terms
Total distinct
words
Trends and
historical
data/charts

21. Search Analytics
What
Metrics?
Who?
How
often?
What to do
with the
results?

22. Action Plan

23. Action Plan
 Start a Discussion!!
 Evaluate your current Search solution
 Evaluate your Content
 Collect Use Cases / Stories
 Identify Users / Audiences
Is Your Search Good Enough???

24. Action Plan
 Collect Feedbacks  Analyze  Take Actions
 Search Analytics Reports  Analyze  Take Actions
 Errors/warnings in the Crawl Log  Analyze  Fix
 Evaluate if any new Content Source is needed
 Evaluate if any new Type of Content is needed
 Improve Metadata
 Improve the UI
Measure and Analyze!
Planning
Implemen
tation
Analysis

25. Keep in Mind: FINDABILITY is more than SEARCH

26. Thank You!
 Web: http://SearchExplained.com
 Yammer: http://yammer.com/SearchExplained
 Twitter: @molnaragnes
 E-mail: agnes.molnar@SearchExplained.com