The document discusses the importance of search analytics for improving search quality. It provides examples of search analytics reports that can help identify issues like zero hit queries, low click-through rates, and high latency. The document emphasizes that analytics allows organizations to measure, monitor and fix search failures and optimize the user experience.

1. Site
Search
Analy.cs
Search
Quality
Ma/ers
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!
1!

2. About
me
Stefan
Thies
stefan.thies@sematext.com
Twi:er:
@se.321
DevOps
Evangelist
@sematext
Copyright 2014 Sematext Group Inc. - All rights reserved.!
! 2!

3. About
Sematext
!
3!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
Performance monitoring, !
anomaly detection and alerting!
Log management!
Site Search Analytics!
Consulting & Support !
for Solr and Elasticsearch!

4. Implemen.ng
Big
Data
&
Search
Systems
!
4!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!
Know-­‐How
Tools

5. Agenda
l What
is
Search
Analy.cs
and
why
it
ma:ers
l Example
reports
and
their
value
!
5!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

6. Why
!
6!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
search
users
search
providers
search
experience
!
This search sucks!!
It takes 17 tries to find anything here!!
F!?@#$%^&?!?!
!
!
Cool, the latest search tweaks !
made our site really sticky!!
Awesome!!
!

7. Fill
in
the
Missing
Piece
!
7!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
Performance
Monitoring
Quality Assurance
Tuning UI!

8. Blind
Leading
the
Blind
!
8!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

9. Analy.cs
as
Compass
Search
logs
are
your
Map
Search
Analy2cs
is
your
Compass
!
9!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

10. The
Bo:om
Line
Why
l Measure
and
monitor
everything.
l Supports
(re)design,
naviga.on
choices
l Helps
with
content
acquisi.on
&
enhancement
l Improve
search
experience
!
10!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

11. The
Moment
of
Truth
• Ques.on
for
the
audience
#1
What
do
you
use
for
Search
Analy.cs?
a)
Home
grown
stuff
b)
Google
Analy.cs
c)
Other
d)
Nothing
!
11!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

12. Search
Analy.cs
Basics
l Collect:
queries
&
clicks
&
interac.ons
&
...
l Analyze:
ac.ons
/
conversions
l Output:
reports
–
over
.me
l Output++:
feedback
loop
l The
means,
not
the
goal
l Ongoing,
not
one-­‐off
!
12!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
remember
this

13. Search
vs.
Web
Analy.cs
l User
intent
and
informa.on
needs
vs.
inferring
l Hand
in
hand
l Ideally
you
can
relate
data
from
both
or
even
unify
it
!
13!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

14. Report
Types
l Failures
vs.
non-­‐failures
l Ac.onable
vs.
non-­‐ac.onable
!
14!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

15. Failures
vs.
Non-­‐Failures
l Zero
hits
l Low
CTR
l Low
MRR
l High
bounce
rate
l Low
conversion
rate
l Deep
paging
l Deep
clicking
l High
latency
!
15!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
l Query rate
l Query volume
l Top seen & clicked
docs
l Top queries
l Terms per query
l Search sessions
l Search users
l Distinct queries

16. Value
of
Failure
Fixes
l Zero
hits
l Low
CTR
l Low
MRR
l High
bounce
rate
l Low
conversion
rate
l Deep
paging
l Deep
clicking
l High
latency
Performance Tuning!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
Re-search
Findability
Relevance Tuning

17. Measure,
then
Fix
If
you
can't
measure,
it
you
can't
fix
it!
!
17!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

18. Metrics
–
e.g.
Mean
Reciprocal
Rank
18!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!
Wikipedia - MRR!

19. Relevance
A/B
Tes.ng
!
19!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

20. Tracking
Zero
Hits
20!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!

21. Report:
Zero
Hit
Queries
• Overall
pct.
(not
raw
count)
vs.
Popular
queries
• Misspellings?
• Synonyms?
• No
matching
content?
• Need
different
tagging?
• Bad
analysis/mapping?
• Mul.lingual
issues?
!
21!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!

22. Tracking
Zero
Hits
22!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!

23. Frequency
of
Zero
Hit
23!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!

24. Minimize:
#
of
Zero
Hit
Queries
• Use
Query
Spellchecker
(DYM)
• Use
Autocomplete
• Using
DYM
ReSearcher,
Query
Relaxer,
Related
Search
• Check
Mapping,
Analyzers,
Queries
• Indexing
process
– Lost
items
while
bulk
indexing?
• Design
a
‚No
Result‘
page
!
24!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!

25. !
25!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!
Ellastic search!
Example
(search-­‐hadoop.com)

26. Watching
Latency
!
26!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

27. Paging
27!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!

28. more
search
metrics
...
28!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
!

29. Search
Analy.cs
&
Measuring
!
29!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
If you can't measure it, you can't fix it!
You can't measure it if you don't have Analytics

30. Ac.onable
vs.
Non-­‐Ac.onable
l Zero
hits
l Low
CTR
l Low
MRR
l High
bounce
rate
l Low
conversion
rate
l Deep
paging
l Deep
clicking
l High
latency
!
30!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
l Query rate
l Query volume
l Top seen & clicked docs
l Top queries
l Terms per query
l Search sessions
l Search users
l Distinct queries

31. More
Fixin'
l Query
rate
l Query
volume
l Search
sessions
l Search
users
l Top
seen
&
clicked
docs
l Top
queries
l Terms
per
query
l Dis.nct
queries
!
31!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
Navigation & Design
Results Shuffling
Diversification,
Recommendations
AutoComplete!
Search box size!

32. Output++:
Data
is
Power
l AutoComplete
-­‐
$MM
improvement
l Be:er
DYM
Spellchecker
l Related
Searches
l Recommenda.ons
l Relevance
Feedback
l ...
!
32!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

33. Closing
the
Loop
!
33!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
search
users
search
providers
search
experience

34. Resources
!
34!
Copyright 2014 Sematext Group Inc. - All rights reserved. !
http://rosenfeldmedia.com/
books/searchanalytics/
Search Analytics for Your Site
Louis Rosenfeld
l http://blog.sematext.com/tag/analytics/
l http://www.slideshare.net/lucenerevolution/gospodnetic-otis-ppt-
search-analytics-what-why-how

35. Key
Take-­‐aways
l Without
Analy5cs
you
are
blind
l If
you
can't
measure
it,
you
can't
fix
it
l Use
Search
Analy5cs
to
understand,
measure
and
improve
search
l Using
Search
Analy5cs
means
having
a
compe55ve
advantage
!
35!
Copyright 2014 Sematext Group Inc. - All rights reserved. !

36. Dig
Search?
Dig
Analy.cs?
Dig
Big
Data?
Dig
Performance?
Dig
Logging?
Dig
working
with
open
–
source?
• We‘re
hiring
planet
-­‐
wide!
h/p://www.sematext.com/about/jobs.html
We are hiring ...!
Copyright 2014 Sematext Group Inc. - All rights reserved.!
! 36!