Search Analytics at Enterprise Search Summit Fall 2011

Search Analytics What ? Why ? How ? Otis Gospodneti ć – Sematext International @otisg ◦ @sematext ◦ sematext.com sematext.com/search-analytics

About Otis Gospodneti ć
ASF Member : Lucene, Solr, Nutch, Mahout
Author : Lucene in Action 1 & 2
Entrepreneur : Sematext , Simpy

Sematext Metrics
100% organic : no GMO, no VC
4 years old
< 10 people
7 countries
3 timezones
2 continents
> 100 customers

About Sematext
Products & Services
Consulting, Development, Tech Support:
Search (Lucene, Solr, ElasticSearch...)
Big Data (Hadoop, HBase, Voldemort...)
Web Crawling (Nutch, Droids)
Machine Learning (Mahout)

Agenda
What is Search Analytics and why it matters
Example reports and their value
Optional: Search Analytics in the Cloud

Communication
twitter.com/ sematext
twitter.com/ otisg
hash tags: # stsa or # stanalytics
http://sematext.com/search-analytics/index.html
Raise your hand !
otis @sematext.com

Why search users search providers search experience

Why Oh Why search providers search experience This search sucks! It takes 17 tries to find anything here! F!?@#$%^&?!? search users Cool, the latest search tweaks made our site really sticky! Awesome!

Fill in the Missing Piece Search Analytics Performance Monitoring Quality Assurance Tuning UI

Blind Leading the Blind

Analytics as Compass
Search logs are your Map
Search Analytics is your Compass

The Bottom Line Why
Measure and monitor everything .
Supports (re)design, navigation choices
Helps with content acquisition & enhancement
Improve search experience
Mula

The Moment of Truth
Question for the audience #1 What do you use for Search Analytics?
a) Home grown stuff b) Google Analytics c) Omniture d) Webtrends e) Other f ) Nothing

Search Analytics Basics
Collect : queries & clicks & interactions & ...
Analyze : actions / xactions / conversions
Output : reports – over time
Output++ : feedback loop
The means, not the goal
Ongoing, not one-off
remember this

Search vs. Web Analytics
User intent and information needs vs. inferring
Hand in hand
Ideally you can relate data from both or even unify it

Report Types
Failures vs. non-failures
Actionable vs. non-actionable
Trends vs. summaries

Failures vs. Non-Failures
Zero hits
Low CTR
Low MRR
High bounce rate
Low conversion rate
Deep paging
Deep clicking
High latency
Query rate
Query volume
Top seen & clicked docs
Top queries
Terms per query
Search sessions
Search users
Distinct queries

Value of Failure Fixes
Zero hits
Low CTR
Low MRR
High bounce rate
Low conversion rate
Deep paging
Deep clicking
High latency
Re-search Findability Relevance Tuning Performance Tuning

Measure, then Fix
If you can't measure , it you can't fix it!

Relevance A/B Testing

Tracking Zero Hits

Watching Latency

Search Analytics & Measuring If you can't measure it, you can't fix it! You can't measure it if you don't have Analytics

Actionable vs. Non-Actionable
Zero hits
Low CTR
Low MRR
High bounce rate
Low conversion rate
Deep paging
Deep clicking
High latency
Query rate
Query volume
Top queries
Terms per query
Search sessions
Search users
Distinct queries

More Fixin'
Query rate
Query volume
Search sessions
Search users
Top queries
Terms per query
Distinct queries
Navigation & Design Results Shuffling Diversification Recommendations AutoComplete Search box size

Output++: Data is Power
AutoComplete - $MM improvement
Better DYM Spellchecker
Related Searches
Recommendations
Relevance Feedback
...

Closing the Loop search users search providers search experience

Resources http://rosenfeldmedia.com/books/searchanalytics/ Search Analytics for Your Site Louis Rosenfeld
Search Analytics What? Why? How?
Search Analytics with Flume and HBase
Search Analytics Business Value & NoSQL Backend http://blog.sematext.com/tag/analytics/

Key Take-aways
Without Analytics you are blind
If you can't measure it, you can't fix it
Use Search Analytics to understand , measure and improve search
Using Search Analytics means having a competitive advantage

Time permitting: Behind the scenes of Sematext Search Analytics
Behind the Scenes

sematext.com
blog.sematext.com
@sematext
@otisg
[email_address] Want SA? Grab me or go to:
sematext.com/search-analytics
Hash tags: # stsa or # stanalytics
Contact

What We've Built
Search Analytics SaaS
Numerous reports (e.g. query volume, rate, latency, term frequencies / comparisons, hit buckets, search origins, etc.)
Trending over time
Comparisons of time periods
Top N reports
Filter , slice and dice

Sematext Search Analytics

Big Dreams
SaaS
Multitenant
Large Scale – Massive Data
Cloud

Storage Choices
RDBMS: MySQL, PostgreSQL
HDFS
Hive
HBase
Cassandra

SaaS vs. In-House
Question for the audience #2
SaaS vs in-house Search Analytics? a) SaaS b) in-house

Sematext Search Analytics

Data Flow
See Search Analytics with Flume and HBase http://blog.sematext.com/2010/10/16/search-analytics-hadoop-world-flume-hbase/

Data Collection
See Search Analytics with Flume and HBase http://blog.sematext.com/2010/10/16/search-analytics-hadoop-world-flume-hbase/

Core Tech
JavaScript Beacons
Metric Capture Web App aka Receiver
Flume Agents, Collectors, Sinks
HBase
MapReduce Aggregations
Search Analytics Reporting Web App

What is Flume
Distributed data/log collection service
Scalable, configurable, extensible
Centrally manageable, open source
Agents get data from app, Collectors save it
Abstractions: Source -> Decorator(s) -> Sink

What is HBase
Scalable, reliable, distributed, column-oriented DB
On top of HDFS
MapReducable

Data Flow, Detailed

Why Flume
Reliable delivery
e.g. queue msgs locally if destination unreachable
Easy, centralized management via Web UI or console
Good community, good progress, now @ASF
But: more complex, more moving parts
On Flume: slideshare.net/cloudera/inside-flume
Alternatives: Kafka, Scribe...

Why HBase
Scalable raw & aggregate data storage
MapReduce data input
Fast scans for time ranges, fast key lookups
Easy storage and compute power expansion
Good looking roadmap, community, progress

Open Sourcing
2 open-source projects:
github.com/sematext/HBaseWD
github.com/sematext/HBaseHUT
See sematext.com/open-source/index.html
Patches for Flume and HBase blog.sematext.com/tag/flume/

Challenges
Data size. Solutions:
Compression (4-5x smaller with lzo)
Data pruning (variable levels)
Query string distribution: very long-tail
Lots of data to process, update, aggregate
Young tools: Flume, HBase
Poor IO on EC2
Hadoop distributions

sematext.com
blog.sematext.com
@sematext
@otisg
[email_address] Want SA? Grab me or go to:
sematext.com/search-analytics
Hash tags: # stsa or # stanalytics
Contact

http://blog.sematext.com	4590
http://feeds.feedburner.com	40
http://sematext.wordpress.com	4
http://webcache.googleusercontent.com	2

Search Analytics at Enterprise Search Summit Fall 2011

by Sematext Group, Inc. on Nov 01, 2011

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 4,636

Statistics

Search Analytics at Enterprise Search Summit Fall 2011 Presentation Transcript