Search Analytics Business Value & NoSQL Backend

Search Analytics Business Value & NoSQL BackendOtis Gospodnetić – Sematext International @otisg ◦ @sematext ◦ sematext.com sematext.com/search-analytics

About Sematext Products & Services Consulting, Development, Tech Support:● Search (Lucene, Solr, ElasticSearch...)● Big Data (Hadoop, HBase, Voldemort...)● Web Crawling (Nutch, Droids)● Machine Learning (Mahout) 4 Copyright 2011 Sematext Intl. All rights reserved.

Communication● twitter.com/sematext● twitter.com/otisg● hash tags: #stsa or #stanalytics● http://sematext.com/search-analytics/index.html● Raise your hand!● otis@sematext.com 6 Copyright 2011 Sematext Intl. All rights reserved.

High Level Why This search sucks! It takes 17 tries to find anything here! F!?@#$%^&?!? search users search experience search providers Cool, the latest search tweaks made our site really sticky! Awesome! 9 Copyright 2011 Sematext Intl. All rights reserved.

More Concrete Why● Measure and monitor everything. Introspection.● Supports (re)design, navigation choices● Helps with content acquisition & enhancement● Improve search experience● Mula 12 Copyright 2011 Sematext Intl. All rights reserved.

The Moment of Truth Question for the audience #1 What do you use for Search Analytics? a) Home grown stuff b) Google Analytics c) Omniture d) Webtrends e) Other f ) Nothing 13 Copyright 2011 Sematext Intl. All rights reserved.

Search Analytics Outline● Collect: queries & clicks & interactions & ...● Analyze: actions / xactions / conversions● Output: reports – over time● Output++: feedback loop remember this● The means, not the goal● Ongoing, not one-off 14 Copyright 2011 Sematext Intl. All rights reserved.

Example Core Reports● Rate & Volume, Latency (mean, avg, 90%)● Click Through Rate, Mean Reciprocal Rank● Top Queries by count, clicks, 0 hits...● Query Trending● Top Seen Docs, Top Clicked Docs (msft)● Page & Click Depth● Facet & Sort Usage● ... 16 Copyright 2011 Sematext Intl. All rights reserved.

What Weve Built● Search Analytics SaaS ● Numerous reports (e.g. query volume, rate, latency, term frequencies / comparisons, hit buckets, search origins, etc.) ● Trending over time ● Comparisons of time periods ● Top N reports ● Filter, slice and dice 19 Copyright 2011 Sematext Intl. All rights reserved.

Core Tech● JavaScript Beacons● Metric Capture Web App aka Receiver● Flume Agents, Collectors, Sinks● HBase● MapReduce Aggregations● Search Analytics Reporting Web App 31 Copyright 2011 Sematext Intl. All rights reserved.

What is Flume● Distributed data/log collection service● Scalable, configurable, extensible● Centrally manageable, open source● Agents get data from app, Collectors save it● Abstractions: Source → Decorator(s) → Sink 32 Copyright 2011 Sematext Intl. All rights reserved.

Why Flume● Reliable delivery ● e.g. queue msgs locally if destination unreachable● Easy, centralized management via Web UI or console● Good community, good progress, now @ASF● But: more complex, more moving parts● On Flume: slideshare.net/cloudera/inside-flume● Alternatives: Kafka, Scribe... 35 Copyright 2011 Sematext Intl. All rights reserved.

Why HBase● Scalable raw & aggregate data storage● MapReduce data input● Fast scans for time ranges, fast key lookups● Easy storage and compute power expansion● Good looking roadmap, community, progress 36 Copyright 2011 Sematext Intl. All rights reserved.

Open Sourcing● 2 open-source projects: github.com/sematext/HBaseWD github.com/sematext/HBaseHUT● See sematext.com/open-source/index.html● Patches for Flume and HBase blog.sematext.com/tag/flume/ 37 Copyright 2011 Sematext Intl. All rights reserved.

Challenges● Data size. Solutions: ● Compression (4-5x smaller with lzo) ● Data pruning (variable levels)● Query string distribution: very long-tail ● Lots of data to process, update, aggregate● Young tools: Flume, HBase● Poor IO on EC2● Hadoop distributions 38 Copyright 2011 Sematext Intl. All rights reserved.

Were Hiring Dig Search? Dig Analytics? Dig Big Data? Dig Performance? Dig working with and in open-source? Were hiring world-wide! http://sematext.com/about/jobs.html 42 Copyright 2011 Sematext Intl. All rights reserved.

Contact sematext.com blog.sematext.com @sematext @otisg otis@sematext.com Want SA? Grab me or go to: sematext.com/search-analytics Hash tags: #stsa or #stanalytics 43 Copyright 2011 Sematext Intl. All rights reserved.

http://blog.sematext.com	4523
http://feeds.feedburner.com	18
http://webcache.googleusercontent.com	4
http://127.0.0.1	1
http://webcache-exp-test.googleusercontent.com	1

Search Analytics Business Value & NoSQL Backend

by Sematext Group, Inc.

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 4,547

Statistics

Search Analytics Business Value & NoSQL Backend Presentation Transcript