ProjectHub

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

ProjectHub Crawling, Indexing, and Searching Software Project Data with Droids, Tika, Solr & friends Otis Gospodneti ć ◦◦ ...

What I Will Cover <ul><li>Who I am </li></ul><ul><li>What Why Where </li></ul><ul><li>Architecture </li></ul><ul><li>Info ...

About Otis Gospodneti ć <ul><li>Lucene/Solr/Nutch/Mahout/... committer </li></ul><ul><li>Lucene in Action 1 & 2 co-author ...

About Sematext <ul><li>Search ( Lucene, Solr, Elastic Search... ) </li></ul><ul><li>Web Crawling (Nutch) </li></ul><ul><...

What <ul><li>Search everything about a Software Project </li></ul><ul><li>Lucene & Hadoop </li></ul><ul><ul><li>All sub-pr...

Why <ul><li>We need it </li></ul><ul><li>Other Hadoop, Lucene, Solr... users need it </li></ul><ul><li>Our own playground ...

Where <ul><li>search-lucene.com </li></ul><ul><li>search-hadoop.com </li></ul><ul><li>Other suggestions / needs? </li></ul...

Architecture

Tool Matrix Data Source Fetch Parse JIRA URLConnection (feed) Digester (feed) DOM (item) ML FileInputStream (fs) URLConnec...

Information Gathering <ul><li>Multiple independent JVM processes (cron) </li></ul><ul><li>Different polling frequencies </...

Thread Detection <ul><li>Email clients are kaput </li></ul><ul><li>SMTP headers are unreliable </li></ul><ul><li>Heuristic...

Indexing <ul><li>Use StreamingUpdateSolrServer </li></ul><ul><li>AutoCommit use-case </li></ul><ul><li>Solr index abuse: t...

Search <ul><li>Facets (multi-select) </li></ul><ul><ul><li>Project </li></ul></ul><ul><ul><li>Data source/type </li></ul><...

Search cont'd <ul><li>Query Spellchecker </li></ul><ul><li>Sematext components: </li></ul><ul><ul><li>ReSearcher & Relaxer...

Dog food #1: Auto-Complete <ul><li>Source: nightly refreshed subject and titles </li></ul><ul><li>Approach: go directly to...

Dog food #2: ReSearcher & Relaxer <ul><li>Avoid “sorry, no/poor matches” </li></ul><ul><li>Multiple algos trigger re-searc...

Dog food #3: Key Phrases <ul><li>Help narrow search results, like facets </li></ul><ul><li>2 types: </li></ul><ul><ul><li>...

Basic Search Analytics <ul><li>Top queries, top terms... </li></ul><ul><li>Daily, weekly, monthly </li></ul><ul><li>MRR </...

Very Basic Search Analytics

Real Search Analytics

Performance & Monitoring: RPM

Availability: Site24x7.com

Operations <ul><li>Small EC2 instance: 1.7 GB RAM </li></ul><ul><li>EBS for data - got burnt once </li></ul><ul><li>Local ...

Statistics <ul><li>search-hadoop.com: </li></ul><ul><ul><li>110K+ documents </li></ul></ul><ul><ul><li>~700 MB optimized <...

Future <ul><li>Field collapsing (threads) </li></ul><ul><li>Bot detection (load) DONE </li></ul><ul><li>Solr duplicate de...

<ul><li>World-wide! </li></ul><ul><li>Search & Data Analytics </li></ul><ul><li>Machine Learning & NLP </li></ul><ul><li>B...

Questions <ul><li>? </li></ul>

Contact <ul><li>sematext.com </li></ul><ul><li>blog.sematext.com </li></ul><ul><li>@ sematext </li></ul><ul><li>@ otisg </...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Faster! Optimize Your Cascade Serve... by hannonhill 2249 views
Large Scale Crawling with Apache Nu... by Julien Nioche 7096 views
Open source enterprise search and r... by mteutelink 1126 views
Populate your Search index, NEST 20... by David Smiley 822 views
Apache Tika end-to-end by gagravarr 4187 views
Content Analysis with Apache Tika by Paolo Mottadelli 7723 views

View on Slideshare

1 of 30 Ad

View on Slideshare

1 of 30 Ad