In Searchsite search
    integrating
                Of...


                      Ian Barber
                     @ianbar...
How Search Works
Integrating Search
 Improving Results
     Using Search
         Questions




                 2
3
Query
Query      Query
 Query
 Query
           Parser


Result
Result
 Result
 Result   Index




          Analyser   Do...
Tokenisation



“  With AT&T’s help, the F.B.I
Miami-Dade office had recovered
$1.1 million from O’Healy’s Ponzi
scheme, 10...
PHP Tokenisation

function tokenise($string) {
    $string = strtolower($string);
    preg_match_all('/w+/', $string,
    ...
Document Term Pairs
Document ID         Term
    1                the
    1               best
    1                of
   ...
Inverted Index
Term              Documents
best    1 (4, 16), 4 (422), 129 (344) ...

what    24 (50, 98), 75 (33, 208) .....
Boolean Query Merge
Query: Best Western Hotel
 best     1    4    129   298   305   338
western   4   95    194   204   29...
Lorem ipsum dolor sit amet,
                                                                 Lorem ipsum dolor sit amet,  ...
TF-IDF

function getWeight($docID, $term, $total) {
  $tf = count($term[$docID]);
  $idf = log($total / count($term), 2);
...
Document Vector
        socket   what   heavy   steel   ...

Doc 1    0.02    0.3    0.001    0      ...

Doc 2     0     ...
Ranked Query Merge
  best     23    42   179   246   333   703

 weight   0.008 0.002 0.023 0.039 0.014 0.001

western    ...
PHP Similarity
function score($queryString, $index) {
  $query = tokenize($queryString);
  $matches = array();
  foreach($...
Integrating Search
                     15
MySQL Full Text Search
CREATE TABLE example (
    id INT(11) NOT NULL auto_increment,
    title VARCHAR(255),
    content ...
MySQL FTI Query
SELECT * FROM example WHERE
MATCH(title,content) AGAINST('loves bacon');

+----+------------------+-------...
Sphinx
http://www.sphinxsearch.com




                        18
Sphinx Configuration
source posts
{
  type             =   mysql
  sql_host         =   localhost
  sql_user         =   us...
index posts
{
  source     = posts
  path       = /var/data/sphinx/example
  morphology = stem_en

    min_word_len     = ...
Stemming
        http://tartarus.org/~martin/PorterStemmer




happening - happen
happened - happen
happens   - happen



...
Command Line Searching
indexer --config /etc/sphinx.conf --all
search --config /etc/sphinx.conf love bacon

displaying mat...
Sphinx From PHP

$cl = new SphinxClient();
$cl->SetServer('localhost', 3312);
$cl->SetMatchMode(SPH_MATCH_ANY);

$result =...
Swish-E
   http://swish-e.org
pecl install swish-beta
                    24
Filesystem Index With Swish-E
/usr/local/bin/swish-e -S fs -c fs-swish-e.conf


fs-swish-e.conf
IndexDir            /var/d...
Crawling Content
/usr/local/bin/swish-e -S prog -c www-swish-e.conf


www-swish-e.conf
IndexDir      /usr/local/lib/swish-...
Swish-E With Multiple Indices
$swish     = new Swish(
   'www-swish-e.index fs-swish-e.index'
);
$search    = $swish->prep...
Lucene




         28
$index = Zend_Search_Lucene::create('idx');
foreach($documents as $title => $content) {
  $doc = new Zend_Search_Lucene_Do...
$results = $index->find('loves bacon');
foreach($results as $result) {
        echo $result->score, " ";
        echo $res...
$file = file_get_contents($url);

$doc = Zend_Search_Lucene_Document_Html::
                           loadHTML($file);

$...
Solr
http://lucene.apache.org/solr/
                                 32
Solr Search Index
$options = array( 'hostname' => 'localhost',
                  'port'     => 8983 );

$client = new Solr...
Solr Search Client
$client = new SolrClient($options);

$query = new SolrQuery('bacon');
$response = $client->query($query...
Xapian
http://xapian.org




              35
Xapian In PHP
$db = new XapianWritableDatabase(
      'idx', Xapian::DB_CREATE_OR_OPEN);
$i = new XapianTermGenerator();
$...
Xapian Search In PHP

$database = new XapianDatabase('idx');
$enquire = new XapianEnquire($database);
$qp = new XapianQuer...
$matches = $enquire->get_mset(0, 10);

$i = $matches->begin();
while(!$i->equals($matches->end())) {
  $n = $i->get_rank()...
Improving Results




                    39
Anchor Text




         40
Parse Anchor Text
$p = file_get_contents('http://phpir.com');

libxml_use_internal_errors(true);
$dom = DomDocument::loadH...
1
         2




         3



    Zone Weighting
                42
$doc = new Zend_Search_Lucene_Document();

$tfield = Zend_Search_Lucene_Field::Text
   ('title', $title);
$tfield->boost =...
Document Authority




                44
Document Weights in ZSL
$doc = new Zend_Search_Lucene_Document();
$doc->addField(
  Zend_Search_Lucene_Field::Text
   ('ti...
Using Search




          46
Summaries & Highlighting




                           47
Sphinx Extract & Highlight
$cl = new SphinxClient();
$cl->SetServer( "localhost", 3312 );
$q = 'bacon';
$r = $cl->Query($q...
Xapian Spelling Correction
Indexer
$indexer = new XapianTermGenerator();
$indexer->set_database($database);
$indexer->set_...
Spelling Correction Output
 php xapsearch.php

Did you mean: str_replace or strcmp

4644 results found for “strreplace or ...
Results Sorting




                  52
Sorting in ZSL

$q = Zend_Search_Lucene_Search_QueryParser::
 parse('search string');

$results = $index->find($q, 'title'...
Faceted Search




                 54
Faceted Search In Solr
$client = new SolrClient($options);
$query = new SolrQuery('bacon');
$response = $client->query($qu...
More Like This




            56
More Like This
$rset = new XapianRset();
$rset->add_document(5959); // str_replace
$e = $enquire->get_eset(40, $rset);

$t...
More Like This Example
 php xapsim.php

1656 results found:
1: 100% docid=5959
    [phpdocs/html/function.str-replace.html...
Image Credits
Title                http://www.flickr.com/photos/generated/2084287794/
What Do You Want     http://www.flickr...
Questions?




             60
Thank You!

                 Ian Barber
                @ianbarber
          http://phpir.com
        ian@ibuildings.com
 ...
In Search Of... (Dutch PHP Conference 2010)
Upcoming SlideShare
Loading in...5
×

In Search Of... (Dutch PHP Conference 2010)

2,687

Published on

DPC version of my In Search Of search introduction talk

Published in: Technology, Design
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,687
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
26
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

In Search Of... (Dutch PHP Conference 2010)

  1. 1. In Searchsite search integrating Of... Ian Barber @ianbarber http://phpir.com ian@ibuildings.com http://joind.in/1556
  2. 2. How Search Works Integrating Search Improving Results Using Search Questions 2
  3. 3. 3
  4. 4. Query Query Query Query Query Parser Result Result Result Result Index Analyser Document Document Document Document 4
  5. 5. Tokenisation “ With AT&T’s help, the F.B.I Miami-Dade office had recovered $1.1 million from O’Healy’s Ponzi scheme, 10-15% more than ” expected. 5
  6. 6. PHP Tokenisation function tokenise($string) { $string = strtolower($string); preg_match_all('/w+/', $string, $matches, PREG_OFFSET_CAPTURE); return $matches[0]; } 6
  7. 7. Document Term Pairs Document ID Term 1 the 1 best 1 of 1 the ... ... 204 and 204 what 204 would 7
  8. 8. Inverted Index Term Documents best 1 (4, 16), 4 (422), 129 (344) ... what 24 (50, 98), 75 (33, 208) ... would 99 (32, 599), 201 (344) .. ... ... 8
  9. 9. Boolean Query Merge Query: Best Western Hotel best 1 4 129 298 305 338 western 4 95 194 204 298 305 working 4 298 305 hotel 2 40 200 298 355 402 Result: Document 298 9
  10. 10. Lorem ipsum dolor sit amet, Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed sit amet ante vitae enim elementum semper sodales quis consectetur adipiscing elit. Sed sit amet ante vitae enim elementum semper sodalesipsum. Aliquam vel condimentum Lorem ipsum dolor sit amet, quis neque. ipsum. Aliquam vel condimentum neque. Curabitur ornare feugiat ornare. Donec consectetur adipiscing elit. Sed sit amet ante consectetur elit metus. Nulla eleifend Curabitur ornare feugiat ornare. Donec vitae enim elementum semper sodales quis consectetur elit metus. Nulla eleifend tincidunt massa et euismod. Vestibulum ipsum. Aliquam vel condimentum neque. vestibulum, justo vel egestas elementum, tincidunt massa et euismod. Vestibulum sit amet, Lorem ipsum dolor Curabitur ornare feugiat ornare. Donec vestibulum, justo consectetur elementum,elit.enim sit ametquam, vel gravida est vel egestas adipiscing purus Sed ornare ante consectetur elit metus. Nulla eleifend purus enim ornarevitae enim elementum sempernibh. quam, vel gravida est vel sodales quis enim tincidunt massa et euismod. Vestibulum Lorem ipsum dolor sit amet, consectetur enim vel nibh. Lorem ipsum dolor ipsum. Aliquam vel condimentum neque. fringillavestibulum, justo vel egestas elementum, sit amet, Nam non eros nisi, eget justo. consectetur adipiscingCurabitur sit ametfeugiat ornare. Donec mauris vehicula enim ornare quam, vel gravida est elit. Sed ornare ante purus adipiscing elit. Sed sit amet ante vitae enim vitae enim elementum consectetur elitjusto.Fusce vel risus vitae Nam non eros nisi,semper sodalesmetus. Nulla eleifend eget fringilla quis enim vel nibh. Fusce vel risus condimentum neque. facilisis sit amet in mi. Nulla ut turpis id ipsum. Aliquam velvitae maurismassa et euismod. Vestibulum tincidunt vehicula elementum semper sodales quis ipsum. Aliquam facilisis sit amet in mi. Nulla ut turpis felis sollicitudin dictum sed nonNam non eros nisi, eget fringilla justo. Curabitur ornare feugiat ornare. Donec velid vestibulum, justo egestas elementum, ipsum. Praesent gravida nulla, sed blandit leo. ut risus est Lorem ipsum dolor sit amet, Lorem ipsum dolor sit amet, consectetur elit metus.purus enim ornare quam, vel volutpat laoreet lacus,Fusce vel risus vitae mauris vehicula felis sollicitudin dictum sed non ipsum. Nulla eleifend vel condimentum neque. Curabitur ornare enim Vestibulum Curabitur ut consectetur adipiscing elit. Sed sit amet ante consectetur adipiscing elit. Sed sit amet ante tincidunt massa risus nulla, sed nibh. leo.consectetur arcu vestibulum vel.facilisis sit amet in mi. Nulla ut turpis id Praesent ut et euismod.vel blandit ut sodales Donec Curabitur volutpat laoreet lacus, vitae enim elementum semper vitae enim elementum semper sodales quis quis felis sollicitudin dictum sed non ipsum. vestibulum, justo vel egestas elementum, dapibus fringilla arcu, et semper lacus feugiat ornare. Donec consectetur elit metus. Nam non vel. ipsum. Aliquam vel condimentumLorem ipsumut risussit amet, blandit leo. consectetur arcu vestibulumeros nisi, eget fringilla justo. purus enim ornare quam, vel gravida est Donec ipsum. Praesent vel condimentum neque. neque. Aliquam dolor nulla, sed arcu, vel risusCurabitur ornare feugiat ornare.consectetur adipiscing elit. Sed Donec ut Curabitur ornare volutpat laoreetsit amet ante Donec enim dapibus fringilla Fusce et sempervitae mauris vehicula vel nibh. lacus Curabitur feugiat ornare. lacus, consectetur elitut turpisNulla eleifendenim elementumNulla eleifend quis metus. id consectetur elit metus. semper sodales Donec Nulla eleifend tincidunt massa et euismod. facilisis sit amet in mi. Nulla vitae consectetur arcu vestibulum vel. tincidunt massa et euismod. Vestibulum massa et euismod. Vestibulum lacus tincidunt Nam non eros nisi, eget fringilla justo. dictum sed non ipsum. felis sollicitudin ipsum. dapibus fringilla arcu, et semper Aliquam vel condimentum neque. vestibulum, justo vel egestas elementum, ornare vel egestas elementum, vestibulum, justo feugiat ornare. Donec Vestibulum vestibulum, justo vel egestas Fusce vel risus vitae mauris vehicula nulla, sed blandit leo. Praesent ut risus purus Curabitur Curabitur volutpat enim ornare quam, vel gravidaenim ornare quam, vel gravida est purus est elit metus. Nulla eleifend facilisis sit amet in mi. Nulla ut turpis id laoreet lacus, ut consectetur enim vel nibh.vel. Donec consectetur arcu vestibulum enim vel nibh. et euismod. Vestibulum elementum, purus enim ornare quam, vel felis sollicitudin dictum sed non ipsum. Praesent ut risus nulla, dapibus fringilla arcu, et semper lacus sed blandit leo. tincidunt massa vestibulum, justo vel egestas elementum, Nam non eros nisi, eget fringilla justo. eros nisi, eget fringilla justo.est Nam non ornare quam, vel gravida gravida est enim vel nibh. Curabitur volutpat laoreet lacus, ut purus enim Fusce vel risus vitae mauris vehicula vel nibh. vitae mauris vehicula Fusce vel risus enim Lorem ipsum dolor sit amet, vel. Donec consectetur arcu vestibulum facilisis sit amet in mi. Nulla ut turpis id amet in mi. Nulla ut turpis id facilisis sit consectetur adipiscing elit.et semper lacus sollicitudin dictum sed non ipsum. dapibus fringilla arcu, Sed sit amet ante felis felis sollicitudin dictum sed non ipsum. Nam non eros nisi, eget fringilla justo. Nam non eros nisi, eget fringilla justo. Fusce vel vitae enim elementum semper sodales quis ipsum. Aliquam vel condimentum neque. Praesent ut risus nulla, sed blandit leo. utrisus vitae mauris vehicula Praesent risus nulla, sed blandit leo. Fusce vel Curabitur volutpat laoreet lacus, ut Curabitur volutpat laoreet lacus, ut Curabitur ornare feugiat ornare. Donec consectetur arcu vestibulum vel. Donec sit arcu vestibulum vel. turpis id facilisis amet in mi. Nulla ut risus vitae mauris vehicula facilisis sit amet in Lorem ipsum dolor sit amet, consectetur consectetur elit metus. Nulla eleifendadipiscing elit. Sed sit amet ante felis sollicitudin dictum sed non ipsum. consectetur Donec Lorem ipsum dolor sit amet, dapibus fringilla arcu, etLorem ipsum dolor sit amet, et semper lacus semper lacus fringilla nulla, sed blandit leo. dapibus ut risus arcu, consectetur adipiscing enimSed vitae elit. elementum ante quis Praesent tincidunt massa et euismod. Vestibulumsit amet semper sodalesconsectetur adipiscing elit. Sed sit amet ante mi. Nulla ut turpis id felis sollicitudin dictum vestibulum, justo vel egestas elementum, vitae enim elementum semper sodales quis vitae Curabitur volutpat laoreet lacus, ut ipsum. Aliquam vel condimentum neque. enim elementum semper sodales quis purus enim ornare quam,vel condimentum feugiat ornare. Donec Curabitur ornare neque. vel gravida est consectetur arcu vestibulum vel. Donec sed non ipsum. Praesent ut risus nulla, sed ipsum. Aliquam enim vel nibh. Curabitur ornare feugiat ornare. metus. ipsum. Aliquam vel condimentum neque. consectetur elit Donec Nulla eleifend Curabiturdapibus feugiat ornare.et semper lacus ornare fringilla arcu, Donec tincidunt massa et euismod. Vestibulum blandit leo. Curabitur volutpat laoreet lacus, ut consectetur elit metus. Nulla eleifend vestibulum,Loremvel egestas elementum, Nam non eros nisi, eget fringilla justo. justo ipsum dolor sit amet, tincidunt massa et euismod. Vestibulum consectetur elit metus. Nulla eleifend tincidunt ipsum dolor sit amet, Lorem massa et euismod. Vestibulum purus enim ornare quam, vel gravidaSed sit amet ante vel egestas elementum, consectetur adipiscing elit. est Fusce vel risus vitaejusto vel egestas elementum, vestibulum, mauris vehicula consectetur arcu vestibulum vel. Donec dapibus enim vel vitae enim est nibh. sit amet in ornare quam, vel id vestibulum, justo consectetur adipiscing elit. Sed sit amet ante facilisis purus enim mi. Nulla ut turpisgravida elementum semper sodales quis purus enim ornare quam, vel gravida est vitae enim elementum semper sodales quis felis sollicitudin dictum sed non ipsum. Aliquam vel condimentum vel nibh. vel condimentum neque. enim vel nibh. ipsum. enim neque. fringilla arcu, et semper lacus egestas non. Praesent ut risus nulla, sed blandit leo. nisi, eget fringilla Nam non eros ipsum. Aliquam Curabitur ornare feugiatjusto. Donec ornare. Curabitur ornare feugiat ornare. Donec consectetur elit metus. Nulla eleifend Curabitur volutpateros nisi,lacus, fringilla vitae mauris vehicula consectetur elit metus. Nulla eleifend Nam non laoreet egetvel risus justo. Fusce ut Nam non eros nisi, eget fringilla justo. Quisque eu purus ut lacus egestas dapibus. consectetur arcu vestibulum vel. Donec inmassa et euismod. Vestibulum tincidunt Fusce vel risus vitae mauris amet mi. Nulla ut turpis tincidunt massavitae mauris Vestibulum facilisis sit vehicula id Fusce vel risus et euismod. vehicula felis sollicitudin dictum vel egestas vestibulum,amet in mi. Nulla elementum, vestibulum, justo dapibus fringilla arcu, et semper lacus turpis id sed non ipsum. facilisis sit amet in mi. Nulla ut elementum, facilisis sit justo vel egestas ut turpis id Integer in velit id est dictum bibendum in id mi. purus enim ornareblandit vel gravida est felis sollicitudin dictum sed non ipsum. sed Praesent ut risus nulla, enim vel nibh. quam, leo. purus enim ornare quam, velnon ipsum. felis sollicitudin dictum sed gravida est Praesent ut risus Curabitur volutpat laoreet lacus, ut enim vel ut risus nulla, sed blandit leo. nulla, sed blandit leo. Praesent nibh. consectetur arcu vestibulum vel. Donec Curabitur volutpat laoreet lacus, ut Curabitur volutpat laoreet lacus, ut dapibus Nam nonarcu, nisi, eget fringilla justo. arcu vestibulum vel. Donec fringilla eros consectetur arcu vestibulum vel. Donec et semper lacus consectetur Nam non eros nisi, eget fringilla justo. Fusce vel risus vitae mauris vehicula dapibus fringilla arcu, et semper lacus Fusce velfringilla arcu, et semper lacus dapibus risus vitae mauris vehicula facilisis sit amet in mi. Nulla ut turpis id facilisis sit amet in mi. Nulla ut turpis id felis sollicitudin dictum sed non ipsum. felis sollicitudin dictum sed non ipsum. Praesent ut risus nulla, sed blandit leo. Praesent ut risus nulla, sed blandit leo. Curabitur volutpat laoreet lacus, ut Curabitur volutpat laoreet lacus, ut consectetur arcu vestibulum vel. Donec consectetur arcu vestibulum vel. Donec dapibus fringilla arcu, et semper lacus dapibus fringilla arcu, et semper lacus
  11. 11. TF-IDF function getWeight($docID, $term, $total) { $tf = count($term[$docID]); $idf = log($total / count($term), 2); return $tf * $idf; } 11
  12. 12. Document Vector socket what heavy steel ... Doc 1 0.02 0.3 0.001 0 ... Doc 2 0 0 0 0 ... Doc 3 0.001 0.2 0 0 ... Doc 4 0 0 0.002 0.003 ... 12
  13. 13. Ranked Query Merge best 23 42 179 246 333 703 weight 0.008 0.002 0.023 0.039 0.014 0.001 western 42 88 120 179 246 798 weight 0.003 0.004 0.023 0.001 0.034 0.004 1 - 246: 0.073 2 - 179: 0.024 3 - 120: 0.023 13
  14. 14. PHP Similarity function score($queryString, $index) { $query = tokenize($queryString); $matches = array(); foreach($query as $qterm) { $postings = $index[$qterm]; foreach($postings as $id => $posting) { $matches[$id] += $posting['score']; } } return arsort($matches); } 14
  15. 15. Integrating Search 15
  16. 16. MySQL Full Text Search CREATE TABLE example ( id INT(11) NOT NULL auto_increment, title VARCHAR(255), content TEXT, PRIMARY KEY(id), FULLTEXT(title,content) ) Engine=MyISAM; INSERT INTO example (title, content) VALUES ('Mikko & Bacon','Mikko loves bacon'), ('Marcello & Bacon','Marcello hates bacon'), ('Jo & Sausages','Johanna loves sausages'), ('Hollywood & Garlic','Lorenzo hates garlic'), ('James & Cheddar','James is keen on cheeses'); 16
  17. 17. MySQL FTI Query SELECT * FROM example WHERE MATCH(title,content) AGAINST('loves bacon'); +----+------------------+------------------------+ | id | title | content | +----+------------------+------------------------+ | 1 | Mikko & Bacon | Mikko loves bacon | | 2 | Marcello & Bacon | Marcello hates bacon | | 3 | Jo & Sausages | Johanna loves sausages | +----+------------------+------------------------+ 3 rows in set (0.00 sec) 17
  18. 18. Sphinx http://www.sphinxsearch.com 18
  19. 19. Sphinx Configuration source posts { type = mysql sql_host = localhost sql_user = user sql_pass = password sql_db = search sql_query = SELECT id, title, content FROM example; sql_attr_multi = uint tag from query; SELECT example_id, tag_id FROM tags; } 19
  20. 20. index posts { source = posts path = /var/data/sphinx/example morphology = stem_en min_word_len = 3 min_prefix_len = 3 min_infix_len = 0 enable_star = 1 } 20
  21. 21. Stemming http://tartarus.org/~martin/PorterStemmer happening - happen happened - happen happens - happen 21
  22. 22. Command Line Searching indexer --config /etc/sphinx.conf --all search --config /etc/sphinx.conf love bacon displaying matches: 1. document=1, weight=3, tag=(1,2) ! id=1 ! title=Mikko & Bacon ! content=Mikko loves bacon words: 1. 'love': 2 documents, 2 hits 2. 'bacon': 2 documents, 4 hits searchd --config /etc/sphinx.conf 22
  23. 23. Sphinx From PHP $cl = new SphinxClient(); $cl->SetServer('localhost', 3312); $cl->SetMatchMode(SPH_MATCH_ANY); $result = $cl->Query('bac*'); $docIDs = array_keys($result["matches"]); $cl->SetFilter('tag', array(1)); $result = $cl->Query('bac*'); $docIDs = array_keys($result["matches"]); 23
  24. 24. Swish-E http://swish-e.org pecl install swish-beta 24
  25. 25. Filesystem Index With Swish-E /usr/local/bin/swish-e -S fs -c fs-swish-e.conf fs-swish-e.conf IndexDir /var/data/documents IndexFile fs-swish-e.index IndexOnly .doc .docx .pdf FuzzyIndexingMode Stemming_en1 FileFilter .pdf /usr/local/bin/swish_filter.pl FileFilter .doc /usr/local/bin/swish_filter.pl
  26. 26. Crawling Content /usr/local/bin/swish-e -S prog -c www-swish-e.conf www-swish-e.conf IndexDir /usr/local/lib/swish-e/spider.pl IndexFile www-swish-e.index SwishProgParameters default http://phpir.com/ FuzzyIndexingMode Stemming_en1 DefaultContents HTML
  27. 27. Swish-E With Multiple Indices $swish = new Swish( 'www-swish-e.index fs-swish-e.index' ); $search = $swish->prepare(); $queryStr = 'search string goes here'; $result = $search->execute($queryStr); $total = $result->hits; while($r = $result->nextResult()) { echo $r->swishdocpath; // url }
  28. 28. Lucene 28
  29. 29. $index = Zend_Search_Lucene::create('idx'); foreach($documents as $title => $content) { $doc = new Zend_Search_Lucene_Document(); $doc->addField( Zend_Search_Lucene_Field::Text( 'title', $title)); $doc->addField( Zend_Search_Lucene_Field::UnStored( 'content', $content)); $index->addDocument($doc); } Build Index 29
  30. 30. $results = $index->find('loves bacon'); foreach($results as $result) { echo $result->score, " "; echo $result->title, "n"; } Output: 0.81656279309067 Mikko and Bacon 0.24800278854758 Marcello & Bacon Query Zend Search Lucene 30
  31. 31. $file = file_get_contents($url); $doc = Zend_Search_Lucene_Document_Html:: loadHTML($file); $doc->addField( Zend_Search_Lucene_Field::Text( 'url', $url ); $index->addDocument($doc) Index HTML 31
  32. 32. Solr http://lucene.apache.org/solr/ 32
  33. 33. Solr Search Index $options = array( 'hostname' => 'localhost', 'port' => 8983 ); $client = new SolrClient($options); $doc = new SolrInputDocument(); $doc->addField('id', $id); $doc->addField('cat', $category); $doc->addField('title', $title); $doc->addField('text', $text); $response = $client->addDocument($doc); $client->commit(); 33
  34. 34. Solr Search Client $client = new SolrClient($options); $query = new SolrQuery('bacon'); $response = $client->query($query); $r = $response->getResponse(); foreach($r['response']['docs'] as $d) { echo $d->title[0] . "n"; } 34
  35. 35. Xapian http://xapian.org 35
  36. 36. Xapian In PHP $db = new XapianWritableDatabase( 'idx', Xapian::DB_CREATE_OR_OPEN); $i = new XapianTermGenerator(); $i->set_stemmer(new XapianStem("english")); $doc = new XapianDocument(); $doc->set_data($content); $doc->add_value(1, $title); $i->set_document($doc); $i->index_text($content); $db->add_document($doc); 36
  37. 37. Xapian Search In PHP $database = new XapianDatabase('idx'); $enquire = new XapianEnquire($database); $qp = new XapianQueryParser(); $qp->set_stemmer(new XapianStem("english")); $qp->set_database($database); $qp->set_stemming_strategy( XapianQueryParser::STEM_SOME); $query = $qp->parse_query($queryString); $enquire->set_query($query); 37
  38. 38. $matches = $enquire->get_mset(0, 10); $i = $matches->begin(); while(!$i->equals($matches->end())) { $n = $i->get_rank() + 1; $data = $i->get_document()->get_data(); $title = $i->get_document()->get_value(1); $score = $i->get_percent(); $i->next(); } 38
  39. 39. Improving Results 39
  40. 40. Anchor Text 40
  41. 41. Parse Anchor Text $p = file_get_contents('http://phpir.com'); libxml_use_internal_errors(true); $dom = DomDocument::loadHTML($p); $links = $dom->getElementsByTagName('a'); foreach($links as $link) { $href = $link->getAttribute('href'); $text = $link->nodeValue; } 41
  42. 42. 1 2 3 Zone Weighting 42
  43. 43. $doc = new Zend_Search_Lucene_Document(); $tfield = Zend_Search_Lucene_Field::Text ('title', $title); $tfield->boost = 1.3; $doc->addField($tfield); $doc->addField( Zend_Search_Lucene_Field::UnStored ('content', $content)); $index->addDocument($doc); ZSL Zone Weighting 43
  44. 44. Document Authority 44
  45. 45. Document Weights in ZSL $doc = new Zend_Search_Lucene_Document(); $doc->addField( Zend_Search_Lucene_Field::Text ('title', $title)); $doc->addField( Zend_Search_Lucene_Field::UnStored ('content', $content)); $doc->boost = 1 + ($numComments / 100); $index->addDocument($doc); 45
  46. 46. Using Search 46
  47. 47. Summaries & Highlighting 47
  48. 48. Sphinx Extract & Highlight $cl = new SphinxClient(); $cl->SetServer( "localhost", 3312 ); $q = 'bacon'; $r = $cl->Query($q); foreach ($r["matches"] as $doc => $info) { $text[$doc] = getTextFromDB($doc); } $e = $cl->BuildExcerpts($text, 'posts', $q); foreach($extracts as $extract) { echo $extract; } 48
  49. 49. Xapian Spelling Correction Indexer $indexer = new XapianTermGenerator(); $indexer->set_database($database); $indexer->set_flags( XapianTermGenerator::FLAG_SPELLING); Searcher $queryString = "strreplace or str_cmp"; $q = new XapianQueryParser(); $q->set_database($database); $query = $q->parse_query($queryString, XapianQueryParser::FLAG_SPELLING_CORRECTION); echo "Did you mean: " . $q->get_corrected_query_string() . "n"; 50
  50. 50. Spelling Correction Output php xapsearch.php Did you mean: str_replace or strcmp 4644 results found for “strreplace or str_cmp”: 1: 2% docid=572 [phpdocs/html/cc.license.html] 2: 2% docid=7169 [phpdocs/html/imagick.constants.html] 3: 2% docid=10086 [phpdocs/html/sqlite3result.fetcharray.html] 4: 2% docid=6132 [phpdocs/html/function.swf-posround.html] 51
  51. 51. Results Sorting 52
  52. 52. Sorting in ZSL $q = Zend_Search_Lucene_Search_QueryParser:: parse('search string'); $results = $index->find($q, 'title'); foreach($results as $result) { echo '<h3>', $result->title, "</h3>n"; $doc = getDocumentFromDB($result->did); echo $q->htmlFragmentHighlightMatches($doc); } 53
  53. 53. Faceted Search 54
  54. 54. Faceted Search In Solr $client = new SolrClient($options); $query = new SolrQuery('bacon'); $response = $client->query($query); $query->setFacet(true); $query->addFacetField('cat'); $r = $response->getResponse(); $f = $r['facet_counts']['facet_fields']; foreach($f['cat'] as $facet => $count) { echo $facet . " " . $count . "n"; } 55
  55. 55. More Like This 56
  56. 56. More Like This $rset = new XapianRset(); $rset->add_document(5959); // str_replace $e = $enquire->get_eset(40, $rset); $t = $e->begin(); for($t; !$t->equals($e->end()); $t->next()){ $qs[] = new XapianQuery($t->get_term(), intval($t->get_weight())); } $query = new XapianQuery( XapianQuery::OP_OR, $qs); 57
  57. 57. More Like This Example php xapsim.php 1656 results found: 1: 100% docid=5959 [phpdocs/html/function.str-replace.html] 2: 47% docid=5956 [phpdocs/html/function.str-ireplace.html] 3: 24% docid=5328 [phpdocs/html/function.preg-replace.html] 4: 18% docid=5958 [phpdocs/html/function.str-repeat.html] 58
  58. 58. Image Credits Title http://www.flickr.com/photos/generated/2084287794/ What Do You Want http://www.flickr.com/photos/the_justified_sinner/ You Are Here 2498066986/ http://www.flickr.com/photos/alecvuijlsteke/2692475420/ Integrating Search http://www.flickr.com/photos/squeaks2569/3700355684/ Sphinx http://www.flickr.com/photos/generated/2084287794/ Lucene http://www.flickr.com/photos/mypanda/7731447/ Swish-e http://www.flickr.com/photos/ryan_fung/2239687100/ Solr http://www.flickr.com/photos/m-j-s/2724756177/ Xapian http://www.flickr.com/photos/olibac/3522056495/ Using Search http://www.flickr.com/photos/eneas/175027945/ Improving Search http://www.flickr.com/photos/x-ray_delta_one/3928200642/ Search Performance http://www.flickr.com/photos/maisonbisson/1634408/ Large Scale Search http://www.flickr.com/photos/zedzap/3663508847/ 59
  59. 59. Questions? 60
  60. 60. Thank You! Ian Barber @ianbarber http://phpir.com ian@ibuildings.com http://joind.in/1556
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×