The document provides a detailed analysis and comparison of various search engines from a developer's perspective, focusing on features, configuration options, and integration capabilities using APIs and frameworks like Apache Lucene, Solr, Elasticsearch, and Sphinx. It discusses aspects such as indexing, searching, data handling, and performance metrics relevant to enterprise-class applications. Additionally, it highlights the requirements and advantages of specific systems along with example code snippets for implementation.

1. An analysis and comparison from a developer’s perspective

3.  Report Buyer product catalogue:
• Text fields: title, subtitle, summary, toc
• Product code and ISBN
• Supplier, category, type and availability
• Publication date and price

4. Enterprise class search engine
Scalable and based on Apache Lucene
REST-ful API or PECL extension
Fast, transactional full-text indexing
Faceted and geospatial search
Rich document indexing
Comes with simple web interface
Built-in caching of queries and responses
Numerous plug-ins

5.  Available as system packages
 Uses Tomcat or Jetty
 Requires a restart on configuration change
 Packages install as a service

6.  Specify database location
 Memory settings
 Query caching options
 Request handler setup
 Search components and plug-ins
 Spell checker configuration

7. <!-- Report Buyer fields -->
<field name="item_guid" type="string" indexed="true" stored="true" required="true" />
<field name="name" type="text" indexed="true" stored="true" required="true" boost="75"
omitNorms="false" />
<field name="subtitle" type="text" indexed="true" stored="true" required="false" boost="25"
omitNorms="false" />
<field name="summary" type="text" indexed="true" stored="false" boost="1" omitNorms="false" />
<field name="toc" type="text" indexed="true" stored="false" boost="1" omitNorms="false" />
<field name="isbn" type="string" indexed="true" stored="false" boost="200" omitNorms="false" />
<field name="product_code" type="string" indexed="true" stored="true" boost="200" omitNorms="false" />
<field name="publish_date" type="tdate" indexed="true" stored="true" />
<field name="price" type="tfloat" indexed="true" stored="true" />
<field name="availability" type="boolean" indexed="true" stored="true" />
<field name="link" type="string" indexed="false" stored="true" />
<field name="text" type="text" indexed="true" stored="false" multiValued="true"/>
<copyField source="name" dest="text"/>
<copyField source="subtitle" dest="text"/>
<copyField source="summary" dest="text"/>
<copyField source="toc" dest="text"/>
<uniqueKey>item_guid</uniqueKey>
<defaultSearchField>text</defaultSearchField>

8.  Data Import Handler
 REST-ful API
 PHP PECL Extension
 Third-party libraries, like Solarium

9. <?php
$solr_options = array('secure' => false, 'hostname' => 'localhost', 'port' => 8080);
$solr = new SolrClient($solr_options);
$doc = new SolrInputDocument();
while ($row = mysql_fetch_array($result, MYSQL_ASSOC))
{
$doc = new SolrInputDocument();
$row['publish_date'] = strftime('%Y-%m-%dT00:00:01Z', strtotime($row['publish_date']));
foreach ($row as $key => $value) {
$doc->addField($key, $value);
}
$updateResponse = $solr->addDocument($doc);
$response = $updateResponse->getResponse();
if ($response->responseHeader->status != 0) {
print "Error importing into Solr: ";
print_r($response);
}
}
$solr->commit();
?>

10. POST to http://localhost:8080/solr/update?commit=true
<add>
<doc>
<field name="item_guid">a34bbff9e17ada79658c72fde90c7369</field>
<field name="name">Research Report on China's Corn Industry</field>
<field name="price">1265</field>
etc
</doc>
</add>

11. $solr_options = array('secure' => false, 'hostname' => 'localhost', 'port' => 8080);
$solr = new SolrClient($solr_options);
$query = new SolrQuery();
$query->setQuery("research in china");
$query->setFacet(true);
$query->addFacetField('availability');
$query->addField('item_guid')->addField('name')->addField('publish_date')->addField('subtitle')->
addField('product_code')->addField('availability')->addField('price');
$query->addSortField('publish_date', SolrQuery::ORDER_DESC);
$query_response = $solr->query($query);
$response = $query_response->getResponse();
print "Found ".$response->response->numFound." results, for {$query_string} in ".$response-
>responseHeader->QTime." ms:nn";
foreach ($response->response->docs as $position=>$doc_data) {
$download = ($doc_data['availability'] == '1') ? 'Yes' : 'No';
print "{$position} - Date:{$pub_date} - {$doc_data['product_code']} - D/L:{$download} £".sprintf("%5d",
$doc_data['price'])." - {$doc_data['name']}n";
}
print "Facets for instant ".$response->facet_counts->facet_fields->availability->false;

12. http://localhost:8080/solr/select/?q=research%20%in%20china&indent=on&hl=true&hl.fl=item_guid,name,
publish_date,subtitle,product_code,availability,price&facet=true&facet.field=availability&wt=json
{
"responseHeader":{
"status":0, "QTime":20,
"params":{
"facet":"true", "indent":"on", "q":"research u0000 china",
"hl.fl":"item_guid,name,publish_date,subtitle,product_code,availability,price",
"facet.field":"availability", "wt":"json", "hl":"true"}},
"response":{"numFound":197481,"start":0,"docs":[
{
"item_guid":"e68cf64921a02e926137d78d2c52da35",
"name":"Market Research Report on China Civil Aero Industry",
"product_code":"SFC00076",
"price":190.0, "availability":false,
"type":10,
"link":
"/industry_manufacturing/plant_heavy_equipment/market_research_report_china_civil_aero_industry.
html",
"publish_date":"2008-07-22T00:00:01Z"
}
}

13.  More features than other products
 Responsive, busy mailing list
 Large team of developers
 Good PHP libraries for integration
 Several books available
 Fairly heavy footprint

14.  Also built on Apache Lucene
 JSON-based
 Distributed, scalable server model
 Easy to configure, or configuration free
 Faceting and highlight support
 Auto type detection
 Multiple indexes
 CouchDB integration

15.  Download and unpack zip file
 Run elasticsearch/bin/elasticsearch

16.  No schema is required - almost
 No configuration is required - almost

17. GET http://localhost:9200/ HTTP/1.0
{
"ok" : true,
"name" : "Test",
"version" : {
"number" : "0.18.7",
"snapshot_build" : false
},
"tagline" : "You Know, for Search",
"cover" : "DON'T PANIC",
"quote" : {
"book" : "The Hitchhiker's Guide to the Galaxy",
"chapter" : "Chapter 27",
"text1" : ""Forty-two," said Deep Thought, with infinite majesty and calm.",
"text2" : ""The Answer to the Great Question, of Life, the Universe and Everything""
}
}

18. curl -XPUT http://localhost:9200/reports/ -d '
{
"index:" {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "standard",
"filter": ["standard", "lowercase", "my_stemmer"]
}
},
"filter": {
"my_stemmer": {
"type": "stemmer",
"name": "english"
}
}
}
}
}'

19. <?php
require_once("ElasticSearch.php");
$es = new ElasticSearch;
$es->index = 'reports';
$type = 'report';
$mappings = array($type => array('properties' => array(
'_id' => array('type' => 'string', 'path' => 'item_guid'),
'item_guid' => array('type' => 'string', 'store' => 'yes', 'index' => 'not_analyzed'),
'name' => array('type' => 'string', 'store' => 'no', 'boost' => 75),
'subtitle' => array('type' => 'string', 'store' => 'yes', 'boost' => 25),
'summary' => array('type' => 'string', 'store' => 'yes', 'boost' => 10),
'toc' => array('type' => 'string', 'store' => 'no'),
'product_code' => array('type' => 'string', 'store' => 'yes', 'boost' => 200, 'index' => 'not_analyzed'),
'isbn' => array('type' => 'string', 'store' => 'yes', 'boost' => 200, 'index' => 'not_analyzed'),
)));
$json = json_encode($mappings);
$es->map($type, $json);
?>

20. <?php
require_once("ElasticSearch.php");
$es = new ElasticSearch;
$es->index = 'reports';
$type = 'report';
$sql = "SELECT `item_guid`, `name`, `subtitle`, `summary`, `toc`, `supplier`,
`product_code`, `isbn`, `category`, `price`, `availibility` as `availability`,
`type`, `link`, `publish_date`
FROM `rb_search`";
$result = read_query($sql);
while ($row = mysql_fetch_array($result, MYSQL_ASSOC))
{
$es->add($type, $row['item_guid'], json_encode($row));
}
?>

21. GET http://localhost:9200/reports/report/_count/
{"count":260349,"_shards":{"total":1,"successful":1,"failed":0}}

22. <?php
require_once("ElasticSearch.php");
$es = new ElasticSearch;
$es->index = 'reports';
$type = 'report';
$query = array(
'fields' => array('item_guid', 'name', 'subtitle'),
'query' => array(
'term' => array('name' => 'research'),
),
'facets' => array(
'availability' => array(
'terms' => array('field' => 'availability')
)
)
);
$result = $es->query($type, json_encode($query));
?>

23.  Nicholas Ruflin's elastica
 Raymond Julin's elasticsearch
 Niranjan Uma Shankar's elasticsearch-php

24.  Very fast indexing
 Auto-scaling architecture
 Elegant REST approach
 Flexible zero configuration model
 Poor documentation
 No feature list, conceptual model or
introduction
 All data is stored, meaning large indices

25.  Indexes MySQL, MSSQL, XML or ODBC
 Querying through Sphinx PHP API
 Searching through SQL queries or API
 Scalable to index 6TB of data in 16bn
documents and 2000 queries/sec
 Used by Craigslist, Boardreader
 Runs as a storage engine in MySQL

26.  Install from system packages or source
 Source tarball is needed to get PHP
SphinxAPI
 No other software needed
 Runs as a service in Ubuntu

27.  Plain index - fast search, slow update
 Real-time index - fast update, less efficient
 Distributed - combination of both methods

28. index rb_test
{
# index type
type = rt
path = /mnt/data_indexed/sphinx/rb_test
# define the fields we're indexing
rt_field = name
rt_field = subtitle
rt_field = summary
rt_field = toc
#define the fields we want to get back out
rt_attr_string = item_guid
rt_attr_string = supplier
rt_attr_string = product_code
rt_attr_string = isbn
rt_attr_string = category
rt_attr_uint = price
rt_attr_string = link
rt_attr_timestamp = publish_date
# morphology preprocessors to apply
morphology = stem_en
html_strip =1
html_index_attrs = img=alt,title; a=title;
html_remove_elements = style, script
}

29. <?php
require_once("mysql.inc.php");
$sql = "SELECT conv(mid(md5(`item_guid`), 1, 16), 16, 10) AS `id`, `item_guid`, `name`,
`subtitle`, `summary`, `toc`, `supplier`, `product_code`, `isbn`, `category`,
`price`, `availibility` as `availability`, `type`, `link`, UNIX_TIMESTAMP(`publish_date`) AS
`publish_date` FROM `rb_search`";
$result = read_query($sql);
$sphinx = mysql_connect("127.0.0.1:9306", "", "", true);
while ($row = mysql_fetch_array($result, MYSQL_ASSOC)) {
foreach ($row as $key=>$value) {
$row[$key] = mysql_escape_string($value);
}
$sql = "REPLACE INTO `rb_search` (`id`, `title`, `subtitle`,`availability`, `type`, `price`, `publish_date`,
`item_guid`, `supplier`, `product_code`, `isbn`, `category`, `link`, `summary`, `toc`)
VALUES
('{$row['id']}', '{$row['name']}', '{$row['subtitle']}', '{$row['availability']}',
'{$row['type']}','{$row['price']}', '{$row['publish_date']}', '{$row['item_guid']}', '{$row['supplier']}',
'{$row['product_code']}', '{$row['isbn']}', '{$row['category']}', '{$row['link']}','{$row['summary']}',
'{$row['toc']}')";
mysql_query($sql, $sphinx);
}
?>

30. mysql --host=127.0.0.1 --port=9306
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 1
Server version: 2.0.3-id64-release (r3043)
mysql> select item_guid, title, subtitle, price from rb_search where match('china pharmaceutical') and price
> 100 and price < 300 limit 2G
************************** 1. row ***************************
id: 5228810066049016302
weight: 6671
price: 220
item_guid: cc74cb075aa37696198e87850f033398
title: North China Pharmaceutical Group Corp-Therapeutic Competitors Report
subtitle:
*************************** 2. row ***************************
id: 3548867347418583847
weight: 6662
price: 190
item_guid: 6ce04df0fb277aa3ff596c2ca00c81a9
title: China Pharmaceutical Industry Report
subtitle: 2006-2007
2 rows in set (0.01 sec)

31.  Fastest indexing of all engines
 Really simple interface via SQL
 Document IDs must be unsigned integers
 No faceting support
 Good support in forums

32.  Deployed as a C++ library
 Bindings provided to connect to PHP
 Available in most package repositories
 Binding need to be compiled separately
 Query Parser, similar to other engines
 Stemming and faceted search
 Server replication

33.  Install from system packages
 Compile PHP bindings from source
 No other software needed
 Runs on demand

34.  No configuration required
 Define-and-go schema
 Documents
 Terms
 Values
 Document data

35. <?php
$xapian_db = new XapianWritableDatabase($xapian, Xapian::DB_CREATE_OR_OVERWRITE);
$xapian_term_generator = new XapianTermGenerator();
$xapian_term_generator->set_stemmer(new XapianStem("english"));
while ($row = mysql_fetch_array($result, MYSQL_ASSOC)) {
$doc = new XapianDocument();
$xapian_term_generator->set_document($doc);
foreach ($xapian_term_weights as $field => $weight) {
$xapian_term_generator->index_text($row[$field], $weight);
}
$xapian_term_generator->index_text($row['name'], 75, 'S:');
$doc->add_boolean_term('CODE:' . $row['product_code']);
$doc->add_value($xapian_value_slots['price'], Xapian::sortable_serialise($row['price']));
$doc->add_value($xapian_value_slots['publish_date'], strftime("%Y%m%d",
strtotime($row['publish_date'])));
// add in additional values that we're going to use for facets
$doc->add_value($xapian_value_slots['availability'], $row['availability']);
$doc->set_data(serialize($doc_data));
$docid = 'Q'.$row['item_guid'];
$xapian_db->replace_document($docid, $doc);
}
?>

36. <?php
$xapian_db = new XapianDatabase($xapian);
$query_parser = new XapianQueryParser();
$query_parser->set_stemmer(new XapianStem("english"));
$query_parser->set_default_op(XapianQuery::OP_AND);
$dvrProcessor = new XapianDateValueRangeProcessor($xapian_value_slots['publish_date'], 'date:');
$query_parser->add_valuerangeprocessor($dvrProcessor);
$query_parser->add_prefix("code", "CODE:");
$query_parser->add_prefix("category", "CATEGORY:");
$query_parser->add_prefix("title", "S:");
$query = $query_parser->parse_query('“Medical devices” NEAR china NOT russian price:10..150 category:medical');
$enquire = new XapianEnquire($xapian_db);
$enquire->set_query($query);
$matches = $enquire->get_mset($offset, $pagesize);
while (!($start->equals($end))) {
$doc = $start->get_document();
$price = Xapian::sortable_unserialise($doc->get_value($xapian_value_slots['price']));
$start->next();
}?>

37.  Only one option available from Xapian
 Requires additional compilation due to
licensing
 Not very well documented API

38.  Reasonably fast indexing
 Very flexible implementation
 Faceting and range searching
 Good Quick Start guide
 Responsive mailing list
 Third-party paid support

39.  Every project has different needs
 Not one search product fits all
 Fastest to index was Sphinx
 Most feature-rich: Solr
 The next steps are up to you