Comparing open source search engines

Comparing open source search engines Richard Boulton @rboulton [email_address]

Search Engine? Document oriented database Inverted index Ranking / weighting algorithm

Lucene Java Apache License Low-level: Java API

Lucene Family Solr: “REST-like” XML/JSON API ElasticSearch: REST API … and many commercial engines

Xapian C++ GPLv2 Low-level: C++ API Python/Ruby/PHP/Perl/Java bindings

Xapian C++ GPLv2 Low-level: C++ API Python/Ruby/PHP/Perl/Java bindings Partiality Risk

Xapian Family Omega: Indexer + CGI interface Flax: REST API Xappy: Python wrapper

Sphinx C++ GPLv2 SQL-like API

Others Riak Search Terrier MySQL Fulltext PostgreSQL FTS Redis Whoosh

Document model Lucene, Xapian: List of terms Solr, Sphinx: Fields in a predefined fixed schema. Flax, Xappy: Fields, with associated modifiable schema. ElasticSearch: Fields, document types, free schema.

Updates Lucene, Xapian + families: Dynamic updates Use batches for fastest updates Sphinx: No updates to existing indexes (“Realtime indexing” in beta with SQL API)

Data structures Lucene: Hash based segments Heirarchical merge Xapian: B-tree, transactional

Scaling / replication
All engines allow searches across databases
Allows sharding
All engines allow replication
Allows spreading load and high availability
Had difficulty with Sphinx
Elastic search does it completely transparently

Commercial Support Lucene: Lucid Imagination, Sematext, … Xapian: Oligarchy Ltd, Flax, me Sphinx: Sphinx Technologies Inc

Lucene / Solr community – revolting (they say)
Xapian – quieter, but steadily growing
Sphinx – popular amongst relational database users (apparently)
Community

http://lanyrd.com	14
http://paper.li	2
https://lanyrd.com	1

Comparing open source search engines

by Richard Boulton on Nov 08, 2010

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 17

Statistics

Comparing open source search engines Presentation Transcript