There are number of players that provide full text search feature, starting from embedded search to dedicated search servers [solr, sphinx, elasticsearch etc], but setting up and configuring them is a time consuming process and requires considerable knowledge of the tools.
What if we could get comparable search results using full text search capabilities of Postgres. Developers already have the working knowledge of the database, so this should come natural. In addition to that, it will be one less tool to manage.
Code: https://github.com/Syerram/postgres_search
2. Search is everywhere
Search expectations
● FAST
● Full Text search
● Linguistic support (“craziness | crazy”)
● Ranking
● Fuzzy Searching
● More like this
3. Django
● SLOW
● `icontains` is dumbed down version of
search
● Searching across tables is pain
● No relevancy, ranking or similar words
unless done manually
● No easy way for fuzzy searching
4. Other Alternatives
● Solr
● ElasticSearch
● AWS CloudSearch
● Sphinx
● etc*
If you’re using any of the above, use Haystack
5. Postgres Search
● FAST
● Simple to implement
● Supports Search features like Full Text,
Ranking, Boosting, Fuzzy etc..
6. Django
Live Example
● Search Students by name or by course
● Use South migration to create tsvector
column
● Store title in Search table
● Update Search table via Celery on Save of
Student data
https://github.com/Syerram/postgres_search
7. GIN, GIST
● GIST is Hash based, GIN is B-trees
● GINs = GISTs * 3 , s = Speed
● GINu = GISTu * 3 , u = update time
● GINkb = GISTkb * 3, kb = size
A gin index
CREATE INDEX student_index ON students USING gin(to_tsvector('english'
name));
Source http://www.postgresql.org/docs/9.2/static/textsearch-indexes.html
8. Full Text Search
● All text should be preprocessed using
tsvector and queried using tsquery
● Both reduce the text to lexemes
SELECT to_tsvector('How much wood would a woodchuck chuck If a woodchuck could
chuck wood?')
"'chuck':7,12 'could':11 'much':2 'wood':3,13 'woodchuck':6,10 'would':4"
● Both are required for searching to work on
normal text
SELECT to_tsvector('How much wood would a woodchucks chucks If a woodchucks could
chucks woods?') @@ 'chucks' -- False
SELECT to_tsvector('How much wood would a woodchucks chucks If a woodchucks could
chucks woods?') @@ to_tsquery('chucks') -- True
9. Full Text Search (Contd.)
● Technically you don’t need index, but for
large tables it will be slow
SELECT * FROM students where to_tsvector('english', name) @@ to_tsquery('english',
'Kirk')
● GIN or GIST Index
CREATE INDEX <index_name> ON <table_name> USING gin(<col_name>);
● Expression Based
CREATE INDEX <index_name> ON <table_name> USING gin(to_tsvector(COALESCE(col_name,'')
|| COALESCE(col_name,'')));
10. Boosting
● Boost certain results over others
● Still matching
● Use ts_rank to boost results
e.g.
…ORDER BY ts_rank(document,
to_tsquery('python')) DESC
11. Ranking
● Importance of search term within document
e.g.
Search term found in title > description > tag
● Use setweight to assign importance to each field
when preparing Document
e.g.
setweight(to_tsvector(‘english’, post.title), 'A') ||
setweight(to_tsvector(‘english’, post.description), 'B') ||
setweight(to_tsvector('english', post.tags), 'C'))
...
--In search query use ‘ts_rank’ to order by ranking
12. Trigram
● Group of 3 consecutive chars from String
● Similarity between strings is matched by # of
trigrams they share
e.g. "hello": "h", "he", "hel", "ell", "llo", "lo", and "o”
"hallo": "h", "ha", "hal", "all", "llo", "lo", and "o”
Number of matches: 4
● Use similarity to find related terms. Returns value
between 0 to 1 where 0 no match and 1 is exact match
13. Soundex/Metaphone
● Oldest and only good for English names
● Converts to a String of Length 4.
e.g. “Anthony == Anthoney” => “A535 ==
A535”
● Create index itself with Soundex or
Metaphone
e.g. CREATE INDEX idx_name ON tb_name USING
GIN(soundex(col_name));
SELECT ... FROM tb_name WHERE soundex(col_name) = soundex(‘...’)
14. Pro & Con
Pros
● Quick implementation
● Lot easier to change document format and call refresh index
● Speed comparable to other search engines
● Cost effective
Cons
● Not as flexible as pure search engines, like Solr
● Not as fast as Solr though pretty fast for humans
● Tied to Postgres
● Indexes can get pretty large, but so can search engine indexes
15. Django ORM
● Implements Full text Search
class StudentCourse(models.Model):
...
search_index = VectorField()
objects = SearchManager(
fields = ('student__user__name', 'course__name'),
config = 'pg_catalog.english', # this is default
search_field = 'search_index', # this is default
auto_update_search_field = True
)
● StudentCourse.objects.search("David")
https://github.com/djangonauts/djorm-ext-pgfulltext
16. Next Steps
● Add Ranking, Boosting, Fuzzy Search to
djorm pgfulltext
e.g. StudentCourse.objects.search("David & Python").rank("Python")
StudentCourse.objects.fuzzy_search("Jython").rank("Python")
StudentCourse.objects.soundex("Davad").rank("Java") & More
● Continue to add examples to
postgres_search
17. Tips
● Use separate DB if necessary or use
Materialized Views
● Don’t index everything. Limit your
searchable data
● Analyze using `Explain` and ts_stat
● Create indexes on fly using concurrently
● Don’t pull Foreign Key objects in search
