Full Text search in Django with Postgres

5,152 views
4,865 views

Published on

There are number of players that provide full text search feature, starting from embedded search to dedicated search servers [solr, sphinx, elasticsearch etc], but setting up and configuring them is a time consuming process and requires considerable knowledge of the tools.

What if we could get comparable search results using full text search capabilities of Postgres. Developers already have the working knowledge of the database, so this should come natural. In addition to that, it will be one less tool to manage.

Code: https://github.com/Syerram/postgres_search

Published in: Technology, Design

Full Text search in Django with Postgres

  1. 1. Full Text Search Django + Postgres
  2. 2. Search is everywhere Search expectations ● FAST ● Full Text search ● Linguistic support (“craziness | crazy”) ● Ranking ● Fuzzy Searching ● More like this
  3. 3. Django ● SLOW ● `icontains` is dumbed down version of search ● Searching across tables is pain ● No relevancy, ranking or similar words unless done manually ● No easy way for fuzzy searching
  4. 4. Other Alternatives ● Solr ● ElasticSearch ● AWS CloudSearch ● Sphinx ● etc* If you’re using any of the above, use Haystack
  5. 5. Postgres Search ● FAST ● Simple to implement ● Supports Search features like Full Text, Ranking, Boosting, Fuzzy etc..
  6. 6. Django Live Example ● Search Students by name or by course ● Use South migration to create tsvector column ● Store title in Search table ● Update Search table via Celery on Save of Student data https://github.com/Syerram/postgres_search
  7. 7. GIN, GIST ● GIST is Hash based, GIN is B-trees ● GINs = GISTs * 3 , s = Speed ● GINu = GISTu * 3 , u = update time ● GINkb = GISTkb * 3, kb = size A gin index CREATE INDEX student_index ON students USING gin(to_tsvector('english' name)); Source http://www.postgresql.org/docs/9.2/static/textsearch-indexes.html
  8. 8. Full Text Search ● All text should be preprocessed using tsvector and queried using tsquery ● Both reduce the text to lexemes SELECT to_tsvector('How much wood would a woodchuck chuck If a woodchuck could chuck wood?') "'chuck':7,12 'could':11 'much':2 'wood':3,13 'woodchuck':6,10 'would':4" ● Both are required for searching to work on normal text SELECT to_tsvector('How much wood would a woodchucks chucks If a woodchucks could chucks woods?') @@ 'chucks' -- False SELECT to_tsvector('How much wood would a woodchucks chucks If a woodchucks could chucks woods?') @@ to_tsquery('chucks') -- True
  9. 9. Full Text Search (Contd.) ● Technically you don’t need index, but for large tables it will be slow SELECT * FROM students where to_tsvector('english', name) @@ to_tsquery('english', 'Kirk') ● GIN or GIST Index CREATE INDEX <index_name> ON <table_name> USING gin(<col_name>); ● Expression Based CREATE INDEX <index_name> ON <table_name> USING gin(to_tsvector(COALESCE(col_name,'') || COALESCE(col_name,'')));
  10. 10. Boosting ● Boost certain results over others ● Still matching ● Use ts_rank to boost results e.g. …ORDER BY ts_rank(document, to_tsquery('python')) DESC
  11. 11. Ranking ● Importance of search term within document e.g. Search term found in title > description > tag ● Use setweight to assign importance to each field when preparing Document e.g. setweight(to_tsvector(‘english’, post.title), 'A') || setweight(to_tsvector(‘english’, post.description), 'B') || setweight(to_tsvector('english', post.tags), 'C')) ... --In search query use ‘ts_rank’ to order by ranking
  12. 12. Trigram ● Group of 3 consecutive chars from String ● Similarity between strings is matched by # of trigrams they share e.g. "hello": "h", "he", "hel", "ell", "llo", "lo", and "o” "hallo": "h", "ha", "hal", "all", "llo", "lo", and "o” Number of matches: 4 ● Use similarity to find related terms. Returns value between 0 to 1 where 0 no match and 1 is exact match
  13. 13. Soundex/Metaphone ● Oldest and only good for English names ● Converts to a String of Length 4. e.g. “Anthony == Anthoney” => “A535 == A535” ● Create index itself with Soundex or Metaphone e.g. CREATE INDEX idx_name ON tb_name USING GIN(soundex(col_name)); SELECT ... FROM tb_name WHERE soundex(col_name) = soundex(‘...’)
  14. 14. Pro & Con Pros ● Quick implementation ● Lot easier to change document format and call refresh index ● Speed comparable to other search engines ● Cost effective Cons ● Not as flexible as pure search engines, like Solr ● Not as fast as Solr though pretty fast for humans ● Tied to Postgres ● Indexes can get pretty large, but so can search engine indexes
  15. 15. Django ORM ● Implements Full text Search class StudentCourse(models.Model): ... search_index = VectorField() objects = SearchManager( fields = ('student__user__name', 'course__name'), config = 'pg_catalog.english', # this is default search_field = 'search_index', # this is default auto_update_search_field = True ) ● StudentCourse.objects.search("David") https://github.com/djangonauts/djorm-ext-pgfulltext
  16. 16. Next Steps ● Add Ranking, Boosting, Fuzzy Search to djorm pgfulltext e.g. StudentCourse.objects.search("David & Python").rank("Python") StudentCourse.objects.fuzzy_search("Jython").rank("Python") StudentCourse.objects.soundex("Davad").rank("Java") & More ● Continue to add examples to postgres_search
  17. 17. Tips ● Use separate DB if necessary or use Materialized Views ● Don’t index everything. Limit your searchable data ● Analyze using `Explain` and ts_stat ● Create indexes on fly using concurrently ● Don’t pull Foreign Key objects in search
  18. 18. Code • https://github.com/Syerram/pos tgres_search • Stack • AngularJS, Django, Celery, Postgres • Feel free to Fork, Pull Request
  19. 19. @agileseeker, github/syerram, syerram.silvrback.com/ Sai

×