Your SlideShare is downloading. ×
Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecker for Classified Ads
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecker for Classified Ads


Published on

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL …

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL

This session aims to explain the implementation and use case for spellchecking in Trovit search engine. Trovit is a classified ads search engine supporting several different sites, one for each on country and vertical. Our search engine supports multiple indexes in multiple languages, each with several millions of indexed ads. Those indexes are segmented in several different sites depending on the type of ads (homes, cars, rentals, products, jobs and deals). We have developed a multi-language spellchecking system using solr and lucene in order to help our users to better find the desired ads and avoid the dreaded 0 results as much as possible. As such our goal is not pure orthographic correction, but also suggestion of correct searches for a certain site.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 2. Outline •  •  •  •  •  •  •  •  Introduction Our approach: Contextual Spellchecking Nature and characteristics of our document corpus Spellcheckers in Solr White-listing and purging: controlling dictionary data Spellchecker configuration Customizing Solr’s SpellcheckComponent Conclusions and Future Work
  • 3. Supporting text for this speech Trovit Engineering Blog post on spellchecking
  • 5. Introduction Trovit: a search engine for classified ads
  • 6. Introduction
  • 7. Introduction: spellchecking in Trovit •  Multi-language spellchecking system using SOLR and Lucene •  Objectives: –  help our users to better find the desired ads –  avoid the dreaded 0 results as much as possible –  Our goal is not pure orthographic correction but also to suggest correct searches for a certain site.
  • 9. Contextual Spellchecking: approach •  The Key element in the spellchecking process is choosing the right dictionary –  one with a relevant vocabulary •  according to the type of information included in each site. •  Approach –  Specializing the dictionaries based on user’s search context. •  Search contexts are composed of: –  country (with a default language) –  vertical (determining the type of ads and vocabulary).
  • 10. Contextual Spellchecking: vocabularies •  Each site’s document corpus has a limited vocabulary –  reduced to the type of information, language and terms included in each site’s ads. •  Using a more generalized approach is not suitable for our needs –  One vocabulary for each language less precise than specialized vocabularies for each site. –  Drastic differences •  type of terms •  semantics of each vertical. –  Terms that are relevant in one context are meaningless in another one •  Different vocabularies for each site, even when supporting the same language. –  Vocabulary is tailored according to context of searches
  • 12. Challenges: Inconsistencies in our corpus •  Document corpus is fed by different third-party sources –  providing the ads for the different sites. •  We can detect incorrect documents and reconcile certain inconsistences –  But we cannot control or modify the content of the ads themselves. Inconsistencies –  hinder any language detection process –  pose challenges to the development of the spellchecking system • 
  • 13. Inconsistencies example •  Spanish homes vertical –  not fully written in Spanish –  Ads in several languages. •  native languages: Spanish, Catalan, Basque and Galician. •  foreign languages: English, German, French, Italian, Russian… even oriental languages like Chinese! •  Multi-language ads –  badly written and misspelled words •  Spanish words badly translated from regional languages •  overtly misspelled words –  e.g. “picina” yields a 1197 docs Vs 1048434 of “piscina”, 0.01% –  “noisy” content •  numbers, postal codes, references, etc.
  • 14. Characteristics of our ads •  Summarizing –  Segmented corpus in different indexes, one per country plus vertical (site) –  3rd party generated –  Ads in national language + other languages (regional and foreign) –  Multi-language content in ads –  Noisy content (numbers, references, postal codes, etc.) –  Small texts (around 3000 characters long) –  Misspellings and incorrect words Corpus unreliable for use as the knowledge base to build any spellchecking dictionary.
  • 15. What/Where search segmentation geolocation data is not mixed with vertical data geolocation data interleaved with vertical data Only vertical data (no geodata) •  Narrower dictionary, less collisons, more controlable Cover all geodata •  Wider dictionary, more collisons, less controlable
  • 17. IndexBasedSpellchecker •  •  It creates a parallel index for the spelling dictionary that is based on an existing Lucene index. –  Depends on index data correctness (misspells) –  Creates additional index from current index (small, MB) –  Supports term frequency parameters –  Must (re)build Even though this component behaves as expected –  it was of no use for Trovit’s use case.
  • 18. IndexBasedSpellchecker •  •  •  It depends on index data –  not an accurate and reliable for the spellchecking dictionary. Continuous builds –  synchronicity between index data and spelling index data. –  If not •  frequency information and hit counting are neither reliable nor accurate. •  false positives/negatives •  suggestions of words with different number of hits, even 0. We cannot risk suffering this situation
  • 19. FileBasedSpellChecker •  It uses a flat file to generate a spelling dictionary in the form of a Lucene spellchecking index. –  Requires a dictionary file –  Creates additional index from dictionary file (small, MB) –  Does not depend on index data (controlled data) –  Build once •  rebuild only if dictionary is updated –  No frequency information used when calculating spelling suggestions
  • 20. FileBasedSpellChecker •  •  •  Requires rebuilds also –  albeit less frequently No frequency related data –  Pure orthographic correction is not our main goal –  We cannot risk suggesting corrections without results. But –  insight on how to approach the final solution we are implementing. –  allows the highest degree of control in dictionary contents •  essential feature for spelling dictionaries.
  • 21. DirectSpellChecker •  Experimental spellchecker that just uses the main Solr index directly –  Build/rebuild is not required. –  Depends on index data correctness (misspells) –  Uses existing index •  field: source of the spelling dictionary. –  Supports term frequency parameters. –  No (re)build. •  Several promising features –  No build + continuously in sync with index data. –  Provides accurate frequency information data.
  • 22. DirectSpellChecker •  •  The real drawback –  lack of control over index data sourcing the spelling dictionary. If we can overcome it, this type would make an ideal candidate for our use case.
  • 23. WordBreakSpellChecker •  Generates suggestions by combining adjacent words and/or breaking words into multiples. –  This spellchecker can be configured with a traditional checker (ie:DirectSolrSpellChecker). –  The results are combined and collations can contain a mix of corrections from both spellcheckers. –  Uses existing index. No build.
  • 24. WordBreakSpellChecker •  •  •  •  Good complement to the other spellcheckers It works really well with well-written concatenated words –  it is able to break them up with great accuracy. Combining split words is not as accurate Drawback: it’s based on index data.
  • 26. White-listing •  •  •  Any spelling system can only be as good as its knowledge base or dictionary is accurate. We need to control the data indexed as dictionary. White-listing approach –  we only index spelling data contained in a controlled dictionary list. –  processes to build a base dictionary specialized for a given site.
  • 27. White-list building process
  • 29. Initial spellchecker configuration •  •  DirectSpellChecker using purged spell field –  Spell field filled with purged content •  Purging according to whitelist •  Whitelist generated from matching dictionary with index words, after purge process Benefits: –  Build is no longer required. –  Spell field is automatically updated via pipeline. –  We can work with term freq. –  No additional index, just an additional field. –  Better relevance and suggestions.
  • 30. Initial spellchecker configuration •  •  Cons: –  Whitelist maintenance and creation for new sites. Features: –  Accurate detection of misspelled words. –  Good detection of concatenated words. •  piscinagarajejardin to piscina garaje jardin •  picina garajejardin to piscina (garaje jardin) –  Able to detect several misspelled words. –  Evolution based on whitelisting fine-tuning.
  • 31. Initial spellchecker configuration •  Issues: –  False negatives: suggestion of corrections when words are correctly spelled. –  Suggestions for all the words in the query, not just those misspelled words. –  Misguiding “correctlySpelled” parameter. •  Parameter dependant on frequency information, making it unreliable for our purposes. •  It returns true/false according to thresholds, –  not really depending on word distance but –  results found, “alternativeTermCount” and “maxResultsForSuggest” thresholds. –  Minor discrepancies if we only index boosted terms (i.e. qf) •  # hits spell< #docs index
  • 33. Hacking SpellcheckComponent •  Lack of reliability of the “correctlySpelled” parameter –  Difficult to know when give a suggestion or not. –  First policy based on document hits •  sliding window –  based on the number of queried terms •  the longer the tail, the smaller the threshold •  inaccurate and prone to collisions. –  Difficult to set up thresholds to a good level of accuracy. We needed a more reliable way.
  • 34. Hacking SpellcheckComponent: correctlySpelled parameter behaviour •  Binary approach to deciding if a word is correctly spelled or not. •  Simpler approach –  any term that appears in our spelling field is a correctly spelled word •  regardless the value of its frequency info or the configured thresholds. –  this way the parameter can be used to control when to start querying the spellchecking index.
  • 35. Hacking SpellcheckComponent •  Other changes to the SpellcheckComponent: –  No suggestions when words are correctly spelled. –  Only makes suggestions for misspelled words, not for all words •  i.e. piscina garage -> piscina garaje •  Spanish-friendly ASCIIFoldingFilter –  modified in order to not fold “ñ” (for Spanish) and “ç” (for Catalan names) characters. •  Avoids collisions with similar words with “n” and “c” –  e.g. “pena” and “peña” –  Still folding accented vowels •  usually omitted by users.
  • 37. Conclusion & Future Work •  •  •  Base code –  expand the spellchecking process to other sites –  design final policy to decide when giving suggestions or not. Geodata in homes verticals –  find ways to avoid collisions in large dictionary sets. Scoring system for spelling dictionary –  Control suggestions based on user input •  Feedback on relevance or quality of our spellchecking suggestions. •  System more accurate and reliable •  Expand whitelists to cover large amounts of geodata –  with acceptable levels of precision.
  • 38. Conclusion & Future Work •  Plural suggester –  suggest alternative searches and corrections using plural or singular variants of the terms in the query. –  Use frequency and scoring information to choose most suitable suggestions.
  • 40. References [1] Lucene/Solr Revolution EU 2013. Dublin, 6-7 November 2013. [2] Trovit – A search engine for classified ads of real estate, jobs, cars and vacation rentals. [3] Apache Software Foundation. “Apache Solr” [4] Apache Software Foundation. “Apache Lucene” [5] Apache Software Foundation. “Spell Checking – Apache Solr Reference Guide – Apache Software Foundation”