I. How we use Solr @ CareerBuilder
II. Traditional Relevancy Scoring
III. Advanced Relevancy through functions
– Factors as a linear function
– Context-aware relevancy parameter weighting
III. Personalization & Recommendations
– Profile and Behavior-based
– Solr as a recommendation engine
– Collaborative Filtering
IV. Semantic Search
Mining user-behavior for synonyms
Uncovering meaning through clustering
Latent Semantic Indexing overview
Foreground vs. Background analysis
TF * IDF
Term Frequency: “How well a term describes a document?”
– Measure: how often a term occurs per document
Inverse Document Frequency: “How important is a term overall?”
– Measure: how rare the term is across all documents
Boosting documents and fields
Certain fields may be more important than other fields:
– The Job Title and Skills may be more relevant than other aspects of the job:
/select?qf=jobtitle^10 skills^5 jobrequirements^2 jobdescription^1
It’s possible to boost documents and fields at both index time and query time
If you need more fine-grained control (such as per-term index-time boosting),
you can make use of payloads
Custom scoring with Payloads
In addition to boosting search terms and fields, content within Fields can also be
boosted differently using Payloads (requires a custom scoring implementation):
design  / engineer  / really [ ] / great [ ] / job [ ] / ten / years /
experience / careerbuilder  / design , …
jobtitle: bucket= boost=10; company: bucket= boost=4;
jobdescription: bucket=[ ] weight=1; experience: bucket= weight=1.5
We can pass in a parameter to solr at query time specifying the boost to apply to each
bucket i.e. …&bucketWeights=1:10;2:4;3:1.5;default:1;
This allows us to map many relevancy buckets to search terms at index time and adjust
the weighting at query time without having to search across hundreds of fields.
By making all scoring parameters overridable at query time, we are able to do A / B
testing to consistently improve our relevancy model
That’s great, but what about domain-specific knowledge?
News search: popularity and freshness drive relevance
Restaurant search: geographical proximity and price range are critical
Ecommerce: likelihood of a purchase is key
Movie search: More popular titles are generally more relevant
Job search: category of job, salary range, and geographical proximity matter
TF * IDF of keywords can’t hold it’s own against good
domain-specific relevance factors!
Beyond domain knowledge… consider per-user knowledge
John lives in Boston but wants to move to New York or possibly another big city.
He is currently a sales manager but wants to move towards business
Irene is a bartender in Dublin and is only interested in jobs within 10KM of her
location in the food service industry.
Irfan is a software engineer in Atlanta and is interested in software engineering
jobs at a Big Data company. He is happy to move across the U.S. for the right job.
Jane is a nurse educator in Boston seeking between $40K and $60K working in
the healthcare industry
Query for Jane
Jane is a nurse educator in Boston seeking between $40K and $60K
working in the healthcare industry
jobtitle:"nurse educator"^25 OR jobtitle:(nurse educator)^10
(city:"Boston" AND state:"MA")^15
AND _val_:"map(salary, 40000, 60000,10, 0)”
*Example from chapter 16 of Solr in Action
Search Results for Jane
(New England/ Boston)",
*Example documents available @ http://github.com/treygrainger/solr-in-action/
What did we just do?
We built a recommendation engine!
What is a recommendation engine?
– A system that uses known information (or derived information from that
known information) to automatically suggest relevant content
Our example was just an attribute based recommendation… we’ll see that
behavioral-based (i.e. collaborative filtering) is also possible.
Redefining “Search Engine”
• “Lucene is a high-performance, full-featured
text search engine library…”
Building up to personalization
Use what you have:
– User’s keywords, IP address, searches, clicks, “likes” (purchases,
job applications, comments, etc.)
– Build up a dossier of information on your users
– If a user gives you a profile (resume, social profile, etc), even better.
For full coverage of building a recommendation engine in Solr…
See my talk from Lucene Revolution 2012 (Boston):
Why limit yourself to JUST explicit search or JUST automated recommendations?
By augmenting your user’s explicit queries with information you know about them, you
can personalize their search results.
– A known software engineer runs a blank job search in New York…
• Why not show software engineering higher in the results?
– A new user runs a keyword-only search for nurse
• Why not use the user’s IP address to boost documents geographically closer?
Not going to talk about…
• Using the SynonymFilter
• Automatic language detection
• Stemming/lemmatization/multi-lingual search
(For all of the above, see the Solr Wiki, Reference Guide, or read Solr in Action)
Instead, we’re going to cover:
– Mining user behavior to discover synonyms/related queries
– Discovering related concepts using document clustering in Solr
– Future work: Latent Semantic Indexing
– Document to Document searching using More Like This
– Foreground/Background corpus analysis
Automatic Synonym Discovery
Our primary approach: Search Co-occurrences
Strategy: Map/Reduce job which computes similar searches run for the same
John searched for “java developer” and “j2ee”
Jane searched for “registered nurse” and “r.n.” and “prn”.
Zeke searched for “java developer” and “scala” and “jvm”
By mining the searches of tens millions of search terms per day, we get a list of top
searches, with the corresponding top co-occurring searches.
We also tie each search term to the top category of jobs (i.e java developer, truck
driver, etc.), so that we know in what context people search for each term.
Future work on building conceptual links
Latent Semantic Indexing
• Concept: Build a matrix of all terms, perform singular value decomposition on that
Matrix to reduce the number of dimensions, and index the meaningful (i.e. blurred)
terms on each document.
Why this matters: if done correctly, the search engine can automatically collapse
terms by meaning, remove the useless and redundant ones, and for it’s own
conceptual model of your domain space. This can be used to infuse more
meaning into a document than just a keyword.
See blog posts and presentations by John Berryman and Doug Turnbull about
their work on this. They’re leading the way on this right now (in the open-source
/solr/clustering/?q=(solr or lucene)
//clustering & grouping don’t currently play nicely
Allows you to dynamically identify “concepts” and their
prevalence within a user’s top search results
Document to Document Searching
Goal: use an entire document as your Solr Query, recommending
other related documents.
Standard approach: More Like This Handler
Alternative Approach: Foreground vs. Background corpus analysis
More Like This (Query)
<requestHandler name="/mlt" class="solr.MoreLikeThisHandler" />
// recommendations based on top scoring doc
mlt.fl=jobtitle,jobdescription& // inspect these fields for interesting terms
mlt.interestingTerms=details& // return the interesting terms
More Like This (passing in external document)
stream.body=Solr is an open source enterprise search platform from the Apache
Lucene project. Its major features include full-text search, hit highlighting, faceted search,
dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling.
Providing distributed search and index replication, Solr is highly scalable. Solr is the most
popular enterprise search engine. Solr 4 adds NoSQL features.
CareerBuilder’s Alternative approach (“enhanced” More Like This)
I. Send document as content stream to Solr
II. Perform Language Identification on the content
III. Do language-specific parts of speech detection
• Keep nouns, remove other parts of speech (removes noise)
IV. Do analysis of additional terms for statistical significance:
tf * idf OR foreground vs. background corpus comparison OR Both
Preferred statistical significance measure:
countFG(x) - totalCountFG * probBG(x)
-------------------------------------------------------sqrt(totalCountFG * probBG(x) * (1 - probBG(x)))
V. Return top scoring terms
Foreground vs. Background Corpus Comparison
fg=category:"software engineer"&bg=*:*&stream.body=java nurse and is are was
were ruby php solr oncology part-time … other text in a really long document”
Terms statistically more likely to appear in foreground query than background query:
Note: This method requires you pre-classify your documents (which we do)… it
doesn’t work with a document that hasn’t already been classified.
Pulling it all together
Lucene’s inverted index is a sparse matrix useful for traditional search
(keywords, locations, etc.), recommendations, and discovering links
Traditional tf * idf keyword search is a good starting point, but the best
relevancy lies in combining your domain knowledge (knowledge of user’s
in aggregate) and user-specific knowledge into your own relevancy
The ability to understand user queries (semantic search) further
enhances the search experience, and you already have many tools at
your fingertips for this.