SlideShare a Scribd company logo
1 of 37
Download to read offline
O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
Implementing Conceptual Search in Solr
Simon Hughes
Chief Data Scientist, Dice.com
3
•  Chief Data Scientist at Dice.com, under Yuri Bykov
•  Key Projects involving Solr:
Who Am I?
•  Recommender Systems – more jobs like this, more seekers like
this (uses custom Solr index)
•  Custom Dice Solr MLT handler (real-time recommendations)
•  Did you mean functionality
•  Title, skills and company type-ahead
•  Relevancy improvements in dice jobs search
4
•  Supply Demand Analysis
•  Dice Skills pages – http://www.dice.com/skills
Other Projects
PhD
•  PhD candidate at DePaul University, studying natural language processing and
machine learning
7
Q. What is the Most Common Relevancy Tuning Mistake?
8
Q. What is the Most Common Relevancy Tuning Mistake?
A. Ignoring the importance of RECALL
9
Relevancy Tuning
•  Key performance metrics to measure:
•  Precision
•  Recall
•  F1 Measure - 2*(P*R)/(P+R)
•  Precision is easier – correct mistakes in the top search results
•  Recall - need to know which relevant documents don’t come back
•  Hard to accurately measure
•  Need to know all the relevant documents present in the index
10
What is Conceptual Search?
•  A.K.A. Semantic Search
•  Two key challenges with keyword matching:
•  Polysemy: Words have more than one meaning
•  e.g. engineer – mechanical? programmer? automation engineer?
•  Synonymy: Many different words have the same meaning
•  e.g. QA, quality assurance, tester; VB, Visual Basic, VB.Net
•  Other related challenges -
•  Typos, Spelling Errors, Idioms
•  Conceptual search attempts to solve these problems by learning concepts
11
Why Conceptual Search?
•  We will attempt to improve recall without diminishing precision
•  Can match relevant documents containing none of the query terms
12
Concepts
•  Conceptual search allows us to retrieve documents by how similar the concepts
in the query are to the concepts in a document
•  Concepts represent important high-level ideas in a given domain (e.g. java
technologies, big data jobs, helpdesk support, etc)
•  Concepts are automatically learned from documents using machine learning
•  Words can belong to multiple concepts, with varying strengths of association
with each concept
13
Traditional Techniques
•  Many algorithms have been used for concept learning, include LSA (Latent
Semantic Analysis), LDA (Latent Dirichlet Allocation) and Word2Vec
•  All involve mapping a document to a low dimensional dense vector (an array
of numbers)
•  Each element of the vector is a number representing how well the document
represents that concept
•  E.g. LSA powers the similar skills found in dice’s skills pages
14
Traditional Techniques Don’t Scale
•  LSALSI, LDA and related techniques rely on factorization of very large term-
document matrices – very slow and computationally intensive
•  Require embedding a machine learning model with the search engine to
map new queries to the concept space (latent or topic space)
•  Query performance is very poor – unable to utilize the inverted index as all
documents have the same number of concepts
•  What we want is a way to map words not documents to concepts. Then we
can embed this in Solr via synonym filters and custom query parsers
15
Word2Vec and ‘Word Math’
•  Word2Vec was developed by google around 2013 for learning vector
representations for words, building on earlier work from Rumelhart, Hinton
and Williams in 1986 (see paper below for citation of this work)
•  Word2Vec Paper:
Efficient Estimation of Word Representations in Vector Space
•  It works by training a machine learning model to predict the words
surrounding a word in a sentence
•  Similar words get similar vector representations
16
“Word Math” Example
•  Using basic vector arithmetic, you get some interesting patterns
•  This illustrates how it represents relationships between words
•  E.g. man – king + woman = queen
17
The algorithm learns to
represent different types
of relationships between
words in vector form
18
At this point you may be thinking…
19
20
Why Do I Care? This is a Search Conference…
21
Why Do I Care? This is a Search Conference…
•  This algorithm can be used to represent documents as vectors of concepts
•  We can them use these representations to do conceptual search
•  This will surface many relevant documents missed by keyword matching
•  This boosts recall
•  This technique can also be used to automatically learn synonyms
22
A Quick Demo
Using our Dice.com active jobs index, some example common user queries:
•  Data Scientist
•  Big Data
•  Information Retrieval
•  C#
•  Web Developer
•  CTO
•  Project Manager
23
How?
GitHub- DiceTechJobs/ConceptualSearch:
1.  Pre-Process documents – parse html, strip noise characters, tokenize words
2.  Define important keywords for your domain, or use my code to auto extract top
terms and phrases
3.  Train Word2Vec model on documents to produce a word2vec model
4.  Using this model, either:
1.  Vectors: Use the raw vectors and embed them in Solr using synonyms + payloads
2.  Top N Similar: Or extract the top n similar terms with similarities and embed these as
weighted synonyms using my custom queryboost parser and tokenizer
3.  Clusters: Cluster these vectors by similarity, and map terms to clusters in a synonym file
24
Define Top Domain Specific Keywords
•  If you have a set of documents belonging to a specific domain, it is
important to define the important keywords for your domain:
•  Use top few thousand search keywords
•  Or use my fast keyword and phrase extraction tool (in GitHub)
•  Or use SolrLucene shingle filter to extract top 1 - 4 word sequences by
document frequency
•  Important to map common phrases to single tokens, e.g. data scientist =>
data_scientist, java developer=>java_developer
25
Do It Yourself
•  All code for this talk is now publicly available on GitHub:
•  https://github.com/DiceTechJobs/SolrPlugins - Solr plugins to work with
conceptual search, and other dice plugins, such as a custom MLT handler
•  https://github.com/DiceTechJobs/SolrConfigExamples - Examples of Solr
configuration file entries to enable conceptual search and other Dice
plugins:
•  https://github.com/DiceTechJobs/ConceptualSearch - Python code to
compute the Word2Vec word vectors, and generate Solr synonym files
26
Some Solr Tricks to Make this Happen
1.  Keyword Extraction: Use the synonym filter to extract key words from your
documents
27
28
Some Solr Tricks to Make this Happen
1.  Keyword Extraction: Use the synonym filter to extract key words from your
documents
2.  Synonym Expansion using Payloads:
•  Use the synonym filter to expand a keyword to multiple tokens
•  Each token has an associated payload – used to adjust relevancy scores at
index or query time
29
30
Synonym File Examples – Vector Method
•  Each keyword maps to a set of tokens via a synonym file
•  Vector Synonym file entry (5 element vector, usually100+ elements):
•  java developer=>001|0.4 002|0.1 003|0.5 005|.9
•  Uses a custom token filter that averages these vectors over the entire
document (see GitHub - DiceTechJobs/SolrPlugins)
•  Relatively fast at index time but some additional indexing overhead
•  Very slow to query
32
Synonym File Examples – Top N Method
•  Each keyword maps to a set of most similar keywords via a synonym file
•  Top N Synonym file entry (top 5):
•  java_developer=>java_j2ee_developer|0.907526 java_architect|0.889903
lead_java_developer|0.867594 j2ee_developer|0.864028 java_engineer|0.861407
•  Can configure this at index time with payloads, a payload aware query parser and a
payload similarity function
•  Or you can configure this at query time with a special token filter that converts
payloads into term boosts, along with a special parser (see
GitHub - DiceTechJobs/SolrPlugins)
•  Fast at index and query time if N is reasonable (10-30)
33
Searching over Clustered Terms
•  After we have learned word vectors, we can use a clustering algorithm to
cluster terms by their vectors to give clusters of related words
•  Can learn several different sizes of cluster, such as 500, 1000, 5000 clusters,
and map each of these to a separate field
•  Apply stronger boosts to the fields containing smaller clusters (e.g. the 5000
cluster field) using the edismax qf parameter - tighter clusters get more
weight
•  Code for clustering vectors in GitHub - DiceTechJobs/ConceptualSearch
34
Synonym File Examples – Clustering Method
•  Each keyword in a cluster maps the same artificial token for that cluster
•  Cluster Synonym file entries:
•  java=>cluster_171
•  java applications=>cluster_171
•  java coding=>cluster_171
•  java design=>cluster_171
•  Doesn’t use payloads so does not require any special plugins
•  No noticeable impact on query or indexing performance
35
Example Clusters Learned from Dice Job Postings
•  Note: Labels in bold are manually assigned for interpretability:
•  Natural Languages: bi lingual, bilingual, chinese, fluent, french, german, japanese,
korean, lingual, localized, portuguese, russian, spanish, speak, speaker
•  Apple Programming Languages: cocoa, swift
•  Search Engine Technologies: apache solr, elasticsearch, lucene, lucene solr,
search, search engines, search technologies, solr, solr lucene
•  Microsoft .Net Technologies: c# wcf, microsoft c#, microsoft.net, mvc web, wcf
web services, web forms, webforms, windows forms, winforms, wpf wcf
36
Example Clusters Learned from Dice Job Postings
AttentionAttitude:
attention, attentive, close attention, compromising, conscientious, conscious, customer
oriented, customer service focus, customer service oriented, deliver results, delivering results,
demonstrated commitment, dependability, dependable, detailed oriented, diligence,
diligent, do attitude, ethic, excellent follow, extremely detail oriented, good attention,
meticulous, meticulous attention, organized, orientated, outgoing, outstanding customer
service, pay attention, personality, pleasant, positive attitude, professional appearance,
professional attitude, professional demeanor, punctual, punctuality, self motivated, self
motivation, superb, superior, thoroughness
37
Summary
•  It’s easy to overlook recall when performing relevancy tuning
•  Conceptual search improves recall while maintaining high precision by matching
documents on concepts or ideas.
•  In reality this involves learning which terms are related to one another
•  Word2Vec is a scalable algorithm for learning related words from a set of documents, that
gives state of the art results in word analogy tasks
•  We can train a Word2Vec model offline, and embed it’s output into Solr by using the in-built
synonym filter and payload functionality, combined with some custom plugins

More Related Content

What's hot

Neural Search Comes to Apache Solr
Neural Search Comes to Apache SolrNeural Search Comes to Apache Solr
Neural Search Comes to Apache Solr
Sease
 
Designing and developing vocabularies in RDF
Designing and developing vocabularies in RDFDesigning and developing vocabularies in RDF
Designing and developing vocabularies in RDF
Open Data Support
 

What's hot (20)

Best practices for highly available and large scale SolrCloud
Best practices for highly available and large scale SolrCloudBest practices for highly available and large scale SolrCloud
Best practices for highly available and large scale SolrCloud
 
Accelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsAccelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered Applications
 
Software architecture for high traffic website
Software architecture for high traffic websiteSoftware architecture for high traffic website
Software architecture for high traffic website
 
Neural Search Comes to Apache Solr
Neural Search Comes to Apache SolrNeural Search Comes to Apache Solr
Neural Search Comes to Apache Solr
 
Berlin Buzzwords 2013 - How does lucene store your data?
Berlin Buzzwords 2013 - How does lucene store your data?Berlin Buzzwords 2013 - How does lucene store your data?
Berlin Buzzwords 2013 - How does lucene store your data?
 
Using OpenNLP with Solr to improve search relevance and to extract named enti...
Using OpenNLP with Solr to improve search relevance and to extract named enti...Using OpenNLP with Solr to improve search relevance and to extract named enti...
Using OpenNLP with Solr to improve search relevance and to extract named enti...
 
JSON in Oracle 18c and 19c
JSON in Oracle 18c and 19cJSON in Oracle 18c and 19c
JSON in Oracle 18c and 19c
 
How Kafka Powers the World's Most Popular Vector Database System with Charles...
How Kafka Powers the World's Most Popular Vector Database System with Charles...How Kafka Powers the World's Most Popular Vector Database System with Charles...
How Kafka Powers the World's Most Popular Vector Database System with Charles...
 
はじめてのOracle Cloud Infrastructure(Oracle Cloudウェビナーシリーズ: 2021年2月3日)
はじめてのOracle Cloud Infrastructure(Oracle Cloudウェビナーシリーズ: 2021年2月3日)はじめてのOracle Cloud Infrastructure(Oracle Cloudウェビナーシリーズ: 2021年2月3日)
はじめてのOracle Cloud Infrastructure(Oracle Cloudウェビナーシリーズ: 2021年2月3日)
 
NoSQL databases
NoSQL databasesNoSQL databases
NoSQL databases
 
Near Real Time Indexing: Presented by Umesh Prasad & Thejus V M, Flipkart
Near Real Time Indexing: Presented by Umesh Prasad & Thejus V M, FlipkartNear Real Time Indexing: Presented by Umesh Prasad & Thejus V M, Flipkart
Near Real Time Indexing: Presented by Umesh Prasad & Thejus V M, Flipkart
 
Building a Knowledge Graph using NLP and Ontologies
Building a Knowledge Graph using NLP and OntologiesBuilding a Knowledge Graph using NLP and Ontologies
Building a Knowledge Graph using NLP and Ontologies
 
Consuming RealTime Signals in Solr
Consuming RealTime Signals in Solr Consuming RealTime Signals in Solr
Consuming RealTime Signals in Solr
 
【Oracle Cloud ウェビナー】WebLogic Serverのご紹介
【Oracle Cloud ウェビナー】WebLogic Serverのご紹介【Oracle Cloud ウェビナー】WebLogic Serverのご紹介
【Oracle Cloud ウェビナー】WebLogic Serverのご紹介
 
Datastax enterprise presentation
Datastax enterprise presentationDatastax enterprise presentation
Datastax enterprise presentation
 
Using pySpark with Google Colab & Spark 3.0 preview
Using pySpark with Google Colab & Spark 3.0 previewUsing pySpark with Google Colab & Spark 3.0 preview
Using pySpark with Google Colab & Spark 3.0 preview
 
XML's validation - DTD
XML's validation - DTDXML's validation - DTD
XML's validation - DTD
 
kubernetes : From beginner to Advanced
kubernetes : From beginner to Advancedkubernetes : From beginner to Advanced
kubernetes : From beginner to Advanced
 
Designing and developing vocabularies in RDF
Designing and developing vocabularies in RDFDesigning and developing vocabularies in RDF
Designing and developing vocabularies in RDF
 
Tutorial develop whatsapp chatbot using php/ mysql
Tutorial develop whatsapp chatbot using php/ mysql Tutorial develop whatsapp chatbot using php/ mysql
Tutorial develop whatsapp chatbot using php/ mysql
 

Similar to Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by Simon Hughes, Dice.com

Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
Lucidworks
 
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
Joaquin Delgado PhD.
 
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
 RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning... RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
S. Diana Hu
 
Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...
Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...
Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...
Lucidworks
 

Similar to Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by Simon Hughes, Dice.com (20)

Dice.com Bay Area Search - Beyond Learning to Rank Talk
Dice.com Bay Area Search - Beyond Learning to Rank TalkDice.com Bay Area Search - Beyond Learning to Rank Talk
Dice.com Bay Area Search - Beyond Learning to Rank Talk
 
Relevance in the Wild - Daniel Gomez Vilanueva, Findwise
Relevance in the Wild - Daniel Gomez Vilanueva, FindwiseRelevance in the Wild - Daniel Gomez Vilanueva, Findwise
Relevance in the Wild - Daniel Gomez Vilanueva, Findwise
 
Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
Vectors in Search – Towards More Semantic Matching - Simon Hughes, Dice.com
 
Vectors in Search - Towards More Semantic Matching
Vectors in Search - Towards More Semantic MatchingVectors in Search - Towards More Semantic Matching
Vectors in Search - Towards More Semantic Matching
 
Haystack 2019 - Search with Vectors - Simon Hughes
Haystack 2019 - Search with Vectors - Simon HughesHaystack 2019 - Search with Vectors - Simon Hughes
Haystack 2019 - Search with Vectors - Simon Hughes
 
Searching with vectors
Searching with vectorsSearching with vectors
Searching with vectors
 
Improving Search in Workday Products using Natural Language Processing
Improving Search in Workday Products using Natural Language ProcessingImproving Search in Workday Products using Natural Language Processing
Improving Search in Workday Products using Natural Language Processing
 
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
 
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
 RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning... RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
 
Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...
Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...
Enriching Solr with Deep Learning for a Question Answering System - Sanket Sh...
 
Apex code (Salesforce)
Apex code (Salesforce)Apex code (Salesforce)
Apex code (Salesforce)
 
Final presentation
Final presentationFinal presentation
Final presentation
 
Extending Solr: Behind CareerBuilder’s Cloud-like Knowledge Discovery Platfor...
Extending Solr: Behind CareerBuilder’s Cloud-like Knowledge Discovery Platfor...Extending Solr: Behind CareerBuilder’s Cloud-like Knowledge Discovery Platfor...
Extending Solr: Behind CareerBuilder’s Cloud-like Knowledge Discovery Platfor...
 
Extending Solr: Building a Cloud-like Knowledge Discovery Platform
Extending Solr: Building a Cloud-like Knowledge Discovery PlatformExtending Solr: Building a Cloud-like Knowledge Discovery Platform
Extending Solr: Building a Cloud-like Knowledge Discovery Platform
 
Candidate selection tutorial
Candidate selection tutorialCandidate selection tutorial
Candidate selection tutorial
 
SIGIR 2017 - Candidate Selection for Large Scale Personalized Search and Reco...
SIGIR 2017 - Candidate Selection for Large Scale Personalized Search and Reco...SIGIR 2017 - Candidate Selection for Large Scale Personalized Search and Reco...
SIGIR 2017 - Candidate Selection for Large Scale Personalized Search and Reco...
 
Evolving The Optimal Relevancy Scoring Model at Dice.com: Presented by Simon ...
Evolving The Optimal Relevancy Scoring Model at Dice.com: Presented by Simon ...Evolving The Optimal Relevancy Scoring Model at Dice.com: Presented by Simon ...
Evolving The Optimal Relevancy Scoring Model at Dice.com: Presented by Simon ...
 
Personalized Search and Job Recommendations - Simon Hughes, Dice.com
Personalized Search and Job Recommendations - Simon Hughes, Dice.comPersonalized Search and Job Recommendations - Simon Hughes, Dice.com
Personalized Search and Job Recommendations - Simon Hughes, Dice.com
 
Enhancing Enterprise Search with Machine Learning - Simon Hughes, Dice.com
Enhancing Enterprise Search with Machine Learning - Simon Hughes, Dice.comEnhancing Enterprise Search with Machine Learning - Simon Hughes, Dice.com
Enhancing Enterprise Search with Machine Learning - Simon Hughes, Dice.com
 
RuleML2015 - Tutorial - Powerful Practical Semantic Rules in Rulelog - Funda...
RuleML2015 - Tutorial -  Powerful Practical Semantic Rules in Rulelog - Funda...RuleML2015 - Tutorial -  Powerful Practical Semantic Rules in Rulelog - Funda...
RuleML2015 - Tutorial - Powerful Practical Semantic Rules in Rulelog - Funda...
 

More from Lucidworks

Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Lucidworks
 

More from Lucidworks (20)

Search is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategySearch is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce Strategy
 
Drive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in SalesforceDrive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in Salesforce
 
How Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsHow Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant Products
 
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
 
Connected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesConnected Experiences Are Personalized Experiences
Connected Experiences Are Personalized Experiences
 
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
 
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
 
Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020
 
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
 
AI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and RosetteAI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and Rosette
 
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentThe Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
 
Webinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeWebinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - Europe
 
Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19
 
Applying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchApplying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 Research
 
Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1
 
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyWebinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
 
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
 
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceApply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise Search
 
Why Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondWhy Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and Beyond
 

Recently uploaded

Recently uploaded (20)

Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptx
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 

Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by Simon Hughes, Dice.com

  • 1. O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
  • 2. Implementing Conceptual Search in Solr Simon Hughes Chief Data Scientist, Dice.com
  • 3. 3 •  Chief Data Scientist at Dice.com, under Yuri Bykov •  Key Projects involving Solr: Who Am I? •  Recommender Systems – more jobs like this, more seekers like this (uses custom Solr index) •  Custom Dice Solr MLT handler (real-time recommendations) •  Did you mean functionality •  Title, skills and company type-ahead •  Relevancy improvements in dice jobs search
  • 4. 4 •  Supply Demand Analysis •  Dice Skills pages – http://www.dice.com/skills Other Projects PhD •  PhD candidate at DePaul University, studying natural language processing and machine learning
  • 5.
  • 6.
  • 7. 7 Q. What is the Most Common Relevancy Tuning Mistake?
  • 8. 8 Q. What is the Most Common Relevancy Tuning Mistake? A. Ignoring the importance of RECALL
  • 9. 9 Relevancy Tuning •  Key performance metrics to measure: •  Precision •  Recall •  F1 Measure - 2*(P*R)/(P+R) •  Precision is easier – correct mistakes in the top search results •  Recall - need to know which relevant documents don’t come back •  Hard to accurately measure •  Need to know all the relevant documents present in the index
  • 10. 10 What is Conceptual Search? •  A.K.A. Semantic Search •  Two key challenges with keyword matching: •  Polysemy: Words have more than one meaning •  e.g. engineer – mechanical? programmer? automation engineer? •  Synonymy: Many different words have the same meaning •  e.g. QA, quality assurance, tester; VB, Visual Basic, VB.Net •  Other related challenges - •  Typos, Spelling Errors, Idioms •  Conceptual search attempts to solve these problems by learning concepts
  • 11. 11 Why Conceptual Search? •  We will attempt to improve recall without diminishing precision •  Can match relevant documents containing none of the query terms
  • 12. 12 Concepts •  Conceptual search allows us to retrieve documents by how similar the concepts in the query are to the concepts in a document •  Concepts represent important high-level ideas in a given domain (e.g. java technologies, big data jobs, helpdesk support, etc) •  Concepts are automatically learned from documents using machine learning •  Words can belong to multiple concepts, with varying strengths of association with each concept
  • 13. 13 Traditional Techniques •  Many algorithms have been used for concept learning, include LSA (Latent Semantic Analysis), LDA (Latent Dirichlet Allocation) and Word2Vec •  All involve mapping a document to a low dimensional dense vector (an array of numbers) •  Each element of the vector is a number representing how well the document represents that concept •  E.g. LSA powers the similar skills found in dice’s skills pages
  • 14. 14 Traditional Techniques Don’t Scale •  LSALSI, LDA and related techniques rely on factorization of very large term- document matrices – very slow and computationally intensive •  Require embedding a machine learning model with the search engine to map new queries to the concept space (latent or topic space) •  Query performance is very poor – unable to utilize the inverted index as all documents have the same number of concepts •  What we want is a way to map words not documents to concepts. Then we can embed this in Solr via synonym filters and custom query parsers
  • 15. 15 Word2Vec and ‘Word Math’ •  Word2Vec was developed by google around 2013 for learning vector representations for words, building on earlier work from Rumelhart, Hinton and Williams in 1986 (see paper below for citation of this work) •  Word2Vec Paper: Efficient Estimation of Word Representations in Vector Space •  It works by training a machine learning model to predict the words surrounding a word in a sentence •  Similar words get similar vector representations
  • 16. 16 “Word Math” Example •  Using basic vector arithmetic, you get some interesting patterns •  This illustrates how it represents relationships between words •  E.g. man – king + woman = queen
  • 17. 17 The algorithm learns to represent different types of relationships between words in vector form
  • 18. 18 At this point you may be thinking…
  • 19. 19
  • 20. 20 Why Do I Care? This is a Search Conference…
  • 21. 21 Why Do I Care? This is a Search Conference… •  This algorithm can be used to represent documents as vectors of concepts •  We can them use these representations to do conceptual search •  This will surface many relevant documents missed by keyword matching •  This boosts recall •  This technique can also be used to automatically learn synonyms
  • 22. 22 A Quick Demo Using our Dice.com active jobs index, some example common user queries: •  Data Scientist •  Big Data •  Information Retrieval •  C# •  Web Developer •  CTO •  Project Manager
  • 23. 23 How? GitHub- DiceTechJobs/ConceptualSearch: 1.  Pre-Process documents – parse html, strip noise characters, tokenize words 2.  Define important keywords for your domain, or use my code to auto extract top terms and phrases 3.  Train Word2Vec model on documents to produce a word2vec model 4.  Using this model, either: 1.  Vectors: Use the raw vectors and embed them in Solr using synonyms + payloads 2.  Top N Similar: Or extract the top n similar terms with similarities and embed these as weighted synonyms using my custom queryboost parser and tokenizer 3.  Clusters: Cluster these vectors by similarity, and map terms to clusters in a synonym file
  • 24. 24 Define Top Domain Specific Keywords •  If you have a set of documents belonging to a specific domain, it is important to define the important keywords for your domain: •  Use top few thousand search keywords •  Or use my fast keyword and phrase extraction tool (in GitHub) •  Or use SolrLucene shingle filter to extract top 1 - 4 word sequences by document frequency •  Important to map common phrases to single tokens, e.g. data scientist => data_scientist, java developer=>java_developer
  • 25. 25 Do It Yourself •  All code for this talk is now publicly available on GitHub: •  https://github.com/DiceTechJobs/SolrPlugins - Solr plugins to work with conceptual search, and other dice plugins, such as a custom MLT handler •  https://github.com/DiceTechJobs/SolrConfigExamples - Examples of Solr configuration file entries to enable conceptual search and other Dice plugins: •  https://github.com/DiceTechJobs/ConceptualSearch - Python code to compute the Word2Vec word vectors, and generate Solr synonym files
  • 26. 26 Some Solr Tricks to Make this Happen 1.  Keyword Extraction: Use the synonym filter to extract key words from your documents
  • 27. 27
  • 28. 28 Some Solr Tricks to Make this Happen 1.  Keyword Extraction: Use the synonym filter to extract key words from your documents 2.  Synonym Expansion using Payloads: •  Use the synonym filter to expand a keyword to multiple tokens •  Each token has an associated payload – used to adjust relevancy scores at index or query time
  • 29. 29
  • 30. 30 Synonym File Examples – Vector Method •  Each keyword maps to a set of tokens via a synonym file •  Vector Synonym file entry (5 element vector, usually100+ elements): •  java developer=>001|0.4 002|0.1 003|0.5 005|.9 •  Uses a custom token filter that averages these vectors over the entire document (see GitHub - DiceTechJobs/SolrPlugins) •  Relatively fast at index time but some additional indexing overhead •  Very slow to query
  • 31.
  • 32. 32 Synonym File Examples – Top N Method •  Each keyword maps to a set of most similar keywords via a synonym file •  Top N Synonym file entry (top 5): •  java_developer=>java_j2ee_developer|0.907526 java_architect|0.889903 lead_java_developer|0.867594 j2ee_developer|0.864028 java_engineer|0.861407 •  Can configure this at index time with payloads, a payload aware query parser and a payload similarity function •  Or you can configure this at query time with a special token filter that converts payloads into term boosts, along with a special parser (see GitHub - DiceTechJobs/SolrPlugins) •  Fast at index and query time if N is reasonable (10-30)
  • 33. 33 Searching over Clustered Terms •  After we have learned word vectors, we can use a clustering algorithm to cluster terms by their vectors to give clusters of related words •  Can learn several different sizes of cluster, such as 500, 1000, 5000 clusters, and map each of these to a separate field •  Apply stronger boosts to the fields containing smaller clusters (e.g. the 5000 cluster field) using the edismax qf parameter - tighter clusters get more weight •  Code for clustering vectors in GitHub - DiceTechJobs/ConceptualSearch
  • 34. 34 Synonym File Examples – Clustering Method •  Each keyword in a cluster maps the same artificial token for that cluster •  Cluster Synonym file entries: •  java=>cluster_171 •  java applications=>cluster_171 •  java coding=>cluster_171 •  java design=>cluster_171 •  Doesn’t use payloads so does not require any special plugins •  No noticeable impact on query or indexing performance
  • 35. 35 Example Clusters Learned from Dice Job Postings •  Note: Labels in bold are manually assigned for interpretability: •  Natural Languages: bi lingual, bilingual, chinese, fluent, french, german, japanese, korean, lingual, localized, portuguese, russian, spanish, speak, speaker •  Apple Programming Languages: cocoa, swift •  Search Engine Technologies: apache solr, elasticsearch, lucene, lucene solr, search, search engines, search technologies, solr, solr lucene •  Microsoft .Net Technologies: c# wcf, microsoft c#, microsoft.net, mvc web, wcf web services, web forms, webforms, windows forms, winforms, wpf wcf
  • 36. 36 Example Clusters Learned from Dice Job Postings AttentionAttitude: attention, attentive, close attention, compromising, conscientious, conscious, customer oriented, customer service focus, customer service oriented, deliver results, delivering results, demonstrated commitment, dependability, dependable, detailed oriented, diligence, diligent, do attitude, ethic, excellent follow, extremely detail oriented, good attention, meticulous, meticulous attention, organized, orientated, outgoing, outstanding customer service, pay attention, personality, pleasant, positive attitude, professional appearance, professional attitude, professional demeanor, punctual, punctuality, self motivated, self motivation, superb, superior, thoroughness
  • 37. 37 Summary •  It’s easy to overlook recall when performing relevancy tuning •  Conceptual search improves recall while maintaining high precision by matching documents on concepts or ideas. •  In reality this involves learning which terms are related to one another •  Word2Vec is a scalable algorithm for learning related words from a set of documents, that gives state of the art results in word analogy tasks •  We can train a Word2Vec model offline, and embed it’s output into Solr by using the in-built synonym filter and payload functionality, combined with some custom plugins