SlideShare a Scribd company logo
From Natural Language to
Structured Solr Queries using
LLMs
BERLIN BUZZWORDS 2024 - 10/06/2024
Speakers: Anna Ruggero, R&D Software Engineer @ Sease
Ilaria Petreti, ML Software Engineer @ Sease
WHO WE ARE
ILARIA PETRETI ANNA RUGGERO
SEArch SErvices
‣ Headquarter in London/distributed
‣ Open-source Enthusiasts
‣ Apache Lucene/Solr experts
‣ Elasticsearch/OpenSearch experts
‣ Community Contributors
‣ Active Researchers
HOT TRENDS:
● Large Language Models Applications
● Vector-based (Neural) Search
● Natural Language Processing
● Learning To Rank
● Document Similarity
● Search Quality Evaluation
● Relevance Tuning
www.sease.io
AGENDA
Use Case Overview
From Natural Language to Structured Queries
Findings
The Road to Production
● Transformers
● Next-token-prediction and masked-
language-modeling
● Estimate the likelihood of each
possible word (in its vocabulary)
given the previous sequence
● Learn the statistical structure of
language
● Pre-trained on huge quantities of text
● Fine-tuned for different tasks
(Following Instructions)
WHAT IS A LARGE LANGUAGE MODEL
https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286
VOCABULARY MISMATCH PROBLEM
● Terms matching between the query and the documents.
○ false positive: docs retrieved (terms match) but no information need
○ false negative: docs not retrieved (terms don’t match) but there was the
information need in the corpus → zero result query
SEMANTIC SIMILARITY
● Same terms different meaning: How old are you? - How are you?
● Different terms same meaning: How old are you? - What is your age?
DISAMBIGUATION
● Same term in two totally different contexts assume totally different meanings
LEXICAL PROBLEMS
There are some lexical solutions to these:
Manually curated
● Synonyms, Hypernyms, Hyponyms
Algorithmic
● Stemming, lemmatization
● Knowledge Base disambiguation
LEXICAL SOLUTIONS
These solutions are
expensive and do not
guarantee high quality
results.
We can do better!
Query/Document
Expansion
(Generative/Extractive)
Retrieval
Augmented
Generation
Generative
Generate synonyms, query reformulations…
Extractive
Select expansion terms from taxonomies
EXPLOIT LLM CAPABILITIES
{
"filters": {
"Country": "European Union (28 countries)#EU28#”,
"Pollutant": "Particulates (PM10)#PM10#",
"Variable": "Total man-made emissions#TOT#|Industrial combustion#STAT_COMB_IND#",
"Time Period": "Second trimester(Q2)",
"Year": "2015"
}
}
NATURAL LANGUAGE QUERY PARSING
PM10 levels produced by industries in the European Community in May 2015
We have been working with some of our clients to exploit an LLM in order to:
● Disambiguate the meaning of a user’s natural language query
● Extract the relevant information
● Use the extracted information to implement a structured Solr query
REAL CASE APPLICATION
● OECD lead initiative
(The Organisation for Economic Co-operation and Development)
● The Statistical Information System Collaboration Community
● .Stat Suite and Apache Solr https://siscc.org/developers/technology/
ONE OF OUR CLIENTS
AGENDA
Use Case Overview
From Natural Language to Structured Queries
Findings
The Road to Production
ARCHITECTURE
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
ARCHITECTURE
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
{"Topic":[
"Economy#ECO#",
"Economy#ECO#|Productivity#ECO_PRO#",
"Agriculture#AGR#",
"Government#GOV#", …],
"Dimension":[
"Reference area",
"Time period",
"Unit of Measure",
"Year", …],
"Reference Area":[
"Australia#AUS#",
"Austria#AUT#", …],
etc… }
FIELD/VALUES RETRIEVAL
List of Fields and Values
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
USER QUERY
What were the sulfur oxide
emissions in Australia in
2013?
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
FILTER EXTRACTION
PROMPT
1) OBJECT REPRESENTATION
Provide input data (e.g. JSON
representation) to the model
1) QUERY PARSING
Request a similar representation
(i.e. subset) based on the input query
1) FORMAL REQUIREMENTS
Specify how the output should be
formatted, including any constraints
or specific criteria to be met.
Query
Answer
LLM Model
Filters
Extraction
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
{'Topic': [
"Environment#ENV#|Air and climate#ENV_AC#"],
'Country': [
"Australia#AUS#"],
'Variable': [
"Total man-made emissions#TOT#"],
'Pollutant': [
"Sulphur Oxides#SOX#"],
'Year': '2013'}
Selected
Filters
FILTER EXTRACTION
What were the sulfur oxide emissions in Australia in 2013?
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
PROMPT
Ask to the model to provide:
- different/additional relevant terms
- synonyms
- variations with same meaning
QUERY REFORMULATION
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
['Sulfur dioxide
emissions',
'Air pollution',
'Environmental impact',
'Fossil fuel combustion',
'Acid rain']
QUERY REFORMULATION
What were the sulfur oxide emissions in Australia in 2013?
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries Structured
Query
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
STRUCTURED QUERY
SOLR QUERY
q=
title:(Sulfur dioxide emissions Air
... Acid rain)
OR topic:"Environment#ENV#|Air and
climate#ENV_AC#"
OR country:"Australia#AUS#"
OR variable:"Total man-made
emissions#TOT#"
OR Pollutant:"Sulphur Oxides#SOX#"
OR 'Year': '2013'
Query
Answer
LLM Model
Filters
Extraction
Selected
Filters
Alternative
queries
Solr
Relevant
Documents
Query
Reformulation
List of Fields and Values
DOC RETRIEVAL
SEARCH RESULTS
"response":{
"numFound":1,
"start":0,
"numFoundExact":true,
"docs":[{
"Title":"Emissions of air pollutants",
"Dimension":["Country", "Pollutant", "Variable", "Year"]
}]
}
● Separates the flow of your program (modules) from the parameters (LM
prompts and weights) of each step
● Introduces new optimizers to tune prompts and/or weights of your LM calls,
given a metric you want to maximise
● LMs and their prompts fade into the background as optimizable pieces of
a larger system that can learn from data
DSPY LIBRARY
https://dspy-docs.vercel.app/docs/intro
https://dspy-docs.vercel.app/docs/intro
DSPY LIBRARY
Is it really as it suggests? Partially!
AGENDA
Use Case Overview
From Natural Language to Structured Queries
Findings
The Road to Production
MODEL CONSIDERATIONS
● [Model Selection]: NOT the most advanced available for this task
● [Model Comparison]: No evaluations or comparisons with
alternative models → Time constraints and limited funding
● [Rationale for Current Choice]: promising capabilities and quick implementation
● [Future Works]:
○ Explore and analyze models that are fine-tuned specifically for our task
○ Potentially undertake our own fine-tuning to optimize model performance
○ Model comparison
PROMISING ASPECTS
● Overcome the lexical matching
land of kangaroos → [Country] AUSTRALIA
tobacco consumption → [Topic] SMOKING/RISK FACTORS FOR HEALTH
● Explainability for selected filters
Analyze input text: "cost per square meter for family houses in italy"
cost per square meter → pricing or valuation → 'Priced unit' or 'Value'
family houses → type of property → 'Real estate type'
italy → location → 'Reference area' or 'Borrowers' country'
PROMISING ASPECTS
● Explainability for selected filters
Analyze input text: "cost per square meter for family houses in italy"
cost per square meter → pricing or valuation → 'Priced unit' or 'Value'
family houses → type of property → 'Real estate type'
italy → location → 'Reference area' or 'Borrowers' country'
PROMISING ASPECTS
Integrate as an "Assistant" feature
to guide users in choosing the most suitable
filters
IDEA!
PROMISING ASPECTS
● Promising potential in early results:
○ challenging and complex task
○ good results (using a commercial out-of-the-box model!)
○ straightforward implementation
○ model's adaptability to the context
LIMITATIONS
FUNCTIONAL
2
1FUNCTIONAL
Retrieval Augmented
Generation
FORMAL
LLM weaknesses in the
language/query semantic comprehension
LLM weaknesses in complying with:
● the problem definition
● the required output format
● Difficult to identify relevant fields when others share the same values
{ "Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...],
"Borrower’s Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...]
}
FUNCTIONAL LIMITATIONS
● Difficult to identify relevant fields when others share the same values
{ "Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...],
"Reporting Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...] }
● Difficult to identify relevant fields when highly specialized domain knowledge
is required
"Marginal lending facility rate" → [Reference Area] Europe
"IMU tax" → [Sector] Real Estate
FUNCTIONAL LIMITATIONS
● Difficult to identify relevant fields when others share the same values
{ "Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...],
"Reporting Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...] }
● Difficult to identify relevant fields when high specialized domain knowledge is required
"Marginal lending facility rate" → [Reference Area] Europe
"IMU tax" → [Sector] Real Estate
● Sometimes right value for a field not selected even if it is present in the user query
User Query → green growth in Rabat
Explainability → "Country": "Morocco" would be the relevant value if it were
listed, but it is not.
Country → ["All countries", "Europe", "G20", "Asia", "Morocco", ...]
FUNCTIONAL LIMITATIONS
EXPECTED FIELD
POSSIBLE SOLUTIONS
● Refinement of the input Solr dictionary
○ Human readable fields’ names
"HEDxkkgkqIr" → "Category"
"INST_NON_EDU" → "Non-educational institutions"
A win for those who always
asked clients to use
understandable Solr fields!
POSSIBLE SOLUTIONS
● Ad-hoc prompt engineering
○ Expand the prompt with ambiguous/difficult examples and solutions
■ "Marginal lending facility rate" → [Reference Area] Europe
■ "IMU tax" → [Sector] Real Estate
○ Break down the prompt
■ One request for topic selection
■ One request for topic values selection
■ One request for dimensions selection
■ One request for dimensions values selection
POSSIBLE SOLUTIONS
● LLM fine-tuning
○ Better disambiguation
○ Learn the specific and domain-related task
FORMAL LIMITATIONS
● LLM hallucinations:
○ Field names
E.g. "Instrument" instead of "Type of instruments"
○ Field values
"Year": "21st century" instead of "2000"
● Returned field-values are mixed up
E.g.
○ "Total emissions per capita" is part of a value and not a dimension
○ "European Union (28 countries)#EU28#" is a valid value present in "Country" but not in
"Reference Area"
FORMAL LIMITATIONS
● Poorly formatted JSON returned
Selected Pairs:
```json
{
"Country": "Australia#AUS#", // land of kangaroos
"Pollutant": "Sulphur Oxides#SOX#",
"Year": "2013"
}
```
These pairs are chosen based on the keywords identified in the
input text and the closest matching dimensions and values from the
provided dictionary.
POSSIBLE SOLUTIONS
● Post-processing to validate and correct the LLM answer.
● DSPy library additional studies.
(Typed Predictors, Optimizers)
● Evaluation of additional libraries and strategies
● Fine-tuning the model for the specific task → Extraction
AGENDA
Use Case Overview
From Natural Language to Structured Queries
Findings
The Road to Production
● [UX] Design the user experience
○ Filtering assistance?
○ Transparent query parsing?
● [LLM] Select the best model to date
○ Can we fine-tune promising models specifically for the task?
● [LLM] Refine the prompts according to the model
○ Can we use only one request to build the structured query?
THE ROAD TO PRODUCTION
● [LLM] Implement integration tests with the most common failures →
LLM/prompt engineering to solve them
● [LLM] Study additional libraries to make the prompt more “programmed” and
“automatically tuned” and less “trial-and-error”
○ Highly depend on the LLM available
● [Performance] Stress test the solution
● [Quality] Set up queries/expected documents
THE ROAD TO PRODUCTION
STAY UP TO DATE
SUBSCRIBE TO THE
INFORMATION
RETRIEVAL NEWSLETTER
https://sease.io/our-blog
THANK YOU!

More Related Content

Similar to From Natural Language to Structured Solr Queries using LLMs

ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing
ISC Cloud13 Sill - Crossing organizational boundaries in cloud computingISC Cloud13 Sill - Crossing organizational boundaries in cloud computing
ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing
Alan Sill
 
Sudipta_Mukherjee_Resume-Nov_2022.pdf
Sudipta_Mukherjee_Resume-Nov_2022.pdfSudipta_Mukherjee_Resume-Nov_2022.pdf
Sudipta_Mukherjee_Resume-Nov_2022.pdf
Sudipta Mukherjee
 
Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases
Yunyao Li
 
Issidoros_Minovgidis_CV_EN
Issidoros_Minovgidis_CV_ENIssidoros_Minovgidis_CV_EN
Issidoros_Minovgidis_CV_EN
Issidoros Minovgidis
 
jayaprakash
jayaprakashjayaprakash
jayaprakash
Jayaprakash T
 
Dissecting SysML v2.pptx
Dissecting SysML v2.pptxDissecting SysML v2.pptx
Dissecting SysML v2.pptx
Elizabeth Steiner
 
Altova NIEM keynote
Altova NIEM keynoteAltova NIEM keynote
Altova NIEM keynote
Alexander Falk
 
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and IstioAdvanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Animesh Singh
 
Shanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_ProfileShanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_Profile
Shanish Jain
 
Usability evaluation of Domain-Specific Languages
Usability evaluation of Domain-Specific LanguagesUsability evaluation of Domain-Specific Languages
Usability evaluation of Domain-Specific Languages
Ankica Barisic
 
OER World Map Project
OER World Map Project OER World Map Project
OER World Map Project
Robert Farrow
 
NoR Webinar 2024 - Introduction to Ellip.pdf
NoR Webinar 2024 - Introduction to Ellip.pdfNoR Webinar 2024 - Introduction to Ellip.pdf
NoR Webinar 2024 - Introduction to Ellip.pdf
terradue
 
Harshit Kothari_CV_ IBM Mainframes
Harshit Kothari_CV_ IBM MainframesHarshit Kothari_CV_ IBM Mainframes
Harshit Kothari_CV_ IBM Mainframes
Harshit Kothari
 
Taming the Wild West of NLP
Taming the Wild West of NLPTaming the Wild West of NLP
Taming the Wild West of NLP
Yunyao Li
 
iMT Language Solutions
iMT Language SolutionsiMT Language Solutions
iMT Language Solutions
SDL
 
History Of C Essay
History Of C EssayHistory Of C Essay
History Of C Essay
Melissa Williams
 
ILAP (held at the 3rd Digital Oilfield Summit)
ILAP (held at the 3rd Digital Oilfield Summit)ILAP (held at the 3rd Digital Oilfield Summit)
ILAP (held at the 3rd Digital Oilfield Summit)
André Torkveen
 
Manigandan_narasimhan_resume
Manigandan_narasimhan_resumeManigandan_narasimhan_resume
Manigandan_narasimhan_resume
manigandan narasimhan
 
Reducing Database Pain & Costs with Postgres
Reducing Database Pain & Costs with PostgresReducing Database Pain & Costs with Postgres
Reducing Database Pain & Costs with Postgres
EDB
 
DI4R 2018 - Ellip: a collaborative workplace for EO Open Science
DI4R 2018 - Ellip: a collaborative workplace for EO Open ScienceDI4R 2018 - Ellip: a collaborative workplace for EO Open Science
DI4R 2018 - Ellip: a collaborative workplace for EO Open Science
terradue
 

Similar to From Natural Language to Structured Solr Queries using LLMs (20)

ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing
ISC Cloud13 Sill - Crossing organizational boundaries in cloud computingISC Cloud13 Sill - Crossing organizational boundaries in cloud computing
ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing
 
Sudipta_Mukherjee_Resume-Nov_2022.pdf
Sudipta_Mukherjee_Resume-Nov_2022.pdfSudipta_Mukherjee_Resume-Nov_2022.pdf
Sudipta_Mukherjee_Resume-Nov_2022.pdf
 
Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases
 
Issidoros_Minovgidis_CV_EN
Issidoros_Minovgidis_CV_ENIssidoros_Minovgidis_CV_EN
Issidoros_Minovgidis_CV_EN
 
jayaprakash
jayaprakashjayaprakash
jayaprakash
 
Dissecting SysML v2.pptx
Dissecting SysML v2.pptxDissecting SysML v2.pptx
Dissecting SysML v2.pptx
 
Altova NIEM keynote
Altova NIEM keynoteAltova NIEM keynote
Altova NIEM keynote
 
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and IstioAdvanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
 
Shanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_ProfileShanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_Profile
 
Usability evaluation of Domain-Specific Languages
Usability evaluation of Domain-Specific LanguagesUsability evaluation of Domain-Specific Languages
Usability evaluation of Domain-Specific Languages
 
OER World Map Project
OER World Map Project OER World Map Project
OER World Map Project
 
NoR Webinar 2024 - Introduction to Ellip.pdf
NoR Webinar 2024 - Introduction to Ellip.pdfNoR Webinar 2024 - Introduction to Ellip.pdf
NoR Webinar 2024 - Introduction to Ellip.pdf
 
Harshit Kothari_CV_ IBM Mainframes
Harshit Kothari_CV_ IBM MainframesHarshit Kothari_CV_ IBM Mainframes
Harshit Kothari_CV_ IBM Mainframes
 
Taming the Wild West of NLP
Taming the Wild West of NLPTaming the Wild West of NLP
Taming the Wild West of NLP
 
iMT Language Solutions
iMT Language SolutionsiMT Language Solutions
iMT Language Solutions
 
History Of C Essay
History Of C EssayHistory Of C Essay
History Of C Essay
 
ILAP (held at the 3rd Digital Oilfield Summit)
ILAP (held at the 3rd Digital Oilfield Summit)ILAP (held at the 3rd Digital Oilfield Summit)
ILAP (held at the 3rd Digital Oilfield Summit)
 
Manigandan_narasimhan_resume
Manigandan_narasimhan_resumeManigandan_narasimhan_resume
Manigandan_narasimhan_resume
 
Reducing Database Pain & Costs with Postgres
Reducing Database Pain & Costs with PostgresReducing Database Pain & Costs with Postgres
Reducing Database Pain & Costs with Postgres
 
DI4R 2018 - Ellip: a collaborative workplace for EO Open Science
DI4R 2018 - Ellip: a collaborative workplace for EO Open ScienceDI4R 2018 - Ellip: a collaborative workplace for EO Open Science
DI4R 2018 - Ellip: a collaborative workplace for EO Open Science
 

More from Sease

Hybrid Search with Apache Solr Reciprocal Rank Fusion
Hybrid Search with Apache Solr Reciprocal Rank FusionHybrid Search with Apache Solr Reciprocal Rank Fusion
Hybrid Search with Apache Solr Reciprocal Rank Fusion
Sease
 
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
Blazing-Fast Serverless MapReduce Indexer for Apache SolrBlazing-Fast Serverless MapReduce Indexer for Apache Solr
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
Sease
 
Hybrid Search With Apache Solr
Hybrid Search With Apache SolrHybrid Search With Apache Solr
Hybrid Search With Apache Solr
Sease
 
Multi Valued Vectors Lucene
Multi Valued Vectors LuceneMulti Valued Vectors Lucene
Multi Valued Vectors Lucene
Sease
 
How To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With KibanaHow To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With Kibana
Sease
 
Introducing Multi Valued Vectors Fields in Apache Lucene
Introducing Multi Valued Vectors Fields in Apache LuceneIntroducing Multi Valued Vectors Fields in Apache Lucene
Introducing Multi Valued Vectors Fields in Apache Lucene
Sease
 
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
Sease
 
How does ChatGPT work: an Information Retrieval perspective
How does ChatGPT work: an Information Retrieval perspectiveHow does ChatGPT work: an Information Retrieval perspective
How does ChatGPT work: an Information Retrieval perspective
Sease
 
How To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With KibanaHow To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With Kibana
Sease
 
Neural Search Comes to Apache Solr
Neural Search Comes to Apache SolrNeural Search Comes to Apache Solr
Neural Search Comes to Apache Solr
Sease
 
Large Scale Indexing
Large Scale IndexingLarge Scale Indexing
Large Scale Indexing
Sease
 
Dense Retrieval with Apache Solr Neural Search.pdf
Dense Retrieval with Apache Solr Neural Search.pdfDense Retrieval with Apache Solr Neural Search.pdf
Dense Retrieval with Apache Solr Neural Search.pdf
Sease
 
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
Sease
 
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdfWord2Vec model to generate synonyms on the fly in Apache Lucene.pdf
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
Sease
 
How to cache your searches_ an open source implementation.pptx
How to cache your searches_ an open source implementation.pptxHow to cache your searches_ an open source implementation.pptx
How to cache your searches_ an open source implementation.pptx
Sease
 
Online Testing Learning to Rank with Solr Interleaving
Online Testing Learning to Rank with Solr InterleavingOnline Testing Learning to Rank with Solr Interleaving
Online Testing Learning to Rank with Solr Interleaving
Sease
 
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
Sease
 
Apache Lucene/Solr Document Classification
Apache Lucene/Solr Document ClassificationApache Lucene/Solr Document Classification
Apache Lucene/Solr Document Classification
Sease
 
Advanced Document Similarity with Apache Lucene
Advanced Document Similarity with Apache LuceneAdvanced Document Similarity with Apache Lucene
Advanced Document Similarity with Apache Lucene
Sease
 
Search Quality Evaluation: a Developer Perspective
Search Quality Evaluation: a Developer PerspectiveSearch Quality Evaluation: a Developer Perspective
Search Quality Evaluation: a Developer Perspective
Sease
 

More from Sease (20)

Hybrid Search with Apache Solr Reciprocal Rank Fusion
Hybrid Search with Apache Solr Reciprocal Rank FusionHybrid Search with Apache Solr Reciprocal Rank Fusion
Hybrid Search with Apache Solr Reciprocal Rank Fusion
 
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
Blazing-Fast Serverless MapReduce Indexer for Apache SolrBlazing-Fast Serverless MapReduce Indexer for Apache Solr
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
 
Hybrid Search With Apache Solr
Hybrid Search With Apache SolrHybrid Search With Apache Solr
Hybrid Search With Apache Solr
 
Multi Valued Vectors Lucene
Multi Valued Vectors LuceneMulti Valued Vectors Lucene
Multi Valued Vectors Lucene
 
How To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With KibanaHow To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With Kibana
 
Introducing Multi Valued Vectors Fields in Apache Lucene
Introducing Multi Valued Vectors Fields in Apache LuceneIntroducing Multi Valued Vectors Fields in Apache Lucene
Introducing Multi Valued Vectors Fields in Apache Lucene
 
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
 
How does ChatGPT work: an Information Retrieval perspective
How does ChatGPT work: an Information Retrieval perspectiveHow does ChatGPT work: an Information Retrieval perspective
How does ChatGPT work: an Information Retrieval perspective
 
How To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With KibanaHow To Implement Your Online Search Quality Evaluation With Kibana
How To Implement Your Online Search Quality Evaluation With Kibana
 
Neural Search Comes to Apache Solr
Neural Search Comes to Apache SolrNeural Search Comes to Apache Solr
Neural Search Comes to Apache Solr
 
Large Scale Indexing
Large Scale IndexingLarge Scale Indexing
Large Scale Indexing
 
Dense Retrieval with Apache Solr Neural Search.pdf
Dense Retrieval with Apache Solr Neural Search.pdfDense Retrieval with Apache Solr Neural Search.pdf
Dense Retrieval with Apache Solr Neural Search.pdf
 
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
 
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdfWord2Vec model to generate synonyms on the fly in Apache Lucene.pdf
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
 
How to cache your searches_ an open source implementation.pptx
How to cache your searches_ an open source implementation.pptxHow to cache your searches_ an open source implementation.pptx
How to cache your searches_ an open source implementation.pptx
 
Online Testing Learning to Rank with Solr Interleaving
Online Testing Learning to Rank with Solr InterleavingOnline Testing Learning to Rank with Solr Interleaving
Online Testing Learning to Rank with Solr Interleaving
 
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
 
Apache Lucene/Solr Document Classification
Apache Lucene/Solr Document ClassificationApache Lucene/Solr Document Classification
Apache Lucene/Solr Document Classification
 
Advanced Document Similarity with Apache Lucene
Advanced Document Similarity with Apache LuceneAdvanced Document Similarity with Apache Lucene
Advanced Document Similarity with Apache Lucene
 
Search Quality Evaluation: a Developer Perspective
Search Quality Evaluation: a Developer PerspectiveSearch Quality Evaluation: a Developer Perspective
Search Quality Evaluation: a Developer Perspective
 

Recently uploaded

Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
Lidia A.
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
HackersList
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
ChristopherTHyatt
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
CEPTES Software Inc
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
ishalveerrandhawa1
 

Recently uploaded (20)

Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
 

From Natural Language to Structured Solr Queries using LLMs

  • 1. From Natural Language to Structured Solr Queries using LLMs BERLIN BUZZWORDS 2024 - 10/06/2024 Speakers: Anna Ruggero, R&D Software Engineer @ Sease Ilaria Petreti, ML Software Engineer @ Sease
  • 2. WHO WE ARE ILARIA PETRETI ANNA RUGGERO
  • 3. SEArch SErvices ‣ Headquarter in London/distributed ‣ Open-source Enthusiasts ‣ Apache Lucene/Solr experts ‣ Elasticsearch/OpenSearch experts ‣ Community Contributors ‣ Active Researchers HOT TRENDS: ● Large Language Models Applications ● Vector-based (Neural) Search ● Natural Language Processing ● Learning To Rank ● Document Similarity ● Search Quality Evaluation ● Relevance Tuning www.sease.io
  • 4. AGENDA Use Case Overview From Natural Language to Structured Queries Findings The Road to Production
  • 5. ● Transformers ● Next-token-prediction and masked- language-modeling ● Estimate the likelihood of each possible word (in its vocabulary) given the previous sequence ● Learn the statistical structure of language ● Pre-trained on huge quantities of text ● Fine-tuned for different tasks (Following Instructions) WHAT IS A LARGE LANGUAGE MODEL https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286
  • 6. VOCABULARY MISMATCH PROBLEM ● Terms matching between the query and the documents. ○ false positive: docs retrieved (terms match) but no information need ○ false negative: docs not retrieved (terms don’t match) but there was the information need in the corpus → zero result query SEMANTIC SIMILARITY ● Same terms different meaning: How old are you? - How are you? ● Different terms same meaning: How old are you? - What is your age? DISAMBIGUATION ● Same term in two totally different contexts assume totally different meanings LEXICAL PROBLEMS
  • 7. There are some lexical solutions to these: Manually curated ● Synonyms, Hypernyms, Hyponyms Algorithmic ● Stemming, lemmatization ● Knowledge Base disambiguation LEXICAL SOLUTIONS These solutions are expensive and do not guarantee high quality results. We can do better!
  • 8. Query/Document Expansion (Generative/Extractive) Retrieval Augmented Generation Generative Generate synonyms, query reformulations… Extractive Select expansion terms from taxonomies EXPLOIT LLM CAPABILITIES
  • 9. { "filters": { "Country": "European Union (28 countries)#EU28#”, "Pollutant": "Particulates (PM10)#PM10#", "Variable": "Total man-made emissions#TOT#|Industrial combustion#STAT_COMB_IND#", "Time Period": "Second trimester(Q2)", "Year": "2015" } } NATURAL LANGUAGE QUERY PARSING PM10 levels produced by industries in the European Community in May 2015
  • 10. We have been working with some of our clients to exploit an LLM in order to: ● Disambiguate the meaning of a user’s natural language query ● Extract the relevant information ● Use the extracted information to implement a structured Solr query REAL CASE APPLICATION
  • 11. ● OECD lead initiative (The Organisation for Economic Co-operation and Development) ● The Statistical Information System Collaboration Community ● .Stat Suite and Apache Solr https://siscc.org/developers/technology/ ONE OF OUR CLIENTS
  • 12. AGENDA Use Case Overview From Natural Language to Structured Queries Findings The Road to Production
  • 15. Query Answer LLM Model Filters Extraction Selected Filters Alternative queries Structured Query Solr Relevant Documents Query Reformulation {"Topic":[ "Economy#ECO#", "Economy#ECO#|Productivity#ECO_PRO#", "Agriculture#AGR#", "Government#GOV#", …], "Dimension":[ "Reference area", "Time period", "Unit of Measure", "Year", …], "Reference Area":[ "Australia#AUS#", "Austria#AUT#", …], etc… } FIELD/VALUES RETRIEVAL List of Fields and Values
  • 17. Query Answer LLM Model Filters Extraction Selected Filters Alternative queries Structured Query Solr Relevant Documents Query Reformulation List of Fields and Values FILTER EXTRACTION PROMPT 1) OBJECT REPRESENTATION Provide input data (e.g. JSON representation) to the model 1) QUERY PARSING Request a similar representation (i.e. subset) based on the input query 1) FORMAL REQUIREMENTS Specify how the output should be formatted, including any constraints or specific criteria to be met.
  • 18. Query Answer LLM Model Filters Extraction Alternative queries Structured Query Solr Relevant Documents Query Reformulation List of Fields and Values {'Topic': [ "Environment#ENV#|Air and climate#ENV_AC#"], 'Country': [ "Australia#AUS#"], 'Variable': [ "Total man-made emissions#TOT#"], 'Pollutant': [ "Sulphur Oxides#SOX#"], 'Year': '2013'} Selected Filters FILTER EXTRACTION What were the sulfur oxide emissions in Australia in 2013?
  • 19. Query Answer LLM Model Filters Extraction Selected Filters Alternative queries Structured Query Solr Relevant Documents Query Reformulation List of Fields and Values PROMPT Ask to the model to provide: - different/additional relevant terms - synonyms - variations with same meaning QUERY REFORMULATION
  • 20. Query Answer LLM Model Filters Extraction Selected Filters Alternative queries Structured Query Solr Relevant Documents Query Reformulation List of Fields and Values ['Sulfur dioxide emissions', 'Air pollution', 'Environmental impact', 'Fossil fuel combustion', 'Acid rain'] QUERY REFORMULATION What were the sulfur oxide emissions in Australia in 2013?
  • 21. Query Answer LLM Model Filters Extraction Selected Filters Alternative queries Structured Query Solr Relevant Documents Query Reformulation List of Fields and Values STRUCTURED QUERY SOLR QUERY q= title:(Sulfur dioxide emissions Air ... Acid rain) OR topic:"Environment#ENV#|Air and climate#ENV_AC#" OR country:"Australia#AUS#" OR variable:"Total man-made emissions#TOT#" OR Pollutant:"Sulphur Oxides#SOX#" OR 'Year': '2013'
  • 22. Query Answer LLM Model Filters Extraction Selected Filters Alternative queries Solr Relevant Documents Query Reformulation List of Fields and Values DOC RETRIEVAL SEARCH RESULTS "response":{ "numFound":1, "start":0, "numFoundExact":true, "docs":[{ "Title":"Emissions of air pollutants", "Dimension":["Country", "Pollutant", "Variable", "Year"] }] }
  • 23. ● Separates the flow of your program (modules) from the parameters (LM prompts and weights) of each step ● Introduces new optimizers to tune prompts and/or weights of your LM calls, given a metric you want to maximise ● LMs and their prompts fade into the background as optimizable pieces of a larger system that can learn from data DSPY LIBRARY https://dspy-docs.vercel.app/docs/intro
  • 25. AGENDA Use Case Overview From Natural Language to Structured Queries Findings The Road to Production
  • 26. MODEL CONSIDERATIONS ● [Model Selection]: NOT the most advanced available for this task ● [Model Comparison]: No evaluations or comparisons with alternative models → Time constraints and limited funding ● [Rationale for Current Choice]: promising capabilities and quick implementation ● [Future Works]: ○ Explore and analyze models that are fine-tuned specifically for our task ○ Potentially undertake our own fine-tuning to optimize model performance ○ Model comparison
  • 27. PROMISING ASPECTS ● Overcome the lexical matching land of kangaroos → [Country] AUSTRALIA tobacco consumption → [Topic] SMOKING/RISK FACTORS FOR HEALTH
  • 28. ● Explainability for selected filters Analyze input text: "cost per square meter for family houses in italy" cost per square meter → pricing or valuation → 'Priced unit' or 'Value' family houses → type of property → 'Real estate type' italy → location → 'Reference area' or 'Borrowers' country' PROMISING ASPECTS
  • 29. ● Explainability for selected filters Analyze input text: "cost per square meter for family houses in italy" cost per square meter → pricing or valuation → 'Priced unit' or 'Value' family houses → type of property → 'Real estate type' italy → location → 'Reference area' or 'Borrowers' country' PROMISING ASPECTS Integrate as an "Assistant" feature to guide users in choosing the most suitable filters IDEA!
  • 30. PROMISING ASPECTS ● Promising potential in early results: ○ challenging and complex task ○ good results (using a commercial out-of-the-box model!) ○ straightforward implementation ○ model's adaptability to the context
  • 31. LIMITATIONS FUNCTIONAL 2 1FUNCTIONAL Retrieval Augmented Generation FORMAL LLM weaknesses in the language/query semantic comprehension LLM weaknesses in complying with: ● the problem definition ● the required output format
  • 32. ● Difficult to identify relevant fields when others share the same values { "Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...], "Borrower’s Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...] } FUNCTIONAL LIMITATIONS
  • 33. ● Difficult to identify relevant fields when others share the same values { "Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...], "Reporting Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...] } ● Difficult to identify relevant fields when highly specialized domain knowledge is required "Marginal lending facility rate" → [Reference Area] Europe "IMU tax" → [Sector] Real Estate FUNCTIONAL LIMITATIONS
  • 34. ● Difficult to identify relevant fields when others share the same values { "Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...], "Reporting Country": ["All countries", "Europe", "G20", "Asia", "Morocco", ...] } ● Difficult to identify relevant fields when high specialized domain knowledge is required "Marginal lending facility rate" → [Reference Area] Europe "IMU tax" → [Sector] Real Estate ● Sometimes right value for a field not selected even if it is present in the user query User Query → green growth in Rabat Explainability → "Country": "Morocco" would be the relevant value if it were listed, but it is not. Country → ["All countries", "Europe", "G20", "Asia", "Morocco", ...] FUNCTIONAL LIMITATIONS EXPECTED FIELD
  • 35. POSSIBLE SOLUTIONS ● Refinement of the input Solr dictionary ○ Human readable fields’ names "HEDxkkgkqIr" → "Category" "INST_NON_EDU" → "Non-educational institutions" A win for those who always asked clients to use understandable Solr fields!
  • 36. POSSIBLE SOLUTIONS ● Ad-hoc prompt engineering ○ Expand the prompt with ambiguous/difficult examples and solutions ■ "Marginal lending facility rate" → [Reference Area] Europe ■ "IMU tax" → [Sector] Real Estate ○ Break down the prompt ■ One request for topic selection ■ One request for topic values selection ■ One request for dimensions selection ■ One request for dimensions values selection
  • 37. POSSIBLE SOLUTIONS ● LLM fine-tuning ○ Better disambiguation ○ Learn the specific and domain-related task
  • 38. FORMAL LIMITATIONS ● LLM hallucinations: ○ Field names E.g. "Instrument" instead of "Type of instruments" ○ Field values "Year": "21st century" instead of "2000" ● Returned field-values are mixed up E.g. ○ "Total emissions per capita" is part of a value and not a dimension ○ "European Union (28 countries)#EU28#" is a valid value present in "Country" but not in "Reference Area"
  • 39. FORMAL LIMITATIONS ● Poorly formatted JSON returned Selected Pairs: ```json { "Country": "Australia#AUS#", // land of kangaroos "Pollutant": "Sulphur Oxides#SOX#", "Year": "2013" } ``` These pairs are chosen based on the keywords identified in the input text and the closest matching dimensions and values from the provided dictionary.
  • 40. POSSIBLE SOLUTIONS ● Post-processing to validate and correct the LLM answer. ● DSPy library additional studies. (Typed Predictors, Optimizers) ● Evaluation of additional libraries and strategies ● Fine-tuning the model for the specific task → Extraction
  • 41. AGENDA Use Case Overview From Natural Language to Structured Queries Findings The Road to Production
  • 42. ● [UX] Design the user experience ○ Filtering assistance? ○ Transparent query parsing? ● [LLM] Select the best model to date ○ Can we fine-tune promising models specifically for the task? ● [LLM] Refine the prompts according to the model ○ Can we use only one request to build the structured query? THE ROAD TO PRODUCTION
  • 43. ● [LLM] Implement integration tests with the most common failures → LLM/prompt engineering to solve them ● [LLM] Study additional libraries to make the prompt more “programmed” and “automatically tuned” and less “trial-and-error” ○ Highly depend on the LLM available ● [Performance] Stress test the solution ● [Quality] Set up queries/expected documents THE ROAD TO PRODUCTION
  • 44. STAY UP TO DATE SUBSCRIBE TO THE INFORMATION RETRIEVAL NEWSLETTER https://sease.io/our-blog