Query expansion


Published on

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Query expansion

  1. 1. Query Expansion Cluster based using N Grams UMA K L (201305514) SPANDAN VEGGALAM (201307674) MAHAVER CHOPRA (201101011) AKSHAT KANDELWAL (201001095) INTERNATIONAL INSTITUTE OF INFORMATION TECHNOLOGY -HYDERABAD
  2. 2. Query Expansion • Key feature of Search Engine • In many cases it is difficult to find the search intent of user • Users do not always formulate query in the best way • Query recommendation is to help users in formulating queries to certain extent • Improves the search retrieval performance, user selects the alternate query input from suggestions which is relevant to his intent. • Increases recall of Information Retrieval System
  3. 3. Expanding Queries Following techniques are used for expanding queries 1. Spell Corrections 2. Finding and searching with “Synonyms” of input query terms 3. Augmenting query with terms In our approach we focus on Augmenting queries and Searching with synonyms
  4. 4. Our Approach Two Phases 1. Offline Phase 1. Add Synonyms to documents 2. Cluster the documents, in order to group similar documents into a cluster 3. Index and Label the clusters Only nouns are indexed. 2. Online Phase 1. Search for clusters as Phrase query 2. Predict words for query augmentation 3. Re-weight the query and suggest top queries as query recommendations Only Nouns are considered as augmented words
  5. 5. Our Approach – Offline Phase Why Clustering? 1. Clustering improves the scope of suggesting queries for different contexts 2. Documents are clustered together, and indexed 3. Search is performed on cluster index. 4. Relevant clusters are considered to find augmented terms 5. Top N query suggestions from each cluster are considered Clustering Parameters Used Algorithm: K-Means Number of Clusters: 150
  6. 6. Our Approach – Offline Phase Adding Synonyms 1. Allows user to search with synonyms as well 2. Ideally system should accept synonyms and is expected to retrieve same relevant documents 3. Top 5% of words from each document are considered, and synonyms are added to these words Labeling Clusters 1. Clusters are tagged with most relevant terms 2. Label contain set of terms which can distinguish it from other clusters
  7. 7. Our Approach – Online Phase 1. Retrieve relevant Clusters for given input query. 2. Select top ‘N’ Clusters 3. If given query can be represented in N Grams 1. As the words are sequential and from same document, intent of user is clear. Next word in document can be suggested as augment word 2. Retrieve next sequential word from the cluster, which is set of documents 3. Augment the query with these predicted words, retrieve top queries are present the user as query recommendations
  8. 8. Our Approach – Online Phase 4. Else if the query terms are separated with some distance 1. Predict next word for each input term, add terms to a list. 2. Identify the tags for clusters, and them to list. Tags are words that gives information about the input words together 3. Here user intent is not clear so, tags which gives category/topic/context of the document are also considered for augmentation 4. Augment the query with these predicted words, retrieve top queries are present the user as query recommendations 5. If the given words are far from each other, it is very difficult to co-relate each word 1. Sequential words cannot be used as augment words, the input words may be from different contexts and is hard to retrieve relevant documents thereby reduces the recall 2. Cluster tags are words which gives some information about the input words together, and are considered as augmented terms
  9. 9. Architecture – Offline Phase
  10. 10. Architecture – Online Phase
  11. 11. Tools & Data Set Used Tools ◦ Word Net: Used to identify synonyms ◦ Cluto: Clusters the documents ◦ Doc2Mat: Represents documents in matrix format ◦ Apache Lucene: Used for indexing and querying. ◦ Stanford NLP POS tagger: Identifying Part of speech of word Data Set ◦ Data set consists of Telegraph Calcutta news paper stories ◦ Stories are categorized into ◦ “FrontPage”, “Nation, “Calcutta”, “Bengal”, “Foreign”, “Business”, “Sports”, “Opinion”, “Metro” ◦ Format of each story in data set is ◦ <DocNo>*</DocNo><Text>*</Text>
  12. 12. Evaluation We have run the augmented queries over the data set to retrieve relevant documents and found considerable increase in recall and precision values. Following bar diagram gives change in precision and recall values for random augmented queries formulated over 30 input queries.
  13. 13. Evaluation Open this file for evaluation results of all augmented queries for 30 input queries.
  14. 14. Future Work 1. Approach can be extended to implement query logs ◦ Query logs can be used as knowledge base for suggesting queries, and also helps in asynchronous way of suggesting queries 2. User preferences can be used to filter the documents based on relevancy 3. Different Mixed versions of Markov models can be used to achieve the best balance among accuracy and coverage both in terms of data (objective) and user (subjective) centric evaluation metrics 4. Different N gram variations can be used to make it ideally suitable for real time Search engine
  15. 15. Conclusion 1. We have explained our approach for query expansions, which uses Clustering method to extend suggestions from various contexts, N gram and Markov model to determine augment terms 2. We have applied sequential probabilistic model as it is suitable for the task of online query recommendation 3. Achieved accuracy and coverage in terms of data. 4. Time and memory complexities of our application is measured and found it is suitable for real time search engine