Intent-aware Temporal Query Modeling for Keyword                            Suggestion                  Fredrik Johansson,...
job openings                      vacation days                      We denote simple (pairwise) graphs G = (V, E P ) and ...
Model                            Error, k = 3    Error, k ≤ 4    θh         Qh                                            ...
Model       Avg. successful suggestions       Avg. recall        [10] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent diric...
Upcoming SlideShare
Loading in …5

Intent-Aware Temporal Query Modeling for Keyword Suggestion


Published on

This paper presents a data-driven approach for capturing the temporal variations in user search behaviour by modeling the dynamic query relationships using query-log data. The dependence between different queries (in terms of the query words and latent user intent) is represented using hypergraphs which allows us to explore more complex relationships compared to graph-based approaches. This time-varying dependence is modeled using the framework of probabilistic graphical models. The inferred interactions are used for query keyword suggestion - a key task in web information retrieval. Preliminary experiments using query logs collected from internal search engine of a large health care organization yield promising results. In particular, our model is able to capture temporal variations between queries relationships that reflect known trends in disease occurrence. Further, hypergraph-based modeling captures relationships significantly better compared to graph-based approaches.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Intent-Aware Temporal Query Modeling for Keyword Suggestion

  1. 1. Intent-aware Temporal Query Modeling for Keyword Suggestion Fredrik Johansson, Vinay Jethava Svetoslav Marinov Tobias Färdig Chalmers University Findwise AB Chalmers University Gothenburg, Sweden Gothenburg, Sweden Gothenburg, Sweden {frejohk,tobiasf}@student.chalmers.seABSTRACT Modeling intent presents two major challenges. First,This paper presents a data-driven approach for capturing while research into intent has been conducted for over athe temporal variations in user search behaviour by mod- decade, no consensus regarding a definition of intent haseling the dynamic query relationships using query-log data. been established. In order to systematically reason aboutThe dependence between different queries (in terms of the intent, a mathetmatical definition is needed. Second, it isquery words and latent user intent) is represented using hy- widely recognized that intent changes with time [37, 27].pergraphs which allows us to explore more complex rela- However, because of the multiple definitions of intent, it istionships compared to graph-based approaches. This time- not clear how to model temporal dynamics.varying dependence is modeled using the framework of prob- Traditionally, intent is modeled as a goal belonging to aabilistic graphical models. The inferred interactions are used small set of categories such as navigational, transactional orfor query keyword suggestion - a key task in web informa- informational [11]. This view has been extended to multi-tion retrieval. Preliminary experiments using query logs col- faceted descriptions [33, 4, 21]. However, scalability is alected from internal search engine of a large health care or- big issue with such classifications, requiring either costly,ganization yield promising results. In particular, our model manual labeling, or machine learning approaches with largeis able to capture temporal variations between queries re- amounts of training data. Recent work, related to proba-lationships that reflect known trends in disease occurrence. bilistic topic models [9], use an implicit, cluster-based rep-Further, hypergraph-based modeling captures relationships resentation of intent [14].significantly better compared to graph-based approaches. As of today, it is well known that user intent changes with time – the same search query may express different goals at different times [37, 27]. This poses a second challenge inCategories and Subject Descriptors intent modeling, requiring temporal dynamics to be takenH.3.3 [Information Storage and Retrieval]: [Informa- into account. While several authors have started applyingtion Search and Retrieval] this knowledge [37, 32] the research into modeling intent dynamics is still nascent.General Terms This paper investigates the problem of systematic mod- eling of dynamic trends within the area of search query in-Algorithms, Experimentation, Theory tent. We use an implicit representation of query intent as a distribution over a set of unknown topics. We presentKeywords a hypergraph-based approach for understanding the time-user intent, keyword suggestion, graphical model, dynamic varying interactions between search queries. A query inter-query interaction, query hypergraph action, represented by a hyperedge consisting of the search queries, expresses commonality in query intent. The varying1. INTRODUCTION strength of interaction, represented by a dynamic weight of Understanding user intent is of great importance in many the hyperedge, reflects the changing user needs over someapplications within the realm of information retrieval (IR), time period. This dependence is modeled using the frame-and more specifically search [37, 36]. A good understanding work of graphical models [26, 38]. Specifically, a genera-of intent can improve the user experience in several appli- tive model is presented wherein the time-varying interactioncations such as result ranking, query suggestion etc [13, 32, strength between two queries is modulated by the (latent)27]. intents of the search queries. The model allows tractable inference of query interactions from query logs using a mod- ified Expectation-Maximization algorithm [16, 30]. This paper applies the modeling of query interactions toPermission to make digital or hard copies of all or part of this work for query keyword suggestion - a key task in web information re-personal or classroom use is granted without fee provided that copies are trieval. More specifically, alternate keywords are suggestednot made or distributed for profit or commercial advantage and that copies for new documents based on queries which strongly interactbear this notice and the full citation on the first page. To copy otherwise, to with test query at current time. Preliminary experiments onrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. synthetic data as well querylogs extracted by the consult-PIKM’12, November 2, 2012, Maui, Hawaii, USA.Copyright 2012 ACM 978-1-4503-1719-1/12/11 ...$15.00.
  2. 2. job openings vacation days We denote simple (pairwise) graphs G = (V, E P ) and hy- 2 x2 = 5 4 x4 = 0 pergraphs H = (V, E). In general, we consider edge to mean job bank employment hyperedge, of cardinality ke ≥ 2 unless otherwise specified. e1 parking We model the query interaction network as a hypergraph, 1 3 e2 x1 = 2 x3 = 4 5 x5 = 1 where each node represents one unique query, and each hy- w1 = 1 w2 = 0 peredge represents a subset of queries expressing similar user intent, see Figure 1. Every edge e is associated with a t weight we , representing the strength of the interaction thatFigure 1: Hypergraph representation of query depends on the time point t. Weights take on values in ainteraction. Nodes represent unique queries and t small set W . If W = {0, 1} we interpret the case of we = 0hyperedges (ellipses) represent interactions of the as the queries making up edge e having no interaction at t.queries inside. The input to our model consists of two components, query usage data x and a base hypergraph H. Query usage data are time sequences x = ((xt )T )N with j the index of the j t=1 j=1ing company, Findwise AB1 , show that a) our hypergraph- query and t the time. Query usage xt is taken to be the jbased approach does significantly better than graph-based number of times a query j has been made at time t. Theapproaches for query keyword suggestion, and b) temporal hypergraph H = (V, E) is a specification of which edges toquery modeling can improve keyword suggestion. infer interactions for. Each node represents a unique query and the edges can be thought of as a representation of which2. RELATED WORK interactions that are expected to occur at all. Various methods for modeling query relationships havebeen explored, for example using measures of query similar- Intent-based query interaction hypergraph.ity and clickthrough URLs [5, 6] and topic models [20] We construct the base hypergraph H by identifying sub- Modeling user intent has been studied in the context of sets of queries which express the same underlying intent. Wequery suggestion [37, 13] and related problems including construct an implicit representation of intent in terms of thequery recommendation [32, 20] and query expansion [34]. click-through documents using LDA [10]. More precisely, weThis problem is related to the problem of keyword extrac- assume that there exists a set of topics H = {h1 , . . . , hNH }tion [24] wherein new documents are assigned a set of key- that documents and queries can belong to. We associatewords when they are entered into a search engine. Baeza- with each click-through document d, a topic distribution PYates et al. [6] analyze a bipartite query-document graph Ad = [Ad,1 , . . . , Ad,H ] , such that H Ad,h = 1, extracted h=1to infer semantic relationships between queries. using LDA [10] with appropriate priors. From the query Recent work has explored query dynamics towards im- logs, each query qi is associated with a set of documents,proving the search experience in applications such as query Di = {di,1 , . . . , di,m }, one document clicked on at each in-suggestions [3] and result ranking [32]. Kulkarni et. al. [27] stance of the query. We compute query topic distributionsstudy the temporal dynamics in queries, their underlying in- i = [κi,1 , . . . , κi,H ] as the average of document topic dis- κtent and the associated documents. However, none of these tributions weighted by the number of times ci,d a documentapproaches consider how queries relate to each other or how has been clicked on after using said query. We build thesuch relationships evolve. base graph by identifying queries of similar topic distribu- Probabilistic graphical models [26, 38] have been widely tions, . We add a node vi to V for each unique query qi . κused in number of fields including study of social networks We add an edge eP = (vi , vj ) to E P if DKL ( i || j ) εKL P κ κ[2], biological networks [22, 35], text streams [1]. where DKL (P || Q) = i P (i) ln(P (i)/Q(i)) [28], and εKL A related model for inferring pair-wise interactions in a is a parameter of the tool. The resulting (pairwise) graphgraph using node observations is Mixed-Membership Stochas- is then converted to a hypergraph H = {V, E} where wetic Blockmodels (MMSB) [2]; which has been extended to take the hyperedges, E, to be the maximal cliques of thetrack temporal dynamics of such interactions [18]. However, pairwise graph. Maximal cliques are computed using theextension to multi-way interactions is highly non-trivial and Bron-Kerbosch algorithm [12], see extended version [23].computationally infeasible since it involves inference of ten-sors rather than matrices. Dependence of query statistics on interactions. Bendersky and Croft [7] model query interactions as a The dependence of observed query statistics t = {xt }v∈V x vhypergraph to capture multi-way interactions. The work on query interactions wt = {we }e∈E present at that time is tindicates that a hypergraph representation of query graphs modeled using the Boltzmann distribution,can be highly beneficial compared to a pairwise approach, „X «but does not consider temporal dynamics. 1 x P (X t = t | W t = wt ) = exp t we φ(e, t) (1) Z(wt ) e∈E3. METHODOLOGY with Z(wt ) the normalization factor. Here, φ(e) is a poten- Q This paper investigates the increasingly recognized view [25, tial function. In this model we use, φ(e, t) = v∈e xt with i(v)19, 7] that in order to successfully capture multi-way interac- i(v) the index of node v.tions, one must treat them accordingly. In order to capturethis effectively, we use hypergraphs [8], a generalization ofgraphs which allow us to express interactions between sev- Modelling query interaction dynamics.eral queries. The base hypergraph H specifies sets of queries (as hy- peredges) which have similar user intent, expressed in terms1 of distribution i over topics H for each query qi . However, κ
  3. 3. Model Error, k = 3 Error, k ≤ 4 θh Qh Graph-based approach [22] 66% 37% H Our model 26% 30% t Qe wet xi t Table 1: Error in inferred weights on synthetic data. λe αe E N E T k is the cardinality of hyperedges.Figure 2: Plate notation representation of the gen-erative model. Letters in bottom right corners of 4. EXPERIMENTSboxes indicate repetitions of the variables inside. Evaluation on synthetic data. We generate random hypergraphs, H = (V, E) of twouser needs vary over time. For example, during periods of types, using the procedure described by Ghosal et. al. [19].economic downturn, more users might be looking for a job; The first type has uniform edge cardinality k = 3 and theand this in turn will be reflected in search queries express- second, random cardinality k ≤ 4 following the distributioning this need. Our model targets to capture such changes as P (k = 2) = 0.70, P (k = 3) = 0.25, P (k = 4) = 0.05. A copy tchanges in the hyperedge weights we . of the hypergraph is then converted to a pairwise (simple) In terms of topic distribution i obtained by LDA, some κ graph G = (V, E P ) by connecting all pair of nodes (nj , nk )of the topics will be more active at each time. These active of every hyperedge e with an edge eP = (j, k) in order totopics govern which query interactions are active at that allow comparison with graph-based approaches [22].time. We note that the active topics at each time are un- Weight sequences We and observations xt are generated t iknown - however, their influence is observed in terms of the for the hypergraph using known (randomly generated) pa-(latent) active interactions and consequently in the observed rameters Qh and αe,h . We construct one instance of ourquery statistics at that time. model using the hypergraph H and one using the pairwise We make the assumption that edge dynamics are condi- graph G. We perform inference on both of the models us-tionally independent conditioned on the active topics at that ing the observations x and the two graphs. This results intime i.e. the interactions in one group of queries is indepen- ˜t ˜ P,t two weight sequences, We and We , one inferred from eachdent of the interactions in other groups if we know which model, to compare to the original weight sequence, W t .topic is influencing each set of queries. We assume that the Since the pairwise graph has more and different edges thanset of active topics changes smoothly over time, motivated the hypergraph used as the ground truth, we need to inferby real-world phenomena such as seasonal changes. Under a weight sequence for the corresponding hypergraph usingthis assumption, we model interactions as a Markov model. the pairwise weight sequence. We do this using one of the t The transition probability of the interaction strengths we simplest decision rules - majority vote. Furthermore, weof an edge e can be written as P (We = wl | W 1 , . . . , W t−1 ) t compute Fleiss’ Kappa[17] , κF as a measure of the agree-= P (We = wl | W t−1 = wk ) = Qe (k, l) , with the constraint t ment between edges in the vote.PK l=1 Qe (k, l) = 1 for all k = 1, . . . , K. We take the the number of elements that differ from the We assume that for each h ∈ H there exist an unknown ˜ truth in sequences We and We as error measure. For thetransition probability matrix Qh which we call topic tran- experiment we use weights W = {−1, 0, 1}, N = 20 nodes,sition probability. We model the dependence of edge tran- T = 100 time points, H = 20 topics, and E = 50 hyperedges.sitions on user intent as follows: if h ∈ H is the active In table 1, we present the results of comparing performancetopic for edge e at time t, then the transition probability for pair-wise graphs and hypergraphs on synthetically gener-for edge e at time t is given by Qt = Qh . Mathematically, e ated data. The average Fleiss kappa was κF = −0.01 whichthis is equivalent to considering a mixture model over the indicates poor agreement in the majority vote [17].set of topic transition probabilities for each edge e. In themixture model, edge transition probabilitis Qe depend on Evaluation on keyword suggestion.topic transition probabilities Qh and mixture proportions In this section we evaluate the performance of our model inαe,h . The parameter αe,h govern the influence of topic h in the task of suggesting such keywords for real-world data. Wethe evolution of edge e and the relationship between Qe and P use querylogs and click-through documents from the internalQh can be written as Qe (k, l) =P h∈H αe,h Qh (k, l) with search-engine of a Swedish county council. The dataset con- h∈H αe,h = 1 for all e ∈ E. The (unknown) parameter α sists of 350 documents and 1 000 000 query instances of 20can be thought of as a matrix with element αe,h at index 000 unique queries. We construct the training and testing(e, h). We impose appropriate priors for α and Qh , see sets by partitioning the data chronologically into trainingextended version [23] for details. The generative model in- set (90%) and test set (10%).corporating the concepts described in this section is shown We suggest as keywords, for document d at time t, the setin Figure 2. t of queries Kd making up the interacting edge with topic dis- The modified expectation-maximization (EM) procedure tribution closest to that of the document in question. De-[16, 31] used for parameter estimation can, because of edge note by Qt = {qd,1 , . . . , qd,n }, for each document d, the set dconditional independence, be parallelized using frameworks of queries used to access d in the time interval t. We con-like MapReduce [15] or GraphLab [29]. t sider Kd a successful suggestion of keywords for d at time t if Qt ⊆ Kd , i.e. if we at least suggest as keywords all d t queries that are used to reach the document. We evaluate suggestions in terms of recall r defined in the usual way, as well as the percentage of documents that
  4. 4. Model Avg. successful suggestions Avg. recall [10] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet Our model 66% 0.62 allocation. JMLR, 3:993–1022, 2003. [11] A. Z. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3–10, 2002.Table 2: Results for keyword suggestion on query [12] C. Bron and J. Kerbosch. Algorithm 457: finding all cliques ofdataset. an undirected graph. Commun. ACM, 16(9):575–577, 1973. [13] H. Cao, D. Jiang, J. Pei, Q. He, Z. Liao, E. Chen, and H. Li. Context-aware query suggestion by mining click-through andare given successful suggestions. This is done for every doc- session data. In Proc. of KDD, pages 875–883, 2008.ument in the test set. Average recall and percentage of [14] A. Celikyilmaz, D. Hakkani-T¨ r, G. T¨ r, A. Fidler, and u u D. Hillard. Exploiting distance based similarity in topic modelssuccessful suggestions are presented in Table 2. for user intent detection. In D. Nahamoo and M. Picheny, An example of a successful suggestion is the set of key- editors, ASRU, pages 425–430, 2011.words [pension, expense, travel expenses, foundation] for a [15] J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Communications of the ACM,document reached by queries [expense, travel expenses]. 51(1):107–113, 2008. [16] A. Dempster, N. Laird, and D. Rubin. Maximum likelihood5. CONCLUSIONS from incomplete data via the em algorithm. J. Royal Statistical Society, Series B, 39(1):1–38, 1977. In this paper we have constructed an implicit hypergraph- [17] J. Fleiss. Measuring nominal scale agreement among manybased model of dynamic search intent using the framework raters. Psychological Bulletin, 76(5):378–382, 1971. [18] W. Fu, L. Song, and E. P. Xing. Dynamic mixed membershipof probabilistic graphical models. We have found through blockmodel for evolving networks. In Proc. of ICML 2009,tests on synthetic as well as real-world data that our model pages 329–336, capable of recovering changing intent expressed as hidden [19] G. Ghoshal, V. Zlatic, G. Caldarelli, and M. E. J. Newman.interaction dynamics in search query networks. Random hypergraphs and their applications. CoRR, abs/0903.0419, 2009. We have also found that our model can be used in an [20] J. Guo, X. Cheng, G. Xu, and X. Zhu. Intent-aware queryapplication to document keyword suggestions, and that the similarity. In Proc. of CIKM, pages 259–268, 2011.suggested keywords are relevant in that they are actually [21] J. Hu, G. Wang, F. Lochovsky, J. T. Sun, and Z. Chen.used as search terms. Understanding user’s query intent with wikipedia. In Proc. of the WWW ’09, pages 471–480, New York, USA, 2009. ACM. This paper is a small piece in long-term research into large [22] V. Jethava, C. Bhattacharyya, D. Dubhashi, and G. N. Vemuri.scale inference of temporal relationships in multivariate de- Netgem: Network embedded temporal generative model forpendence models. The implicit representation of intent as gene expression data. BMC Bioinformatics, 12(327), 2011.a topic distribution used in the paper allows for parallel- [23] F. Johansson and T. F¨rdig. Query concept interaction over a time. Master’s thesis, Chalmers University of Technology, 2012.lizable inference. This fact makes the model suitable for [24] J. Kaur and V. Gupta. Effective approaches for extraction oflarge-scale applications using existing frameworks such as keywords. Journal of Computer Science, 7(6):144–148, 2010.MapReduce [15] or GraphLab [29]. In future work, the im- [25] S. Klamt, U. Haus, and F. Theis. Hypergraphs and cellularpact of the temporal modeling on real-world applications networks. PLoS computational biology, 5(5):e1000385, 2009. [26] D. Koller and N. Friedman. Probabilistic Graphical Models -needs to be demonstrated fully. Principles and Techniques. MIT Press, 2009. [27] A. Kulkarni, J. Teevan, K. M. Svore, and S. T. Dumais.6. ACKNOWLEDGMENTS Understanding temporal query dynamics. In Proc. of WSDM 2011, pages 167–176, 2011. The authors would like to thank Devdatt Dubhashi and [28] S. Kullback and R. A. Leibler. On information and sufficiency.Chiranjib Bhattacharyya for helpful suggestions during the Ann. Math. Statist., 22(1):79–86, 1951. [29] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, andcourse of this project. We would also like to thank Findwise J. Hellerstein. Graphlab: A new parallel framework forAB, a consulting company in search solutions, and their em- machine learning. In Conf. on Uncertainty in Artificialployees for their insight and help and for providing data. Intelligence (UAI), Catalina Island, California, 2010. [30] R. Neal and G. E. Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in7. REFERENCES Graphical Models, pages 355–368. Kluwer Academic Publishers, 1998. [1] A. Ahmed and E. Xing. Timeline: A dynamic hierarchical dirichlet process model for recovering birth/death and [31] R. M. Neal. Probabilistic inference using Markov chain Monte evolution of topics in text stream. In Proc. of UAI, 2010. Carlo methods. Technical Report CRG-TR-93-1, Dept. of Computer Science, University of Toronto, 1993. [2] E. M. Airoldi, D. M. Blei, S. E. Fienberg, and E. P. Xing. Mixed membership stochastic blockmodels. JMLR, [32] K. Radinsky, K. Svore, S. Dumais, J. Teevan, A. Bocharov, 9:1981–2014, 2008. and E. Horvitz. Modeling and predicting behavioral dynamics on the web. In Proc. of WWW 2012, pages 599–608, 2012. [3] E. Alfonseca, M. Ciaramita, and K. Hall. Gazpacho and summer rash: lexical relationships from temporal patterns of [33] D. E. Rose and D. Levinson. Understanding user goals in web web search queries. In Proc. of EMNLP 2009, pages search. In Proc. of WWW 2004, pages 13–19, 2004. 1046–1055, 2009. [34] E. Sadikov, J. Madhavan, L. Wang, and A. Halevy. Clustering [4] R. Baeza-Yates, L. Calder´n-Benavides, and C. Gonz´lez-Caro. o a query refinements by user intent. In Proc. of WWW 2010, The intention behind web queries. In Proc. of SPIRE, volume pages 841–850, 2010. 409 of LNCS, pages 98–109, 2006. [35] L. Song, M. Kolar, and E. P. Xing. KELLER: estimating [5] R. Baeza-Yates, C. Hurtado, and M. Mendoza. Query time-varying interactions between genes. Bioinformatics, recommendation using query logs in search engines. In Proc. of 25:i128–i136, 2009. EDBT, pages 588–596, 2004. [36] J. Teevan, S. Dumais, and E. Horvitz. Potential for [6] R. Baeza-Yates and A. Tiberi. Extracting semantic relations personalization. ACM Trans. on Computer, Jan 2010. from query logs. In Proc. of KDD, pages 76–85, 2007. [37] J. Teevan, S. Dumais, and D. Liebling. To personalize or not to [7] M. Bendersky and W. B. Croft. Modeling higher-order term personalize: modeling queries with variation in user intent. dependencies in information retrieval using query hypergraphs. Proc. of SIGIR 2008, pages 163–170, 2008. In SIGIR, pages 941–950, 2012. [38] M. Wainwright and M. Jordan. Graphical Models, Exponential [8] C. Berge. Graphs and hypergraphs. Elsevier, 1976. Families, and Variational Inference. Foundations and Trends in ML, 1(1-2):1–305, 2008. [9] D. M. Blei. Introduction to probabilistic topic models. Commun. ACM, 2011.