Taxonomy-driven lumping for sequence mining

Taxonomy-driven lumping for sequence mining Francesco Bonchi, Carlos Castillo, Debora Donato, and Aristides Gionis Yahoo! Research, Barcelona

a day in barcelona wakeup la-padrera desingual siete-puertas goto-sleep wakeup parc-guell corte-ingles cal-pep fianna goto-sleep wakeup parc-guell sagrada-familia zara camber txakoli euskal-etxea goto-sleep wakeup la-padrera casa-batllo corte-ingles camber cal-pep txakoli fianna razzmatazz bella-istanbul goto-sleep wakeup casa-batllo parc-quell sagrada-familia quatre-gats goto-sleep wakeup casa-batllo rambla desingual siete-puertas goto-sleep wakeup parc-guell la-padrera zara corte-ingles diesel euskal-etxea no-se goto-sleep ...

a day in barcelona assume that we given a hierarchy on the basic events something tourism eating sightseeing shopping restaurant bar

a day in barcelona research question: what is a good model for a tourist’s day in barcelona?

a day in barcelona ^ something $

a day in barcelona ^ tourism eating $

a day in barcelona ^ sightseeing shopping eating $

a day in barcelona ^ sightseeing shopping restaurant bar $

a day in barcelona observation: deciding the model is equivalent with deciding where to cut the hierarchy

a day in barcelona ^ something something tourism eating sightseeing shopping restaurant bar $

a day in barcelona ^ something tourism tourism eating eating sightseeing shopping restaurant bar $

a day in barcelona ^ something sightseeing shopping tourism eating eating sightseeing shopping restaurant bar $

a day in barcelona ^ something sightseeing shopping tourism eating restaurant bar sightseeing shopping restaurant bar $

the problem we are solving input: something wakeup la-padrera desingual siete-puertas goto-sleep wakeup parc-guell corte-ingles cal-pep goto-sleep + tourism eating ... sightseeing shopping restaurant bar output: ^ something sightseeing shopping tourism eating eating sightseeing shopping restaurant bar $

motivating scenarios access logs at a corporate website access logs of interaction with a software system query-log mining trajectory data in all the above cases a hierarchy is given

contribution tons of related work Markov models, hidden Markov models, state aggregation of Markov models, sequence clustering, and more some references in the paper, by no means complete however, to our knowledge we are the ﬁrst to address this particular problem argue that it is a useful problem and propose eﬃcient algorithms

a bit more formally... alphabet on a set of symbols Σ = {α1 , . . . , αm } a hierarchy T over Σ set of input sequences D = {σ1 , . . . , σr } over Σ a set of states X = {x1 , . . . , xs } (Markov Model) transition probabilities px,y a set of symbols A(x) ⊆ Σ associated with each state x i.e., collection {A(x)} partitions Σ emission probabilities qx,α overall the model is represented by M = (X , A, p, q)

is it a hidden Markov model? no: there are no hidden states for each observed symbol there is unique emission state avoid computational challenges of HMM; eg. Baum Welch but not as ﬁne-grained as a Markov model ability to generalize from input data eg. {sightseeing, shopping} → tourism

problem statement given set of input sequences D and hierarchy T ﬁnd model M that minimizes the likelihood score function SL (D | M) SL is computed using both transition and emission probabilities

facts about the problem given A(x) (partition of the Σ) we can compute M = (X , A, p, q) that optimizes SL (D | M) i.e., compute transition and emission probabilities by counting transitions and occurrences in the data (maximum-likelihood estimation) ∗ let it be SL (D | M) — depends only on the partition of the states so, the complexity of our problem comes from computing the optimal partition

facts about the problem Theorem. Consider M1 and M2 such that the states of M1 is a sub-partition of the states of M2 . Then ∗ ∗ SL (D | M1 ) ≤ SL (D | M2 )

adapting the problem statement given set of input sequences D and hierarchy T ﬁnd model M that minimizes the function 2 2 ∗ ∗ 2 s − smin SL − SL,min Dist (M) = +w · ∗ ∗ , smax − smin SL,max − SL,min where M has s states and achieves likelihood score ∗ ∗ SL = SL (D | M)

algorithms

algorithm a generic bottom-up state-merging search algorithm with the following components: (i) an objective function g to evaluate goodness of a model (ii) a search policy p to compare models (iii) a priority queue Q of candidate models to evaluate (iv ) a set E of models already evaluated

algorithm 1 start with the leaf-level model and add it in the queue 2 extract from the queue the best model M according to p and evaluate its value according to g 3 consider all models that result from M with one merging (along the hierarchy) and add them in the queue 4 if has not reached maximum number of iterations go to step 2 possible to evaluate the a model eﬃciently from its ancestor model and precomputed counters

policies to compare models (i) ProbabilityFirst: p(vi , si ) > p(vj , sj ) if and only if vi < vj or (vi = vj and si < sj ); (ii) StatesFirst: p(vi , si ) > p(vj , sj ) if and only if si < sj or (si = sj and vi < vj ); (iii) DistSqFirst: p(vi , si ) > p(vj , sj ) if and only if g (vi , si ) < g (vj , sj ).

clustering EM-type algorithm to partition the input sequences into k clusters start with a random partition evaluate a model M for each partition reassign each sequence to the best model repeat

experiments

taxomo (taxonomy-driven markovian modeler) software written in Java provides the following features: model inference given a taxonomy and a set of sequences EM-based clustering computation of log-likelihood of a given set of sequences from a given model generation of a set of sequences given a model

dataset I — synthetic binary hierarchy with 64 leaves generative model by a random cut generate 10 000 random sequences

dataset II — query log sample of 85 000 queries from Yahoo! queries are assigned by a classiﬁer to the leafs of a hierarchy “ferrari photos” and “motorcycles” assigned to “recreation/automotive/autos” and “recreation/automotive/motorcycle” 100 leafs and max depth 6

dataset III — trajectories Brinkhoﬀ’s network-based synthetic generator of moving objects 500 000 spatio-temporal points for 80 000 trajectories hierarchy is a quad-tree formed by a 32 × 32 grid (e.g., the whole city is the root and recursively divide the area in quadrants)

results — synthetic dataset Synthetic data set 1.07 States DistSq 1 1.065 DistSq 10 1.06 log−likelihood ratio 1.055 1.05 1.045 1.04 1.035 0 2000 4000 6000 8000 10000 Number of iterations approximation error vs number of iterations

results — diﬀerent policies 6 Trajectories dataset 5 Query−log dataset x 10 x 10 States 2.8 3.55 Probability 2.6 3.5 DistSq 1 States DistSq 10 2.4 3.45 Probability Minus log likelihood DistSq 1 Minus log likelihood 2.2 3.4 DistSq 10 2 3.35 3.3 1.8 3.25 1.6 3.2 1.4 3.15 1.2 3.1 1 20 30 40 50 60 70 80 90 100 0 200 400 600 800 1000 Number of states Number of states search space proﬁle: probability of the best model found by the search algorithms, at each number of states, within 10,000 iterations

results — convergence 6 Trajectories dataset 5 Query−log dataset x 10 x 10 3.6 2.8 DistSq 10 −− 100 iter DistSq 10 −− 100 iter DistSq 10 −− 1K iter 3.55 DistSq 10 −− 1K iter 2.6 DistSq 10 −− 10K iter DistSq 10 −− 10K iter 3.5 2.4 Minus log likelihood 3.45 2.2 Minus log likelihood 3.4 2 3.35 1.8 3.3 1.6 3.25 1.4 3.2 1.2 3.15 1 3.1 0 20 40 60 80 100 0 200 400 600 800 1000 Number of states Number of states proﬁle of search space explored by DistSqFirst-10 policy for diﬀerent number of iterations

trajectories

trajectories — clustering

future directions incorporate time no hierarchy more applications

thank you!

Taxonomy-driven lumping for sequence mining

by Carlos Castillo on Sep 17, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Taxonomy-driven lumping for sequence mining — Presentation Transcript