Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Generalization of Tensor Factorization and Applications

12,347 views

Published on

Published in: Technology
  • Be the first to comment

Generalization of Tensor Factorization and Applications

  1. 1. Generalization of Tensor Factorization and Applications Kohei Hayashi Collaborators:T. Takenouchi, T. Shibata, Y. Kamiya, D. Kato, K. Kunieda, K. Yamada, K. Ikeda, R. Tomioka, H. Kashima May 14, 2012 1 / 29
  2. 2. Relational datais a collection of relationships among multiple objects. relationships of pair ⇔ matrix 2 / 29
  3. 3. Relational datais a collection of relationships among multiple objects. relationships of pair ⇔ matrix relationships of 3-tuple ⇔ 3 dim. array 3 / 29
  4. 4. Relational datais a collection of relationships among multiple objects.  relationships of pair ⇔ matrix  relationships of 3-tuple ⇔ 3 dim. array tensor . . . .  . .can represent as a tensor with missing values. 4 / 29
  5. 5. Issue of tensor representationTensor representation is generally high-dimensional andlarge-scale • e.g. 1, 000 users × 1, 000 items × 1, 000 times = total 1, 000, 000, 000 relationshipsDimensional reduction techniques such as tensorfactorization is used. 5 / 29
  6. 6. Tensor factorizationTucker decomposition [Tucker 1966]: a tensor factorizationmethod assuming 1 observation noise is Gaussian . 2 underlying tensor is low-dimensional .These assumptions are general ... but are not alwaystrue. 6 / 29
  7. 7. ContributionsGeneralize Tucker decomposition and propose two newmodels: 1 Exponential family tensor factorization (ETF) . [Joint work with Takenouchi, Shibata, Kamiya, Kunieda, Yamada, and Ikeda] • Generalize the noise distribution. • Can handle a tensor containing mixed discrete and continuous values. 2. Full-rank tensor completion (FTC) [Joint work with Tomioka and Kashima] • Kernelize Tucker decomposition • Complete missing values without reducing the dimensionality. 7 / 29
  8. 8. Outline1. Tucker decomposition2. Exponential family tensor factorization (ETF)3. Full-rank tensor completion (FTC)4. Conclusion 8 / 29
  9. 9. Outline1. Tucker decomposition2. Exponential family tensor factorization (ETF)3. Full-rank tensor completion (FTC)4. Conclusion 9 / 29
  10. 10. Tucker decomposition K1 K2 K3 (1) (2) (3) xijk = uiq ujr uks zqrs + εijk (1) q=1 r=1 s=1• ε: i.i.d Gaussian noise 10 / 29
  11. 11. Tucker decomposition K1 K2 K3 (1) (2) (3) xijk = uiq ujr uks zqrs + εijk (1) q=1 r=1 s=1• ε: i.i.d Gaussian noise 11 / 29
  12. 12. Tucker decomposition K1 K2 K3 (1) (2) (3) xijk = uiq ujr uks zqrs + εijk (1) q=1 r=1 s=1• ε: i.i.d Gaussian noise 12 / 29
  13. 13. Vectorized formLet x ∈ RD and z ∈ RK denote vectorized X and Z,resp., then x = Wz + ε where W ≡ U(3) ⊗ U(2) ⊗ U(1) . • D ≡ D1 D2 D3 , K ≡ K1 K2 K3 • ⊗: the Kronecker productTucker decomposition = A linear Gaussian model x ∼ N (x | Wz, σ 2 I) 13 / 29
  14. 14. Rank of tensorCall dimensionalities of the core tensor rank of tensor . • Rank of X is (K1 , K2 , K3 ).Rank of tensor represents its complexity • low-rank tensor has less information 14 / 29
  15. 15. Outline1. Tucker decomposition2. Exponential family tensor factorization (ETF)3. Full-rank tensor completion (FTC)4. Conclusion 15 / 29
  16. 16. Motivation: heterogeneous tensorTucker decomposition assumes Gaussian noise • not appropriate for such data 16 / 29
  17. 17. Motivation: heterogeneous tensorTucker decomposition assumes Gaussian noise • not appropriate for such data.Approach.generalize Tucker decomposition, assuming a differentdistribution for each element. 17 / 29
  18. 18. Exponential family tensor factorization.Likelihood. D x∼ Expond (xd | θd ), θ ≡ Wz d=1 exp. family Tucker decomp.. where Expon(x | θ) ≡ exp [xθ − ψ(θ) + F (x)]exponential family : a class of distributions Gaussian Poisson Bernoulli 0.4 0.7 Density Density 0.2 0.2 0.5 (θ = 0) 0.0 0.0 0.3 ψ(θ) θ2−4 −2 /2 0 2 4 exp[θ] 4 0 2 6 8 ln(10.0 0.5 1.0 1.5 −0.5 + exp[θ]) x 18 / 29
  19. 19. .Priors. • for z: a Gaussian prior N (0, I) (m) −1. • for U : a Gaussian prior N (0, αm I).Joint log-likelihood. D L =x Wz − ψhd (wd z) d=1 M 1 αm 2 − ||z||2 − U(m) + const. 2 m=1 2 Fro.Estimate parameters by Bayesian inference. 19 / 29
  20. 20. Bayesian inferenceMarginal-MAP estimator: argmax exp[L(z, U(1) , . . . , U(M ) )]dz U(1) ,...,U(M ) • The integral is not analytical.Develop efficient yet accurate approximation withLaplace approximation and Gaussian process. • Computational cost is still higher than Tucker dcomp. • For a long and thin tensor (e.g. time series), online algorithm is applicable (see thesis.) 20 / 29
  21. 21. Experiments 21 / 29
  22. 22. Anomaly detection by ETF Purpose Find irregular parts of tensor data Method Apply distance-based outlier DB (p, D) [Knorr+ VLDB’00] to the estimated factor matrix U(m).Definition of DB (p, D).“An object O is a DB (p, D) outlier if at least fraction p. the objects lies at a distance greater than D from O.”of 22 / 29
  23. 23. Data setA time series of multisensor measurements • Each sensor recorded the human behavior (e.g. position) of researchers in NEC lab. for 8 month. • 6 (sensors) × 21 (persons) × 1927 (hours) Sensor Type Min Max X1 # of sent emails Count 0.00 14.00 X2 # of received emails Count 0.00 15.00 X3 # of typed keys Count 0.00 50422.00 X4 X coordinate Real −550.17 3444.15 X5 Y coordinate Real 128.71 2353.55 X6 Movement distance Non-negative 0.00 203136.96 23 / 29
  24. 24. EvaluationList of irregular events are provided. Examples of irregular events Date Time Description Dec 21 All day Private incident Dec 22 15:00 Monthly seminar 16:00 Jun 15 13:00 Visiting tour . . . . . . . . . • Evaluate as a binary classification • 50% are missing, 10 trials • Apply ETF with online algorithm 24 / 29
  25. 25. Result: classification performance 0.7 method 0.6 PARAFAC AUC Tucker pTucker (EM) 0.5 ETF online 0.4 2x2x2 3x3x3 4x4x4 5x5x5 6x6x6 6x7x7 Rank 25 / 29
  26. 26. Result: classification performance 0.7 method 0.6 PARAFAC AUC Tucker pTucker (EM) 0.5 ETF online 0.4 2x2x2 3x3x3 4x4x4 5x5x5 6x6x6 6x7x7 Rank ETF well detect anomaly events 26 / 29
  27. 27. Skip the rest 27 / 29

×