Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Factorization Machines with libFM

13,135 views

Published on

Published in: Technology
  • Be the first to comment

Factorization Machines with libFM

  1. 1. Factorization Machines with libFMSteffen RendleUniversity of KonstanzACM TIST, May 2012 WUME Reading Group Liangjie Hong
  2. 2. Outline• Motivations• Model• Experiments
  3. 3. MotivationFactorization models show superior performance• Collaborative filtering ▫ Movie recommendation ▫ Tag recommendation• Link prediction
  4. 4. Motivation• (Too) many factorization models ▫ General Form  matrix factorization [Srebro and Jaakkola 2003]  tensor factorization [Tucker 1966, Harshman 1970] ▫ Specific Tasks  SVD++ [Koren 2008]  STE [Ma et al. 2011]  timeSVD++ [Koren 2009b]  BPTF [Xiong et al. 2010]
  5. 5. Motivation• Each task requires re-design ▫ model ▫ inference algorithm
  6. 6. Motivation• What we want! ▫ Simple, easy to use, like libSVM, Weka… ▫ Feed in feature vectors ▫ Keep factorizations!
  7. 7. Proposed method• Factorization Machines ▫ like libSVM… ▫ enjoy the benefits of factorized interactions between variables  2-n order interactions… ▫ can mimic many successful models ▫ three major inference algorithms  SGD  ALS  MCMC
  8. 8. Proposed method• Similar approaches ▫ Regression-based latent factor models ▫ SVD-feature model ▫ MF with Gaussian process/Dirichlet mixture process
  9. 9. Roadmap• Model ▫ properties• Probabilistic Interpretation• Relationships with other Factorization models ▫ matrix factorizations ▫ pairwise interaction tensor factorization ▫ SVD++ and FPMC ▫ BPFT and TimeSVD++ ▫ NN ▫ attribute-aware models• Inference algorithms ▫ SGD, ALS, MCMC
  10. 10. Model
  11. 11. Model
  12. 12. Model• Factorization model with degree = 2
  13. 13. Model• Factorization model with degree = 2 global “bias” pairwise interaction factorization! regression coefficients strength of j-th variable
  14. 14. Model• Factorization model with degree = 2
  15. 15. Model
  16. 16. Model
  17. 17. Model: Properties• Expressiveness
  18. 18. Model: Properties•
  19. 19. Model: Properties• Multi-linearity
  20. 20. Model: Properties• Multi-linearity
  21. 21. Model: Properties• Multi-linearity
  22. 22. Model: Properties• Multi-linearity
  23. 23. Model: Properties• Multi-linearity
  24. 24. Model: Properties• Complexity
  25. 25. Model: Properties• Complexity
  26. 26. Model: Properties• Complexity
  27. 27. Model: Higher-order
  28. 28. Model: Higher-order
  29. 29. Relationships to other models• Matrix factorization• Pairwise interaction tensor factorization• SVD++ and FPMC• BPTF and TimeSVD++• NN• Attribute-aware models• SVM• Others
  30. 30. Relationships to other models• Matrix factorization• Pairwise interaction tensor factorization• SVD++ and FPMC• BPTF and TimeSVD++• NN• Attribute-aware models• SVM• Others
  31. 31. Relationships to other models• Matrix factorization
  32. 32. Relationships to other models• Pairwise Interaction Tensor Factorization ▫ [Rendle and Schmidt-Thieme 2010]
  33. 33. Relationships to other models• Pairwise Interaction Tensor Factorization
  34. 34. Relationships to other models• Pairwise Interaction Tensor Factorization ▫ Tucker Decomposition
  35. 35. Relationships to other models• Pairwise Interaction Tensor Factorization ▫ Canonical Decomposition (CD)
  36. 36. Relationships to other models• Pairwise Interaction Tensor Factorization ▫ Pairwise Decomposition
  37. 37. Relationships to other models• Pairwise Interaction Tensor Factorization
  38. 38. Relationships to other models• Pairwise Interaction Tensor Factorization
  39. 39. Relationships to other models• Pairwise Interaction Tensor Factorization
  40. 40. Relationships to other models• SVD++ ▫ SVD++ [Koren 2008]
  41. 41. Relationships to other models• SVD++
  42. 42. Relationships to other models• SVD++
  43. 43. Relationships to other models• Bayesian Probabilistic Tensor Factorization ▫ [Xiong et al. 2010]• TimeSVD++ ▫ [Koren 2009b]• Capture temporal effects
  44. 44. Relationships to other models
  45. 45. Relationships to other models• Nearest neighbor Models ▫ Factorized nearest neighbor model  [Koren 2010] ▫ Non-factorized nearest neighbor model  [Koren 2008b]
  46. 46. Relationships to other models• Nearest neighbor Models
  47. 47. Relationships to other models• Nearest neighbor Models
  48. 48. Relationships to other models• Attribute-aware models
  49. 49. Relationships to other models• Attribute-aware models ▫ [Agarwal and Chen 2009] ▫ [Gantner et al. 2010]• Cold-start problem
  50. 50. Relationships to other models• Attribute-aware models
  51. 51. Relationships to other models• Attribute-aware models
  52. 52. Relationships to other models• Attribute-aware models
  53. 53. Relationships to other models• SVM
  54. 54. Relationships to other models• SVM ▫ Linear kernel
  55. 55. Relationships to other models• SVM ▫ Linear kernel• identical to 1st order FM
  56. 56. Relationships to other models• SVM ▫ Polynomial kernel
  57. 57. Relationships to other models• SVM ▫ Polynomial kernel
  58. 58. Relationships to other models• SVM V.S.
  59. 59. Relationships to other models• SVM V.S.
  60. 60. Experiments• Rating prediction ▫ Netflix data ▫ RMSE• Context-aware recommendation ▫ Yahoo! Webscope data ▫ RMSE• Tag recommendation ▫ ECML/PKDD data ▫ F1 measure
  61. 61. Experiments• Rating prediction
  62. 62. Experiments• Rating prediction
  63. 63. Experiments• Context-aware Rec.
  64. 64. Experiments• Tag Rec.
  65. 65. Conclusion• libFM is available.• (potentially) integrate many more models.• A simple way to combine features & latent factors
  66. 66. Conclusion• libFM is available.• (potentially) integrate many more models.• A simple way to combine features & latent factors• Both 4th position in KDD Cup 2012 T1/T2
  67. 67. Reference• Steffen Rendle. Factorization machines with libfm. ACM Transactions on Intelligent Systems and Technology, 3(3):57:1–57:22, May 2012• Steffen Rendle. Factorization machines. In Proceedings of the 2010 IEEE International Conference on Data Mining, pages 995– 1000, Washington, DC, USA, 2010. IEEE Computer Society.• Steffen Rendle, Zeno Gantner, Christoph Freudenthaler, and Lars Schmidt- Thieme. Fast context-aware recommendations with factorization machines. In Proceedings of the 34th international ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 635–644, New York, NY, USA, 2011. ACM• Christoph Freudenthaler, Lars Schmidt-Thieme, and Steffen Rendle. Bayesian factorization machines. In Workshop on Sparse Representation and Low-rank Approximation, Neural Information Processing Systems (NIPS), Granada, Spain, 2011

×