This document introduces Factorization Machines, a general model that can mimic many successful factorization models. Factorization Machines allow feature vectors to be easily input and enjoy benefits of factorizing interactions between variables. The model has properties like expressiveness, multi-linearity, and scalable complexity. It relates to models like matrix factorization, tensor factorization, SVD++, and nearest neighbor models. Experiments show Factorization Machines outperform other models on rating prediction, context-aware recommendation, and tag recommendation tasks.

1. Factorization Machines with libFM
Steffen Rendle
University of Konstanz
ACM TIST, May 2012
WUME Reading Group
Liangjie Hong

2. Outline
• Motivations
• Model
• Experiments

3. Motivation
Factorization models show superior performance
• Collaborative filtering
▫ Movie recommendation
▫ Tag recommendation
• Link prediction

4. Motivation
• (Too) many factorization models
▫ General Form
 matrix factorization [Srebro and Jaakkola 2003]
 tensor factorization [Tucker 1966, Harshman 1970]
▫ Specific Tasks
 SVD++ [Koren 2008]
 STE [Ma et al. 2011]
 timeSVD++ [Koren 2009b]
 BPTF [Xiong et al. 2010]

5. Motivation
• Each task requires re-design
▫ model
▫ inference algorithm

6. Motivation
• What we want!
▫ Simple, easy to use, like libSVM, Weka…
▫ Feed in feature vectors
▫ Keep factorizations!

7. Proposed method
• Factorization Machines
▫ like libSVM…
▫ enjoy the benefits of factorized interactions
between variables
 2-n order interactions…
▫ can mimic many successful models
▫ three major inference algorithms
 SGD
 ALS
 MCMC

8. Proposed method
• Similar approaches
▫ Regression-based latent factor models
▫ SVD-feature model
▫ MF with Gaussian process/Dirichlet mixture
process

9. Roadmap
• Model
▫ properties
• Probabilistic Interpretation
• Relationships with other Factorization models
▫ matrix factorizations
▫ pairwise interaction tensor factorization
▫ SVD++ and FPMC
▫ BPFT and TimeSVD++
▫ NN
▫ attribute-aware models
• Inference algorithms
▫ SGD, ALS, MCMC

10. Model

11. Model

12. Model
• Factorization model with degree = 2

13. Model
• Factorization model with degree = 2
global “bias”
pairwise interaction factorization!
regression coefficients
strength of j-th variable

14. Model
• Factorization model with degree = 2

15. Model

16. Model

17. Model: Properties
• Expressiveness

18. Model: Properties
•

19. Model: Properties
• Multi-linearity

20. Model: Properties
• Multi-linearity

21. Model: Properties
• Multi-linearity

22. Model: Properties
• Multi-linearity

23. Model: Properties
• Multi-linearity

24. Model: Properties
• Complexity

25. Model: Properties
• Complexity

26. Model: Properties
• Complexity

27. Model: Higher-order

28. Model: Higher-order

29. Relationships to other models
• Matrix factorization
• Pairwise interaction tensor factorization
• SVD++ and FPMC
• BPTF and TimeSVD++
• NN
• Attribute-aware models
• SVM
• Others

30. Relationships to other models
• Matrix factorization
• Pairwise interaction tensor factorization
• SVD++ and FPMC
• BPTF and TimeSVD++
• NN
• Attribute-aware models
• SVM
• Others

31. Relationships to other models
• Matrix factorization

32. Relationships to other models
• Pairwise Interaction Tensor Factorization
▫ [Rendle and Schmidt-Thieme 2010]

33. Relationships to other models
• Pairwise Interaction Tensor Factorization

34. Relationships to other models
• Pairwise Interaction Tensor Factorization
▫ Tucker Decomposition

35. Relationships to other models
• Pairwise Interaction Tensor Factorization
▫ Canonical Decomposition (CD)

36. Relationships to other models
• Pairwise Interaction Tensor Factorization
▫ Pairwise Decomposition

37. Relationships to other models
• Pairwise Interaction Tensor Factorization

38. Relationships to other models
• Pairwise Interaction Tensor Factorization

39. Relationships to other models
• Pairwise Interaction Tensor Factorization

40. Relationships to other models
• SVD++
▫ SVD++ [Koren 2008]

41. Relationships to other models
• SVD++

42. Relationships to other models
• SVD++

43. Relationships to other models
• Bayesian Probabilistic Tensor Factorization
▫ [Xiong et al. 2010]
• TimeSVD++
▫ [Koren 2009b]
• Capture temporal effects

44. Relationships to other models

45. Relationships to other models
• Nearest neighbor Models
▫ Factorized nearest neighbor model
 [Koren 2010]
▫ Non-factorized nearest neighbor model
 [Koren 2008b]

46. Relationships to other models
• Nearest neighbor Models

47. Relationships to other models
• Nearest neighbor Models

48. Relationships to other models
• Attribute-aware models

49. Relationships to other models
• Attribute-aware models
▫ [Agarwal and Chen 2009]
▫ [Gantner et al. 2010]
• Cold-start problem

50. Relationships to other models
• Attribute-aware models

51. Relationships to other models
• Attribute-aware models

52. Relationships to other models
• Attribute-aware models

53. Relationships to other models
• SVM

54. Relationships to other models
• SVM
▫ Linear kernel

55. Relationships to other models
• SVM
▫ Linear kernel
• identical to 1st order FM

56. Relationships to other models
• SVM
▫ Polynomial kernel

57. Relationships to other models
• SVM
▫ Polynomial kernel

58. Relationships to other models
• SVM
V.S.

59. Relationships to other models
• SVM
V.S.

60. Experiments
• Rating prediction
▫ Netflix data
▫ RMSE
• Context-aware recommendation
▫ Yahoo! Webscope data
▫ RMSE
• Tag recommendation
▫ ECML/PKDD data
▫ F1 measure

61. Experiments
• Rating prediction

62. Experiments
• Rating prediction

63. Experiments
• Context-aware Rec.

64. Experiments
• Tag Rec.

65. Conclusion
• libFM is available.
• (potentially) integrate many more models.
• A simple way to combine features & latent factors

66. Conclusion
• libFM is available.
• (potentially) integrate many more models.
• A simple way to combine features & latent factors
• Both 4th position in KDD Cup 2012 T1/T2

67. Reference
• Steffen Rendle. Factorization machines with libfm. ACM Transactions on
Intelligent Systems and Technology, 3(3):57:1–57:22, May 2012
• Steffen Rendle. Factorization machines. In Proceedings of the 2010 IEEE
International Conference on Data Mining, pages 995–
1000, Washington, DC, USA, 2010. IEEE Computer Society.
• Steffen Rendle, Zeno Gantner, Christoph Freudenthaler, and Lars Schmidt-
Thieme. Fast context-aware recommendations with factorization machines. In
Proceedings of the 34th international ACM SIGIR Conference on Research
and Development in Information Retrieval (SIGIR), pages 635–644, New
York, NY, USA, 2011. ACM
• Christoph Freudenthaler, Lars Schmidt-Thieme, and Steffen Rendle. Bayesian
factorization machines. In Workshop on Sparse Representation and Low-rank
Approximation, Neural Information Processing Systems
(NIPS), Granada, Spain, 2011