Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Talk@rmit 09112017


Published on

With the explosive growth of online information, recommender system has been an effective tool to overcome information overload and promote sales. In recent years, deep learning's revolutionary advances in speech recognition, image analysis and natural language processing have gained significant attention. Meanwhile, recent studies also demonstrate its efficacy in coping with information retrieval and recommendation tasks. Applying deep learning techniques into recommender system has been gaining momentum due to its state-of-the-art performance. In this talk, I will present recent development of deep learning based recommender models and highlight some future challenges and open issues of this research field.

Published in: Internet
  • Be the first to comment

Talk@rmit 09112017

  1. 1. Deep Learning for Recommender Systems Presenter: Shuai Zhang, PhD student, CSE, UNSW Email:
  2. 2. Biography • Oct 2016 – Present, PhD Student at School of Computer Science and Engineering of UNSW, and Data61, CSIRO • 2010 – 2014, Bachelor degree from • 2014 – 2016, Software Engineer, • Lina Yao, UNSW Xiwei Xu, Data61, CSIRO Liming Zhu, Data61, CSIRO Recommender Systems Deep Learning Internet of Things
  3. 3. Content • Introduction to Recommender Systems • Overview of Deep Learning based Recommender Systems • State-of-the-art Models • Future Research Directions • My PhD Research Topic
  4. 4. Content Deep Learning based Recommender Systems: A Survey and New Perspectives • Shuai Zhang, University of New South Wales • Lina Yao, University of New South Wales • Aixin Sun, Nanyang Technological University
  5. 5. Introduction to Recommender Systems Recommender Systems are software tools and techniques that provide suggestions for items that are most likely of interest to a particular user [9] Promote Sales Overcome Information Overload Business Customer
  6. 6. Introduction to Recommender Systems To find relevant things for a user based on some kind of feedback: explicit feedback (ratings) or implicit feedback (interaction) Products Content Services Offer etc. User Preferences Interests Intentions etc. What is relevant
  7. 7. Introduction to Recommender Systems 30 percent of's page views were from recommendations, 2015 80 percent of movies watched on Netflix came through recommendations, 2016 60 percent of video clicks came from homepage recommendations, 2010
  8. 8. Overview Deep learning has achieved great success in many application domains • Computer Vision • Speech Recognition • Natural Language Processing
  9. 9. Overview Can deep learning techniques solve recommendation problems?
  10. 10. Overview In recent years, there has been a surge of interest in applying deep learning techniques to recommender systems • DLRS, workshop of Recsys • Google, Yahoo, Hulu, Alibaba Deep learning based models achieved the best performances and is a promising tool for recommender problems
  11. 11. Overview • The number of research publications on deep learning based recommendation models has increased exponentially in these years. Deep learning has driven a remarkable revolution in recommender applications
  12. 12. Overview: Advantages Advantages of employing deep learning techniques for recommendations:  Representation Learning • Structural features: genres • Text: storyline • Images: posters of movies • Audio: audio signals of music • Video
  13. 13. Overview: Advantages Advantages of employing deep learning techniques for recommendations:  Non-linearity • Sigmoid • Tanh • Rectified linear unit
  14. 14. Overview: Advantages Advantages of employing deep learning techniques for recommendations:  Generalization Many classical model can be extended with neural network to more generalized models • Neural Collaborative Filtering [6] • Neural Factorization Machine [12]
  15. 15. Overview: Advantages Advantages of employing deep learning techniques for recommendations:  Make use of the GPU Computing Resources • Involves a lot of matrix processing • GPU is better at intensive computing
  16. 16. Overview • More than 100 research papers • Different tasks: rating prediction, top-n recommendations, cross-domain recommendation, session-based recommendation • Different deep learning techniques: MLP, CNN, RNN, Autoencoder, etc • To help us better understand deep learning based recommender systems: Current status, Future trends, Open issues
  17. 17. Overview: Categories Two-dimensional Classification Scheme for deep learning based recommender systems • Neural Network Model: Classify the existing studies in accordance with the types of employed deep learning techniques • Integration Model: Whether it integrates traditional recommendation models (neighbourhood model, Matrix factorization, factorization machine, etc.) with deep learning or relies solely on deep learning
  18. 18. Overview: Categories Neural Autoregressive Distribution Estimation Generative Adversarial Network
  19. 19. Overview: Categories Model based on single deep learning technique, eight subcategories: 1. Multilayer Perceptron(MLP): scalable, and can easily introduce non-linearity for recommender systems; Neural Collaborative Filtering Wide & Deep Learning Deep Factorization Machine 2. Autoencoder(AE): learn salient feature representations; AutoSVD++ (by myself, SIGIR 2017): learn low dimensional feature representations to enhance recommendation accuracy AutoRec
  20. 20. Overview: Categories 3. Convolutional Neural Network(CNN): extract local and global representations heterogenous data sources; Extracting features from images Audio (Music Recommendations) Texts 4. Recurrent Neural Network(RNN):temporal dynamics and sequential influences; Session-based Recommendation Recurrent Recommender Networks
  21. 21. Overview: Categories 5. Deep Semantic Similarity Model(DSSM): perform semantic matching between users and items; Tag-aware Recommendation Cross-domain Recommendation 6. Restricted Boltzmann Machine(RBM): was first applied to recommendation tasks; 7. Neural Autoregressive Distribution Estimation: it is a tractable and efficient estimator for modelling data distributions; 8. Generative Adversarial Network: combine discriminative and generative models together. IRGAN (web search, item recommendation, question answering)
  22. 22. Overview: Categories Deep Composite Models: deep learning techniques can complement one another and enable a powerful hybrid model. Existing deep composite models Every deep composite model should be carefully designed and suitable for specific tasks
  23. 23. Overview: Categories • Integrate Deep Learning with Traditional Recommendation Model (matrix factorization, probabilistic matrix factorization, factorization machine); Based on how tightly the two approaches are integrated. Loosely Coupled Model: Learn parameters of deep learning and conventional recommendation algorithms separately Tightly Coupled Model: Learning process are done simultaneously • Recommend Rely Solely on Deep Learning • Without any forms of help from traditional recommendation models
  24. 24. State-of-the-art Models: NCF Neural Collaborative Filtering WWW 2017 Xiangnan He, National University of Singapore Lizi Liao, National University of Singapore Hanwang Zhang, Columbia University Liqiang Nie, Shandong University Xia Hu, Texas A&M University Tat-Seng Chua, National University of Singapore
  25. 25. State-of-the-art Models: NCF 𝑦 𝑢𝑖 ^ = 𝑓𝑜𝑢𝑡 𝑓𝑋 … 𝑓2 𝑓1 𝑃 𝑇 𝑖𝑑 𝑢, 𝑄 𝑇 𝑖𝑑𝑖 … 𝑓𝑜𝑢𝑡, 𝑓𝑋: mapping function for output layer and x-th neural CF layer Explicit feedback: Implicit feedback:
  26. 26. State-of-the-art Models: NCF 𝑦 𝑢𝑖 ^ = 𝑓𝑜𝑢𝑡(𝑓1 𝑃 𝑇 𝑖𝑑 𝑢 ⊙ 𝑄 𝑇 𝑖𝑑𝑖 ) 𝑓1 𝑎𝑛𝑑 𝑓𝑜𝑢𝑡(): are identity mapping functions Matrix Factorization: 𝑦 𝑢𝑖 ^ = 𝑃𝑢 𝑇 ⊙ 𝑄𝑖 𝑇 Squared Loss: 𝐿 = 𝑢,𝑖 ∈𝑌 𝑦 𝑢𝑖 − 𝑦 𝑢𝑖 ^ 2
  27. 27. State-of-the-art Models: NCF Matrix Factorization Multilayer Perceptron
  28. 28. State-of-the-art Models: AutoSVD++ AutoSVD++: an efficient hybrid collaborative filtering model via contractive auto-encoders SIGIR 2017 Shuai Zhang, University of New South Wales Lina Yao, University of New South Wales Xiwei Xu, Data61, CSIRO
  29. 29. State-of-the-art Models: AutoSVD++ we propose utilizing Contractive Autoencoder (CAE) to extract salient feature representations, and then integrating them into traditional matrix factorization model, SVD and SVD++. CAE + SVD  AutoSVD CAE + SVD++  AutoSVD++ Illustration of AutoSVD (remove the implicit feedback) and AutoSVD++.
  30. 30. • Our model leverages latent factor model and auto-encoder in a coupled manner with high scalability. The proposed efficient AutoSVD++ algorithm significantly improves the computational efficiency. • By integrating the Contractive Auto-encoder (CAE), our model catches the non-trivial and non-linear characteristics from item content information. Compared to other autoencoder variants, contractive autoencoder captures the infinitesimal input variations. • Our model effectively makes use of implicit feedback to further improve the accuracy. State-of-the-art Models: AutoSVD++
  31. 31. Contractive Autoencoder • 𝐽𝑐𝑎𝑒 𝜃 = 𝑥 ∈𝐷 𝑛 (𝐿(𝑥, 𝑔 𝑓 𝑥 ) + 𝜆 ∥ 𝐽 𝑓 𝑥 ∥ 𝐹 2 ) • Reconstruction: 𝑔 𝑓 𝑥 = 𝑠𝑔 𝑊′ 𝑠𝑓 𝑊𝑥 + 𝑏ℎ + 𝑏 𝑦 • Bottleneck layer: 𝑐𝑎𝑒 𝐶𝑖 = 𝑠𝑓 (𝑊 ⋅ 𝐶𝑖 +𝑏ℎ) Prediction Rule of AutoSVD • 𝑟𝑢𝑖 = 𝜇 + 𝑏 𝑢 + 𝑏𝑖 + 𝛽 ⋅ 𝑐𝑎𝑒 𝐶𝑖 + 𝜖𝑖 𝑇 𝑈 𝑢 Prediction Rule of AutoSVD++ • 𝑟𝑢𝑖 = 𝜇 + 𝑏 𝑢 + 𝑏𝑖 + • 𝛽 ⋅ 𝑐𝑎𝑒 𝐶𝑖 + 𝜖𝑖 𝑇 (𝑈 𝑢+|𝑁 𝑢 |− 1 2 𝑗 ∈𝑁(𝑢) 𝑦𝑗) State-of-the-art Models: AutoSVD++
  32. 32. Idea: group the data of same users who share same implicit feedback information. State-of-the-art Models: AutoSVD++
  33. 33. Deep Collaborative Filtering Framework [16] State-of-the-art Models: Extensions
  34. 34. Collaborative Deep Learning for Recommender System [15] State-of-the-art Models: Extensions
  35. 35. Convolutional Matrix Factorization [17] State-of-the-art Models: Extensions
  36. 36. State-of-the-art Models: Neural Rating Tips Neural Rating Regression with abstractive Tips Generation for Recommendation [14] Deep Composite Model : MLP and RNN Multitask Framework • Main Task: Rating prediction with Multilayer perceptron • Auxiliary Task: Tips Generation
  37. 37. State-of-the-art Models: Neural Rating Tips Examples of the predicted ratings and the generated tips. The first line of each group shows the generated rating and tips. The second line shows the ground truth.
  38. 38. Future Research Directions
  39. 39. Future Research Directions: Feature Learning Get a deep understanding of users and items from various side information with deep learning • Video • Audio • Image • User’s footprints • Etc…
  40. 40. Future Research Directions: Temporal Dynamics • Timestamp • Changes of Item and User preferences • Session based Recommendation  Tracking user’s long term interaction is inapplicable for many applications and mobile apps  Short term session can be usually collected and utilized
  41. 41. Future Research Directions: Multitask Learning • Learning several tasks at a time can prevent overfitting • Auxiliary task provides interpretable output for explaining the recommendation • Alleviating the sparsity problem implicitly Neural Rating Tips
  42. 42. Future Research Directions: Cross Domain • User’s preferences on one domain will influence her preferences on other domains somehow • Many companies offer diversified products or services • Deep learning is well suited to transfer learning as it can learn high-level abstractions that disentangle the variation of different domains
  43. 43. My PhD Research Topic Intention-aware Recommender Systems User’s intention inferred from other data sources can help improve the recommendation quality What is intention? Intention  Intend to do something (a determination to act in a certain way)  Demands / Needs  Influence later action somehow, e.g. consumption
  44. 44. My PhD Research Topic Twitter users you followed Meetup groups Books in Amazon
  45. 45. My PhD Research Topic What is Intention Mining (Intention Modeling) ? • Intention Mining is the problem of determining a user’s intention from their activities [4] • It has been the key research issue for providing personalized experiences and services [5] Application Fields: • Web Search • Software Engineering, etc.
  46. 46. My PhD Research Topic
  47. 47. My PhD Research Topic Infer user’s interests (long-lasting intention) from Twitter • Users she follows • Twitter lists • Descriptions Recommend Special interest groups • Meetup Groups
  48. 48. References [1] Zhang, Shuai, Lina Yao, and Aixin Sun. "Deep Learning based Recommender System: A Survey and New Perspectives." arXiv preprint arXiv:1707.07435 (2017). [2] [3] Li, Piji, et al. "Neural Rating Regression with Abstractive Tips Generation for Recommendation." (2017). [4] Khodabandelou, Ghazaleh, et al. "Supervised intentional process models discovery using hidden markov models." Research Challenges in Information Science (RCIS), 2013 IEEE Seventh International Conference on. IEEE, 2013. [5] Chen, Zheng, et al. "User intention modeling in web applications using data mining." World Wide Web 5.3 (2002): 181-191. [6] He, Xiangnan, et al. "Neural collaborative filtering." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017. [7] Zhang, Shuai, Lina Yao, and Xiwei Xu. "AutoSVD++: An Efficient Hybrid Collaborative Filtering Model via Contractive Auto-encoders." arXiv preprint arXiv:1704.00551 (2017). [8] Zhang, Shuai, and Lina Yao. "Dynamic Intention-Aware Recommendation System." arXiv preprint arXiv:1703.03112(2017). [9] Ricci, Francesco, Lior Rokach, and Bracha Shapira. "Recommender systems: introduction and challenges." Recommender systems handbook. Springer US, 2015. 1-34. [10]
  49. 49. References [11] learning-ai/ [12] He, Xiangnan, and Tat-Seng Chua. "Neural Factorization Machines for Sparse Predictive Analytics." (2017). [13] [14] Li, Piji, et al. "Neural Rating Regression with Abstractive Tips Generation for Recommendation." (2017). [15] Wang, Hao, Naiyan Wang, and Dit-Yan Yeung. "Collaborative deep learning for recommender systems." Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2015. [16] Li, Sheng, Jaya Kawale, and Yun Fu. "Deep collaborative filtering via marginalized denoising auto- encoder." Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM, 2015. [17] Kim, Donghyun, et al. "Convolutional matrix factorization for document context-aware recommendation." Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 2016.
  50. 50. Thank you Q & A