Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Immersive Recommendation Workshop, NYC Media Lab'17

267 views

Published on

The rapid evolution of deep learning technologies and the explosion of diverse user interaction traces have brought significant challenges and opportunities to recommendation and personalized systems. In this workshop, we discussed recent trends and techniques in user modeling and presented our work on immersive recommendation systems. These systems learn users’ preferences from diverse digital trace modalities (text, image and unstructured data streams) in a wide range of recommendation domains (creative art, food, news, and events). The workshop included a light tutorial on OpenRec, an open source framework that enables quick prototyping of complex recommender systems via modularization.

This workshop is based on research and development done at Cornell Tech as part of the Connected Experiences Lab, supported by Oath and NSF.

Published in: Engineering
  • Be the first to comment

Immersive Recommendation Workshop, NYC Media Lab'17

  1. 1. Immersive Recommendation Deep User and Content Modeling for Personalization Longqi Yang, Ph.D. student Connected Experiences Lab, Small Data Lab Cornell Tech
  2. 2. Collaborators Faculty Interns Industry Collaborators Ph.D. students PostDocs
  3. 3. What we will be talking about today Recommendation Systems: Past, Present and Future Immersive Recommendation. OpenRec.
  4. 4. Recommendation Systems Research
  5. 5. The Matrix From item-based filtering to Netflix Challenge
  6. 6. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 4 5 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001. 1 5 1
  7. 7. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 1 5 4 5 4 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.
  8. 8. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 1 5 4 5 4 0.9 0.1 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.
  9. 9. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 1 5 4 5 4 0.9 0.1 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.
  10. 10. … … … … … … … … … Item-based filtering (WWW 2001) 3.74 1 1 1 5 4 5 4 0.9 0.1 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.
  11. 11. Early Adoption by Amazon.com
  12. 12. Netflix Challenge (Prize) We’re quite curious, really. To the tune of one million dollars… … To help customers find movies, we’ve developed our world-class movie recommendation system: Cinematch. Its job is to predict whether someone will enjoy a movie based on how much they liked or disliked other movies … … We provide you with a lot of anonymous rating data, and a prediction accuracy bar that is 10% better than what Cinematch can do on the same training data set… If you develop a system that we judge most beats that bar on the qualifying test set we provide, you get serious money and the bragging rights …
  13. 13. Netflix Challenge (Prize) Cinematch score - RMSE = 0.9525 2007 Progress Prize - RMSE = 0.8723 8.42% 2008 Progress Prize - RMSE = 0.8627 9.27% 2009 Grand Prize - RMSE = 0.8567 10.06% (AT&T Research)
  14. 14. Rating-based recommendations
  15. 15. Beyond Matrix Implicit feedback Deep Content Modeling Beyond “black-box items” Beyond Accuracy Diversity and Fairness Rich Context and Modality Learning preference from auxiliary channels
  16. 16. Beyond Matrix Implicit feedback Deep Content Modeling Beyond “black-box items” Beyond Accuracy Diversity and Fairness Rich Context and Modality Learning preference from auxiliary channels
  17. 17. Click-through Upvote/Like View/Watch/Visit Listen/Play Implicit Feedback
  18. 18. Main Challenge of Implicit Feedback Only “positive signal” is observed Does “rating estimation” still work?
  19. 19. … … … … … … … … … ?1 1 1 1 1 1 1 1 Main Challenge of Implicit Feedback
  20. 20. … … … … … … … … … 11 1 1 1 1 1 1 1 Main Challenge of Implicit Feedback
  21. 21. … … … … … … … … … 11 1 1 1 1 1 1 1 No matter what this item is! Main Challenge of Implicit Feedback
  22. 22. Feature Learning Framework for Implicit Feedback [0.1 -0.2 0.35 … 0.15]𝒖 𝟏 = 𝒗 𝟏 = [-0.05 0.5 0.1 … -0.3] 𝒖 𝟐 𝒖 𝑵 … 𝒗 𝟐 𝒗 𝑴 … Optimization
  23. 23. One Example – Bayesian Personalized Ranking (BPR) 𝒖𝒊, 𝒗 𝒑, 𝒗 𝒏For all m𝑎𝑥 ln 𝜎 𝒖𝒊 ∙ 𝒗 𝒑 − 𝒖𝒊 ∙ 𝒗 𝒏 Rendle, Steffen, et al. "BPR: Bayesian personalized ranking from implicit feedback." Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence. AUAI Press, 2009. user i An item that the user “click” An item that the user does not “click”
  24. 24. Algorithms (Incomprehensive List) • Weighted Regularized Matrix Factorization (WRMF) • Probabilistic Matrix Factorization (PMF) “Shallow” Models: • Weighted Approximately Ranked Pairwise Loss (WARP) “Deep” Models: Hsieh, Cheng-Kang, et al. "Collaborative metric learning." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017.• Collaborative Metric Learning (CML) He, Xiangnan, et al. "Neural collaborative filtering." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017.• Neural Collaborative Filtering Mnih, Andriy, and Ruslan R. Salakhutdinov. "Probabilistic matrix factorization." Advances in neural information processing systems. 2008. • Wide and Deep Learning for Recommender Systems Hu, Yifan, Yehuda Koren, and Chris Volinsky. "Collaborative filtering for implicit feedback datasets." Data Mining, 2008. ICDM'08. Eighth IEEE International Conference on. Ieee, 2008. Cheng, Heng-Tze, et al. "Wide & deep learning for recommender systems." Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM, 2016. Weston, Jason, Samy Bengio, and Nicolas Usunier. "Wsabie: Scaling up to large vocabulary image annotation." IJCAI. Vol. 11. 2011.
  25. 25. Beyond Matrix Implicit feedback Deep Content Modeling Beyond “black-box items” Beyond Accuracy Diversity and Fairness Rich Context and Modality Learning preference from auxiliary channels
  26. 26. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! - Articles Is it appropriate to recommend these two articles together?
  27. 27. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! - Food Random (Most healthy) Trattner, Christoph, and David Elsweiler. "Investigating the healthiness of internet- sourced recipes: implications for meal planning and recommender systems." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017
  28. 28. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! - Food “Users in general tend to interact most often with the least healthy recipes. Recommender algorithms tend to score popular items highly and thus on average promote unhealthy items.” Trattner, Christoph, and David Elsweiler. "Investigating the healthiness of internet- sourced recipes: implications for meal planning and recommender systems." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017
  29. 29. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! – Cold Start A new fashion cloth A new online course A new job post Recommendations without user feedback
  30. 30. Deep Content Modeling for Recommendations (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization
  31. 31. Beyond Matrix Implicit feedback Deep Content Modeling Beyond “black-box items” Beyond Accuracy Diversity and Fairness Rich Context and Modality Learning preference from auxiliary channels
  32. 32. Are ratings/clicks/views enough for recommendations? Context matters! – Music Recommendation Schedl, Markus, et al. "Music recommender systems." Recommender Systems Handbook. Springer US, 2015. 453-492. Schedl, Markus, Peter Knees, and Fabien Gouyon. "New Paths in Music Recommender Systems Research." Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 2017. time location weather Environmental Context Individual Context emotion activity social context Schedl, Markus, Georg Breitschopf, and Bogdan Ionescu. "Mobile Music Genius: Reggae at the Beach, Metal on a Friday Night?." Proceedings of International Conference on Multimedia Retrieval. ACM, 2014.
  33. 33. Recommendations are not always “a list”: Rich modality Sun, Yu, et al. "Contextual intent tracking for personal assistants." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016. Kang, Jie, et al. "Understanding How People Use Natural Language to Ask for Recommendations." Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 2017.
  34. 34. Rich Context and Modality (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization (C/R/Res/Adversarial/Rei nforcement) NN (C/R/Res/Adversarial/Rei nforcement) NN
  35. 35. Beyond Matrix Implicit feedback Deep Content Modeling Beyond “black-box items” Beyond Accuracy Diversity and Fairness Rich Context and Modality Learning preference from auxiliary channels
  36. 36. Accuracy Popularity & Similarity
  37. 37. Diversity – Filtering bubble and echo chamber
  38. 38. Fairness – Long tail and Minority # views (attention) popular unpopular
  39. 39. Fairness – Long tail and Minority Recommender system better worse Yao, Sirui, and Bert Huang. "Beyond Parity: Fairness Objectives for Collaborative Filtering." arXiv preprint arXiv:1705.08804 (2017).
  40. 40. Incorporating diversity and fairness into recommendations (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization (C/R/Res/Adversarial/Rei nforcement) NN (C/R/Res/Adversarial/Rei nforcement) NN Penalize homogeneous and unfair recommendations
  41. 41. Immersive Recommendation
  42. 42. Immersive Recommendation Deep understanding of users’ diverse digital traces Deep modeling of heterogeneous contents + = News Events Food Art Spoken word
  43. 43. Immersive Recommendation Deep understanding of users’ diverse digital traces Deep modeling of heterogeneous contents + = News Events Food Art Spoken word
  44. 44. Yum-me Bringing healthiness into the recommendation of food Yang, Longqi, et al. "Yum-Me: A Personalized Nutrient-Based Meal Recommender System." ACM Transactions on Information Systems (TOIS) 36.1 (2017): 7.
  45. 45. *Number of Americans Living with Diet-and Inactivity-Related Diseases Obesity HBP Diabetes 113M 50M 15M Critical Issue of Food
  46. 46. The problem is not awareness, but adherence How can we (efficiently) find meals that are healthy but also cater to people’s tastes? - Bringing the notion of healthiness into recommendations!
  47. 47. Yum-me: An interactive healthy meal recommendation system Take a look at the food below and tap all that look delicious to you. http:// http:// Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste 2iters + 13iters 2iters + 13iters 2iters + 13iters Browser Mobile Wearable Personal Dietary Profile (Food Preferences) … … Healthy meal recommendations based on dietary restrictions Re-ranking Personalized healthy meal recommendations …... …... Phase I Phase II Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste http:// Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian⌾ Vegan ⌾ Kosher ⌾ Halal Identify your health goals. ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase Calories Protein Fat + Survey Choose the closest diet type to you. Identify your health goals. ⌾Reduce ⌾ Maintain Calories ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal + + Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal
  48. 48. Yum-me: An interactive healthy meal recommendation system Take a look at the food below and tap all that look delicious to you. http:// http:// Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste 2iters + 13iters 2iters + 13iters 2iters + 13iters Browser Mobile Wearable Personal Dietary Profile (Food Preferences) … … Healthy meal recommendations based on dietary restrictions Re-ranking Personalized healthy meal recommendations …... …... Phase I Phase II Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste http:// Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian⌾ Vegan ⌾ Kosher ⌾ Halal Identify your health goals. ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase Calories Protein Fat + Survey Choose the closest diet type to you. Identify your health goals. ⌾Reduce ⌾ Maintain Calories ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal + + Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal
  49. 49. Interactive Learning Process propagate
  50. 50. Interactive Learning Process selection
  51. 51. Similarity between Food Items 0.9151 0.6471 0.9652 1.3484 1.3410 1.3484 1.1476 Siamese Network
  52. 52. Similarity between Food Items (a) No restrictions (b) Vegetarian
  53. 53. Yum-me: An interactive healthy meal recommendation system Take a look at the food below and tap all that look delicious to you. http:// http:// Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste 2iters + 13iters 2iters + 13iters 2iters + 13iters Browser Mobile Wearable Personal Dietary Profile (Food Preferences) … … Healthy meal recommendations based on dietary restrictions Re-ranking Personalized healthy meal recommendations …... …... Phase I Phase II Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste http:// Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian⌾ Vegan ⌾ Kosher ⌾ Halal Identify your health goals. ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase Calories Protein Fat + Survey Choose the closest diet type to you. Identify your health goals. ⌾Reduce ⌾ Maintain Calories ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal + + Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal
  54. 54. Re-ranking healthy unhealthy Re-ranked by an user’s preference Top N recommendations
  55. 55. User study Step 1. Users identify their diet types and health goals. Step 2. Users use visual interace to express their fine-grained food preferences. Step 3. Users identify each of recommended meals as either Yummy or No way. (The order of the items is randomized) Top 500 healthy items that meet users’ diet types and health goals. Select top 10 items ranked by user’s fine-grained dietary preference. Randomly select 10 food items from 500 healthy meal pool. …... …... …...…...
  56. 56. Acceptance Rate 51% 72.5% Traditional Approach Yum-me
  57. 57. User study Goal: reduce calories (25 users) Goal: maintain calories (21 users) Goal: maintain protein (36 users) Goal: increase protein (12 users) Goal: reduce fat (17 users) Goal: increase calories (2 users) Goal: maintain fat (30 users) users’ 20 favorite meals meals recommended by Yum-me and accepted by users. Averageamountofnutrients perserving(normalized) Averageamountofnutrients perserving(normalized) Averageamountofnutrients perserving(normalized)
  58. 58. Creative Content Recommendation Bringing unstructured command traces into the recommendation of art Yang, Longqi, et al. "Personalizing Software and Web Services by Integrating Unstructured Application Usage Traces." Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 2017.
  59. 59. Cold-start creative content recommendation Day to day work activities (Commands performed in professional design software)
  60. 60. … FC FC FC FC Two-step recommendation algorithm
  61. 61. utilization-to-vector (util2vec) action action action action action Representation Learning Paradigm
  62. 62. utilization-to-vector (util2vec) - sliding window sliding window
  63. 63. utilization-to-vector (util2vec) - sliding window sliding window
  64. 64. utilization-to-vector (util2vec) - inside each window prediction target 2n+1 actions (n=4) predictor inputs
  65. 65. utilization-to-vector (util2vec) - inside each window prediction target 2n+1 actions (n=4) predictor inputs
  66. 66. Concatenation/Average Softmax Predictor utilization-to-vector (util2vec) - predictor
  67. 67. Evaluation
  68. 68. OpenRec An open source and modular framework for extensible recommendation algorithms
  69. 69. Recap (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization (C/R/Res/Adversarial/Rei nforcement) NN (C/R/Res/Adversarial/Rei nforcement) NN
  70. 70. Real World Systems can only be more complex User Item Interaction Optimization … … … … … … User Item Interaction User Item Interaction
  71. 71. A Traditional (Monolithic) View of Recommender Systems … RS 1 RS 2 RS 3 RS n
  72. 72. OpenRec: Experimentation and innovation through Extension rating item text item textitem image rating user text item image item text rating rating user text user demogr item text item image integrator module extractor module interaction module R1 R2 R4R3
  73. 73. OpenRec: Architecture Module Extractor IntegratorInteraction BPR WARP PMF CML NeuMF … LF ResNet MLP LSTM FoodDist Concatenation Average Weighted sum …… Recommender News recommender system with users’ click history, Twitter posts and news topic modeling. Music recommender system with users’ listen history, lyrics and audio analysis … Utility Sampler Pairwise sampler Triplet sampler … Evaluator … AUC Recall@K
  74. 74. OpenRec: Examples input buff input holders user extractors item extractors interactions optimizerVisualCML Modules VBPR input buff input holders user extractors item extractors interactions optimizer reusable modules reusable functions build from scratch CML Interactionmodule … input buff input holders user extractors item extractors interactions optimizerVanillaBPR Modules reusable modules build from scratch LatentFactor BPR Extractor module Interactionmodule … …
  75. 75. Stay tuned, Late Fall: https://github.com/ylongqi/OpenRec OpenRec
  76. 76. Thank you! Longqi Yang Ph.D. Student, Computer Science, Cornell Tech, Cornell University Email: ylongqi@cs.cornell.edu Web: bit.ly/longqi Twitter: @ylongqi Connected Experiences Lab: http://cx.jacobs.cornell.edu/ Small Data Lab: http://smalldata.io/

×