SlideShare a Scribd company logo
1 of 42
Download to read offline
| 0
Daniel Kershaw (@danjamker)
Building Recommenders
20th September 2017
| 1
Mendeley
• Reference Manager
• Social Network
• Publication Catalogue
| 2
Science Direct
• Scientific publication database
• Used by the majority of
university and research
institutions
• Contains 12 million articles of
content from 3,500 academic
journals and 34,000 e-books
| 3
Why Recommendations
Pull
Allow users to discover more content
Make it easier to navigate catalogue
| 4
Why Recommendations
Pull
Allow users to discover more content
Make it easier to navigate catalogue
Push
Highlight new content to users
Bring users back to service
| 5
The five core components
Data Collection
Recommender Model
Recommendation
Post Processing
Online
Modules
User Interface
| 6
Outline
Developed Algorithms – keeping it simple
Practical Considerations – don’t look stupid
Implementation – how to scale a system
Evaluation – what is good enough
Evolution – what’s changed over time
Future Direction – the future’s bright the future’s is deep
| 7
Developed Algorithms
| 8
Available Data
Implicit
User libraries (Mendeley)
User article interactions (Science Direct)
Content
Abstracts
Titles
References
| 9
Content Based
Similarity between what users
have read
Similarity in references
Collaborative Collaborative
Matrix Factorization
KNN
LDA
Potential Methods
| 10
User item interaction matrix
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 11
Similarity between query users and other readers
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 12
Similarity between all users
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 13
Generating recommendations for user
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 14
• Ability to scale
• Matrix incredibly sparse
Why not Matrix Factorization
| 15
Practical Considerations
| 16
Explore/Exploit (Dithering)
Recommendations generated in batch
Users want an interactive experience
Slight shuffles give the impression of
freshness
Allow for the exploration of the list if only
a proportion shown
𝑠𝑐𝑜𝑟𝑒 𝑑𝑖𝑡ℎ𝑒𝑟𝑒𝑑 = log 𝑟𝑎𝑛𝑘 + 𝑁 0, log 𝜖
where 𝜀 =
∆ 𝑟𝑎𝑛𝑘
𝑟𝑎𝑛𝑘
and tipically 𝜀 ∈ [1.5,2]
| 17
Impression Discounting
• Experience deteriorates if exposed to the same information
• Push recommendations seen before down the list
Rank
Impressions
| 18
Impression Discounting
• Experience deteriorates if exposed to the same information
• Push recommendations seen before down the list
𝑠𝑐𝑜𝑟𝑒 𝑛𝑒𝑤 = scoreoriginal ∗ (w1 ∗ g impCount + w2 ∗ g lastSeen )
See Lee, P. et. al
| 19
Business Logic (Pre and Post Filtering)
Don’t show items they already have (bought, added, consumed)
Don’t feed the recommender positive feedback from recommender
Don’t recommend out of stock items
• A bad recommender has a cost
- Can be greater than not receiving a recommendation
| 20
Implementation
| 21
Systems Architecture
Impression
Discounting
API
Front End
AWS
Dithering
Candidate Selection
Content
Based
Item2Item
CF
Online
Offline
Logs
| 22
The unbundled mess
| 23
System
• Which run generated the
recommendation
• What was served to the user
• How was the score modified
• What was removed from the
recommendations
User (Feedback loop)
• What was displayed
• What was clicked
• When were they served
• Where the recommendations
displayed
Logging
Used for both debugging and feeding information to recommender
| 24
Evolutions
| 25
• User to Item CF
• Impression Discounting
Mendeley – Desktop Application
| 26
Mendeley – Online
• Implicit – serves
recommendations based on
user libraries
• Recent Activity – based off
recent additions to a users
library
• Research Interests - based on
user generated tags
• Discipline – based on their
self identified discipline
Most Personalized
Least Personalized
See Hristakeva, M et. Al (2017)
| 27
• Remove carousels
• Focus on implicit
recommendations
• Fall back to content based
solution
Mendeley – Online
| 28
• Recommendation based of the
complete library of the user
• Don’t send the same
recommendations twice
Mendeley - Email
| 29
• Item to Item
• Take user reading history
• Get recommendations for each
item
• Interleave recommendations
• Don’t send same
recommendations twice
Science Direct - Email
| 30
Science Direct – Article Page
Item to Item
Dither
recommendations
every 30 minutes
| 31
Evaluation
| 32
Off-line Methodology
Train model Query
Ground
truth
Time, user interactions
Test
| 33
Off-line evaluation - Mendeley
From Hristakeva, M et. al
| 34
Science Direct – Item-to-item
| 35
• Infrastructure takes a long time
to build
• Need feedback from users to
learn
1. Generate recommendations
off-line
2. Send to users via email (A/A)
3. Modify method based on
feedback
4. Send second set of users split
into A/B buckets
Static Recommendations for quick learnings
Email to users
Modify
Recommender
Email to users
| 36
Future Direction
| 37
Learning to rank (LtR)
Currently only using implicit feedback
No content used
Use CF as candidate selection
Re-rank results based on learnt model
optimised for CtR
Use item and user features
| 38
Deep Learning
Use to learn more complex features
Use as features in LtR
Build on the existing framework developed
Use pre-trained models before developing own
| 39
Conclusion (Take Homes)
• Log EVERYTHING
• Start Simple
• Iterate quickly
• Get recommendations out quickly to learn
• Don’t look stupid
• CTR ≇ Off-line Evaluation
| 40
www.elsevier.com/rd-solutions
Thank you,
Book chapter being written based on the content in this presentation
| 41
References
Hristakeva, M., Kershaw, D., Rossetti, M., Knoth, P., Pettit, B., Vargas, S., & Jack, K. (2017). Building
recommender systems for scholarly information. the 1st Workshop (pp. 25–32). New York, New York,
USA: ACM. http://doi.org/10.1145/3057148.3057152
Rossetti, M., Stella, F., & Zanker, M. (2016). Contrasting Offline and Online Results when Evaluating
Recommendation Algorithms (pp. 31–34). Presented at the Proceedings of the 10th ACM Conference
on Recommender Systems, New York, NY, USA: ACM. http://doi.org/10.1145/2959100.2959176
Lee, P., Lakshmanan, L. V. S., Tiwari, M., & Shah, S. (2014). Modeling impression discounting in
large-scale recommender systems (pp. 1837–1846). Presented at the Proceedings of the ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, New York,
USA: ACM Press. http://doi.org/10.1145/2623330.2623356
Koren, Y. (2010). Collaborative filtering with temporal dynamics. Communications of the ACM, 53(4),
89–97. http://doi.org/10.1145/1721654.1721677

More Related Content

Similar to Building Recommender Systems - Mendeley and Science Direct

Modern Perspectives on Recommender Systems and their Applications in Mendeley
Modern Perspectives on Recommender Systems and their Applications in MendeleyModern Perspectives on Recommender Systems and their Applications in Mendeley
Modern Perspectives on Recommender Systems and their Applications in MendeleyKris Jack
 
Metaphor: A system for related searches recommendations
Metaphor: A system for related searches recommendationsMetaphor: A system for related searches recommendations
Metaphor: A system for related searches recommendationsMitul Tiwari
 
Use of data science in recommendation system
Use of data science in  recommendation systemUse of data science in  recommendation system
Use of data science in recommendation systemAkashPatil334
 
Exploring Generative Models of Tripartite Graphs for Recommendation in Social...
Exploring Generative Models of Tripartite Graphs for Recommendation in Social...Exploring Generative Models of Tripartite Graphs for Recommendation in Social...
Exploring Generative Models of Tripartite Graphs for Recommendation in Social...Charalampos Chelmis
 
Beyond Collaborative Filtering: Learning to Rank Research Articles
Beyond Collaborative Filtering: Learning to Rank Research ArticlesBeyond Collaborative Filtering: Learning to Rank Research Articles
Beyond Collaborative Filtering: Learning to Rank Research ArticlesMaya Hristakeva
 
WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...
WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...
WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...Dominik Kowald
 
Tag based recommender system
Tag based recommender systemTag based recommender system
Tag based recommender systemKaren Li
 
Structure, Personalization, Scale: A Deep Dive into LinkedIn Search
Structure, Personalization, Scale: A Deep Dive into LinkedIn SearchStructure, Personalization, Scale: A Deep Dive into LinkedIn Search
Structure, Personalization, Scale: A Deep Dive into LinkedIn SearchC4Media
 
Recommender Systems @ Scale - PyData 2019
Recommender Systems @ Scale - PyData 2019Recommender Systems @ Scale - PyData 2019
Recommender Systems @ Scale - PyData 2019Sonya Liberman
 
A task-based scientific paper recommender system for literature review and ma...
A task-based scientific paper recommender system for literature review and ma...A task-based scientific paper recommender system for literature review and ma...
A task-based scientific paper recommender system for literature review and ma...Aravind Sesagiri Raamkumar
 
Who's Afraid of Qualitative Analysis?
Who's Afraid of Qualitative Analysis?Who's Afraid of Qualitative Analysis?
Who's Afraid of Qualitative Analysis?BrigitteScott
 
Using Social Network Analysis to Assess Organizational Development Initiatives
Using Social Network Analysis to Assess Organizational Development InitiativesUsing Social Network Analysis to Assess Organizational Development Initiatives
Using Social Network Analysis to Assess Organizational Development InitiativesStephanie Richter
 
Recommender systems for E-commerce
Recommender systems for E-commerceRecommender systems for E-commerce
Recommender systems for E-commerceAlexander Konduforov
 
NLM Update by Dianne Babski, 18th June 2019
NLM Update by Dianne Babski, 18th June 2019NLM Update by Dianne Babski, 18th June 2019
NLM Update by Dianne Babski, 18th June 2019EAHILPHIG
 
productionising-recommenders
productionising-recommendersproductionising-recommenders
productionising-recommendersLudovik Coba
 
What Students Want: Redesigning Research Guides Based on Student Needs
What Students Want: Redesigning Research Guides Based on Student NeedsWhat Students Want: Redesigning Research Guides Based on Student Needs
What Students Want: Redesigning Research Guides Based on Student NeedsAmy Gratz Barker
 
How Does the USA Today Network Provide Its Readers With Meaningful Content? -...
How Does the USA Today Network Provide Its Readers With Meaningful Content? -...How Does the USA Today Network Provide Its Readers With Meaningful Content? -...
How Does the USA Today Network Provide Its Readers With Meaningful Content? -...Lucidworks
 
Utilizing additional information in factorization methods (research overview,...
Utilizing additional information in factorization methods (research overview,...Utilizing additional information in factorization methods (research overview,...
Utilizing additional information in factorization methods (research overview,...Balázs Hidasi
 

Similar to Building Recommender Systems - Mendeley and Science Direct (20)

Modern Perspectives on Recommender Systems and their Applications in Mendeley
Modern Perspectives on Recommender Systems and their Applications in MendeleyModern Perspectives on Recommender Systems and their Applications in Mendeley
Modern Perspectives on Recommender Systems and their Applications in Mendeley
 
Metaphor: A system for related searches recommendations
Metaphor: A system for related searches recommendationsMetaphor: A system for related searches recommendations
Metaphor: A system for related searches recommendations
 
Use of data science in recommendation system
Use of data science in  recommendation systemUse of data science in  recommendation system
Use of data science in recommendation system
 
Exploring Generative Models of Tripartite Graphs for Recommendation in Social...
Exploring Generative Models of Tripartite Graphs for Recommendation in Social...Exploring Generative Models of Tripartite Graphs for Recommendation in Social...
Exploring Generative Models of Tripartite Graphs for Recommendation in Social...
 
Beyond Collaborative Filtering: Learning to Rank Research Articles
Beyond Collaborative Filtering: Learning to Rank Research ArticlesBeyond Collaborative Filtering: Learning to Rank Research Articles
Beyond Collaborative Filtering: Learning to Rank Research Articles
 
WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...
WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...
WWW'15: A Hybrid Resource Recommender Mimicking Attention-Interpretation Dyna...
 
Tag based recommender system
Tag based recommender systemTag based recommender system
Tag based recommender system
 
lms final ppt.pptx
lms final ppt.pptxlms final ppt.pptx
lms final ppt.pptx
 
Structure, Personalization, Scale: A Deep Dive into LinkedIn Search
Structure, Personalization, Scale: A Deep Dive into LinkedIn SearchStructure, Personalization, Scale: A Deep Dive into LinkedIn Search
Structure, Personalization, Scale: A Deep Dive into LinkedIn Search
 
Recommender Systems @ Scale - PyData 2019
Recommender Systems @ Scale - PyData 2019Recommender Systems @ Scale - PyData 2019
Recommender Systems @ Scale - PyData 2019
 
A task-based scientific paper recommender system for literature review and ma...
A task-based scientific paper recommender system for literature review and ma...A task-based scientific paper recommender system for literature review and ma...
A task-based scientific paper recommender system for literature review and ma...
 
Who's Afraid of Qualitative Analysis?
Who's Afraid of Qualitative Analysis?Who's Afraid of Qualitative Analysis?
Who's Afraid of Qualitative Analysis?
 
Using Social Network Analysis to Assess Organizational Development Initiatives
Using Social Network Analysis to Assess Organizational Development InitiativesUsing Social Network Analysis to Assess Organizational Development Initiatives
Using Social Network Analysis to Assess Organizational Development Initiatives
 
Recommender systems for E-commerce
Recommender systems for E-commerceRecommender systems for E-commerce
Recommender systems for E-commerce
 
NLM Update by Dianne Babski, 18th June 2019
NLM Update by Dianne Babski, 18th June 2019NLM Update by Dianne Babski, 18th June 2019
NLM Update by Dianne Babski, 18th June 2019
 
productionising-recommenders
productionising-recommendersproductionising-recommenders
productionising-recommenders
 
What Students Want: Redesigning Research Guides Based on Student Needs
What Students Want: Redesigning Research Guides Based on Student NeedsWhat Students Want: Redesigning Research Guides Based on Student Needs
What Students Want: Redesigning Research Guides Based on Student Needs
 
How Does the USA Today Network Provide Its Readers With Meaningful Content? -...
How Does the USA Today Network Provide Its Readers With Meaningful Content? -...How Does the USA Today Network Provide Its Readers With Meaningful Content? -...
How Does the USA Today Network Provide Its Readers With Meaningful Content? -...
 
Utilizing additional information in factorization methods (research overview,...
Utilizing additional information in factorization methods (research overview,...Utilizing additional information in factorization methods (research overview,...
Utilizing additional information in factorization methods (research overview,...
 
L1-intro(2).pptx
L1-intro(2).pptxL1-intro(2).pptx
L1-intro(2).pptx
 

Recently uploaded

UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 

Recently uploaded (20)

UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 

Building Recommender Systems - Mendeley and Science Direct

  • 1. | 0 Daniel Kershaw (@danjamker) Building Recommenders 20th September 2017
  • 2. | 1 Mendeley • Reference Manager • Social Network • Publication Catalogue
  • 3. | 2 Science Direct • Scientific publication database • Used by the majority of university and research institutions • Contains 12 million articles of content from 3,500 academic journals and 34,000 e-books
  • 4. | 3 Why Recommendations Pull Allow users to discover more content Make it easier to navigate catalogue
  • 5. | 4 Why Recommendations Pull Allow users to discover more content Make it easier to navigate catalogue Push Highlight new content to users Bring users back to service
  • 6. | 5 The five core components Data Collection Recommender Model Recommendation Post Processing Online Modules User Interface
  • 7. | 6 Outline Developed Algorithms – keeping it simple Practical Considerations – don’t look stupid Implementation – how to scale a system Evaluation – what is good enough Evolution – what’s changed over time Future Direction – the future’s bright the future’s is deep
  • 9. | 8 Available Data Implicit User libraries (Mendeley) User article interactions (Science Direct) Content Abstracts Titles References
  • 10. | 9 Content Based Similarity between what users have read Similarity in references Collaborative Collaborative Matrix Factorization KNN LDA Potential Methods
  • 11. | 10 User item interaction matrix User base CF – (kNN) https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
  • 12. | 11 Similarity between query users and other readers User base CF – (kNN) https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
  • 13. | 12 Similarity between all users User base CF – (kNN) https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
  • 14. | 13 Generating recommendations for user User base CF – (kNN) https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
  • 15. | 14 • Ability to scale • Matrix incredibly sparse Why not Matrix Factorization
  • 17. | 16 Explore/Exploit (Dithering) Recommendations generated in batch Users want an interactive experience Slight shuffles give the impression of freshness Allow for the exploration of the list if only a proportion shown 𝑠𝑐𝑜𝑟𝑒 𝑑𝑖𝑡ℎ𝑒𝑟𝑒𝑑 = log 𝑟𝑎𝑛𝑘 + 𝑁 0, log 𝜖 where 𝜀 = ∆ 𝑟𝑎𝑛𝑘 𝑟𝑎𝑛𝑘 and tipically 𝜀 ∈ [1.5,2]
  • 18. | 17 Impression Discounting • Experience deteriorates if exposed to the same information • Push recommendations seen before down the list Rank Impressions
  • 19. | 18 Impression Discounting • Experience deteriorates if exposed to the same information • Push recommendations seen before down the list 𝑠𝑐𝑜𝑟𝑒 𝑛𝑒𝑤 = scoreoriginal ∗ (w1 ∗ g impCount + w2 ∗ g lastSeen ) See Lee, P. et. al
  • 20. | 19 Business Logic (Pre and Post Filtering) Don’t show items they already have (bought, added, consumed) Don’t feed the recommender positive feedback from recommender Don’t recommend out of stock items • A bad recommender has a cost - Can be greater than not receiving a recommendation
  • 22. | 21 Systems Architecture Impression Discounting API Front End AWS Dithering Candidate Selection Content Based Item2Item CF Online Offline Logs
  • 24. | 23 System • Which run generated the recommendation • What was served to the user • How was the score modified • What was removed from the recommendations User (Feedback loop) • What was displayed • What was clicked • When were they served • Where the recommendations displayed Logging Used for both debugging and feeding information to recommender
  • 26. | 25 • User to Item CF • Impression Discounting Mendeley – Desktop Application
  • 27. | 26 Mendeley – Online • Implicit – serves recommendations based on user libraries • Recent Activity – based off recent additions to a users library • Research Interests - based on user generated tags • Discipline – based on their self identified discipline Most Personalized Least Personalized See Hristakeva, M et. Al (2017)
  • 28. | 27 • Remove carousels • Focus on implicit recommendations • Fall back to content based solution Mendeley – Online
  • 29. | 28 • Recommendation based of the complete library of the user • Don’t send the same recommendations twice Mendeley - Email
  • 30. | 29 • Item to Item • Take user reading history • Get recommendations for each item • Interleave recommendations • Don’t send same recommendations twice Science Direct - Email
  • 31. | 30 Science Direct – Article Page Item to Item Dither recommendations every 30 minutes
  • 33. | 32 Off-line Methodology Train model Query Ground truth Time, user interactions Test
  • 34. | 33 Off-line evaluation - Mendeley From Hristakeva, M et. al
  • 35. | 34 Science Direct – Item-to-item
  • 36. | 35 • Infrastructure takes a long time to build • Need feedback from users to learn 1. Generate recommendations off-line 2. Send to users via email (A/A) 3. Modify method based on feedback 4. Send second set of users split into A/B buckets Static Recommendations for quick learnings Email to users Modify Recommender Email to users
  • 38. | 37 Learning to rank (LtR) Currently only using implicit feedback No content used Use CF as candidate selection Re-rank results based on learnt model optimised for CtR Use item and user features
  • 39. | 38 Deep Learning Use to learn more complex features Use as features in LtR Build on the existing framework developed Use pre-trained models before developing own
  • 40. | 39 Conclusion (Take Homes) • Log EVERYTHING • Start Simple • Iterate quickly • Get recommendations out quickly to learn • Don’t look stupid • CTR ≇ Off-line Evaluation
  • 41. | 40 www.elsevier.com/rd-solutions Thank you, Book chapter being written based on the content in this presentation
  • 42. | 41 References Hristakeva, M., Kershaw, D., Rossetti, M., Knoth, P., Pettit, B., Vargas, S., & Jack, K. (2017). Building recommender systems for scholarly information. the 1st Workshop (pp. 25–32). New York, New York, USA: ACM. http://doi.org/10.1145/3057148.3057152 Rossetti, M., Stella, F., & Zanker, M. (2016). Contrasting Offline and Online Results when Evaluating Recommendation Algorithms (pp. 31–34). Presented at the Proceedings of the 10th ACM Conference on Recommender Systems, New York, NY, USA: ACM. http://doi.org/10.1145/2959100.2959176 Lee, P., Lakshmanan, L. V. S., Tiwari, M., & Shah, S. (2014). Modeling impression discounting in large-scale recommender systems (pp. 1837–1846). Presented at the Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, New York, USA: ACM Press. http://doi.org/10.1145/2623330.2623356 Koren, Y. (2010). Collaborative filtering with temporal dynamics. Communications of the ACM, 53(4), 89–97. http://doi.org/10.1145/1721654.1721677