The document discusses the application of recommender systems at Quora, highlighting various challenges and approaches to recommend content like answers, writers, and topics based on user engagement and preferences. It details the use of machine learning models, such as logistic regression and neural networks, to personalize recommendations while addressing issues like user feedback and optimization trade-offs. The conclusion emphasizes the need for algorithms to understand complex aspects like relevance and user expertise, alongside a notification of job openings at Quora.

1. Recommending the World’s Knowledge
Application of Recommender Systems at Quora
Lei Yang (leiyang@quora.com)

2. Quora: Share and grow the world’s knowledge
Millions of questions & answers, millions of users, thousands of topics

3. Data and Scale

4. Topic Question
User
Answer
Actions

5. Lots of data relations

6. Complex network propagation effects

7. Complex topics network & semantics

8. Recommendation
Problems

9. Recommending the digest email
Recommend 10 most interesting answers in an
email to get users to re-engage on Quora
● Requirements
○ Relevance
○ Freshness
○ Quality
● Challenges
○ Many candidates
○ Personalized frequency and send times
● Personalized LTR model
○ Predict probability of click on answers
shown

10. Recommending the digest email
● Candidates
○ User/topic follows
○ Globally popular/viral content
○ Collaborative filtering suggestions
■ Relevant writers
■ Relevant topics
■ Relevant answers
● Features
○ Text features
○ User follow graph
○ Historic data
○ ...

11. Recommending writers for a question
Recommend experts who are best qualified
to answer a question
● Ask to Answer (A2A)
● Given a questions and a viewer, predict
how “well-suited” a candidate user is for an
A2A suggestion
● Well-suited
○ viewer send a request &
○ candidate write a good answer

12. Recommending writers for a question
● Candidates
○ Users who have written at least an
answer on one of the topics of the
question
○ A number of filters
● Features
○ Candidate history features
○ Viewer history features
○ Candidate-viewer relevance
○ ...

13. Recommending the home feed
Present most interesting stories for a user at
a given time
● Stories: questions + answers
● Relevance-ordered vs time-ordered = big
gains in engagement
● Challenges
○ Many candidate stories
○ Real-time ranking
○ Objective function

14. Recommending the home feed
● Candidates
○ Different temporal windows
○ Multi-stage solution with different
“streams”
● Features
○ Quality of question/answer
○ Topics the user is interested in
or knows about
○ Users the user is following
○ What is trending/popular
○ ...

15. Recommending topics and users
Recommend new topics and users for
the user to follow, based on
● Topics you already follow
● Users you already follow
● Interactions with
questions/answers/users
● Topic-related features
● Social graph
● …..

16. Many more problems…
Related Questions Sessions
Trending
Topic feed

17. What We’ve Learned

18. Learning to recommend
● ML can often solve recommendation
problems
○ Select candidates
○ ML model to score each candidate
○ Top scored candidates are returned
● Workflow
○ Build data pipeline
○ Define training label
○ Choose your model and metric
○ Create features

19. ML models used at Quora
● Logistic Regression
● Elastic Nets
● Gradient Boosted Decision Trees
● Random Forests
● (Deep) Neural Networks
● LambdaMART
● Matrix Factorization
● LDA
● ...

20. Open source project -- QMF
Quora Matrix Factorization
https://github.com/quora/qmf
● Currently BPR and WALS
● Multithreaded implementation in
C++14

21. Challenges
● Personalization
● Explicit v.s. implicit user feedback
● Explore v.s. exploit and the filter bubble
● Business logic in your system
● Trade-offs between different optimization goals
● Explain the recommendations
● ...

22. Conclusions

23. Conclusions
● At Quora we have not only big, but also “rich” data
● Many interesting recommendation problems and unsolved challenges
● We use ML to solve many of our recommendation problems
● Our algorithms need to understand and optimize complex aspects such
as quality, relevance, or user expertise

24. We are hiring!
www.quora.com/careers