Beyond Accuracy: Goal-Driven Recommender Systems Design

  • 232 views
Uploaded on

Tamas Jambor @UCL talk at @ds_dln #strataconf 02/10/12

Tamas Jambor @UCL talk at @ds_dln #strataconf 02/10/12

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
232
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
7
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. BEYOND ACCURACY:GOAL-DRIVEN RECOMMENDERSYSTEM DESIGNTamas Jambor (@jamborta)Data Scientist at Big Data Partnership (@bigdataexperts)Ph.D. student at UCL
  • 2. Evaluating Recommender Systems• General error metrics • Mean absolute error (MAE) • Root mean squared error (RMSE)• General rank based metrics • Precision/Recall • Mean Reciprocal Rank (MRR)• Directly/indirectly optimised• What does improving these metrics mean?
  • 3. Goal driven vs Metric driven design• Metric driven (error or rank based) • Metric(s) is abstract and general (e.g. RMSE) • Objective of the algorithm is abstract • Competitions (e.g. Kaggle, Netflix) encourage this approach• Goal driven • Metric(s) depends on the goal • More focused algorithm design • Fitting goal to data (not model to data)
  • 4. Goal driven design• User/system perspective • Define and consider user satisfaction or system related objectives as priority• Internal/external goals • Consider algorithmic or non-algorithmic solutions• Time dependent goals • Identify various objectives and find the optimal solution
  • 5. User/System perspective• System perspective • Performance • Cost • Update time • Profit margin• User perspective • User satisfaction (how to measure?) • Diversity • Novelty • Serendipity • Context
  • 6. User vs System perspective System- User- General focused focused metric metric metric Collaborative system-user goals System- General focused metric metric User- focused metric Opposite system-user goals
  • 7. External/Internal goal• External goals • System as a black box • Plugin any algorithm • Post/pre-filtering or independent algorithmic solution • Easier to evaluate (modularised)• Internal goals • Goal is built in the algorithm • Goal is directly optimised • Difficult to evaluate different components
  • 8. Approach to goal-driven design Recommender system External goal External goal Internal goaldata data prediction prediction
  • 9. Time dependent goals• User side • Change in taste • Seasonal trends• System side • Cost over time (e.g. peak vs off-peak) • Demand
  • 10. Time dependent algorithms• Internal • Exponential decay (model temporal change) • Survival analysis • Time-series model• External • Time sensitive post filtering • Control theory• Online learning approaches
  • 11. Approach to goal-driven design User or External or Time Performance Goal(s) System Internal dependency Measures
  • 12. User perspective Static TemporalInternal goals Diversification/long tail items Optimal Control Theory (promote diverse items) (cold-start problem)External goals Nudging and Serendipity Balanced Control Theory (promote serendipitous items) (improve prediction per user)
  • 13. System perspective Static TemporalInternal goals Stock control Optimal Control Theory (promote items that are in stock) (estimate/maximise profit)External goals Optimised content delivery Balanced Control Theory (pre-cache liked items) (stabilise resource allocation)
  • 14. Example: Diversification/long tail itemsGoal: Promote diverse itemsChallenges: How to measure diversity?Scope: Goal is optimised within the algorithmAlgorithm: Matrix factorisation with convex optimisationEvaluation: Measure diverse items in top position
  • 15. Example: Stock availabilityGoal: Recommend items that are in stockChallenges: Up-to-date stock availability?Scope: Goal is optimised within the algorithmAlgorithm: Matrix factorisation with convex optimisationEvaluation: Measure waiting list for items
  • 16. Example: Improve prediction per userGoal: Request more data for difficult users Use only useful data to train modelChallenges: Define noise/signal for data pointsScope: Goal is optimised over time, for each user independentlyAlgorithm: Control theoryEvaluation: Performance measured per user basis
  • 17. Example: Resource allocationGoal: Improve resource allocation (e.g. CPU time) Maximise available resources (e.g. fixed cluster)Challenges: Define system dynamicsScope: Goal is optimised over timeAlgorithm: Control theoryEvaluation: Stability, divergence from reference
  • 18. Summary• Goal defines algorithm and performance measure• Multiple goals can be integrate into a single system• Modularity to evaluate goals separately (external)• Optimisation to merge goals (internal)