The Search for the Best Live
Recommender System
Torben Brodt
plista GmbH
Keynote
SIGIR Conference 2013, Dublin
BARS Worksh...
recommendations
where
● news websites
● below the article
different types
● content
● advertising
quality is win win
● happy user
● happy advertiser
● happy publisher
● happy plista*
* company i am working
some years ago
one recommender
● collaborative
filtering
○ well known algorithm
○ more data means
more knowledge
● parameter tuning
○ tim...
one recommender = good result
2008
● finished studies
● publication
● plista was born
today
● 5k recs/second
● many publis...
netflix prize
" use as many
recommenders as
possible! "
more recommenders
lost in serendipity
● we have one score
● lucky success? bad
loose?
● we needed to keep
track on different
recommenders
su...
how to measure success
number of
● clicks
● orders
● engages
● time on site
● money
BAD
GOOD
evaluation technology
● features
○ SUM
○ INCR
● big data (!!)
● real time
● in memory
evaluation technology
impressions
collaborative filtering 500 +1
most popular 500
text similarity 500
ZINCRBY
"impressions...
evaluation technology
impressions
collaborative filtering 500
most popular 500
text similarity 500
clicks
collaborative fi...
evaluation results
● CF is "always" the best
recommender
● but "always" is just avg
of all context
lets check on context!
...
evaluation context
● our context is limited to the web
● we have URL + HTTP Headers
○ user agent -> device
○ IP address ->...
evaluation context
we use ~60 context attributes
publisher = welt.de
collaborative filtering 689 +1
most popular 420
text ...
evaluation context
publisher = welt.de
collaborative filterin 689
most popular 420
text similarity 135
weekday = sunday
co...
evaluation context
recap
● added 3rd dimension
result
● better for news:
Collaborative Filtering
● better for content: Tex...
now breathe!
what did we get?
● possibly many recommenders
● know how to measure success
● technology to see success
now breathe!
what is the link to the workshop?
“.. novel, personalization-centric benchmarking
approaches to evaluate adap...
the ensemble
● realtime evaluation
technology exists
● to choose best
algorithm for
current context we
need to learn
○ mul...
multi armed bandit
temporary
success?
No. 1 getting most
local minima?
Interested? Look for Ted Dunning + Bayesian Bandit
the ensemble = better results
● new total / avg is
much better
● thx bandit
● thx ensemble
t = time
s = success
try and error
● minimum pre-
testing
● no risk if
recommender
crashs
● "bad" code might
find its context
collaboration
● now plista
developers can try
ideas
● and allow
researchers to do
same
big pool of algorithms
Ensemble is able to choose
researcher has idea
.. needs to start the server
... probably hosted by
university, plista or
any cloud provider?
.. api implementation
"message bus"
● event notifications
○ impression
○ click
● error notifications
● item updates
train ...
{ // json
"type": "impression",
"context": {
"simple": {
"27": 418, // publisher
"14": 31721, // widget
...
},
"lists": {
...
.. reply to recommendation requests
{ // json
"recs": {
"int": {
"3": [13010630, 84799192]
// 3 refers to content
recommen...
quality is win win #2
● happy user
● happy researcher
● happy plista
research can profit
● real user feedback
● real bench...
quick and fast
● no movies!
● news articles will outdate!
● visitors need the recs NOW
● => handle the data very fast
srch...
"send quickly" technologies
● fast web server
● fast network protocol
● fast message queue
● fast storage
or Apache Kafka
"learn quickly" technologies
● use common
frameworks
src http://en.wikipedia.org/wiki/Pac-Man
comparison to plista
"real-time features feel better in a
real-time world"
we don't need batch! see http://goo.gl/AJntul
o...
Overview
Questions?
Torben
http://goo.gl/pvXm5 (Blog)
torben.brodt@plista.com
http://lnkd.in/MUXXuv
xing.com/profile/Torben_Brodt
w...
Upcoming SlideShare
Loading in …5
×

SIGIR 2013 BARS Keynote - the search for the best live recommender system

1,181 views

Published on

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,181
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
15
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

SIGIR 2013 BARS Keynote - the search for the best live recommender system

  1. 1. The Search for the Best Live Recommender System Torben Brodt plista GmbH Keynote SIGIR Conference 2013, Dublin BARS Workshop - Benchmarking Adaptive Retrieval and Recommender Systems August 1st, 2013
  2. 2. recommendations where ● news websites ● below the article different types ● content ● advertising
  3. 3. quality is win win ● happy user ● happy advertiser ● happy publisher ● happy plista* * company i am working
  4. 4. some years ago
  5. 5. one recommender ● collaborative filtering ○ well known algorithm ○ more data means more knowledge ● parameter tuning ○ time ○ trust ○ mainstream
  6. 6. one recommender = good result 2008 ● finished studies ● publication ● plista was born today ● 5k recs/second ● many publishers
  7. 7. netflix prize " use as many recommenders as possible! "
  8. 8. more recommenders
  9. 9. lost in serendipity ● we have one score ● lucky success? bad loose? ● we needed to keep track on different recommenders success: 0.31 %
  10. 10. how to measure success number of ● clicks ● orders ● engages ● time on site ● money BAD GOOD
  11. 11. evaluation technology ● features ○ SUM ○ INCR ● big data (!!) ● real time ● in memory
  12. 12. evaluation technology impressions collaborative filtering 500 +1 most popular 500 text similarity 500 ZINCRBY "impressions" "collaborative_filtering" "1" ZREVRANGEBYSCORE "impressions"
  13. 13. evaluation technology impressions collaborative filtering 500 most popular 500 text similarity 500 clicks collaborative filtering 100 most popular 10 ... 1 needs division ZREVRANGEBYSCORE "clicks" ZREVRANGEBYSCORE "impressions"
  14. 14. evaluation results ● CF is "always" the best recommender ● but "always" is just avg of all context lets check on context! t = time s = success
  15. 15. evaluation context ● our context is limited to the web ● we have URL + HTTP Headers ○ user agent -> device ○ IP address -> geolocation ○ time -> weekday
  16. 16. evaluation context we use ~60 context attributes publisher = welt.de collaborative filtering 689 +1 most popular 420 text similarity 135 weekday = sunday collaborative filtering 400 +1 most popular 200 ... 100 category = archive text similarity 200 collaborative filtering 10 +1 ... 5
  17. 17. evaluation context publisher = welt.de collaborative filterin 689 most popular 420 text similarity 135 weekday = sunday collaborative filtering 400 most popular 200 ... 100 category = archive text similarity 200 collaborative filtering 10 ... 5 ZUNION clk ... WEIGHTS p:welt.de:clk 4 w:sunday:clk 1 c:archive:clk 1 ZREVRANGEBYSCORE "clk" ZUNION imp ... WEIGHTS p:welt.de:imp 4 w:sunday:imp 1 c:archive:imp 1 ZREVRANGEBYSCORE "imp"
  18. 18. evaluation context recap ● added 3rd dimension result ● better for news: Collaborative Filtering ● better for content: Text Similarity t = time s = success c = context
  19. 19. now breathe! what did we get? ● possibly many recommenders ● know how to measure success ● technology to see success
  20. 20. now breathe! what is the link to the workshop? “.. novel, personalization-centric benchmarking approaches to evaluate adaptive retrieval and recommender systems” ● Functional: focus on user-centered utility metrics ● Non-functional: scalability and reactivity
  21. 21. the ensemble ● realtime evaluation technology exists ● to choose best algorithm for current context we need to learn ○ multi-armed bayesian bandit
  22. 22. multi armed bandit temporary success? No. 1 getting most local minima? Interested? Look for Ted Dunning + Bayesian Bandit
  23. 23. the ensemble = better results ● new total / avg is much better ● thx bandit ● thx ensemble t = time s = success
  24. 24. try and error ● minimum pre- testing ● no risk if recommender crashs ● "bad" code might find its context
  25. 25. collaboration ● now plista developers can try ideas ● and allow researchers to do same
  26. 26. big pool of algorithms Ensemble is able to choose
  27. 27. researcher has idea
  28. 28. .. needs to start the server ... probably hosted by university, plista or any cloud provider?
  29. 29. .. api implementation "message bus" ● event notifications ○ impression ○ click ● error notifications ● item updates train model from it plista API API research
  30. 30. { // json "type": "impression", "context": { "simple": { "27": 418, // publisher "14": 31721, // widget ... }, "lists": { "10": [100, 101] // channel } ... } .. package content api specs hosted at https://sites.google. com/site/newsrec2013/ long term URL to be announced plista API API research Context + Kind
  31. 31. .. reply to recommendation requests { // json "recs": { "int": { "3": [13010630, 84799192] // 3 refers to content recommendations } ... } generated by researchers to be shown to real user api specs hosted at https://sites.google. com/site/newsrec2013/ long term URL to be announced recs API real user researcher
  32. 32. quality is win win #2 ● happy user ● happy researcher ● happy plista research can profit ● real user feedback ● real benchmark recs plista real user researcher
  33. 33. quick and fast ● no movies! ● news articles will outdate! ● visitors need the recs NOW ● => handle the data very fast srchttp://en.wikipedia.org/wiki/Flash_(comics)
  34. 34. "send quickly" technologies ● fast web server ● fast network protocol ● fast message queue ● fast storage or Apache Kafka
  35. 35. "learn quickly" technologies ● use common frameworks src http://en.wikipedia.org/wiki/Pac-Man
  36. 36. comparison to plista "real-time features feel better in a real-time world" we don't need batch! see http://goo.gl/AJntul our setup ● php, its easy ● redis, its fast ● r, its well known
  37. 37. Overview
  38. 38. Questions? Torben http://goo.gl/pvXm5 (Blog) torben.brodt@plista.com http://lnkd.in/MUXXuv xing.com/profile/Torben_Brodt www.plista.com News Recommender Challenge https://sites.google.com/site/newsrec2013/ #sigir2013 #bars2013 @torbenbrodt @plista @BARSws

×