SlideShare a Scribd company logo
The Search for the Best Live
Recommender System
Torben Brodt
plista GmbH
Keynote
SIGIR Conference 2013, Dublin
BARS Workshop - Benchmarking Adaptive
Retrieval and Recommender Systems
August 1st, 2013
recommendations
where
● news websites
● below the article
different types
● content
● advertising
quality is win win
● happy user
● happy advertiser
● happy publisher
● happy plista*
* company i am working
some years ago
one recommender
● collaborative
filtering
○ well known algorithm
○ more data means
more knowledge
● parameter tuning
○ time
○ trust
○ mainstream
one recommender = good result
2008
● finished studies
● publication
● plista was born
today
● 5k recs/second
● many publishers
netflix prize
" use as many
recommenders as
possible! "
more recommenders
lost in serendipity
● we have one score
● lucky success? bad
loose?
● we needed to keep
track on different
recommenders
success: 0.31 %
how to measure success
number of
● clicks
● orders
● engages
● time on site
● money
BAD
GOOD
evaluation technology
● features
○ SUM
○ INCR
● big data (!!)
● real time
● in memory
evaluation technology
impressions
collaborative filtering 500 +1
most popular 500
text similarity 500
ZINCRBY
"impressions"
"collaborative_filtering"
"1"
ZREVRANGEBYSCORE
"impressions"
evaluation technology
impressions
collaborative filtering 500
most popular 500
text similarity 500
clicks
collaborative filtering 100
most popular 10
... 1
needs division
ZREVRANGEBYSCORE
"clicks"
ZREVRANGEBYSCORE
"impressions"
evaluation results
● CF is "always" the best
recommender
● but "always" is just avg
of all context
lets check on context!
t = time
s = success
evaluation context
● our context is limited to the web
● we have URL + HTTP Headers
○ user agent -> device
○ IP address -> geolocation
○ time -> weekday
evaluation context
we use ~60 context attributes
publisher = welt.de
collaborative filtering 689 +1
most popular 420
text similarity 135
weekday = sunday
collaborative filtering 400 +1
most popular 200
... 100
category = archive
text similarity 200
collaborative filtering 10 +1
... 5
evaluation context
publisher = welt.de
collaborative filterin 689
most popular 420
text similarity 135
weekday = sunday
collaborative filtering 400
most popular 200
... 100
category = archive
text similarity 200
collaborative filtering 10
... 5
ZUNION clk ... WEIGHTS
p:welt.de:clk 4
w:sunday:clk 1
c:archive:clk 1
ZREVRANGEBYSCORE
"clk"
ZUNION imp ... WEIGHTS
p:welt.de:imp 4
w:sunday:imp 1
c:archive:imp 1
ZREVRANGEBYSCORE
"imp"
evaluation context
recap
● added 3rd dimension
result
● better for news:
Collaborative Filtering
● better for content: Text
Similarity
t = time
s = success
c = context
now breathe!
what did we get?
● possibly many recommenders
● know how to measure success
● technology to see success
now breathe!
what is the link to the workshop?
“.. novel, personalization-centric benchmarking
approaches to evaluate adaptive retrieval and
recommender systems”
● Functional: focus on user-centered
utility metrics
● Non-functional: scalability and
reactivity
the ensemble
● realtime evaluation
technology exists
● to choose best
algorithm for
current context we
need to learn
○ multi-armed
bayesian bandit
multi armed bandit
temporary
success?
No. 1 getting most
local minima?
Interested? Look for Ted Dunning + Bayesian Bandit
the ensemble = better results
● new total / avg is
much better
● thx bandit
● thx ensemble
t = time
s = success
try and error
● minimum pre-
testing
● no risk if
recommender
crashs
● "bad" code might
find its context
collaboration
● now plista
developers can try
ideas
● and allow
researchers to do
same
big pool of algorithms
Ensemble is able to choose
researcher has idea
.. needs to start the server
... probably hosted by
university, plista or
any cloud provider?
.. api implementation
"message bus"
● event notifications
○ impression
○ click
● error notifications
● item updates
train model from it
plista
API
API
research
{ // json
"type": "impression",
"context": {
"simple": {
"27": 418, // publisher
"14": 31721, // widget
...
},
"lists": {
"10": [100, 101] // channel
}
...
}
.. package content
api specs hosted at https://sites.google.
com/site/newsrec2013/
long term URL to be announced
plista
API
API
research
Context
+ Kind
.. reply to recommendation requests
{ // json
"recs": {
"int": {
"3": [13010630, 84799192]
// 3 refers to content
recommendations
}
...
}
generated by researchers
to be shown to real user
api specs hosted at https://sites.google.
com/site/newsrec2013/
long term URL to be announced
recs
API
real user
researcher
quality is win win #2
● happy user
● happy researcher
● happy plista
research can profit
● real user feedback
● real benchmark
recs
plista
real user
researcher
quick and fast
● no movies!
● news articles will outdate!
● visitors need the recs NOW
● => handle the data very fast
srchttp://en.wikipedia.org/wiki/Flash_(comics)
"send quickly" technologies
● fast web server
● fast network protocol
● fast message queue
● fast storage
or Apache Kafka
"learn quickly" technologies
● use common
frameworks
src http://en.wikipedia.org/wiki/Pac-Man
comparison to plista
"real-time features feel better in a
real-time world"
we don't need batch! see http://goo.gl/AJntul
our setup
● php, its easy
● redis, its fast
● r, its well known
Overview
Questions?
Torben
http://goo.gl/pvXm5 (Blog)
torben.brodt@plista.com
http://lnkd.in/MUXXuv
xing.com/profile/Torben_Brodt
www.plista.com
News Recommender Challenge
https://sites.google.com/site/newsrec2013/
#sigir2013 #bars2013
@torbenbrodt @plista @BARSws

More Related Content

Similar to SIGIR 2013 BARS Keynote - the search for the best live recommender system

Living Labs Challenge Workshop
Living Labs Challenge WorkshopLiving Labs Challenge Workshop
Living Labs Challenge Workshop
Torben Brodt
 
Pinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at PinterestPinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at Pinterest
Alluxio, Inc.
 
The right path to making search relevant - Taxonomy Bootcamp London 2019
The right path to making search relevant  - Taxonomy Bootcamp London 2019The right path to making search relevant  - Taxonomy Bootcamp London 2019
The right path to making search relevant - Taxonomy Bootcamp London 2019
OpenSource Connections
 
Recommender Hackathon @plista 2013/04
Recommender Hackathon @plista 2013/04Recommender Hackathon @plista 2013/04
Recommender Hackathon @plista 2013/04
Torben Brodt
 
Partner Webinar: Recommendation Engines with MongoDB and Hadoop
 Partner Webinar: Recommendation Engines with MongoDB and Hadoop Partner Webinar: Recommendation Engines with MongoDB and Hadoop
Partner Webinar: Recommendation Engines with MongoDB and Hadoop
MongoDB
 
Google Developers Overview Deck 2015
Google Developers Overview Deck 2015Google Developers Overview Deck 2015
Google Developers Overview Deck 2015
Houssem Eddine LASSOUED
 
Tweak Geeks #FOS15
Tweak Geeks #FOS15Tweak Geeks #FOS15
Tweak Geeks #FOS15
Pascal Fantou
 
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks FusionWebinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
Lucidworks
 
I Know It Was MEAN, But I Cut the Cord to LAMP Anyway
I Know It Was MEAN, But I Cut the Cord to LAMP AnywayI Know It Was MEAN, But I Cut the Cord to LAMP Anyway
I Know It Was MEAN, But I Cut the Cord to LAMP Anyway
POSSCON
 
Making the Most of Customer Data
Making the Most of Customer DataMaking the Most of Customer Data
Making the Most of Customer Data
WSO2
 
Logs & Visualizations at Twitter
Logs & Visualizations at TwitterLogs & Visualizations at Twitter
Logs & Visualizations at Twitter
Krist Wongsuphasawat
 
Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...
Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...
Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...
Krist Wongsuphasawat
 
Are API Services Taking Over All the Interesting Data Science Problems?
Are API Services Taking Over All the Interesting Data Science Problems?Are API Services Taking Over All the Interesting Data Science Problems?
Are API Services Taking Over All the Interesting Data Science Problems?
IDEAS - Int'l Data Engineering and Science Association
 
Opticon18: Developer Night
Opticon18: Developer NightOpticon18: Developer Night
Opticon18: Developer Night
Optimizely
 
GDayX - Advanced Angular.JS
GDayX - Advanced Angular.JSGDayX - Advanced Angular.JS
GDayX - Advanced Angular.JS
Nicolas Embleton
 
Node in Production at Aviary
Node in Production at AviaryNode in Production at Aviary
Node in Production at Aviary
Aviary
 
MCOE Masterclass - Creating Helpful Content.pdf
MCOE Masterclass - Creating Helpful Content.pdfMCOE Masterclass - Creating Helpful Content.pdf
MCOE Masterclass - Creating Helpful Content.pdf
Lane Houk
 
Introduction to Google Cloud platform technologies
Introduction to Google Cloud platform technologiesIntroduction to Google Cloud platform technologies
Introduction to Google Cloud platform technologies
Chris Schalk
 
Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...
Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...
Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...
Spark Summit
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature Delivery
Optimizely
 

Similar to SIGIR 2013 BARS Keynote - the search for the best live recommender system (20)

Living Labs Challenge Workshop
Living Labs Challenge WorkshopLiving Labs Challenge Workshop
Living Labs Challenge Workshop
 
Pinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at PinterestPinterest - Big Data Machine Learning Platform at Pinterest
Pinterest - Big Data Machine Learning Platform at Pinterest
 
The right path to making search relevant - Taxonomy Bootcamp London 2019
The right path to making search relevant  - Taxonomy Bootcamp London 2019The right path to making search relevant  - Taxonomy Bootcamp London 2019
The right path to making search relevant - Taxonomy Bootcamp London 2019
 
Recommender Hackathon @plista 2013/04
Recommender Hackathon @plista 2013/04Recommender Hackathon @plista 2013/04
Recommender Hackathon @plista 2013/04
 
Partner Webinar: Recommendation Engines with MongoDB and Hadoop
 Partner Webinar: Recommendation Engines with MongoDB and Hadoop Partner Webinar: Recommendation Engines with MongoDB and Hadoop
Partner Webinar: Recommendation Engines with MongoDB and Hadoop
 
Google Developers Overview Deck 2015
Google Developers Overview Deck 2015Google Developers Overview Deck 2015
Google Developers Overview Deck 2015
 
Tweak Geeks #FOS15
Tweak Geeks #FOS15Tweak Geeks #FOS15
Tweak Geeks #FOS15
 
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks FusionWebinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
 
I Know It Was MEAN, But I Cut the Cord to LAMP Anyway
I Know It Was MEAN, But I Cut the Cord to LAMP AnywayI Know It Was MEAN, But I Cut the Cord to LAMP Anyway
I Know It Was MEAN, But I Cut the Cord to LAMP Anyway
 
Making the Most of Customer Data
Making the Most of Customer DataMaking the Most of Customer Data
Making the Most of Customer Data
 
Logs & Visualizations at Twitter
Logs & Visualizations at TwitterLogs & Visualizations at Twitter
Logs & Visualizations at Twitter
 
Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...
Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...
Using Visualizations to Monitor Changes and Harvest Insights from a Global-sc...
 
Are API Services Taking Over All the Interesting Data Science Problems?
Are API Services Taking Over All the Interesting Data Science Problems?Are API Services Taking Over All the Interesting Data Science Problems?
Are API Services Taking Over All the Interesting Data Science Problems?
 
Opticon18: Developer Night
Opticon18: Developer NightOpticon18: Developer Night
Opticon18: Developer Night
 
GDayX - Advanced Angular.JS
GDayX - Advanced Angular.JSGDayX - Advanced Angular.JS
GDayX - Advanced Angular.JS
 
Node in Production at Aviary
Node in Production at AviaryNode in Production at Aviary
Node in Production at Aviary
 
MCOE Masterclass - Creating Helpful Content.pdf
MCOE Masterclass - Creating Helpful Content.pdfMCOE Masterclass - Creating Helpful Content.pdf
MCOE Masterclass - Creating Helpful Content.pdf
 
Introduction to Google Cloud platform technologies
Introduction to Google Cloud platform technologiesIntroduction to Google Cloud platform technologies
Introduction to Google Cloud platform technologies
 
Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...
Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...
Digital Attribution Modeling Using Apache Spark-(Anny Chen and William Yan, A...
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature Delivery
 

More from Torben Brodt

Recommender Trends 2014
Recommender Trends 2014Recommender Trends 2014
Recommender Trends 2014
Torben Brodt
 
Paper the plista dataset
Paper  the plista datasetPaper  the plista dataset
Paper the plista dataset
Torben Brodt
 
Algorithmus, Good School, Camp Digital
Algorithmus, Good School, Camp DigitalAlgorithmus, Good School, Camp Digital
Algorithmus, Good School, Camp Digital
Torben Brodt
 
Realtime Recommender with Redis: Hands on
Realtime Recommender with Redis: Hands onRealtime Recommender with Redis: Hands on
Realtime Recommender with Redis: Hands on
Torben Brodt
 
RecSys2012 inside the plista contest
RecSys2012   inside the plista contestRecSys2012   inside the plista contest
RecSys2012 inside the plista contest
Torben Brodt
 
Webhacks am Beispiel PHP + MySQL
Webhacks am Beispiel PHP + MySQLWebhacks am Beispiel PHP + MySQL
Webhacks am Beispiel PHP + MySQL
Torben Brodt
 
GIT / SVN
GIT / SVNGIT / SVN
GIT / SVN
Torben Brodt
 
Collaborative Filtering.. für automatische Empfehlungen
Collaborative Filtering.. für automatische EmpfehlungenCollaborative Filtering.. für automatische Empfehlungen
Collaborative Filtering.. für automatische Empfehlungen
Torben Brodt
 
Google Web Toolkit
Google Web ToolkitGoogle Web Toolkit
Google Web Toolkit
Torben Brodt
 
Geld Verdienen Mit Adsense
Geld Verdienen Mit AdsenseGeld Verdienen Mit Adsense
Geld Verdienen Mit Adsense
Torben Brodt
 
AJAX
AJAXAJAX
Web 2.0 - "Fluch oder Segen"
Web 2.0 - "Fluch oder Segen"Web 2.0 - "Fluch oder Segen"
Web 2.0 - "Fluch oder Segen"
Torben Brodt
 

More from Torben Brodt (12)

Recommender Trends 2014
Recommender Trends 2014Recommender Trends 2014
Recommender Trends 2014
 
Paper the plista dataset
Paper  the plista datasetPaper  the plista dataset
Paper the plista dataset
 
Algorithmus, Good School, Camp Digital
Algorithmus, Good School, Camp DigitalAlgorithmus, Good School, Camp Digital
Algorithmus, Good School, Camp Digital
 
Realtime Recommender with Redis: Hands on
Realtime Recommender with Redis: Hands onRealtime Recommender with Redis: Hands on
Realtime Recommender with Redis: Hands on
 
RecSys2012 inside the plista contest
RecSys2012   inside the plista contestRecSys2012   inside the plista contest
RecSys2012 inside the plista contest
 
Webhacks am Beispiel PHP + MySQL
Webhacks am Beispiel PHP + MySQLWebhacks am Beispiel PHP + MySQL
Webhacks am Beispiel PHP + MySQL
 
GIT / SVN
GIT / SVNGIT / SVN
GIT / SVN
 
Collaborative Filtering.. für automatische Empfehlungen
Collaborative Filtering.. für automatische EmpfehlungenCollaborative Filtering.. für automatische Empfehlungen
Collaborative Filtering.. für automatische Empfehlungen
 
Google Web Toolkit
Google Web ToolkitGoogle Web Toolkit
Google Web Toolkit
 
Geld Verdienen Mit Adsense
Geld Verdienen Mit AdsenseGeld Verdienen Mit Adsense
Geld Verdienen Mit Adsense
 
AJAX
AJAXAJAX
AJAX
 
Web 2.0 - "Fluch oder Segen"
Web 2.0 - "Fluch oder Segen"Web 2.0 - "Fluch oder Segen"
Web 2.0 - "Fluch oder Segen"
 

Recently uploaded

Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
CEPTES Software Inc
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
moinahousna
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
LINUS PROJECTS (INDIA)
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 

Recently uploaded (20)

Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 

SIGIR 2013 BARS Keynote - the search for the best live recommender system

  • 1. The Search for the Best Live Recommender System Torben Brodt plista GmbH Keynote SIGIR Conference 2013, Dublin BARS Workshop - Benchmarking Adaptive Retrieval and Recommender Systems August 1st, 2013
  • 2. recommendations where ● news websites ● below the article different types ● content ● advertising
  • 3. quality is win win ● happy user ● happy advertiser ● happy publisher ● happy plista* * company i am working
  • 5. one recommender ● collaborative filtering ○ well known algorithm ○ more data means more knowledge ● parameter tuning ○ time ○ trust ○ mainstream
  • 6. one recommender = good result 2008 ● finished studies ● publication ● plista was born today ● 5k recs/second ● many publishers
  • 7. netflix prize " use as many recommenders as possible! "
  • 9. lost in serendipity ● we have one score ● lucky success? bad loose? ● we needed to keep track on different recommenders success: 0.31 %
  • 10. how to measure success number of ● clicks ● orders ● engages ● time on site ● money BAD GOOD
  • 11. evaluation technology ● features ○ SUM ○ INCR ● big data (!!) ● real time ● in memory
  • 12. evaluation technology impressions collaborative filtering 500 +1 most popular 500 text similarity 500 ZINCRBY "impressions" "collaborative_filtering" "1" ZREVRANGEBYSCORE "impressions"
  • 13. evaluation technology impressions collaborative filtering 500 most popular 500 text similarity 500 clicks collaborative filtering 100 most popular 10 ... 1 needs division ZREVRANGEBYSCORE "clicks" ZREVRANGEBYSCORE "impressions"
  • 14. evaluation results ● CF is "always" the best recommender ● but "always" is just avg of all context lets check on context! t = time s = success
  • 15. evaluation context ● our context is limited to the web ● we have URL + HTTP Headers ○ user agent -> device ○ IP address -> geolocation ○ time -> weekday
  • 16. evaluation context we use ~60 context attributes publisher = welt.de collaborative filtering 689 +1 most popular 420 text similarity 135 weekday = sunday collaborative filtering 400 +1 most popular 200 ... 100 category = archive text similarity 200 collaborative filtering 10 +1 ... 5
  • 17. evaluation context publisher = welt.de collaborative filterin 689 most popular 420 text similarity 135 weekday = sunday collaborative filtering 400 most popular 200 ... 100 category = archive text similarity 200 collaborative filtering 10 ... 5 ZUNION clk ... WEIGHTS p:welt.de:clk 4 w:sunday:clk 1 c:archive:clk 1 ZREVRANGEBYSCORE "clk" ZUNION imp ... WEIGHTS p:welt.de:imp 4 w:sunday:imp 1 c:archive:imp 1 ZREVRANGEBYSCORE "imp"
  • 18. evaluation context recap ● added 3rd dimension result ● better for news: Collaborative Filtering ● better for content: Text Similarity t = time s = success c = context
  • 19. now breathe! what did we get? ● possibly many recommenders ● know how to measure success ● technology to see success
  • 20. now breathe! what is the link to the workshop? “.. novel, personalization-centric benchmarking approaches to evaluate adaptive retrieval and recommender systems” ● Functional: focus on user-centered utility metrics ● Non-functional: scalability and reactivity
  • 21. the ensemble ● realtime evaluation technology exists ● to choose best algorithm for current context we need to learn ○ multi-armed bayesian bandit
  • 22. multi armed bandit temporary success? No. 1 getting most local minima? Interested? Look for Ted Dunning + Bayesian Bandit
  • 23. the ensemble = better results ● new total / avg is much better ● thx bandit ● thx ensemble t = time s = success
  • 24. try and error ● minimum pre- testing ● no risk if recommender crashs ● "bad" code might find its context
  • 25. collaboration ● now plista developers can try ideas ● and allow researchers to do same
  • 26. big pool of algorithms Ensemble is able to choose
  • 28. .. needs to start the server ... probably hosted by university, plista or any cloud provider?
  • 29. .. api implementation "message bus" ● event notifications ○ impression ○ click ● error notifications ● item updates train model from it plista API API research
  • 30. { // json "type": "impression", "context": { "simple": { "27": 418, // publisher "14": 31721, // widget ... }, "lists": { "10": [100, 101] // channel } ... } .. package content api specs hosted at https://sites.google. com/site/newsrec2013/ long term URL to be announced plista API API research Context + Kind
  • 31. .. reply to recommendation requests { // json "recs": { "int": { "3": [13010630, 84799192] // 3 refers to content recommendations } ... } generated by researchers to be shown to real user api specs hosted at https://sites.google. com/site/newsrec2013/ long term URL to be announced recs API real user researcher
  • 32. quality is win win #2 ● happy user ● happy researcher ● happy plista research can profit ● real user feedback ● real benchmark recs plista real user researcher
  • 33. quick and fast ● no movies! ● news articles will outdate! ● visitors need the recs NOW ● => handle the data very fast srchttp://en.wikipedia.org/wiki/Flash_(comics)
  • 34. "send quickly" technologies ● fast web server ● fast network protocol ● fast message queue ● fast storage or Apache Kafka
  • 35. "learn quickly" technologies ● use common frameworks src http://en.wikipedia.org/wiki/Pac-Man
  • 36. comparison to plista "real-time features feel better in a real-time world" we don't need batch! see http://goo.gl/AJntul our setup ● php, its easy ● redis, its fast ● r, its well known
  • 38. Questions? Torben http://goo.gl/pvXm5 (Blog) torben.brodt@plista.com http://lnkd.in/MUXXuv xing.com/profile/Torben_Brodt www.plista.com News Recommender Challenge https://sites.google.com/site/newsrec2013/ #sigir2013 #bars2013 @torbenbrodt @plista @BARSws