Recommender Systems in TEL Nikos Manouselis Greek Research & Technology Network (GRNET) nikosm@ieee.org

about me Computer Engineer MSc on Operational Research PhD from Informatics Lab of an Agricultural University working on services for agricultural & rural communities learning repositories social information retrieval Organic.Edunet e Content plus

Computer Engineer

MSc on Operational Research

PhD from Informatics Lab of an Agricultural University

working on services for agricultural & rural communities

learning repositories

social information retrieval

Organic.Edunet e Content plus

(promised)aim of this lecture introduce recommender systems discuss how they relate to TEL identify open research issues

introduce recommender systems

discuss how they relate to TEL

identify open research issues

(actual)aim of this lecture share some concerns about TEL and recommender systems

share some concerns about TEL and recommender systems

structure tale of 3 friends tasks modeling & techniques evaluation wrap up

tale of 3 friends

tasks

modeling & techniques

evaluation

wrap up

intro: tale of 3 friends

which movie?

lets ask some friend “ Guys, heard about the last Batman movie… should I watch it?” “ You will definitely like it” “ Maybe not, the scenario is too weak”

“ Guys, heard about the last Batman movie… should I watch it?”

lets ask some friend “ Wait – did you like the previous one?”

“ Wait – did you like the previous one?”

… so, which movie? taking advantage of knowledge or experience from people in the social circle or network e.g. colleagues, friends, peers need to answer several questions how to identify like-minded people? on which dimensions? for which types of items? does context matter? …

taking advantage of knowledge or experience from people in the social circle or network

e.g. colleagues, friends, peers

need to answer several questions

how to identify like-minded people?

on which dimensions?

for which types of items?

does context matter?

…

recommender systems

using the opinions of a community of users to help individuals in that community to identify more effectively content of interest from a potentially overwhelming set of choices Resnick P. & Varian H.R., “Recommender Systems”, Communications of the ACM, 40(3),1997 definition (1/2)

using the opinions of a community of users

to help individuals in that community to identify more effectively content of interest

from a potentially overwhelming set of choices

Resnick P. & Varian H.R., “Recommender Systems”, Communications of the ACM, 40(3),1997

definition (2/2) any system that produces individualized recommendations as output or has the effect of guiding the user in a personalized way to interesting or useful objects in a large space of possible options Burke R. “Hybrid Recommender Systems: Survey and Experiments”, User Modeling & User-Adapted Interaction, 12, 331-370, 2002

any system that

produces individualized recommendations as output

or has the effect of guiding the user in a personalized way to interesting or useful objects in a large space of possible options

Burke R. “Hybrid Recommender Systems: Survey and Experiments”, User Modeling & User-Adapted Interaction, 12, 331-370, 2002

why do we need them? A trip to a local supermarket [F. Ricci] : 85 different varieties and brands of crackers 285 varieties of cookies 165 varieties of “juice drinks” 75 iced teas 275 varieties of cereal 120 different pasta sauces 80 different pain relievers 40 options for toothpaste 95 varieties of snacks (chips, pretzels, etc.) 61 varieties of sun tan oil and sunblock 360 types of shampoo, conditioner, gel, and mousse. 90 different cold remedies and decongestants. 230 soups, including 29 different chicken soups 175 different salad dressings

A trip to a local supermarket [F. Ricci] :

85 different varieties and brands of crackers

285 varieties of cookies

165 varieties of “juice drinks”

75 iced teas

275 varieties of cereal

120 different pasta sauces

80 different pain relievers

40 options for toothpaste

95 varieties of snacks (chips, pretzels, etc.)

61 varieties of sun tan oil and sunblock

360 types of shampoo, conditioner, gel, and mousse.

90 different cold remedies and decongestants.

230 soups, including 29 different chicken soups

175 different salad dressings

wait a second is TEL like a super market??

is TEL like a super market??

large number of options

tasks for recommender systems

tasks usually supported annotation in context find good items find all good items receive sequence of items (+some less important ones) Herlocker et al., “Evaluating Collaborative Filtering Recommender Systems” ACM Transactions on Information Systems, 22(1), 5-53, 2004.

annotation in context

find good items

find all good items

receive sequence of items

(+some less important ones)

Herlocker et al., “Evaluating Collaborative Filtering Recommender Systems” ACM Transactions on Information Systems, 22(1), 5-53, 2004.

1. annotation in context integrated in existing working environment to provide additional support or information, e.g. predicted usefulness of an item that the user is currently viewing links within a Web page that the user is recommended to follow

integrated in existing working environment to provide additional support or information, e.g.

predicted usefulness of an item that the user is currently viewing

links within a Web page that the user is recommended to follow

annotation in context Screenshot/example

Screenshot/example

2. find good items suggesting specific item(s) to a user characterized as core recommendation task, since occurring in most systems e.g. presenting a ranked list of recommended items

suggesting specific item(s) to a user

characterized as core recommendation task, since occurring in most systems

e.g. presenting a ranked list of recommended items

find good items Screenshot/example

Screenshot/example

3. find all good items user wants to identify all items that might be interesting when its important not to overlook any potentially relevant case e.g. medical or legal cases

user wants to identify all items that might be interesting

when its important not to overlook any potentially relevant case

e.g. medical or legal cases

find all good items

4. sequence of items sequence of related items is recommended to the user e.g. entertainment applications such as TV or radio programs

sequence of related items is recommended to the user

e.g. entertainment applications such as TV or radio programs

sequence of items

and what about TEL? informal reminder: technology enhanced learning is generally dealing with the ways ICT can be used to support learning , teaching , and competence development [http://cordis.europa.eu/fp7/ict/telearn-digicult/telearn_en.html]

informal reminder:

technology enhanced learning is generally dealing with the ways ICT can be used to support learning , teaching , and competence development

[http://cordis.europa.eu/fp7/ict/telearn-digicult/telearn_en.html]

break2think bring yourself in one typical learning situation that occurs very often to YOU

bring yourself in one typical learning situation that occurs very often to YOU

break2think imagine that some magic TEL system is there to support you it could make some great suggestions about something to you name one learning task where a recommender system would be useful

imagine that some magic TEL system is there to support you

it could make some great suggestions about something to you

name one learning task where a recommender system would be useful

modeling & techniques

typical classification content-based: information needs of user and characteristics of items are represented in some (usually textual) form collaborative filtering: user is recommended items that people with similar tastes and preferences liked hybrid: methods that combine content-based and collaborative methods … other categorizations also exist (Burke, 2002)

content-based: information needs of user and characteristics of items are represented in some (usually textual) form

collaborative filtering: user is recommended items that people with similar tastes and preferences liked

hybrid: methods that combine content-based and collaborative methods

… other categorizations also exist (Burke, 2002)

example: content-based

example: collaborative filtering

generally speaking: some user has a profile with some user characteristics, e.g. past ratings [collaborative filtering] keywords describing past selections [content-based recommendation]

has a profile with some user characteristics, e.g.

past ratings [collaborative filtering]

keywords describing past selections [content-based recommendation]

generally speaking: some items are represented using some dimensions, e.g. satisfaction over one (or more) criteria [collaborative filtering] item attributes/features [content-based recommendation]

are represented using some dimensions, e.g.

satisfaction over one (or more) criteria [collaborative filtering]

item attributes/features [content-based recommendation]

generally speaking: a mechanism is taking advantage of the user profile and the item representations it provides personalised recommendations of items to users

is taking advantage of the user profile and the item representations

it provides personalised recommendations of items to users

rings some bell? for TEL, this sounds so… adaptive educational hypermedia systems ( AEHS )

for TEL, this sounds so…

adaptive educational hypermedia systems ( AEHS )

a generic architecture [Karampiperis & Sampson, 2005]

an example [Karampiperis & Sampson, 2005]

enhanced version of [Hanani et al., "Information Filtering: Overview of Issues, Research and Systems", User Modeling and User-Adapted Interaction, 11, 2001] classification/analysis

enhanced version of [Hanani et al., "Information Filtering: Overview of Issues, Research and Systems", User Modeling and User-Adapted Interaction, 11, 2001]

recommend in TEL based on what? on learner models/profiles e.g. learning styles, competence gaps … other ideas? on item characteristics e.g. interactivity, granularity, accessibility … other ideas?

on learner models/profiles

e.g. learning styles, competence gaps

… other ideas?

on item characteristics

e.g. interactivity, granularity, accessibility

… other ideas?

evaluation

evaluating recommendation currently based on performance “ how good are your algorithms?” e.g. how accurate are they in predictions? for how many unknown items can they produce a prediction? … mainly information retrieval evaluation approaches [Herlocker et al., “Evaluating Collaborative Filtering Recommender Systems” ACM Transactions on Information Systems, 22(1), 5-53, 2004]

currently based on performance

“ how good are your algorithms?”

e.g.

how accurate are they in predictions?

for how many unknown items can they produce a prediction?

… mainly information retrieval evaluation approaches

[Herlocker et al., “Evaluating Collaborative Filtering Recommender Systems” ACM Transactions on Information Systems, 22(1), 5-53, 2004]

typical results means that a prediction could be 4,6 stars instead of 4 or 5 … does this really matter in TEL?

other issues live experiments vs. offline analyses synthesized vs. natural data sets properties of data sets existing data sets

live experiments vs. offline analyses

synthesized vs. natural data sets

properties of data sets

existing data sets

metrics (popular) accuracy predictive accuracy (MAE) classification accuracy Precision and Recall probability that a selected item is relevant probability that a relevant item will be selected ad hoc Rank Accuracy Metrics Prediction-Rating Correlation coverage percentage of items for which prediction is possible

accuracy

predictive accuracy (MAE)

classification accuracy

Precision and Recall

probability that a selected item is relevant

probability that a relevant item will be selected

ad hoc

Rank Accuracy Metrics

Prediction-Rating Correlation

coverage

percentage of items for which prediction is possible

metrics (not popular) novelty serendipity confidence user evaluation explicit (ask) vs. implicit (observe) laboratory studies vs. field studies outcome vs. process short-term vs. long-term

novelty

serendipity

confidence

user evaluation

explicit (ask) vs. implicit (observe)

laboratory studies vs. field studies

outcome vs. process

short-term vs. long-term

evaluation in TEL recommenders few systems actually evaluated even fewer actually tried with users recent analysis of 15 TEL recommender systems: half of the systems (8/15) still at design or prototyping stage only 5 systems evaluated through trials with human users [N.Manouselis, H.Drachsler, R.Vuorikari, H.Hummel, R.Koper, “Recommender Systems in Technology Enhanced Learning”, Handbook of Recommender Systems (under review)]

few systems actually evaluated

even fewer actually tried with users

recent analysis of 15 TEL recommender systems:

half of the systems (8/15) still at design or prototyping stage

only 5 systems evaluated through trials with human users

[N.Manouselis, H.Drachsler, R.Vuorikari, H.Hummel, R.Koper, “Recommender Systems in Technology Enhanced Learning”, Handbook of Recommender Systems (under review)]

example: Altered Vista evaluate the effectiveness and usefulness system usability and performance predictive accuracy of recommender engine extent to which reviewing Web resources within a community of users supports and promotes collaborative and c ommunity-building activities extent to which critical review of Web resources leads to improvements in user’s information literacy skills [Walker et al., “ Collaborative Information Filtering: a review and an educational application”, International Journal of Artificial Intelligence in Education 14, 2004 ]

evaluate the effectiveness and usefulness

system usability and performance

predictive accuracy of recommender engine

extent to which reviewing Web resources within a community of users supports and promotes collaborative and c ommunity-building activities

extent to which critical review of Web resources leads to improvements in user’s information literacy skills

[Walker et al., “ Collaborative Information Filtering: a review and an educational application”, International Journal of Artificial Intelligence in Education 14, 2004 ]

another look at it e.g. using Kirckpatrick’s model on evaluating training programs reaction of student - what they thought and felt about the training learning - the resulting increase in knowledge or capability behaviour - extent of behaviour and capability improvement and implementation/application results - the effects on the business or environment resulting from the trainee's performance

e.g. using Kirckpatrick’s model on evaluating training programs

reaction of student - what they thought and felt about the training

learning - the resulting increase in knowledge or capability

behaviour - extent of behaviour and capability improvement and implementation/application

results - the effects on the business or environment resulting from the trainee's performance

what else could be evaluated? when deploying a recommender system in a TEL setting … what could we evaluate and how to measure it?

when deploying a recommender system in a TEL setting

… what could we evaluate and how to measure it?

wrap up & directions

basic conclusion assuming an information overload problem in TEL recommender systems are good need to think out of the box connect with existing research focus on TEL particularities explore alternative uses integrate with existing theories

assuming an information overload problem in TEL

recommender systems are good

need to think out of the box

connect with existing research

focus on TEL particularities

explore alternative uses

integrate with existing theories

interesting (?) issues recommendation of peers criteria for expressing learner satisfaction (no more 5-stars) study actual usage/acceptance assess performance/learning improvement … implement, deploy, pilot!

recommendation of peers

criteria for expressing learner satisfaction (no more 5-stars)

study actual usage/acceptance

assess performance/learning improvement

… implement, deploy, pilot!

but do they exist?? http://www.oerrecommender.org

http://www.oerrecommender.org

interested in more? Journal of Digital Information (JoDI) Special Issue on Social Information Retrieval for Technology-Enhanced Learning, 10(2), 2009 Workshop on Social Information Retrieval for Technology Enhanced Learning (SIRTEL) SIRTEL 2007 (http://ceur-ws.org/Vol-307) SIRTEL 2008 (http://ceur-ws.org/Vol-382) SIRTEL 2009 (http://celstec.org/sirtel) co-located with ICWL’09, Aachen, Germany, August 21 st - deadline: 12/6

Journal of Digital Information (JoDI)

Special Issue on Social Information Retrieval for Technology-Enhanced Learning, 10(2), 2009

Workshop on Social Information Retrieval for Technology Enhanced Learning (SIRTEL)

SIRTEL 2007 (http://ceur-ws.org/Vol-307)

SIRTEL 2008 (http://ceur-ws.org/Vol-382)

SIRTEL 2009 (http://celstec.org/sirtel)

co-located with ICWL’09, Aachen, Germany, August 21 st - deadline: 12/6

thank you! questions? ideas?