Your SlideShare is downloading. ×
0
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems

317

Published on

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
317
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. © author(s) of these slides including research results from the KOM research network and TU Darmstadt; otherwise it is specified at the respective slide 21-Sep-13 Prof. Dr.-Ing. Ralf Steinmetz KOM - Multimedia Communications Lab Rensing_Crowdsourcing_ECTELmeetsECSCW__2013.pptx Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Systems Mojisola Erdt Florian Jomrich Katja Schüler Christoph Rensing Source: http://www.digitalvisitor.com/cultural-differences-in-online-behaviour-and-customer-reviews/
  • 2. KOM – Multimedia Communications Lab 2 Motivation & Context Workplace Learning  to solve a particular work related problem or to prepare for a new task  Informal: Community of practices & using resources from the Web or companies Intranet or created from the employees Train and practice necessary competences in vocational training Learning Application  Social tagging application  Help to structure resources  Help to retrieve resources  Reflection  … see Proceedings of EC-TEL 2011 (TEL) Recommender Systems  Recommend relevant resources, tags or users
  • 3. KOM – Multimedia Communications Lab 3 Quality Measures for Recommender Systems in TEL:  Relevance: “Relevance refers to the ability of a RS (Recommender System) to provide items that fit the user’s preferences” [Epifania 2012].  Novelty: “Novelty (or discovery) is the extent to which users receive new and interesting recommendations” [Pu 2011].  Diversity: “Diversity measures the diversity level of items in the recommendation list” [Pu 2011]. Quality Measures for a User Experiment
  • 4. KOM – Multimedia Communications Lab 4 Evaluation Approach Method Advantages Disadvantages Offline Evaluation  Cross-validation using historical datasets  Fast  Less effort  Repeatable  New, unknown resources cannot be evaluated  Cross-validation using synthetically generated data  For testing and analysing  Might be biased towards an algorithm  Artificial scenarios Online Evaluation User Experiments (Lab & Field)  User’s perspective  A lot of effort and time  Few users Real-life testing  Real-life setting  Needs a substantial amount of users Crowdsourcing  Fast  Less effort  Enough users  Spamming Evaluation Methods for TEL Recommender Systems
  • 5. KOM – Multimedia Communications Lab 6 Experiment Generate a basis graph structure (extended Folksonomy)  5 Experts research on the topic of climate change for one hour  Using CROKODIL to create an extended folksonomy  Ca. 70 resources were attached to 8 activities Generate Recommendations  Recommendations from AScore and FolkRank algorithms (see Proceedings of EC-TEL 2012) Three Hypotheses: 1. AScore gives more relevant resources than baseline (FolkRank) 2. AScore gives more novel resources than baseline (FolkRank) 3. AScore gives more diverse resources than baseline (FolkRank) Questionnaire is created  10 questions per recommendation  3 questions to each hypothesis  1 control question to detect spammers
  • 6. KOM – Multimedia Communications Lab 7 Crowdsouring Evaluation Concept Jobs on 2 crowdsourcing platforms  Microworkers.com  Active Users and good service quality  CrowdFlower.com  Gives access to other platforms (e.g. Amazon MTurk  a lot of crowdworkers)
  • 7. KOM – Multimedia Communications Lab 9 Overview of Participants in the Evaluation Jobs on 2 crowdsourcing platforms  Microworkers.com (Active Users and good service quality)  CrowdFlower.com (Gives access to other platforms (e.g. Amazon MTurk  a lot of crowdworkers)) 125 completed Questionnaires were evaluated
  • 8. KOM – Multimedia Communications Lab 11 Summary: Results Hyp. 1 Relevance Hyp. 2 Novelty Hyp. 3 Diversity p=0.0065 < 0.5 p=0.0042 < 0.5 p=0.677 > 0.5 supported supported not supported
  • 9. KOM – Multimedia Communications Lab 15 Conclusion & Future Work Crowdsourcing can be used as an Evaluation Method for TEL Further Analysis of Data collected  Comparing results between Crowdworkers and Non-Crowdworkers Improve Crowdsourcing Evaluation Concept  Apply a more complex evaluation  Apply to further scenarios
  • 10. KOM – Multimedia Communications Lab 16 Workshop related research questions How has a recommender to been designed to support resource-based knowledge acquisition at the workplace?  Relevance and novelty valid quality measures? How has a recommender to been designed to support resource-based Learning?  Which are the valid quality measures? Aggregation of relevance, novelty, diversity, … ?  How can the recommendation been adapted to the current need of a learner? What do we know from the learner  Learning Analytics How can we support teachers to train and practice self-regulated learning competences in vocational training?
  • 11. KOM – Multimedia Communications Lab 17 Questions & Contact
  • 12. KOM – Multimedia Communications Lab 18 Hypothesis 1: Relevance Q1. The given internet resource supports me very well in my research about the topic. Q2. If I could only use this resource, my research would still be very successful. Q3. Without this resource just by using my own resources, my research about the given topic would still be very good. Hypothesis 2: Novelty Q4. The internet resource gives me new insights and/ or information for my task. Q5. I would have found this resource on my own/ anyway/ during my research. Q6. There are lots of important aspects about the topic described in this resource that lack in other resources. Hypothesis 3: Diversity Q7. This internet resource differs strongly from my other resources. Q8. This resource informs me comprehensively about my topic. Q9. This resource covers the whole spectrum of research about the given topic. Questions to Hypothesis 1 – 3
  • 13. KOM – Multimedia Communications Lab 19 Control Questions in Questionnaire Q10a. How many pictures and tables that are relevant to the given research topic does the given resource contain? Q10b. Give a short summary of the recommended resource above by giving 4 keywords describing its content. Q10c. Describe the content of the given resource in two sentences. Control Questions
  • 14. KOM – Multimedia Communications Lab 20
  • 15. KOM – Multimedia Communications Lab 21
  • 16. KOM – Multimedia Communications Lab 22 Quality Measures for Recommender Systems in TEL:  Relevance: “Relevance refers to the ability of a RS (Recommender System) to provide items that fit the user’s preferences” [Epifania 2012].  Novelty: “Novelty (or discovery) is the extent to which users receive new and interesting recommendations” [Pu 2011].  Diversity: “Diversity measures the diversity level of items in the recommendation list” [Pu 2011]. Quality Measures for a User Experiment
  • 17. KOM – Multimedia Communications Lab 23 Crowdworkers: Age distribution

×