Ratings, tags, bookmarks andother species: some examples ofquantitative research oninformation filtering in TELSalvador Sá...
About this talk    Some context        about me, my group and my research    Research coordinates    Revision of succe...
About me    Remember Pecha-kucha? JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
About IE    Research lines    Projects    Doctorate studies    Publications    Journals and conferencesJTEL Summer sc...
About our PhD program    Not officially online but definitely not face-to-face!    High performance-oriented        No ...
Objectives of this talk    Find research opportunities in quantitative TEL     research    Learn from our experience   ...
Assumptions    You are familiar with        The concept of Learning object        The concept of metadata        Learn...
Research coordinatesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
The process    Develop a research question related to some     functionality and state a hypothesis (not formally yet)   ...
DataJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
TechniquesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
Social Network Analysis (SNA)    Social network: a graph made up of nodes (individuals,     organizations...) and edges r...
Collaborative filtering    Informally: a form of automating the process of "word-of-mouth"        But you would rather l...
Ontologies    Formal representation of knowledge    Concepts, relations and properties are represented in     an ontolog...
Statistical profiling    A set of techniques that allow to discover patterns or     correlations in large quantities of d...
CasesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
Assesing LO                                          reusability from                                           their meta...
Hypothesis     • It is possible to find an aprioristic reusability       evaluation based on LO metadata           • This ...
Functionalities     • Estimating reusability would provide useful information       when it comes to selecting reusable ob...
The process in a nutshell    “Polish” the hypothesis    Gather data from 2 repositories: MERLOT and eLERA    Find the m...
Method                                                            Cohesion                               Learning object  ...
Factors, metrics and metadata   elementsJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
Results         – Statistically significant           correlation between           estimated reusability and           co...
Publications    Sanz-Rodriguez, J., Dodero, J. and Sanchez-Alonso, S.     (2010). Metrics-based evaluation of learning ob...
Expanding queries                                 in LO repositories                                    with ontologies   ...
HypothesisEvaluación y resultados                           Query expansion can help repository users to find            ...
Procedure                          Query          Query                     Search                                        ...
Procedure    24 Official courses in genetics        In Universities and Higher education institutions        Syllabi pu...
Method    Experts in the field evaluated the results (3     experts each query).        Topical relevance        Expert...
Results    Rater agreement moderate according to     Cohen’s Kappa and Spearman correlation    COVERAGE: More than half ...
Results         0,90         0,80         0,70         0,60         0,50         0,40         0,30         0,20         0,...
ConclusionsConclusiones               Results of expanded queries are affected by:                 The quality of the on...
Publications    Segura, A., Sánchez-Alonso, S., Garcia-Barriocanal, E.     and Prieto, M. (2011). An empirical analysis o...
Exploring affiliation          network models as a          collaborative filtering       mechanism in e-learning         ...
Hypothesis    Social network analysis of relations in learning     environments will make it possible to re-configure…   ...
Method    Modeling and analysing learners participation in     activities organized around communication forums,     (ver...
Results    We identified two     different groups, one     interested in the     learning tools used     during the cours...
Results    33-slice cluster with     introductory topics (most     people are interested)    As long as course     progr...
Publications    Rodríguez, D., Sicilia, MA., Sánchez-Alonso, S., Lezcano,     L. and García-Barriocanal, E. (2009) Explor...
Automated quality                                    assessment of                                  Learning Objects      ...
Hypothesis Some intrinsic features of learning objects stored in  existing repositories may present significant  differen...
FunctionalityDeveloping models for automatically assessing quality of learning objects inside repositories based on the i...
Method Identification of intrinsic metrics of learning resources  that could serve as potential indicators of quality Da...
Metrics  Class of Measure                                        Metric    Link Measures           Number of Links, Number...
Method Learning objects were classified into three groups  (good, average, poor) according to their ratings. A Mann-Whit...
Preliminary ResultsThe two groups of ratings available on MERLOT (i.e. peer-reviewers and user comments ratings) differ s...
Preliminary Results We developed a Linear Discriminant Analysis (LDA) to  build models in order to distinguish 1.   good ...
Preliminary Results    The pursuit for an automated model for the quality     evaluation of learning objects must conside...
Publications    Cechinel, C., Sánchez-Alonso, S. and García-Barriocanal,     E. (2011). Statistical profiles of highly-ra...
Can 3D platforms                             improve training of                            trainers programs?            ...
Hypothesis    Using Massively Multiuser Online Learning     environments (MMOL) platforms in virtual courses can     impr...
Method    MMOL session using the     collaborative LORI approach        A group of users contribute their         indivi...
The experiments    A prototype of 3D MMOL platform was created in a     realXtend server with an interactive space called...
Results             •Case «A».                                 Data              •LCMS Experim.                           ...
Publications    Manuscript submitted to Computer & Education (April     2011). Still waiting…                            ...
Recommenders                  inside learning object                            repositories:                       requir...
Hypothesis   Implicit communities found via SNA    blockmodeling & component analysis    have a potential for recommendin...
Application   Implicit communities found via SNA    blockmodeling & component analysis    have a potential for recommendi...
Method       Evaluate parameters for Collaborative Filtering        Algorithms for two datasets from MERLOT         1. Re...
Method       Compare the results generated by the different        algorithms for the two datasets       Contrast the re...
Results       Very high Precision values (varying from 20% to        100%) and not so high Recall percentages (with a    ...
Future work         Involving users’ opinions in the process         Contrasting if recommendations for a given user    ...
Publications    Sánchez Alonso, S., Sicilia, MA., García, E., Pagés, C. and     Lezcano, L. (2011) Social models in open ...
Conclussions and                                      open reseach                                          directionsJTEL...
Lessons learned    Quantitative research is usually well received by     impact factor journals editorial boards    It i...
Lessons learned          LOR data should be shared!          Evangelize repository owners to share data for           re...
Open research opportunities          More in-depth study of social interaction in LCMS           (software ready for SNA ...
Your turn!    5 next minutes [individually]: think of an experiment     similar to those reported in my talk    15 minut...
To be written down    Salvador: salvador.sanchez@uah.es    Want to know more about our distance PhD program     @ IE-UAH...
Upcoming SlideShare
Loading in …5
×

Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL

1,139 views

Published on

In the last few years, the Information Engineering research unit @ University of Alcala has developed a productive research line on the use of these data to identify underlying knowledge such as social relations, user preferences or hints on the quality of the resources to create applications for recommendation, filtering an item or learner clustering, to name just a few. Successes and pitfalls of this research are illustrated from the past experiences of one member of the group.

Published in: Education, Technology
1 Comment
3 Likes
Statistics
Notes
  • A guided tour on what my research group (IE@UAH) achieved in the last 4 years, our methods and results, as well as a definition of open research opportunities for PhD students. This talk was given during the first day of the 7th JTEL Summer School that takes place from May 30th to Friday June 3rd, in Chania, Crete, Greece.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
1,139
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
19
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL

  1. 1. Ratings, tags, bookmarks andother species: some examples ofquantitative research oninformation filtering in TELSalvador Sánchez-Alonso
  2. 2. About this talk  Some context  about me, my group and my research  Research coordinates  Revision of successful cases  Conclussions and open research directions  Practical exerciseJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  3. 3. About me  Remember Pecha-kucha? JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  4. 4. About IE  Research lines  Projects  Doctorate studies  Publications  Journals and conferencesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  5. 5. About our PhD program  Not officially online but definitely not face-to-face!  High performance-oriented  No more “read this” or “have a look at…”  Lots of autonomous work but with REAL help/guidance  Procedure:  Presentation (including CV)  Finding a few ideas PhD candidate likes  Writing objectives  Usually avoiding conferences unless veeeeeery junior  Paper accepted in an impact factor journal  PhD finished.  2-3 years usually enoughJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  6. 6. Objectives of this talk  Find research opportunities in quantitative TEL research  Learn from our experience  See how TEL research can target high impact factor journalsJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  7. 7. Assumptions  You are familiar with  The concept of Learning object  The concept of metadata  Learning objects repositories  … and of course with IEEE LOM standard  You have (ideally) visited one or more learning object repositories (e.g. MERLOT, CNX, Organic.Edunet…)JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  8. 8. Research coordinatesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  9. 9. The process  Develop a research question related to some functionality and state a hypothesis (not formally yet)  Identify the data source  Build a software engine to collect the data  Find the more apropriate technique(s) to analyse the data and apply it on the dataset  Use statistics to assess if the hypothesis holdsJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  10. 10. DataJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  11. 11. JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  12. 12. JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  13. 13. TechniquesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  14. 14. Social Network Analysis (SNA)  Social network: a graph made up of nodes (individuals, organizations...) and edges representing relationships between nodes (friendship, work, membership...)  Social Network Analysis: a set of techniques to discover features of a network by means of its numerical or visual representation.  Find network measures such as betweenness and centrality  Most SNA software uses a plain text ASCII data format to represent the networkJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  15. 15. Collaborative filtering  Informally: a form of automating the process of "word-of-mouth"  But you would rather like to hear the opinions of those who have interests similar to your own!  Basic mechanism:  A large group of peoples preferences are registered  Using a similarity metric, a subgroup of people is selected whose preferences are similar to the preferences of the person who seeks advice;  A (possibly weighted) average of the preferences for that subgroup is calculated;  The resulting preference function is used to recommend options on which the advice-seeker has expressed no personal opinion as yet.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  16. 16. Ontologies  Formal representation of knowledge  Concepts, relations and properties are represented in an ontology language (eg OWL)  Ontologies can be used to  Enhance information retrieval  Power advanced services such as more accurate web search  Communicate between systems  Evaluate the correctness of a conceptual model  …JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  17. 17. Statistical profiling  A set of techniques that allow to discover patterns or correlations in large quantities of data  Helps in dealing with the increasing data overload, helping to discriminate information from noise  Metrics:  Precision: the fraction of correct instances among those that the algorithm believes to belong to the relevant subset  Recall: the fraction of correct instances among all instances that actually belong to the relevant subsetJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  18. 18. CasesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  19. 19. Assesing LO reusability from their metadata PhD. Javier Sanz Timeline: August 2008 -April 2010 Status: FinalJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  20. 20. Hypothesis • It is possible to find an aprioristic reusability evaluation based on LO metadata • This metric would span all the factors affecting the reusability of a learning object • It is possible to compute reusability in an automated wayJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  21. 21. Functionalities • Estimating reusability would provide useful information when it comes to selecting reusable objects • This measure of reusability might constitute an indicator of quality which would allow for search results to be ordered, with those which have greater possibilities of being reused taking priority.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  22. 22. The process in a nutshell  “Polish” the hypothesis  Gather data from 2 repositories: MERLOT and eLERA  Find the metadata elements having an impact on reusability  Create the metrics  Adjust them with empirical data  Assess their effectivenessJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  23. 23. Method Cohesion Learning object Reusability Size + Expert Metadata Technological Portability Agregation Educational Portability Repository > Aggregation methods: Weighted mean, Choquet’s integral, Multiple linear regression > Evaluation of the efficiency of the model: Average absolute error, Average relative error, Correlation between the real and the estimated value, Quality of the prediction > Expert questionnaire + LORIJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  24. 24. Factors, metrics and metadata elementsJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  25. 25. Results – Statistically significant correlation between estimated reusability and content quality evaluated by the experts. – Statistically significant correlation between estimated reusability and Correlation K endall’s Spearman’s Rho Tau effectiveness as a Content quality 0.228 0.307 learning tool and ease of Effectiveness 0.153 0.217 use. Ease of use 0.190 0.250JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  26. 26. Publications  Sanz-Rodriguez, J., Dodero, J. and Sanchez-Alonso, S. (2010). Metrics-based evaluation of learning object JCR reusability. Software Quality Journal 19(1), pp. 121-140.  Sanz-Rodríguez, Dodero, Sánchez-Alonso. (2010) Ranking Learning Objects through Integration of Different Quality Indicators, IEEE Transactions on Learning Technologies, 2008 3(4), pp. 358-363.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  27. 27. Expanding queries in LO repositories with ontologies PhD. Alejandra Segura Timeline: June 2009 - December 2010 Status: FinalJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  28. 28. HypothesisEvaluación y resultados  Query expansion can help repository users to find additional relevant resources not retrieved using the regular built-in search  Scenario of application: A teacher searches for educational resources to design a new course or to create a new resource JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  29. 29. Procedure Query Query Search other Synonym Original Part of Is a extraction Expanded concepts Query is a Expansion part of • Exists • Aproximate Remove • Doesnt exist other duplicates synonyms List of LO Filter results A  Contrast B +/- Relevance results C  evaluationJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  30. 30. Procedure  24 Official courses in genetics  In Universities and Higher education institutions  Syllabi published in the web  Academic period 2009  711 different concepts identified (lists of contents)  91 test queries (concept retrieval frequency >1)JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  31. 31. Method  Experts in the field evaluated the results (3 experts each query).  Topical relevance  Expert profile: medical practitioners and genetics specialists, 5 years experience in teaching and practice extrictly necessary.  Expert correlation analysis using rater agreement metrics  Precission and recall where used to state relevance and noveltyJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  32. 32. Results  Rater agreement moderate according to Cohen’s Kappa and Spearman correlation  COVERAGE: More than half the results (54%) retrieved from non-expanded queries are also retrieved when expanding the query.  NOVELTY: 53% of the relevant LOs retrieved by the expanded queries are new (e.g. different from those retrieved without expansion).JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  33. 33. Results 0,90 0,80 0,70 0,60 0,50 0,40 0,30 0,20 0,10 0,00 isa hermanos isa hijos isa padres par todo par partes syn exacto Novedad CoberturaJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  34. 34. ConclusionsConclusiones Results of expanded queries are affected by:  The quality of the ontology  Built-in retrieval mechanism  Intrinsic characteristics of the learning objects collection Best novelty results when:  Polysemic queries  Results from all types of expansions are merged in a unique list JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  35. 35. Publications  Segura, A., Sánchez-Alonso, S., Garcia-Barriocanal, E. and Prieto, M. (2011). An empirical analysis of ontology- JCR based query expansion for learning resource searches using MERLOT and the Gene ontology. Knowledge Based Systems, 24(1), pp. 119-133.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  36. 36. Exploring affiliation network models as a collaborative filtering mechanism in e-learning Not linked to any PhD Status: ready for anyone interestedJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  37. 37. Hypothesis  Social network analysis of relations in learning environments will make it possible to re-configure…  A) the learning contents and/or activities  E.g. including new activities, changing the future course structure or taking other kind of actions.  B) the learning environment  E.g. group formation, rearranging groups once the course is being delivered.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  38. 38. Method  Modeling and analysing learners participation in activities organized around communication forums, (very common in e-learning environments)  Forum participation as an affiliation network (a kind of social network with different types of nodes)  One of the possible applications: identifying common interests of groups of learners.  Technique: Blockmodeling (aimed at transforming an apparently non-coherent network into a more easily comprehensible arrangement)JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  39. 39. Results  We identified two different groups, one interested in the learning tools used during the course and the other group more interested in the theoretical aspects of the course.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  40. 40. Results  33-slice cluster with introductory topics (most people are interested)  As long as course progresses interest is less cohesive  16-slice cluster for topics on general LO definitions and concepts  8-slice for highly technical issues (SCORM + LD)JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  41. 41. Publications  Rodríguez, D., Sicilia, MA., Sánchez-Alonso, S., Lezcano, L. and García-Barriocanal, E. (2009) Exploring affiliation JCR network models as a collaborative filtering mechanism in e- learning, Interactive Learning Environments. DOI: 10.1080/10494820903148610JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  42. 42. Automated quality assessment of Learning Objects PhD. Cristian Cechinel Timeline: from July 2009 Status: Almost finalJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  43. 43. Hypothesis Some intrinsic features of learning objects stored in existing repositories may present significant difference between highly rated (good) learning objects and poorly rated (not-good) learning objects. These quantitative measures could serve as the basis for the development of models for quality prediction.
  44. 44. FunctionalityDeveloping models for automatically assessing quality of learning objects inside repositories based on the intrinsic features of the resources
  45. 45. Method Identification of intrinsic metrics of learning resources that could serve as potential indicators of quality Database gathered from the MERLOT repository by using a web crawler. In total, 35 metrics were extracted from 6,740 learning objects. From these resources, only 1,765 (27.27%) had at least one peer review or one user rating and were used in the analysis, the rest were discarded
  46. 46. Metrics Class of Measure Metric Link Measures Number of Links, Number of Unique Links, Number of Internal Links, Number of Unique Internal Links, Number of External Links, Number of Unique External Links Text Measures Number of Words, Number of words that are linksGraphic, Interactive and Number of Images, Total Size of the Images (in bytes), Number of Multimedia Measures Scripts, Number of Applets, Number of Audio Files, Number of Video Files, Number of Multimedia Files Site Architecture Size of the Page (in bytes), Number of Files for downloading, Total Measures Number of Pages Evaluation Metadata Number of Personal Collections (contrast metric)
  47. 47. Method Learning objects were classified into three groups (good, average, poor) according to their ratings. A Mann-Whitney (Wilcoxon) test was performed to evaluate whether the selected metrics presented significant difference in their medians between the groups of good and not-good (average + poor) resources Kolmogorov-Smirnov test was performed to evaluate differences regarding the distributions.
  48. 48. Preliminary ResultsThe two groups of ratings available on MERLOT (i.e. peer-reviewers and user comments ratings) differ substantially regarding the intrinsic characteristics of the resources.The tested metrics present different profiles and tendencies between good and not-good materials depending on the category of discipline and the type of material to which the resource belong
  49. 49. Preliminary Results We developed a Linear Discriminant Analysis (LDA) to build models in order to distinguish 1. good from not-good resources, 2. good from average resources, and 3. good from poor resources For the Science and Technology discipline intersected with the Simulation material type in the context of peer- reviews thresholds. The third model achieved 91.49% of overall accuracy, with a squared canonical correlation of 0.81130 (significant at the 99% level)
  50. 50. Preliminary Results  The pursuit for an automated model for the quality evaluation of learning objects must consider the development of profiles taking into account the intersection of the categories of disciplines and material types, as well as the distinct groups of raters.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  51. 51. Publications  Cechinel, C., Sánchez-Alonso, S. and García-Barriocanal, E. (2011). Statistical profiles of highly-rated learning JCR objects. Computers & Education, 57(1), pp. 1255-1269.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  52. 52. Can 3D platforms improve training of trainers programs? PhD. Carlos M. Lorenzo Timeline: from April 2010 Status: in progressJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  53. 53. Hypothesis  Using Massively Multiuser Online Learning environments (MMOL) platforms in virtual courses can improve training-of-trainers program  The aim is to explore how a specific MMOL Platform facilitates online tutor’s tasks in a rich virtual learning environment with a pedagogical framework, and to identify essential issues of interactivity in this contextJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  54. 54. Method  MMOL session using the collaborative LORI approach  A group of users contribute their individual evaluations on a learning object and try to reach a consensus after hearing everyone else’s opinion.  2D session in LCMS  Comparing results in terms of satisfaction and efectivenessJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  55. 55. The experiments  A prototype of 3D MMOL platform was created in a realXtend server with an interactive space called “MadriPolis”. Case A: 11 master students Both cases consist in training- of-trainers experiences about • LCMS on-line tutor experiment collaborative Learning Object • MMOL on-line tutor experiment evaluation based on Learning Object Review Instrument Case B: 10 graduated (LORI) with the Convergent students Participation Model (CPM) (Vargo et al., 2003) to • LCMS on-line tutor experiment determinate the quality of e- • MMOL on-line tutor experiment learning resourcesJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  56. 56. Results •Case «A». Data •LCMS Experim. • Data Analisys Collection and SNA: •MMOL Experim. •Case «B» •Log events. • Density •MMOL Experim. •On-line surveys • Centrality •LCMS Experim. •Direct Observations •Triangulation Case studies EvaluationJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  57. 57. Publications  Manuscript submitted to Computer & Education (April 2011). Still waiting… JCRJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  58. 58. Recommenders inside learning object repositories: requirements for meaningful datasets Not linked to any PhD Status: ready for anyone interestedJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  59. 59. Hypothesis Implicit communities found via SNA blockmodeling & component analysis have a potential for recommending learning objects to repository usersJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  60. 60. Application Implicit communities found via SNA blockmodeling & component analysis have a potential for recommending learning objects to repository usersJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  61. 61. Method  Evaluate parameters for Collaborative Filtering Algorithms for two datasets from MERLOT 1. Resources including ratings given by users 2. Resources present in the users’ Personal Collections  Generating recommendations for the datasets using optimized parametersJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  62. 62. Method  Compare the results generated by the different algorithms for the two datasets  Contrast the results of the recommendations generated by the algorithms with existing endorsement mechanisms of the repositoryJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  63. 63. Results  Very high Precision values (varying from 20% to 100%) and not so high Recall percentages (with a maximum of 18%).  Recommendations generated are related to other endorsement mechanisms in MERLOT  Big differences between the recommendations generated using the two distinct datasets  Reinforcement of the initial idea that these two datasets represent very distinct informationJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  64. 64. Future work  Involving users’ opinions in the process  Contrasting if recommendations for a given user fall in the disciplinary area of that user or are crossing disciplines  Evaluating if users are already familiar with the recommended resources, and if they would recommend such resources to their fellowsJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  65. 65. Publications  Sánchez Alonso, S., Sicilia, MA., García, E., Pagés, C. and Lezcano, L. (2011) Social models in open learning object repositories: A simulation approach for sustainable JCR collections. Simulation Modelling Practice and Theory 19(1): 110-120  Sicilia, M.-Á., García-Barriocanal, E., Sánchez-Alonso, S., & Cechinel, C. (2010). Exploring user-based recommender results in large learning object repositories: the case of MERLOT. Procedia Computer Science, 1(2), 2859-2864.JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  66. 66. Conclussions and open reseach directionsJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  67. 67. Lessons learned  Quantitative research is usually well received by impact factor journals editorial boards  It is feasible to have a PhD ready in about 2 years  Respect repositories’ policies on acceptable use  Collecting data, either manually or through automated processes may be not permittedJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  68. 68. Lessons learned  LOR data should be shared!  Evangelize repository owners to share data for research and to include that in their conditions for use.  A common dataset sharing format for LOR neededJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  69. 69. Open research opportunities  More in-depth study of social interaction in LCMS (software ready for SNA in Moodle)  Open courseware research studies similar to those presented (OCW Finder crawler ready)  Several project-related research (e.g. Assessment of automated translation mechanisms in Organic.Edunet)  Any other research extending previous cases...JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  70. 70. Your turn!  5 next minutes [individually]: think of an experiment similar to those reported in my talk  15 minutes [all]: share your ideas with us  [In pairs] Write down a summary with at least:  Hypothesis  Functionality  Techniques  Assessment metodJTEL Summer school 2011 - Ratings, tags, bookmarks and other species
  71. 71. To be written down  Salvador: salvador.sanchez@uah.es  Want to know more about our distance PhD program @ IE-UAH?  Talk to me today or email me at your wish  Fancy to work with us in a EU project?  Contact me or prof. Sicilia: msicilia@uah.esJTEL Summer school 2011 - Ratings, tags, bookmarks and other species

×