Doctoral Consortium, RecSys 2007

Can Social Information Retrieval Enhance
       the Discovery and Reuse
                ...
Outline of the presentation
    Context of the dissertation work
●




    Main research questions
●




    Experimental ...
Context of the dissertation work
Context
    European education, especially that of K-12
●

    education, is inherently multilingual and
    multicultural...
EUN partners..image
Context
    Resources
●

        In many different languages
    –
        For different national and regional curriculum
...
Challenge for users
    End-users (e.g. teachers) have difficulties to
●

    discover and find resources from educational...
Challenges for repositories
    Users become more demanding and expect
●

    services that are seen elsewhere (own
    co...
My Main Question

Can Social Information Retrieval
           Enhance
   the Discovery and Reuse
              of
     Lea...
Social Information Retrieval
                (SIR)
    Refers to a family of techniques that assist
●

    users in obtain...
What is SIR for education?
    Is education as a field of implementation that
●

    different from other fields (e.g. mus...
Context of this dissertation


                          Education
      Social                               Information
...
Main research questions
Main research questions 1
Teachers, tagging, languages:

    How do teachers tag and use social
●

    bookmarking in a mu...
Main research questions 2
SIR aspect:

    Can bookmarks and tags be used to connect
●

    like-minded teachers cross cou...
Main research questions 3
Information Seeking aspect:

    What are the main information seeking tasks
●

    that teacher...
Experimental design
Data source 1
    Calibrate project (http://calibrate.eun.org),
●

    now to end of 2007

    K-12 digital learning resou...
Implementation area and data
             source 2
    MELT project (http://info.melt-project.eu),
●

    from now to Marc...
Data gathering
    Diverse data collection methods to allow
●

    triangulation of collected data.

        log files fro...
Experimental Design

                Independent   Social Condition
                Condition



    Salganik, M., Dodds, ...
Experimental Design

           Independent    Social Condition
           Condition
Tag input No tags shown   Tags shown ...
Some early analysis
Analysis of User Behavior on Multi-
    lingual Tagging of Learning Objects
    January 24 to April 21 2007
●




    77 t...
Cross-border and language use
                                                                              5
            ...
Language should not divide..

                                                5
                                       Tag...
..but bring like-minded people
            together

                              2
                    Tag
             ...
Visualisation tool for cross-
      country use of bookmarks
    Prototype tool to visualise
●

        Bookmarks (title, ...
Distribution of bookmarks
                 Average: 6 bookmarks
             ●

                 Wide distribution:
      ...
Language analysis
    Out of 417 tags many were with multiple
●

    terms, when separated we found 585 terms

    1/3 in ...
Language analysis
    The language was right in about 70% of
●

    cases (from the interface), and found out
    that...
...
Btw, what do others do?
    del.icio.us, Yahoo.fr, MyWeb.Yahoo.uk,
●

    blogmarks.net, MisterWong.de...

    Two differe...
Does the language matter?




    Need for better ways to identify the language
●

        Give rules (if the user first p...
Semantic analysis
    Factual tags 63%
●

    (Golder: item topics, kinds of
    item, category refinements)

    Subjecti...
Why tag categories?
    In Sen et al. (2006)
●

    it was found that
    tags of different
    categories can be
    usef...
“Travel well” tags
    About 13% of tags contain a general term, a
●

    name, place

    e.g. EU, Euroopa, Europa, europ...
What's the point of travel well
                tags?
    If those tags need no translation or language
●

    filtering t...
Do users find tags useful?
Usefulness of tags..
    Overall, the thesaurus terms performed
●

    better than the tags,

    However, it can be argue...
So what is needed?
    HIDE ALL BUT THE RIGHT STUFF!
●




    In the tagging interface (guided tagging)
●

        Show t...
Future studies
    Similar language and semantic analysis are
●

    planned for a more thorough data in 2008

    Moreove...
User engagement
    Inspired by Yahoo!'s START
●

        rating shows the first level of engagement;
    –
        then t...
User engagement
    In our case these
●

    look very different:
        views the page
    –
        views metadata
    ...
That's it for now!


http://www.cs.kuleuven.be/~riina

    riina.vuorikari@eun.org
riina.vuorikari@cs.kuleuven.be
RecSys 07 Doctoral Consortium Presentation
RecSys 07 Doctoral Consortium Presentation
RecSys 07 Doctoral Consortium Presentation
RecSys 07 Doctoral Consortium Presentation
RecSys 07 Doctoral Consortium Presentation
Upcoming SlideShare
Loading in …5
×

RecSys 07 Doctoral Consortium Presentation

1,526 views

Published on

My PhD research topics and experimental set-up.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

RecSys 07 Doctoral Consortium Presentation

  1. 1. Doctoral Consortium, RecSys 2007 Can Social Information Retrieval Enhance the Discovery and Reuse of Learning Resources? Riina Vuorikari Katholieke Universiteit Leuven, Department of Computer Science European Schoolnet, Belgium
  2. 2. Outline of the presentation Context of the dissertation work ● Main research questions ● Experimental design ● First evaluations so far: ● Multi-lingual use of tags – Levels of user engagement –
  3. 3. Context of the dissertation work
  4. 4. Context European education, especially that of K-12 ● education, is inherently multilingual and multicultural. European teachers have access to multiple ● repositories of digital learning resources by Educational Authorities, – publishers, – other teachers,.. –
  5. 5. EUN partners..image
  6. 6. Context Resources ● In many different languages – For different national and regional curriculum – Contain metadata (e.g.title, keywords, language) – Of varying quality – Repositories have formed federations to ● make resources available Federated search based on metadata – Harvesting of metada –
  7. 7. Challenge for users End-users (e.g. teachers) have difficulties to ● discover and find resources from educational repositories Metadata does not always match search terms – Locating content across linguistic and ● national borders within Europe has proven hard Despite the use of a multilingual Thesaurus and – controlled vocabularies
  8. 8. Challenges for repositories Users become more demanding and expect ● services that are seen elsewhere (own collections, pedagogical hints, ..) European Schoolnet leading projects that ● build services on top of federation of European repositories Social bookmarking tool – Tags – My networks –
  9. 9. My Main Question Can Social Information Retrieval Enhance the Discovery and Reuse of Learning Resources?
  10. 10. Social Information Retrieval (SIR) Refers to a family of techniques that assist ● users in obtaining information to meet their information needs by harnessing the knowledge or experience of other users. Examples of SIR techniques include: ● sharing of queries, – collaborative filtering, – social network analysis, – social bookmarking, – subjective relevance judgements such as – tags, annotations, ratings and evaluations, etc.
  11. 11. What is SIR for education? Is education as a field of implementation that ● different from other fields (e.g. music, movies)? What are the domain specific requirements, where ● does the data come from and what are its semantics? What are objects of recommendation? ● SIR TEL http://ariadne.cs.kuleuven.be/sirtel/ ● My audience are teachers. Metaphor: it's like ● recommending for DJs?
  12. 12. Context of this dissertation Education Social Information Information seeking theories Digital Retrieval Digital libraries content (SIR) methods To empower the social and contextual aspects of teachers' work
  13. 13. Main research questions
  14. 14. Main research questions 1 Teachers, tagging, languages: How do teachers tag and use social ● bookmarking in a multi-lingual environment? Are those bookmarks and tags useful for ● discovery of resources? How about tags in multiple languages? ●
  15. 15. Main research questions 2 SIR aspect: Can bookmarks and tags be used to connect ● like-minded teachers cross country and linguistic borders? ...and thus used for social information ● retrieval? What are the levels of user engagement ● with the system?
  16. 16. Main research questions 3 Information Seeking aspect: What are the main information seeking tasks ● that teachers have? What are the main SIR retrieval methods that ● they use for them? Can we match a task to a SIR method? ●
  17. 17. Experimental design
  18. 18. Data source 1 Calibrate project (http://calibrate.eun.org), ● now to end of 2007 K-12 digital learning resources ● Personal collections and tags (not shared) ● 78 pilot schools in Hungary, Austria, Estonia, ● Czech Republic, Lithuania and Poland
  19. 19. Implementation area and data source 2 MELT project (http://info.melt-project.eu), ● from now to March 2009 K-12 digital learning resources from a ● federation of about 10 repositories Implementation of a social bookmarking tool, ● annotations and my networks About 70 teachers from Austria, Belgium, ● Finland and Hungary
  20. 20. Data gathering Diverse data collection methods to allow ● triangulation of collected data. log files from the portals to see the grand lines, – patterns, etc complimented by some questionnaires to – understand groups or communities possible interviews, thinking alouds, observation, – etc. on some few users to understand individual behaviour.
  21. 21. Experimental Design Independent Social Condition Condition Salganik, M., Dodds, P., & Watts, D. ● Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market. Science, 311(5762), (2006), 854- 856.
  22. 22. Experimental Design Independent Social Condition Condition Tag input No tags shown Tags shown Tags shown when tagging within users in all spoken language languages Social Ranking of Social navigation based on Information resources bookmarks, tags, annotations Retrieval and my networks
  23. 23. Some early analysis
  24. 24. Analysis of User Behavior on Multi- lingual Tagging of Learning Objects January 24 to April 21 2007 ● 77 teachers /173 total participating ● 459 bookmarks ● 417 multilingual tags ● 320 different learning resources ●
  25. 25. Cross-border and language use 5 Tag Tag 1 fr fi 3 Tag de 4 Tag Tag 1 fr fi 2 Tag en LO 2 in Fr LO 1 in Fi fr de fi fi fi
  26. 26. Language should not divide.. 5 Tag fr 2 Tag 2 Tag en de Tag 1 4 Tag fi fr Tag 1 fi LO 2 LO 2 LO 2 LO 1 LO 1 in Fr in Fr in Fr in Fi in Fi fi de fr fi fi
  27. 27. ..but bring like-minded people together 2 Tag en Tag 1 Tag 5 fi fr Tag 1 Tag 2 fi de LO 2 LO 1 in Fr in Fi de fr fi fi fi
  28. 28. Visualisation tool for cross- country use of bookmarks Prototype tool to visualise ● Bookmarks (title, classification keyword, country) – Tags (language) – Users (name, country, language) – – Wanna play around with it? ● http://www.cs.kuleuven.ac.be/~hmdb/infovis/ calibrate/calibrate.html
  29. 29. Distribution of bookmarks Average: 6 bookmarks ● Wide distribution: ● 10% “Super users” – more than 20 15% 20-6 bookmarks – 45% 6-2 bookmarks – About 30% only – experimented (1)
  30. 30. Language analysis Out of 417 tags many were with multiple ● terms, when separated we found 585 terms 1/3 in Hungarian ● 26% in English, even though none of the ● users were native English speakers 1/3 in German and Polish ●
  31. 31. Language analysis The language was right in about 70% of ● cases (from the interface), and found out that... ...users tag in many different languages: ● at the same time (e.g. Baum, arbre, tree) – at different times (once in Pl, other times in En) – use the interface in different languages (seems – like not only to test)
  32. 32. Btw, what do others do? del.icio.us, Yahoo.fr, MyWeb.Yahoo.uk, ● blogmarks.net, MisterWong.de... Two different ways to deal with multiple ● languages can be observed; ones taken care of by users (i.e. crowd- – sourcing”) others where the system supports multiple – languages to certain extent
  33. 33. Does the language matter? Need for better ways to identify the language ● Give rules (if the user first preferred languages is.., then..) – Automate the recognition of languages – Out-source it to users –
  34. 34. Semantic analysis Factual tags 63% ● (Golder: item topics, kinds of item, category refinements) Subjective tags 29% ● ( Golder: item qualities) Personal tags 3% ● (Golder: item ownership, self- reference, tasks organisation) 5% other ● Sen et al. (2006). ●
  35. 35. Why tag categories? In Sen et al. (2006) ● it was found that tags of different categories can be useful for different tasks In our case it is too ● early to say anything, but ...we'll have an eye on it!
  36. 36. “Travel well” tags About 13% of tags contain a general term, a ● name, place e.g. EU, Euroopa, Europa, europe, ● geograafia, Pythagoras, etc.
  37. 37. What's the point of travel well tags? If those tags need no translation or language ● filtering to be understood, and .. ..if they can be identified ● We can be sure to show at least some tags ● to users whose language preferences we don't know, and – in which language there are no tags or keywords – available.
  38. 38. Do users find tags useful?
  39. 39. Usefulness of tags.. Overall, the thesaurus terms performed ● better than the tags, However, it can be argued that tags, after all ● being produced with no outlay, showed an overall encouraging and potential gain in overall usefulness!
  40. 40. So what is needed? HIDE ALL BUT THE RIGHT STUFF! ● In the tagging interface (guided tagging) ● Show tags in all languages? – Show only travel well tags? – Show only tags in users' preferred languages – While viewing the tags ● In a tag cloud – For social navigation (resource-user-tag) – Q: does the system translate tags or only when a – user-given translation exist?
  41. 41. Future studies Similar language and semantic analysis are ● planned for a more thorough data in 2008 Moreover, our goals are to find out: ● How do users use the tags (e.g. language and – tag convergence) ? How are tags and the relation resource-tag-user – used for discovery? Identify teachers information seeking tasks and a – best fit for a retrieval system.
  42. 42. User engagement Inspired by Yahoo!'s START ● rating shows the first level of engagement; – then tags it; – user views a page; – forwards it to friends, – and finally writing a review – How can this be used for recommending ● purpose?
  43. 43. User engagement In our case these ● look very different: views the page – views metadata – bookmarks and tags – rates – actual use? –
  44. 44. That's it for now! http://www.cs.kuleuven.be/~riina riina.vuorikari@eun.org riina.vuorikari@cs.kuleuven.be

×