0
Preliminary Discussion on a Digital Curation Framework for Learning Repositories Nikos Palavitsinis 1,2 , Nikos Manouselis...
Structure <ul><li>background – definitions </li></ul><ul><li>quality in practice </li></ul><ul><li>experiments </li></ul><...
background & definitions
problem <ul><li>Quality of the metadata provided by annotators of the resources </li></ul><ul><li>Experiences from Organic...
background <ul><li>PhD topic:  Metadata Quality Issues  in Learning Object Repositories </li></ul><ul><li>Behind the words...
concepts <ul><li>learning resource/object </li></ul><ul><ul><li>from information/audiovisual assets… </li></ul></ul><ul><u...
define: metadata <ul><li>“ Metadata is structured information that describes, explains, locates, or otherwise makes it eas...
define: curation <ul><li>“ Curation includes verification and additions to the existing  documentation  for objects.” </li...
define: quality <ul><li>Level of excellence; A property or attribute that differentiates a thing or person </li></ul><ul><...
“ quality intersections” Repository Level Portal Level Resources Level content/course creator end-user portal owner
quality in practice
Learning Resource Lifecycle Van Assche, Vuorikari, 2006 Retract Expose Describe Discovery Search Social Rec. Alerts Evalua...
Organic.Edunet <ul><li>Project that makes digital content on topics of Organic Agriculture & Agroecology available </li></...
Organic.Edunet approach <ul><li>overall  quality strategy </li></ul><ul><ul><li>quality guide for the  creation   of learn...
levels of quality considerations <ul><li>individual </li></ul><ul><ul><li>contribution by an individual  (teacher, learner...
Learning Resource Lifecycle Van Assche, Vuorikari, 2006 Retract Expose Describe Discovery Search Social Rec. Alerts Evalua...
Organic.Edunet AP <ul><li>With more than 10.000 resources </li></ul><ul><li>With 11 repositories </li></ul><ul><li>With pa...
Development of AP Evaluation phase Results’ analysis AP modifications Definition of own requirements Selection of LOM elem...
Experiment 1: Evaluation Study by subject matter experts  to elaborate on the process of describing resources with metadata
Experiment Details <ul><li>Participants : 20 </li></ul><ul><ul><li>Experts in Organic Agriculture, ICT, Education </li></u...
Is this element easy to understand?
Is this element easy to understand? <ul><li>Best rated elements: </li></ul><ul><ul><li>General.Keyword   </li></ul></ul><u...
Is this element useful?
Is this element useful? <ul><li>Best rated elements: </li></ul><ul><ul><li>General.Identifier </li></ul></ul><ul><ul><li>G...
Are the values clear & appropriate?
Are the values clear & appropriate? <ul><li>Best rated elements: </li></ul><ul><ul><li>General.Description   </li></ul></u...
Status of elements -8,3% -19% +31% % 11 12 Optional 21 25 Post-evaluation 26 19 Pre-evaluation Recommended Mandatory Status
Experiment 2: Usage Data Analysis  of data produced by subject matter experts using an annotation tool  to provide metadata
Experiment Details <ul><li>Participants : 30 </li></ul><ul><ul><li>Experts in Organic Agriculture, Education </li></ul></u...
Results <ul><li>Metadata element:  Keyword </li></ul>55% 100% 611 TOTAL 1,9 3,5 21 More than 9 8,8 15,9 97 From 7 to 9 17,...
Results <ul><li>Metadata element:  TaxonId </li></ul>41,2% 100% 455 TOTAL 6,6 16 73 More than 6 4,2 10,1 46 5 or 6 9 21,8 ...
Results <ul><li>Metadata element:  End User Role </li></ul>51% 100% 565 TOTAL 3,3 6,6 37 5 or 6 12,9 25,3 143 3 or 4 34,8 ...
Mandatory Elements <ul><li>Not all mandatory elements were used in the expected degree </li></ul>
Recommended Elements <ul><li>Recommended elements present similar problems </li></ul>
Experiment 3: Consultation with experts Discussion on Quality Considerations in Learning Repositories and Portals
Budapest, 17/9/2010 <ul><li>Quality Considerations for Learning Portals and Repositories in Agriculture, Food & Environmen...
 
next: online consultation <ul><li>e-Conferenence </li></ul><ul><ul><li>From 6/10 to 20/10/2010 </li></ul></ul><ul><ul><li>...
future directions:  towards digital cu-ality?
DCC Curation Lifecycle Model
Description & Preservation Information Preservation Planning Community Watch & Participation Preserve Curate Create & Rece...
other issues <ul><li>Criteria for selecting LOs to curate </li></ul><ul><li>Aggregation level is important for curation </...
next step Van Assche, Vuorikari, 2006 Retract Expose Describe Discovery Search Social Rec. Alerts Evaluate Select  Procure...
some thoughts <ul><li>Dice:  http ://www.vatsgroup.com/Quality.htm </li></ul><ul><li>Stool:  http ://www.codinghorror.com/...
Upcoming SlideShare
Loading in...5
×

Preliminary Discussion on a Digital Curation Framework for Learning Repositories

1,416

Published on

Presentation made at SE@M workshop in the context of EC-TEL 2010 in Barcelona, Spain (Tuesday, 28/9/2010)

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,416
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Preliminary Discussion on a Digital Curation Framework for Learning Repositories"

  1. 1. Preliminary Discussion on a Digital Curation Framework for Learning Repositories Nikos Palavitsinis 1,2 , Nikos Manouselis 1 , Salvador Sanchez-Alonso 2 1 Greek Research & Technology Network 2 University of Alcala 4 th International Workshop on Search and Exchange of e-le@rning Materials 27-28 September, 2010 Barcelona, Spain
  2. 2. Structure <ul><li>background – definitions </li></ul><ul><li>quality in practice </li></ul><ul><li>experiments </li></ul><ul><li>towards digital cura-lity </li></ul><ul><li>experts – users opinion </li></ul><ul><li>e-Conference </li></ul><ul><li>conclusions </li></ul>
  3. 3. background & definitions
  4. 4. problem <ul><li>Quality of the metadata provided by annotators of the resources </li></ul><ul><li>Experiences from Organic.Edunet Project </li></ul><ul><li>Relevant experiences coming from: </li></ul><ul><ul><li>Ochoa & Duval (2006) </li></ul></ul><ul><ul><li>Zschocke & Beniest (2009) </li></ul></ul>
  5. 5. background <ul><li>PhD topic: Metadata Quality Issues in Learning Object Repositories </li></ul><ul><li>Behind the words: Trying to find ways of improving the quality of metadata in learning object repositories & portals </li></ul><ul><li>Really… Behind the words: Can we introduce mechanisms to ensure/control/assess quality of metadata in learning repositories & portals? </li></ul>
  6. 6. concepts <ul><li>learning resource/object </li></ul><ul><ul><li>from information/audiovisual assets… </li></ul></ul><ul><ul><li>… to complete educational programs </li></ul></ul><ul><li>collection of learning resources </li></ul><ul><ul><li>+metadata=learning repository </li></ul></ul><ul><li>online services </li></ul><ul><ul><li>learning portals </li></ul></ul>
  7. 7. define: metadata <ul><li>“ Metadata is structured information that describes, explains, locates, or otherwise makes it easier to retrieve, use, or manage an information resource” </li></ul><ul><li>Metadata must be always considered in a specific context, i.e. education, research, etc. </li></ul>
  8. 8. define: curation <ul><li>“ Curation includes verification and additions to the existing documentation for objects.” </li></ul><ul><ul><li>Documentation = metadata </li></ul></ul><ul><li>Digital Curation </li></ul><ul><ul><li>… to maintain & add value to digital materials over their entire life-cycle and over time for current and future use </li></ul></ul>
  9. 9. define: quality <ul><li>Level of excellence; A property or attribute that differentiates a thing or person </li></ul><ul><li>Quality is the suitability of procedures, processes and systems in relation to the strategic objectives </li></ul>
  10. 10. “ quality intersections” Repository Level Portal Level Resources Level content/course creator end-user portal owner
  11. 11. quality in practice
  12. 12. Learning Resource Lifecycle Van Assche, Vuorikari, 2006 Retract Expose Describe Discovery Search Social Rec. Alerts Evaluate Select Procure Gather Metadata Enrich Resolution Get Reference or LO Create Integrate Adapt & Reuse Disaggregate Aggregate Modify Use/Play Integrate Local delete
  13. 13. Organic.Edunet <ul><li>Project that makes digital content on topics of Organic Agriculture & Agroecology available </li></ul><ul><li>Using IEEE LOM Metadata </li></ul><ul><ul><li>Organic.Edunet AP </li></ul></ul><ul><li>Through the Organic.Edunet Portal </li></ul><ul><ul><li>With a well-defined Quality Process </li></ul></ul>
  14. 14. Organic.Edunet approach <ul><li>overall quality strategy </li></ul><ul><ul><li>quality guide for the creation of learning resources </li></ul></ul><ul><ul><li>reflecting quality in metadata </li></ul></ul><ul><ul><li>quality procedures within the repositories (Quality Check, Peer-Review, …) </li></ul></ul><ul><ul><li>quality procedures on a portal level (User rating, Featured Resources, …) </li></ul></ul><ul><ul><li>quality of educational activities </li></ul></ul>
  15. 15. levels of quality considerations <ul><li>individual </li></ul><ul><ul><li>contribution by an individual (teacher, learner, learning material designer, etc.) </li></ul></ul><ul><li>institutional contribution </li></ul><ul><ul><li>contribution through an institutional provider (public/ private content provider, educational organisation, another repository) </li></ul></ul>
  16. 16. Learning Resource Lifecycle Van Assche, Vuorikari, 2006 Retract Expose Describe Discovery Search Social Rec. Alerts Evaluate Select Procure Gather Metadata Enrich Resolution Get Reference or LO Create Integrate Adapt & Reuse Disaggregate Aggregate Modify Use/Play Integrate Local delete
  17. 17. Organic.Edunet AP <ul><li>With more than 10.000 resources </li></ul><ul><li>With 11 repositories </li></ul><ul><li>With partners from 10 different countries </li></ul><ul><li>We needed a Metadata Application Profile </li></ul><ul><ul><li>Multilingual </li></ul></ul><ul><ul><li>Ontology support </li></ul></ul>
  18. 18. Development of AP Evaluation phase Results’ analysis AP modifications Definition of own requirements Selection of LOM elements Semantics Refinement Multiplicity constraints and values Relationships and dependencies Required extensions Application Profile Binding Evaluation of AP
  19. 19. Experiment 1: Evaluation Study by subject matter experts to elaborate on the process of describing resources with metadata
  20. 20. Experiment Details <ul><li>Participants : 20 </li></ul><ul><ul><li>Experts in Organic Agriculture, ICT, Education </li></ul></ul><ul><li>Date : January 2009 </li></ul><ul><li>Object : IEEE LOM AP Elements </li></ul><ul><li>Tool : Questionnaire </li></ul><ul><ul><li>5 point scale for most questions and a 3-value multiple choice in one of them </li></ul></ul>
  21. 21. Is this element easy to understand?
  22. 22. Is this element easy to understand? <ul><li>Best rated elements: </li></ul><ul><ul><li>General.Keyword </li></ul></ul><ul><ul><li>Technical.Format </li></ul></ul><ul><ul><li>Technical.Size </li></ul></ul><ul><li>Worst rated elements: </li></ul><ul><ul><li>Classification.Taxon </li></ul></ul><ul><ul><li>Relation.Resource </li></ul></ul><ul><ul><li>Educational.Semantic Density </li></ul></ul>
  23. 23. Is this element useful?
  24. 24. Is this element useful? <ul><li>Best rated elements: </li></ul><ul><ul><li>General.Identifier </li></ul></ul><ul><ul><li>General.Description </li></ul></ul><ul><ul><li>Technical.Format </li></ul></ul><ul><li>Worst rated elements: </li></ul><ul><ul><li>Classification.Taxon </li></ul></ul><ul><ul><li>Annotation.Entity </li></ul></ul><ul><ul><li>Annotation.Date </li></ul></ul>
  25. 25. Are the values clear & appropriate?
  26. 26. Are the values clear & appropriate? <ul><li>Best rated elements: </li></ul><ul><ul><li>General.Description </li></ul></ul><ul><ul><li>Rights.Cost </li></ul></ul><ul><ul><li>Format.Size </li></ul></ul><ul><li>Worst rated elements: </li></ul><ul><ul><li>Classification.Taxon </li></ul></ul><ul><ul><li>Classification.Purpose </li></ul></ul><ul><ul><li>General.Identifier </li></ul></ul>
  27. 27. Status of elements -8,3% -19% +31% % 11 12 Optional 21 25 Post-evaluation 26 19 Pre-evaluation Recommended Mandatory Status
  28. 28. Experiment 2: Usage Data Analysis of data produced by subject matter experts using an annotation tool to provide metadata
  29. 29. Experiment Details <ul><li>Participants : 30 </li></ul><ul><ul><li>Experts in Organic Agriculture, Education </li></ul></ul><ul><li>Date : January 2009 – March 2009 </li></ul><ul><li>Object : Actual usage of IEEE LOM AP </li></ul><ul><li>Tool : Log files analysis </li></ul>
  30. 30. Results <ul><li>Metadata element: Keyword </li></ul>55% 100% 611 TOTAL 1,9 3,5 21 More than 9 8,8 15,9 97 From 7 to 9 17,8 32,2 197 From 4 to 6 26,8 48,4 296 From 1 to 3 % of total % of filled Count Keyword
  31. 31. Results <ul><li>Metadata element: TaxonId </li></ul>41,2% 100% 455 TOTAL 6,6 16 73 More than 6 4,2 10,1 46 5 or 6 9 21,8 99 3 or 4 21,4 52,1 237 1 or 2 % of total % of filled Count TaxonPathTaxonId
  32. 32. Results <ul><li>Metadata element: End User Role </li></ul>51% 100% 565 TOTAL 3,3 6,6 37 5 or 6 12,9 25,3 143 3 or 4 34,8 68,1 385 1 or 2 % of total % of filled Count Intended End User Role
  33. 33. Mandatory Elements <ul><li>Not all mandatory elements were used in the expected degree </li></ul>
  34. 34. Recommended Elements <ul><li>Recommended elements present similar problems </li></ul>
  35. 35. Experiment 3: Consultation with experts Discussion on Quality Considerations in Learning Repositories and Portals
  36. 36. Budapest, 17/9/2010 <ul><li>Quality Considerations for Learning Portals and Repositories in Agriculture, Food & Environment </li></ul><ul><ul><li>40 participants – mixed audience </li></ul></ul><ul><ul><li>17/9/2010 </li></ul></ul><ul><ul><li>Budapest, Hungary </li></ul></ul><ul><ul><li>Organic.Edunet Final Conference </li></ul></ul>
  37. 38. next: online consultation <ul><li>e-Conferenence </li></ul><ul><ul><li>From 6/10 to 20/10/2010 </li></ul></ul><ul><ul><li>e-Agriculture.org platform (>3.000 experts) </li></ul></ul><ul><li>Topics: </li></ul><ul><ul><li>building quality in the resource annotation, curation & preservation life cycles; </li></ul></ul><ul><ul><li>quality processes on a repository level; </li></ul></ul><ul><ul><li>quality criteria and processes on a web portal level </li></ul></ul>
  38. 39. future directions: towards digital cu-ality?
  39. 40. DCC Curation Lifecycle Model
  40. 41. Description & Preservation Information Preservation Planning Community Watch & Participation Preserve Curate Create & Receive Access, Use & Reuse Ingest Appraise & Select Transform Preservation Action Store Conceptualize Dispose Migrate Reappraise LR & Metadata LR & Metadata <ul><li>Quality of metadata provided by subject matter experts </li></ul><ul><li>Variability in quality amongst the metadata provided </li></ul><ul><li>It requires skilled, professional curators </li></ul><ul><li>Human resources do not scale well with many resources </li></ul><ul><li>Lack of a unanimous definition of a learning resource </li></ul><ul><li>How can you preserve anything without defining it first? </li></ul>
  41. 42. other issues <ul><li>Criteria for selecting LOs to curate </li></ul><ul><li>Aggregation level is important for curation </li></ul><ul><li>Ingest resources  Access rights? Owner? </li></ul>
  42. 43. next step Van Assche, Vuorikari, 2006 Retract Expose Describe Discovery Search Social Rec. Alerts Evaluate Select Procure Gather Metadata Enrich Resolution Get Reference or LO Create Integrate Adapt & Reuse Disaggregate Aggregate Modify Use/Play Integrate Local delete
  43. 44. some thoughts <ul><li>Dice: http ://www.vatsgroup.com/Quality.htm </li></ul><ul><li>Stool: http ://www.codinghorror.com/blog/archives/000708.html </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×