Preliminary Discussion on a Digital
Curation Framework for Learning
Repositories
Nikos Palavitsinis1,2
, Nikos Manouselis1...
Structure
• background – definitions
• quality in practice
• experiments
• towards digital cura-lity
• experts – users opi...
background & definitions
problem
• Quality of the metadata provided by
annotators of the resources
• Experiences from Organic.Edunet Project
• Rele...
background
• PhD topic: Metadata Quality Issues
in Learning Object Repositories
• Behind the words: Trying to find ways of...
concepts
• learning resource/object
• from information/audiovisual assets…
• …to complete educational programs
• collectio...
define: metadata
• “Metadata is structured information that
describes, explains, locates, or otherwise
makes it easier to ...
define: curation
• “Curation includes verification and additions
to the existing documentation for objects.”
– Documentati...
define: quality
• Level of excellence; A property or attribute
that differentiates a thing or person
• Quality is the suit...
“quality intersections”
Repository Level
Portal Level
Resources Level
content/course
creator
end-user portal owner
quality in practice
Learning Resource Lifecycle
Retract
Expose
Describe
Discovery
Search
Social Rec.
Alerts
Evaluate
Select
Procure
Gather
Met...
Organic.Edunet
• Project that makes digital content on topics of
Organic Agriculture & Agroecology available
• Using IEEE ...
Organic.Edunet approach
• overall quality strategy
• quality guide for the creation of learning
resources
• reflecting qua...
levels of quality considerations
• individual
•contribution by an individual (teacher,
learner, learning material designer...
Learning Resource Lifecycle
Retract
Expose
Describe
Discovery
Search
Social Rec.
Alerts
Evaluate
Select
Procure
Gather
Met...
Organic.Edunet AP
• With more than 10.000 resources
• With 11 repositories
• With partners from 10 different countries
• W...
Development of APDefinition of own requirements
Selection of LOM elements
Semantics Refinement
Multiplicity constraints an...
Experiment 1:
Evaluation Study
by subject matter experts
to elaborate on the process of
describing resources with metadata
Experiment Details
• Participants: 20
–Experts in Organic Agriculture, ICT, Education
• Date: January 2009
• Object: IEEE ...
Is this element easy to understand?
Easy to understand
33%
42%
21%
4%
0%
Very Easy Fairly Easy Mediocre
Fairly Difficult V...
Is this element easy to understand?
• Best rated elements:
– General.Keyword
– Technical.Format
– Technical.Size
• Worst r...
Is this element useful?
Useful for your content
14%
41%
33%
12% 0%
Very Useful Fairly Useful Indifferent
Fairly Useless Co...
Is this element useful?
• Best rated elements:
– General.Identifier
– General.Description
– Technical.Format
• Worst rated...
Are the values clear & appropriate?
Clear Values
9%
50%
37%
4%
0%
Very Clear Fairly Clear Mediocre Fairly Confusing Very C...
Are the values clear & appropriate?
• Best rated elements:
– General.Description
– Rights.Cost
– Format.Size
• Worst rated...
Status of elements
Status Mandatory Recommended Optional
Pre-
evaluation
19 26 12
Post-
evaluation
25 21 11
% +31% -19% -8...
Experiment 2:
Usage Data Analysis
of data produced by subject matter
experts using an annotation tool
to provide metadata
Experiment Details
• Participants: 30
–Experts in Organic Agriculture, Education
• Date: January 2009 – March 2009
• Objec...
Results
• Metadata element: Keyword
Keyword Count % of filled % of total
From 1 to 3 296 48,4 26,8
From 4 to 6 197 32,2 17...
TaxonPathTaxonId Count % of filled % of total
1 or 2 237 52,1 21,4
3 or 4 99 21,8 9
5 or 6 46 10,1 4,2
More than 6 73 16 6...
Intended End User
Role
Count % of filled % of total
1 or 2 385 68,1 34,8
3 or 4 143 25,3 12,9
5 or 6 37 6,6 3,3
TOTAL 565 ...
Mandatory Elements
• Not all mandatory elements were used in the
expected degree
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
10...
Recommended Elements
• Recommended elements present similar problems
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
Resou...
Experiment 3:
Consultation with experts
Discussion on Quality Considerations in
Learning Repositories and Portals
Budapest, 17/9/2010
• Quality Considerations for Learning Portals
and Repositories in Agriculture, Food &
Environment
– 40...
next: online consultation
• e-Conferenence
– From 6/10 to 20/10/2010
– e-Agriculture.org platform (>3.000 experts)
• Topic...
future directions:
towards digital cu-ality?
DCC Curation Lifecycle Model
Description & Preservation Information
Preservation Planning
Community Watch & Participation
Preserve CurateCurate
Create
...
other issues
• Criteria for selecting LOs to curate
• Aggregation level is important for curation
• Ingest resources  Acc...
next step
Retract
Expose
Describe
Discovery
Search
Social Rec.
Alerts
Evaluate
Select
Procure
Gather
Metadata
Enrich Resol...
some thoughts
Dice: http://www.vatsgroup.com/Quality.htm
Stool: http://www.codinghorror.com/blog/archives/000708.html
Preliminary Discussion on a Digital Curation Framework for Learning Repositories
Upcoming SlideShare
Loading in …5
×

Preliminary Discussion on a Digital Curation Framework for Learning Repositories

1,683 views

Published on

Presentation made at SE@M workshop in the context of EC-TEL 2010 in Barcelona, Spain (Tuesday, 28/9/2010)

Published in: Education
  • Be the first to comment

  • Be the first to like this

Preliminary Discussion on a Digital Curation Framework for Learning Repositories

  1. 1. Preliminary Discussion on a Digital Curation Framework for Learning Repositories Nikos Palavitsinis1,2 , Nikos Manouselis1 , Salvador Sanchez-Alonso2 1 Greek Research & Technology Network 2 University of Alcala 4th International Workshop on Search and Exchange of e-le@rning Materials 27-28 September, 2010 Barcelona, Spain
  2. 2. Structure • background – definitions • quality in practice • experiments • towards digital cura-lity • experts – users opinion • e-Conference • conclusions
  3. 3. background & definitions
  4. 4. problem • Quality of the metadata provided by annotators of the resources • Experiences from Organic.Edunet Project • Relevant experiences coming from: – Ochoa & Duval (2006) – Zschocke & Beniest (2009)
  5. 5. background • PhD topic: Metadata Quality Issues in Learning Object Repositories • Behind the words: Trying to find ways of improving the quality of metadata in learning object repositories & portals • Really… Behind the words: Can we introduce mechanisms to ensure/control/assess quality of metadata in learning repositories & portals?
  6. 6. concepts • learning resource/object • from information/audiovisual assets… • …to complete educational programs • collection of learning resources • +metadata=learning repository • online services • learning portals
  7. 7. define: metadata • “Metadata is structured information that describes, explains, locates, or otherwise makes it easier to retrieve, use, or manage an information resource” • Metadata must be always considered in a specific context, i.e. education, research, etc.
  8. 8. define: curation • “Curation includes verification and additions to the existing documentation for objects.” – Documentation = metadata • Digital Curation – …to maintain & add value to digital materials over their entire life-cycle and over time for current and future use
  9. 9. define: quality • Level of excellence; A property or attribute that differentiates a thing or person • Quality is the suitability of procedures, processes and systems in relation to the strategic objectives
  10. 10. “quality intersections” Repository Level Portal Level Resources Level content/course creator end-user portal owner
  11. 11. quality in practice
  12. 12. Learning Resource Lifecycle Retract Expose Describe Discovery Search Social Rec. Alerts Evaluate Select Procure Gather Metadata Enrich Resolution Get Reference or LO Create Integrate Adapt & Reuse Disaggregate Aggregate Modify Use/PlayIntegrate Local delete Van Assche, Vuorikari, 2006
  13. 13. Organic.Edunet • Project that makes digital content on topics of Organic Agriculture & Agroecology available • Using IEEE LOM Metadata – Organic.Edunet AP • Through the Organic.Edunet Portal – With a well-defined Quality Process
  14. 14. Organic.Edunet approach • overall quality strategy • quality guide for the creation of learning resources • reflecting quality in metadata • quality procedures within the repositories (Quality Check, Peer-Review, …) • quality procedures on a portal level (User rating, Featured Resources, …) • quality of educational activities
  15. 15. levels of quality considerations • individual •contribution by an individual (teacher, learner, learning material designer, etc.) • institutional contribution •contribution through an institutional provider (public/ private content provider, educational organisation, another repository)
  16. 16. Learning Resource Lifecycle Retract Expose Describe Discovery Search Social Rec. Alerts Evaluate Select Procure Gather Metadata Enrich Resolution Get Reference or LO Create Integrate Adapt & Reuse Disaggregate Aggregate Modify Use/PlayIntegrate Local delete Van Assche, Vuorikari, 2006
  17. 17. Organic.Edunet AP • With more than 10.000 resources • With 11 repositories • With partners from 10 different countries • We needed a Metadata Application Profile – Multilingual – Ontology support
  18. 18. Development of APDefinition of own requirements Selection of LOM elements Semantics Refinement Multiplicity constraints and values Relationships and dependencies Required extensions Application Profile Binding Evaluation of AP Evaluation phase Results’ analysis AP modifications
  19. 19. Experiment 1: Evaluation Study by subject matter experts to elaborate on the process of describing resources with metadata
  20. 20. Experiment Details • Participants: 20 –Experts in Organic Agriculture, ICT, Education • Date: January 2009 • Object: IEEE LOM AP Elements • Tool: Questionnaire –5 point scale for most questions and a 3- value multiple choice in one of them
  21. 21. Is this element easy to understand? Easy to understand 33% 42% 21% 4% 0% Very Easy Fairly Easy Mediocre Fairly Difficult Very Difficult
  22. 22. Is this element easy to understand? • Best rated elements: – General.Keyword – Technical.Format – Technical.Size • Worst rated elements: – Classification.Taxon – Relation.Resource – Educational.Semantic Density
  23. 23. Is this element useful? Useful for your content 14% 41% 33% 12% 0% Very Useful Fairly Useful Indifferent Fairly Useless Completely Useless
  24. 24. Is this element useful? • Best rated elements: – General.Identifier – General.Description – Technical.Format • Worst rated elements: – Classification.Taxon – Annotation.Entity – Annotation.Date
  25. 25. Are the values clear & appropriate? Clear Values 9% 50% 37% 4% 0% Very Clear Fairly Clear Mediocre Fairly Confusing Very Confusing
  26. 26. Are the values clear & appropriate? • Best rated elements: – General.Description – Rights.Cost – Format.Size • Worst rated elements: – Classification.Taxon – Classification.Purpose – General.Identifier
  27. 27. Status of elements Status Mandatory Recommended Optional Pre- evaluation 19 26 12 Post- evaluation 25 21 11 % +31% -19% -8,3%
  28. 28. Experiment 2: Usage Data Analysis of data produced by subject matter experts using an annotation tool to provide metadata
  29. 29. Experiment Details • Participants: 30 –Experts in Organic Agriculture, Education • Date: January 2009 – March 2009 • Object: Actual usage of IEEE LOM AP • Tool: Log files analysis
  30. 30. Results • Metadata element: Keyword Keyword Count % of filled % of total From 1 to 3 296 48,4 26,8 From 4 to 6 197 32,2 17,8 From 7 to 9 97 15,9 8,8 More than 9 21 3,5 1,9 TOTAL 611 100% 55%
  31. 31. TaxonPathTaxonId Count % of filled % of total 1 or 2 237 52,1 21,4 3 or 4 99 21,8 9 5 or 6 46 10,1 4,2 More than 6 73 16 6,6 TOTAL 455 100% 41,2% Results • Metadata element: TaxonId
  32. 32. Intended End User Role Count % of filled % of total 1 or 2 385 68,1 34,8 3 or 4 143 25,3 12,9 5 or 6 37 6,6 3,3 TOTAL 565 100% 51% Results • Metadata element: End User Role
  33. 33. Mandatory Elements • Not all mandatory elements were used in the expected degree 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Resources Title Description Language Copyright and Other Restrictions Rights Cost
  34. 34. Recommended Elements • Recommended elements present similar problems 0,00% 10,00% 20,00% 30,00% 40,00% 50,00% 60,00% Resources Keyword Learning Resource Type Intended End User Role Contribute Role Contribute Date TaxonPathTaxonId Contribute Entity Context Structure Rights Description Typical Age Range
  35. 35. Experiment 3: Consultation with experts Discussion on Quality Considerations in Learning Repositories and Portals
  36. 36. Budapest, 17/9/2010 • Quality Considerations for Learning Portals and Repositories in Agriculture, Food & Environment – 40 participants – mixed audience – 17/9/2010 – Budapest, Hungary – Organic.Edunet Final Conference
  37. 37. next: online consultation • e-Conferenence – From 6/10 to 20/10/2010 – e-Agriculture.org platform (>3.000 experts) • Topics: – building quality in the resource annotation, curation & preservation life cycles; – quality processes on a repository level; – quality criteria and processes on a web portal level
  38. 38. future directions: towards digital cu-ality?
  39. 39. DCC Curation Lifecycle Model
  40. 40. Description & Preservation Information Preservation Planning Community Watch & Participation Preserve CurateCurate Create & Receive Access, Use & Reuse Access, Use & Reuse IngestAppraise & Select Appraise & Select Transfor m Preservatio n Action Preservatio n Action StoreConceptualizeConceptualize Dispose MigrateMigrate Reappraise LR & Metadata LR & Metadata -Quality of metadata provided by subject matter experts -Variability in quality amongst the metadata provided -It requires skilled, professional curators -Human resources do not scale well with many resources -Lack of a unanimous definition of a learning resource -How can you preserve anything without defining it first?
  41. 41. other issues • Criteria for selecting LOs to curate • Aggregation level is important for curation • Ingest resources  Access rights? Owner?
  42. 42. next step Retract Expose Describe Discovery Search Social Rec. Alerts Evaluate Select Procure Gather Metadata Enrich Resolution Get Reference or LO Create Integrate Adapt & Reuse Disaggregate Aggregate Modify Use/PlayIntegrate Local delete Van Assche, Vuorikari, 2006
  43. 43. some thoughts Dice: http://www.vatsgroup.com/Quality.htm Stool: http://www.codinghorror.com/blog/archives/000708.html

×