Your SlideShare is downloading. ×
0
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Generating educational assessment items from Linked Open Data
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Generating educational assessment items from Linked Open Data

500

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
500
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Generating educational assessment items from Linked Open Data The case of DBpedia Muriel Foulonneau [email_address]
  • 2. “ To Really Learn, Quit Studying and Take a Test” (NYT, Jan, 2011)
    • Formative assessment
    • Self-assessment
    • Items are expensive
    • Creating, reusing, sharing test items
    05/2011 ESWC 2011
  • 3. Why generating items?
    • Security issue
    • Adding variability to an item
    • no expected variation of the construct
    • Model-based learning
    • Generating items from knowledge represented as a model
    • the construct is modified for each item
    27/05/11 Presentation Tudor
  • 4. Assumption on model-based learning
    • INTERESTING BECAUSE
    • Can enable adaptive learning paths
    • Independent from particular representations of learning resources
    • CONSTRAINTS
    • A domain model must exist
    • Can enable adaptive learning paths
    • Bring experts together to design a model of what learners should learn
    • LIMITATIONS
    • Experts are difficult to mobilize for a long modeling exercise
    • What about specialized /professional knowledge?
    • How to ensure the evolution of the model?
    27/05/11 Presentation Tudor
  • 5. The LoD Cloud as a source of knowledge
    • Existing data sources
    • no need to gather experts
    • Including knowledge which is not well codified in curricula
    • Knowledge gathered from experts as well as non experts
    • Many datasets added or modified all the time
    • Can reflect evolution of the knowledge
    27/05/11 Presentation Tudor
  • 6. Using LoD for model-based learning 27/05/11 Presentation Tudor Limitations of model-based learning LoD as a source of knowledge
    • Experts are difficult to mobilize for a long modeling exercise
    • Existing data sources
    • No need to gather experts
    • What about specialized /professional knowledge?
    • Including knowledge which is not well codified in curricula
    • Knowledge gathered from experts as well as non experts
    • How to ensure the evolution of the model?
    • Many datasets added or modified all the time
    • Can reflect evolution of the knowledge
  • 7. Objectives of the experimentation
    • Are there limitations to the use of Linked open Data as knowledge model for learning ?
      • Is this feasible?
      • Are the datasets relevant?
      • How much quality control is needed?
    • Test on factual knowledge for simple choice items
    27/05/11 Presentation Tudor
  • 8. Semi-automatic item generation
    • Manual definition of an item template
    • Automatic generation of variables
    27/05/11 ESWC 2011 Stem variables options key Auxiliary information
  • 9. Existing strategies
    • Algorithms
      • X: Value range: 3 to 18 by 3
    • Natural language processing
      • vocabulary questions and cloze questions
    • Structured datasets
      • Vocabulary questions from the WordNet dataset
    • Model extraction then question generation
      • From natural language (or model creation by experts)
    • Mostly used in mathematics and scientific subjects
      • where algorithmic definition of variables is easier
    • And for L2 learning
      • Challenge to generate other types of variables
        • Additional information, historical knowledge, feedback…
    27/05/11 Presentation Tudor
  • 10. The QTI item generation process 27/05/11 Presentation Tudor
  • 11. QTI Item template
    • IMS Question & Test Interoperability Specification
    • XML serialization using JSON templates
    27/05/11 ESWC 2011 <choiceInteraction responseIdentifier=&quot;RESPONSE&quot; shuffle=&quot;false&quot; maxChoices=&quot;1&quot;> <prompt>What is the capital of {prompt}?</prompt> <simpleChoice identifier=&quot;{responseCode1}&quot;>{responseOption1}</simpleChoice> <simpleChoice identifier=&quot;{responseCode2}&quot;>{responseOption2}</simpleChoice> <simpleChoice identifier=&quot;{responseCode3}&quot;>{responseOption3}</simpleChoice> </choiceInteraction>
  • 12. Get the knowledge from LoD 27/05/11 ESWC 2011 SPARQL query to generate capitals in Europe Never possible to generate an item from a single triple because of constraint to find appropriate labels Label SELECT ?country ?capital WHERE { ?c <http://dbpedia.org/property/commonName> ?country . ?c <http://dbpedia.org/property/capital> ?capital } LIMIT 30
  • 13. Generating item distractors
    • i.e., incorrect answer options
    • Strategies
    • - Instances of the same class
    • Creation of a variable store
      • Random selection of distractors
    • Next step: Attribute-based resource similarity (can be instances of a different class)
    • => use of semantic recommender system
    27/05/11 ESWC 2011
  • 14. Item data dictionary 27/05/11 ESWC 2011
  • 15. Generation of the QTI-XML item 27/05/11 ESWC 2011
  • 16. Publication on the TAO platform
    • TAO is an open source e-assessment platform based on semantic technologies.
    • Used for diagnostic, formative, large-scale assessment, including national school monitoring, OECD PISA/PIIAC surveys, competence assessment for unemployed ….
    • Supports imports
    • of IMS-QTI items
    27/05/11 Presentation Tudor
  • 17. Different types of questions
    • Q1: queries uncontrolled datasets
    • Q2: queries revised ontology
    • Q3: queries in History
    • Q4: queries a linked data set to add item feedback
    • Q5: queries medical information
    27/05/11 ESWC 2011
  • 18. Q1: What is the capital of { Azerbaijan }?
    • Infobox dataset
    • 3 were not generated for a country (Neuenburg am Rhein, Wain, and Offenburg)
    • “ Managua right|20px”
    • Two distinct capitals were found for Swaziland (Mbabane, the administrative capital and Lobamba, the royal and legislative capital)
    27/05/11 ESWC 2011
  • 19. Q2: Which country is represented by this flag ?
    • Use of FOAF and YAGO
    • Transactional closures
    • <http://dbpedia.org/class/yago/EuropeanCountries> < http://dbpedia.org/class/yago/Country108544813>
    • Out of 30 items including pictures of flags used as stimuli, 6 URIs did not resolve to a usable picture (HTTP 404 errors or encoding problem).
    27/05/11 ESWC 2011
  • 20. Q3: Who succeeded to { Charles VII the Victorious } as ruler of France ?
    • YAGO ontology
    • 1 was incorrect ( The three Musketeers )
    • Multiple labels for the same king
    • Louis IX , Saint Louis , Saint Louis IX
    • One item generated with options having inconsistent naming:
    • Charles VII the Victorious , Charles 09 Of France , Louis VII
    27/05/11 ESWC 2011
  • 21. Q4: What is the capital of { Argentina }? With feedback
    • Uses the linkage of the DBpedia dataset with the Flickr wrapper dataset
    • The Flickr wrapper data source was unavailable
    • No IPR information
    27/05/11 ESWC 2011
  • 22. Q5: Which category does { Asthma } belong to?
    • Retrieves diseases and their categories
    • SKOS and Dublin Core, Inforbox dataset for labels
    • SKOS concepts are not related to a specific SKOS scheme
    • Categories retrieved range from Skeletal disorders to childhood .
      • => the correct answer to the question on Obesity is childhood .
    27/05/11 ESWC 2011
  • 23. Data quality challenges
    • From Q1, 53,33% were directly usable
    • neither a defective prompt nor a defective correct answer nor a defective distractor .
    • Benchmark from unstructured content between 3,5% and 21%.
    • Issues
      • Ontology issue
      • Labels
      • Inaccurate statements
      • Data linkage (resolvable URIs)
      • Missing inferences
    27/05/11 ESWC 2011 Chance that an item will have a defective distractor =
  • 24. Data selection
    • Item difficulty
    • can change even with variables not related to the construct (cognitive issues)
    • Can change according to the distractors
    • => need to establish a framework to assess the difficulty of the construct AND of the item in general (including the relevance of the distractors for instance)
      • Psychometric model: what do we know about previous test takers? What can we infer from their performance?
      • Ad hoc model: can a
    27/05/11 ESWC 2011
  • 25. Future work
    • Assessing models on Linked Open Data as a source of knowledge for supporting formative assessment and the learning process
    • Improving the selection of distractors by integrating dedicated similarity approach (from a semantic recommender system)
    • A wider variety of assessment item models
    • Authoring interface for item templates
    27/05/11 ESWC 2011

×