Generating educational assessment items from Linked Open Data


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Generating educational assessment items from Linked Open Data

  1. 1. Generating educational assessment items from Linked Open Data The case of DBpedia Muriel Foulonneau [email_address]
  2. 2. “ To Really Learn, Quit Studying and Take a Test” (NYT, Jan, 2011) <ul><li>Formative assessment </li></ul><ul><li>Self-assessment </li></ul><ul><li>Items are expensive </li></ul><ul><li>Creating, reusing, sharing test items </li></ul>05/2011 ESWC 2011
  3. 3. Why generating items? <ul><li>Security issue </li></ul><ul><li>Adding variability to an item </li></ul><ul><li>no expected variation of the construct </li></ul><ul><li>Model-based learning </li></ul><ul><li>Generating items from knowledge represented as a model </li></ul><ul><li>the construct is modified for each item </li></ul>27/05/11 Presentation Tudor
  4. 4. Assumption on model-based learning <ul><li>INTERESTING BECAUSE </li></ul><ul><li>Can enable adaptive learning paths </li></ul><ul><li>Independent from particular representations of learning resources </li></ul><ul><li>CONSTRAINTS </li></ul><ul><li>A domain model must exist </li></ul><ul><li>Can enable adaptive learning paths </li></ul><ul><li>Bring experts together to design a model of what learners should learn </li></ul><ul><li>LIMITATIONS </li></ul><ul><li>Experts are difficult to mobilize for a long modeling exercise </li></ul><ul><li>What about specialized /professional knowledge? </li></ul><ul><li>How to ensure the evolution of the model? </li></ul>27/05/11 Presentation Tudor
  5. 5. The LoD Cloud as a source of knowledge <ul><li>Existing data sources </li></ul><ul><li>no need to gather experts </li></ul><ul><li>Including knowledge which is not well codified in curricula </li></ul><ul><li> Knowledge gathered from experts as well as non experts </li></ul><ul><li>Many datasets added or modified all the time </li></ul><ul><li>Can reflect evolution of the knowledge </li></ul>27/05/11 Presentation Tudor
  6. 6. Using LoD for model-based learning 27/05/11 Presentation Tudor Limitations of model-based learning LoD as a source of knowledge <ul><li>Experts are difficult to mobilize for a long modeling exercise </li></ul><ul><li>Existing data sources </li></ul><ul><li>No need to gather experts </li></ul><ul><li>What about specialized /professional knowledge? </li></ul><ul><li>Including knowledge which is not well codified in curricula </li></ul><ul><li>Knowledge gathered from experts as well as non experts </li></ul><ul><li>How to ensure the evolution of the model? </li></ul><ul><li>Many datasets added or modified all the time </li></ul><ul><li>Can reflect evolution of the knowledge </li></ul>
  7. 7. Objectives of the experimentation <ul><li>Are there limitations to the use of Linked open Data as knowledge model for learning ? </li></ul><ul><ul><li>Is this feasible? </li></ul></ul><ul><ul><li>Are the datasets relevant? </li></ul></ul><ul><ul><li>How much quality control is needed? </li></ul></ul><ul><li>Test on factual knowledge for simple choice items </li></ul>27/05/11 Presentation Tudor
  8. 8. Semi-automatic item generation <ul><li>Manual definition of an item template </li></ul><ul><li>Automatic generation of variables </li></ul>27/05/11 ESWC 2011 Stem variables options key Auxiliary information
  9. 9. Existing strategies <ul><li>Algorithms </li></ul><ul><ul><li>X: Value range: 3 to 18 by 3 </li></ul></ul><ul><li>Natural language processing </li></ul><ul><ul><li>vocabulary questions and cloze questions </li></ul></ul><ul><li>Structured datasets </li></ul><ul><ul><li>Vocabulary questions from the WordNet dataset </li></ul></ul><ul><li>Model extraction then question generation </li></ul><ul><ul><li>From natural language (or model creation by experts) </li></ul></ul><ul><li>Mostly used in mathematics and scientific subjects </li></ul><ul><ul><li>where algorithmic definition of variables is easier </li></ul></ul><ul><li>And for L2 learning </li></ul><ul><ul><li>Challenge to generate other types of variables </li></ul></ul><ul><ul><ul><li>Additional information, historical knowledge, feedback… </li></ul></ul></ul>27/05/11 Presentation Tudor
  10. 10. The QTI item generation process 27/05/11 Presentation Tudor
  11. 11. QTI Item template <ul><li>IMS Question & Test Interoperability Specification </li></ul><ul><li>XML serialization using JSON templates </li></ul>27/05/11 ESWC 2011 <choiceInteraction responseIdentifier=&quot;RESPONSE&quot; shuffle=&quot;false&quot; maxChoices=&quot;1&quot;> <prompt>What is the capital of {prompt}?</prompt> <simpleChoice identifier=&quot;{responseCode1}&quot;>{responseOption1}</simpleChoice> <simpleChoice identifier=&quot;{responseCode2}&quot;>{responseOption2}</simpleChoice> <simpleChoice identifier=&quot;{responseCode3}&quot;>{responseOption3}</simpleChoice> </choiceInteraction>
  12. 12. Get the knowledge from LoD 27/05/11 ESWC 2011 SPARQL query to generate capitals in Europe Never possible to generate an item from a single triple because of constraint to find appropriate labels Label SELECT ?country ?capital WHERE { ?c <> ?country . ?c <> ?capital } LIMIT 30
  13. 13. Generating item distractors <ul><li>i.e., incorrect answer options </li></ul><ul><li>Strategies </li></ul><ul><li>- Instances of the same class </li></ul><ul><li>Creation of a variable store </li></ul><ul><ul><li>Random selection of distractors </li></ul></ul><ul><li>Next step: Attribute-based resource similarity (can be instances of a different class) </li></ul><ul><li>=> use of semantic recommender system </li></ul>27/05/11 ESWC 2011
  14. 14. Item data dictionary 27/05/11 ESWC 2011
  15. 15. Generation of the QTI-XML item 27/05/11 ESWC 2011
  16. 16. Publication on the TAO platform <ul><li>TAO is an open source e-assessment platform based on semantic technologies. </li></ul><ul><li>Used for diagnostic, formative, large-scale assessment, including national school monitoring, OECD PISA/PIIAC surveys, competence assessment for unemployed …. </li></ul><ul><li>Supports imports </li></ul><ul><li>of IMS-QTI items </li></ul>27/05/11 Presentation Tudor
  17. 17. Different types of questions <ul><li>Q1: queries uncontrolled datasets </li></ul><ul><li>Q2: queries revised ontology </li></ul><ul><li>Q3: queries in History </li></ul><ul><li>Q4: queries a linked data set to add item feedback </li></ul><ul><li>Q5: queries medical information </li></ul>27/05/11 ESWC 2011
  18. 18. Q1: What is the capital of { Azerbaijan }? <ul><li>Infobox dataset </li></ul><ul><li>3 were not generated for a country (Neuenburg am Rhein, Wain, and Offenburg) </li></ul><ul><li>“ Managua right|20px” </li></ul><ul><li>Two distinct capitals were found for Swaziland (Mbabane, the administrative capital and Lobamba, the royal and legislative capital) </li></ul>27/05/11 ESWC 2011
  19. 19. Q2: Which country is represented by this flag ? <ul><li>Use of FOAF and YAGO </li></ul><ul><li>Transactional closures </li></ul><ul><li><> <> </li></ul><ul><li>Out of 30 items including pictures of flags used as stimuli, 6 URIs did not resolve to a usable picture (HTTP 404 errors or encoding problem). </li></ul>27/05/11 ESWC 2011
  20. 20. Q3: Who succeeded to { Charles VII the Victorious } as ruler of France ? <ul><li>YAGO ontology </li></ul><ul><li>1 was incorrect ( The three Musketeers ) </li></ul><ul><li>Multiple labels for the same king </li></ul><ul><li>Louis IX , Saint Louis , Saint Louis IX </li></ul><ul><li>One item generated with options having inconsistent naming: </li></ul><ul><li>Charles VII the Victorious , Charles 09 Of France , Louis VII </li></ul>27/05/11 ESWC 2011
  21. 21. Q4: What is the capital of { Argentina }? With feedback <ul><li>Uses the linkage of the DBpedia dataset with the Flickr wrapper dataset </li></ul><ul><li>The Flickr wrapper data source was unavailable </li></ul><ul><li>No IPR information </li></ul>27/05/11 ESWC 2011
  22. 22. Q5: Which category does { Asthma } belong to? <ul><li>Retrieves diseases and their categories </li></ul><ul><li>SKOS and Dublin Core, Inforbox dataset for labels </li></ul><ul><li>SKOS concepts are not related to a specific SKOS scheme </li></ul><ul><li>Categories retrieved range from Skeletal disorders to childhood . </li></ul><ul><ul><li>=> the correct answer to the question on Obesity is childhood . </li></ul></ul>27/05/11 ESWC 2011
  23. 23. Data quality challenges <ul><li>From Q1, 53,33% were directly usable </li></ul><ul><li>neither a defective prompt nor a defective correct answer nor a defective distractor . </li></ul><ul><li>Benchmark from unstructured content between 3,5% and 21%. </li></ul><ul><li>Issues </li></ul><ul><ul><li>Ontology issue </li></ul></ul><ul><ul><li>Labels </li></ul></ul><ul><ul><li>Inaccurate statements </li></ul></ul><ul><ul><li>Data linkage (resolvable URIs) </li></ul></ul><ul><ul><li>Missing inferences </li></ul></ul>27/05/11 ESWC 2011 Chance that an item will have a defective distractor =
  24. 24. Data selection <ul><li>Item difficulty </li></ul><ul><li>can change even with variables not related to the construct (cognitive issues) </li></ul><ul><li>Can change according to the distractors </li></ul><ul><li>=> need to establish a framework to assess the difficulty of the construct AND of the item in general (including the relevance of the distractors for instance) </li></ul><ul><ul><li>Psychometric model: what do we know about previous test takers? What can we infer from their performance? </li></ul></ul><ul><ul><li>Ad hoc model: can a </li></ul></ul>27/05/11 ESWC 2011
  25. 25. Future work <ul><li>Assessing models on Linked Open Data as a source of knowledge for supporting formative assessment and the learning process </li></ul><ul><li>Improving the selection of distractors by integrating dedicated similarity approach (from a semantic recommender system) </li></ul><ul><li>A wider variety of assessment item models </li></ul><ul><li>Authoring interface for item templates </li></ul>27/05/11 ESWC 2011