Towards automatic generation of  e-assessment using semantic web technologies   *   † <ul><li>Marija Cubric,  [email_addre...
Abstract <ul><li>Semantic Web  technologies have been increasingly used as a tool for generating, organizing and personali...
Content <ul><li>Terminology ……….. Research Problem  ………... ……….. Related Work……….. Methodology ……….. </li></ul><ul><li>Ou...
E-learning <ul><li>… is  the use of  innovative   technologies  and  learning   models  to transform the way individuals a...
E-learning . Information Systems Education
This research . Education Assessment Information Systems Information Technology
Objective tests and CAA Objective tests require a user to choose or provide a response to a question whose correct answer ...
Motivation <ul><li>Students ’ needs </li></ul><ul><ul><li>Prompt and frequent feedback (NSS) </li></ul></ul><ul><ul><li>Co...
Research  Problem Develop algorithms and tools for automated generation of objective tests, leveraging on increasing body ...
Semantic Web Primer <ul><li>Semantic Web  -  “The Web of data with meaning in the sense that a computer program can learn ...
An Example Ontology from Business Domain (e-commerce)
E-commerce ontology (class view) classes properties instances annotations
Related Work-  Ontologies in Learning <ul><li>Ontological formalization of learning content, instructional processes and l...
<ul><li>Chung, Niemi & Bewley (2003): assessment authoring  support system  for  “aiding assessment authors to populate th...
<ul><li>Papasalouros A., Kotis K., Kanaris K. (2008): defined 11 ontology-based strategies for automatic generation of ‘di...
Our Contribution <ul><li>Focus on  ontology -based  automatic  generation of assessment of an  arbitrary  knowledge domain...
@code.google.com opensemcq.org
Design as a Methodology <ul><li>Design  can be thought of as a mapping from function space - a functional requirement cons...
Design Science Research Method <ul><li>Design science research  </li></ul><ul><li>is most effective when its practitioners...
Philosophical assumptions of Design Science Research <ul><li>(source: DESRIST.org) </li></ul>
Protégé Plugin(“suggestions”)  <ul><li>Features </li></ul><ul><li>Variety of questions generated based on different plugga...
MCQ Ontology (IMS, 2002; CAA,2010) CAA Centre Resources (2002) http://www.caacentre.ac.uk/resources/ (accessed 12/5/2010) ...
Generation Process <ul><li>Input  : Any domain ontology </li></ul><ul><li>Output : MCQ ontology populated with instances <...
Stems, question templates & semantic interpretation <ul><li>Stems using ontology annotations </li></ul><ul><li>Stems gener...
Strategies for generating distracters <ul><li>Based on ‘text similarity measures’ - the more similar distracters are, the ...
Which of the following definitions describe the concept <A> : (x,y,…) Which of the following definitions describes the con...
Which one of the following examples demonstrates the concept <A>:(a,b,c,...) Which one of the following examples demonstra...
Read the paragraph <x> and decide which one of the following concepts generalize the concept defined by <x>: (A,B,C,D,E,F,...
Read the paragraph <x> and decide which one of the following concepts it defines: (A,B,…) Read the paragraph and decide wh...
Which one of the following response pairs relates in the same way as <a> and <b> in the relation <R>? :  (c,d),(c,b),(a,d)...
Analyze the text <x> and decide which one of the following words is a correct replacement for the blank space in <x>: (B,C...
Implementation: opensemcq: generate test
Implementation: opensemcq: view test
Implementation: opensemcq: evaluate test
Benefits <ul><li>Adding annotations  </li></ul><ul><ul><li>Four more question types  and three more strategies  </li></ul>...
Issues and Limitations <ul><li>Lack of annotated (test) ontologies </li></ul><ul><li>Lack of questions templates for asses...
Current Work <ul><li>Invitation for evaluation to be sent to authors of ontologies used for testing  </li></ul><ul><li>Fur...
Future Work <ul><li>Integrate text similarity with more advanced ontological approaches to generation of distracters </li>...
Upcoming SlideShare
Loading in …5
×

Introduction to OpenSemcq

1,440 views

Published on

Semantic Web technologies have been increasingly used as a tool for generating, organizing and personalizing e-learning content.  In this presentation we will discuss and demonstrate an innovative approach to automated generation of computer-assisted assessment (CAA) from Semantic Web–based domain ontologies.  The primary application domain of this work is in the automated assessment,  and in particular, the development of intelligent CAA systems and question banks, but the ideas can be further generalized in the context of ontology engineering and evaluation. Prototype is implemented and available online at http://www.opensemcq.org

Published in: Education
  • Be the first to comment

  • Be the first to like this

Introduction to OpenSemcq

  1. 1. Towards automatic generation of e-assessment using semantic web technologies * † <ul><li>Marija Cubric, [email_address] </li></ul><ul><li>Milorad Tosic, [email_address] </li></ul>*Presentation based on Cubric,M.&Tosic,M.(2010/2011) ‘Towards automatic generation of e- assessment using semantic web technologies; International Conference on CAA, University of Southampton, July 2010; Also to appear in International Journal of E-assessment 2011 † University of Hertfordshire Business School research Seminar 2/11/2011
  2. 2. Abstract <ul><li>Semantic Web technologies have been increasingly used as a tool for generating, organizing and personalizing e-learning content.  In this presentation we will discuss and demonstrate an innovative approach to automated generation of computer-assisted assessment (CAA) from Semantic Web–based domain ontologies .  The primary application domain of this work is in the automated assessment,  and in particular, the development of intelligent CAA systems and question banks, but the ideas can be further generalized in the context of ontology engineering and evaluation. </li></ul>
  3. 3. Content <ul><li>Terminology ……….. Research Problem  ………... ……….. Related Work……….. Methodology ……….. </li></ul><ul><li>Our Contribution ……….. Implementation ………. </li></ul><ul><li>Benefits and Limitations ……….. Future Work …..……….. Other research ……….. Q&A </li></ul>
  4. 4. E-learning <ul><li>… is the use of innovative technologies and learning models to transform the way individuals and organisations acquire new skills and access knowledge (Moeng, 2004) </li></ul>. … comprises all forms of electronically supported learning and teaching. The information and communication systems , whether networked learning or not, serve as specific media to implement the learning process (Tavangarian et al.,2004) … a broad combination of processes, content, and infrastructure to use computers and networks to scale and/or improve one or more significant parts of a learning value chain , including management and delivery (Aldrich, 2004) Adrich, C. (2004) Simulations and the Future of Learning. San Francisco: Pfeiffer, p.240 Moeng, B. (2004). IBM tackles learning in the workplace. IBM Management Development Solutions , Nov 8, 2004. Tavangarian D., Leypold M., Nölting K., Röser M.,(2004). Is e-learning the Solution for Individual Learning? Journal of e-learning
  5. 5. E-learning . Information Systems Education
  6. 6. This research . Education Assessment Information Systems Information Technology
  7. 7. Objective tests and CAA Objective tests require a user to choose or provide a response to a question whose correct answer is predetermined. Such a question might require a student to : - select a solution from a set of choices (MCQ, true-false, matching) - identify an object or position (graphical hotspot) or - supply brief numeric or text responses (text input). Because the correct answers to objective test questions are pre-determined, they are well suited to the many forms of CAA (Computer-assisted assessment) or e-assessment ( CAA Centre Resources )
  8. 8. Motivation <ul><li>Students ’ needs </li></ul><ul><ul><li>Prompt and frequent feedback (NSS) </li></ul></ul><ul><ul><li>Constant simulation, fun, new stuff (Bean, 2010) </li></ul></ul><ul><li>Universities ’ agendas </li></ul><ul><ul><li>Increase opportunities for flexible learning ( “anytime anywhere”) </li></ul></ul><ul><ul><li>Cost reduction ( “more for less”) </li></ul></ul><ul><li>Good teaching practice ( Chickering and Gamson, 1987) </li></ul><ul><ul><li>Encourages active learning </li></ul></ul><ul><ul><li>Gives prompt feedback </li></ul></ul><ul><ul><li>Emphasises time on task </li></ul></ul><ul><ul><li>Respects diverse talents and ways of learning </li></ul></ul><ul><li>Teachers ’ needs </li></ul><ul><ul><li>Questions are difficult and time-consuming to write </li></ul></ul><ul><ul><li>Professional writers plan one hour or more per item (Van Hoozer quoted in Collins, 2006) </li></ul></ul>Bean M (2010) JISC 2010 Opening keynote: The learning journey: From informal to formal Chickering A.W. and Gamson Z.F (1987) &quot;Seven principles for good practice in undergraduate education. The American Association for Higher Education Bulletin, March 1987 Van Hoozer H, et al. The teaching process: theory and practice in nursing. Norwalk, CT: Appleton-Century-Crofts, 1987:279-280
  9. 9. Research Problem Develop algorithms and tools for automated generation of objective tests, leveraging on increasing body of domain ontologies (“knowledge conceptualisations”) and other advances in the area of semantic web. . Information Systems Education Information Technology Assessment
  10. 10. Semantic Web Primer <ul><li>Semantic Web - “The Web of data with meaning in the sense that a computer program can learn enough about what the data means to process it” (&quot;Weaving the Web&quot; by Tim Berners-Lee, 1999) </li></ul><ul><li>Ontology - A formal specification of a conceptualization of a knowledge domain in terms of classes, instances, relations, properties and annotations relevant to modeling of the domain (Gruber, 1993) </li></ul><ul><li>W3C standards define specific formalisms for encoding ontologies, such as RDF (Resource Description Framework) and OWL (Ontology Web Language) </li></ul>http://www.w3.org/ Gruber T. R. (1993), A Translation Approach to Portable Ontology Specifications. Knowledge Acquisition , 5(2):199-220
  11. 11.
  12. 12. An Example Ontology from Business Domain (e-commerce)
  13. 13. E-commerce ontology (class view) classes properties instances annotations
  14. 14. Related Work- Ontologies in Learning <ul><li>Ontological formalization of learning content, instructional processes and learning designs for the purpose of automated discovery, composition and presentation of knowledge </li></ul><ul><ul><li>Holohan & Pahl , 2003; Sicilia & Barriocanal, 2005; Knight, Gaševic & Richards,2006 </li></ul></ul><ul><li>Related data mining techniques for the discovery of ontologies from arbitrary text corpora </li></ul><ul><ul><li>Montoyo, et al. 2005 </li></ul></ul><ul><li>Ontologies for fostering collaborative learning (Allert, Markkanen, Richter 2006) </li></ul><ul><ul><li>Collaborative concept mapping </li></ul></ul><ul><ul><li>Using ontologies to organize or annotate shared artifacts </li></ul></ul><ul><ul><li>Using existing ontologies to carry out an activity </li></ul></ul><ul><ul><li>Collaborative ontology development as part of an overarching task </li></ul></ul>
  15. 15. <ul><li>Chung, Niemi & Bewley (2003): assessment authoring support system for “aiding assessment authors to populate the assessment ontology with values specific to the users’ purposes” </li></ul><ul><li>Holohan, et al, (2005): semi-automatic generation of simple learning objects such as slide shows and objective tests, in the context of an adaptive learning environment </li></ul><ul><li>Holohan, et al, (2006) & Zualkernan et al, (2009) dynamic problem generation, using domain-specific algorithms : the examples considered was from the domain of relational databases and software engineering and the resulting problems were database queries and questions related to the UML models of software artifacts </li></ul><ul><li>Mitkov, R. et al, (2006): NLP techniques such as shallow parsing, term extraction, sentence transformation and computation of semantic distance, aided by WordNet </li></ul>Related Work- Automated Generation of E-assessment
  16. 16. <ul><li>Papasalouros A., Kotis K., Kanaris K. (2008): defined 11 ontology-based strategies for automatic generation of ‘distractors’ for Multiple Choice Questions (MCQ), from arbitrary knowledge domains. </li></ul><ul><li>The generation is based on the basic meta-ontology relations between a ‘class’ and ‘individual’, as well as between two individuals (binary ‘role’) </li></ul>Related Work- Automated Generation of E-assessment (cont) A(a) and not A(b) R(a,b) and not R(c,b) A<B and not D<B
  17. 17. Our Contribution <ul><li>Focus on ontology -based automatic generation of assessment of an arbitrary knowledge domain </li></ul><ul><li>Developed Protégé-based plugin (Tosic&Cubric,2009) </li></ul><ul><ul><li>Extended and optimized strategies introduced by Papasalouros et al (2008) </li></ul></ul><ul><li>In Cubric & Tosic (2010) : </li></ul><ul><li>Added annotations to the meta-ontology used for question generation </li></ul><ul><li>Added question templates for generating question text </li></ul><ul><li>Added semantic interpretation to the question templates in terms of learning theory </li></ul><ul><ul><ul><li>- different types of questions can be generated based on different semantic interpretation </li></ul></ul></ul><ul><li>Most recent: : Developed open source prototype for evaluation purposes ( www.opensemcq.org ) </li></ul>
  18. 18. @code.google.com opensemcq.org
  19. 19. Design as a Methodology <ul><li>Design can be thought of as a mapping from function space - a functional requirement constituting a point in this multidimensional space - to attribute space, where an artifact satisfying the mapping constitutes a point in that space (Takeda, et al, 1990). Design then, is knowledge in the form of techniques and methods for performing this mapping – the know-how for implementing an artifact that satisfies a set of functional requirements. (DESRIST.org) </li></ul><ul><li>Design Science Research – “learning through building - artifact construction” and “ using design as a research method or technique ” (DESRIST.org) </li></ul><ul><li>Subset of Design Research (research into or about design) whereas DSR is research using design as a research method or technique </li></ul>Takeda, H., Veerkamp, P., Tomiyama, T., Yoshikawam, H. (1990). &quot;Modeling Design Processes.&quot; AI MagazineWinter: 37-48. Information Systems Information Technology
  20. 20. Design Science Research Method <ul><li>Design science research </li></ul><ul><li>is most effective when its practitioners shift between pragmatic </li></ul><ul><li>and critical realist perspectives, guided by a pragmatic assessment of progress in the design cycle( Bunge,1984) </li></ul>Bunge, M. (1984). Philosophical Inputs and Outputs of Technology. History and Philosophy of Technology. G. Bugliarello and D. Donner. Urbana, IL, University of Illinois Press : 263-281. Knowledge gained from construction
  21. 21. Philosophical assumptions of Design Science Research <ul><li>(source: DESRIST.org) </li></ul>
  22. 22. Protégé Plugin(“suggestions”) <ul><li>Features </li></ul><ul><li>Variety of questions generated based on different pluggable ontologies (19 test ontologies) </li></ul><ul><li>Optimized existing 11 (Papasalouros et al, 2008) and added 4 new strategies for generating distracters </li></ul><ul><li>Adjustable number of questions and types of strategies used. </li></ul><ul><li>Adjustable difficulty level of the generated question set. </li></ul><ul><li>Architecture </li></ul>
  23. 23. MCQ Ontology (IMS, 2002; CAA,2010) CAA Centre Resources (2002) http://www.caacentre.ac.uk/resources/ (accessed 12/5/2010) IMS Global Learning Consortium, Inc. IMS Question and Test Interoperability Version 2.1 Public Draft n m * *
  24. 24. Generation Process <ul><li>Input : Any domain ontology </li></ul><ul><li>Output : MCQ ontology populated with instances </li></ul><ul><li>of MultipleChoiceQuestion class </li></ul>(any) domain ontology target (MCQ) ontology question template
  25. 25. Stems, question templates & semantic interpretation <ul><li>Stems using ontology annotations </li></ul><ul><li>Stems generated from pre-defined parameterized question templates </li></ul><ul><li>Templates based on different levels of Bloom’s (1956) taxonomy of cognitive domains </li></ul>
  26. 26. Strategies for generating distracters <ul><li>Based on ‘text similarity measures’ - the more similar distracters are, the more difficult question becomes (Mitkov et al, 2008) </li></ul>Mitkov, R. et al, 2006. A computer-aided environment for generating multiple-choice test items. Natural Language Engineering, Vol. 12, No. 2, pp. 177-194.
  27. 27. Which of the following definitions describe the concept <A> : (x,y,…) Which of the following definitions describes the concept Quantitative value?   a) This is a conceptual entity that holds together all aspects of the n-ary relation hasWarrantyPromise. A ________ is an entity representing the duration and scope of services that will be provided to a customer free of charge in case of a defect or malfunction of theProductOrService. b) An instance of this class is an actual _________ for a quantitative property of a product. This instance is usually characterized by a minimal value, a maximal value, and a unit of measurement. c) A _____ is a numerical interval that represents the range of a certain quantitative Product Or Service Property in terms of the lower and upper bounds for a particular Product Or Service . It is to be interpreted in combination with the respective unit of measurement . d) An instance of this class is an actual _______for a quantitative property of a product. This instance is usually characterized by a minimal value, a maximal value, and a unit of measurement.Examples: The intervals between 10.0 and 25.4 kilogramms&quot; or &quot;10.2 and 15.5 milimeters&quot;. e) A __________ is a predefined value for a product characteristic. Examples: the color &quot;green&quot; or the power cord plug type &quot;US&quot;; the garment sizes &quot;S&quot;, &quot;M&quot;, &quot;L&quot;, and &quot;XL”. Strategies for distractors: Most similar annotation Class A x Annotation Class B y High Similarity Annotation Annotation Annotation ✔ ✖
  28. 28. Which one of the following examples demonstrates the concept <A>:(a,b,c,...) Which one of the following examples demonstrates the concept Payment method credit card?   a) Check in advance b) Direct debit c) Pay Swarm d) JCB e) Google Checkout Strategies for distractors : Instance of a Parent Class, Instance of a Sibling Class … a Class A ✔ instance instance b Class B ✖ instance instance c instance instance Class C ✖
  29. 29. Read the paragraph <x> and decide which one of the following concepts generalize the concept defined by <x>: (A,B,C,D,E,F,…) Read the paragraph and decide which one of the following concepts generalize the concept defined by it:   &quot;A _____ is a conceptual entity that specifies the additional costs asked for settling the payment after accepting a given Offering using a particular Payment Method . A ____ is characterized by (1) a monetary amount per order specified as a literal value of type float in combination with a Currency , (2) the payment method , and (3) a whether this charge includes local sales taxes , namely VAT” a) Product or service b) Actual product or service instance c) Business function d) Price specification e) Location of sales or service provisioning Strategies for distractors : Sibling concept, Child Concept, Parent’s sibling, Grandparent… Class B Class A x ✔ ✖ Class E Class F ✖ ✖ Class C Class D ✖ ✖
  30. 30. Read the paragraph <x> and decide which one of the following concepts it defines: (A,B,…) Read the paragraph and decide which one of the following concepts it defines: &quot;The ________ represents types of services that will be provided free of charge by the vendor or manufacturer in the case of a defect (e .g . labor and parts , just parts ), as part of the warranty included in an Offering . The actual services may be provided by the Business Entity making the offering , by the manufacturer of the product , or by a third party . Examples: Parts and Labor , Parts&quot; <d> a) Quantitative value b) Delivery method c) Payment method d) Warranty scope e) Warranty promise Strategies for distractors: Most similar concept Class A x Annotation Class B Annotation High Similarity Annotation Annotation Annotation ✔ ✖
  31. 31. Which one of the following response pairs relates in the same way as <a> and <b> in the relation <R>? : (c,d),(c,b),(a,d),… <ul><li>Which one of the following response pairs relates in the same way as Sunday and Saturday in the relation has previous? </li></ul><ul><li>Wednesday and Tuesday </li></ul><ul><li>Friday and Saturday </li></ul><ul><li>c) Wednesday and Sunday </li></ul><ul><li>d) Sunday and Monday </li></ul><ul><li>e) Monday and Wednesday </li></ul>Strategies for distractors : Non-existent relation with the same domain, Non-existent relation with the same co-domain a b R c d R ✔ ✖ ✖
  32. 32. Analyze the text <x> and decide which one of the following words is a correct replacement for the blank space in <x>: (B,C,…) Analyze the following text and decide which one of the following words is a correct replacement for the blank space in the text:   &quot;This subproperty specifies that the upper and lower limit of the given ________ are identical and have the respective value . It is a shortcut for such cases where a quantitative property is (at least practically ) a single point value and not an interval .&quot; a) Warranty promise b) Quantitative value integer c) Quantitative value d) Quantitative value float e) Qualitative value Strategies for distractors : Most Similar Annotation Class B annotation annotation annotation Class C annotation annotation annotation High Similarity Class A annotation x annotation Describes ✔ ✖
  33. 33. Implementation: opensemcq: generate test
  34. 34. Implementation: opensemcq: view test
  35. 35. Implementation: opensemcq: evaluate test
  36. 36. Benefits <ul><li>Adding annotations </li></ul><ul><ul><li>Four more question types and three more strategies </li></ul></ul><ul><li>Adding questions templates </li></ul><ul><ul><li>Avoiding the NLP complexity problem </li></ul></ul><ul><li>Adding semantic interpretation </li></ul><ul><ul><li>Questions ordered according to increased level of educational objectives (Bloom, 1956) </li></ul></ul><ul><ul><li>Enables adaptive test creation (Lilley & Barker, 2002) </li></ul></ul><ul><li>Decoupling semantic interpretation from question generation </li></ul><ul><ul><li>Enables change of the underlying learning theory </li></ul></ul><ul><ul><li>Kolb ’s experiential learning (Kolb, 1984) as a basis for defining question templates (Barker, 2008) </li></ul></ul><ul><li>Useful ‘seed’ for further question enhancements </li></ul>Bloom, B. S., Krathwohl, D. R. (1956). Taxonomy of educational objectives. Handbook 1. Cognitive domain. New York: Addison-Wesley. Lilley, M., & Barker, T. (2002). The development and evaluation of a computer-adaptive testing application for English language. 6th Computer assisted assessment conference, July 2002, Loughborough. Kolb, D. (1984) Experiential Learning. Englewood Cliffs. N.J.: Prentice Hall
  37. 37. Issues and Limitations <ul><li>Lack of annotated (test) ontologies </li></ul><ul><li>Lack of questions templates for assessing higher order skills (e.g. synthesis and evaluation) </li></ul><ul><li>Generation of distracters based on text similarity bear no necessary semantically structured relation to the subject matter domain </li></ul><ul><ul><li>Should be extended with meta-ontological relations, such as: subclass, superclass, ‘friend’ class, ‘related’ classes </li></ul></ul><ul><li>Scope of assessment limited to testing the knowledge described in the ontology and the quality of the ontology. </li></ul>
  38. 38. Current Work <ul><li>Invitation for evaluation to be sent to authors of ontologies used for testing </li></ul><ul><li>Further invitations to be sent to University of Hertfordshire academics </li></ul><ul><li>Improvements to be implemented based on the feedback from the evaluation. </li></ul><ul><li>Want to participate? </li></ul><ul><ul><li>Go to www.opensemcq.org </li></ul></ul><ul><ul><li>Select a Test to evaluate </li></ul></ul><ul><ul><li>Click on Evaluate ! </li></ul></ul><ul><li>Want to develop a new ontology? </li></ul><ul><ul><li>Talk to us! </li></ul></ul>
  39. 39. Future Work <ul><li>Integrate text similarity with more advanced ontological approaches to generation of distracters </li></ul><ul><ul><li>“ The distance between concepts in the hierarchy can be used to generate challenging answers covering closely related false answers (distracters)” (Pahl & Holohan, 2009) </li></ul></ul><ul><li>Extend and enrich the template base </li></ul><ul><li>Add other ontology components, such as rules, axioms, restrictions, events etc. to the meta ontology used for question generations </li></ul><ul><ul><li>Explore Horridge’s et al. (2009) ideas on ontology justifications, for generating question feedback </li></ul></ul><ul><li>Extend the empirical dataset to include some larger ‘real-life’ ontologies </li></ul><ul><li>Generalize the MCQ ontology to include other types of objective tests e.g. Multiple Response Questions </li></ul><ul><li>Further collaborative work planned with the Semantic Web groups from University of Southampton and Manchester; CAA group group University of Dublin , Agean and CMU, and Ontology engineering group from Stanford … </li></ul>

×