An Adaptive Assessment System with Knowledge         Representation and Visual Feedback                                   ...
Several adaptive assessment systems are presented next,             The iAdaptTest [8] is a desktop-based tool for adaptiv...
and actually answered correctly, along with the questions the          Figure 1 represents the general components of the A...
After the first experiment, the case study will continue till                                                             ...
Upcoming SlideShare
Loading in …5

Contribution300 a


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Contribution300 a

  1. 1. An Adaptive Assessment System with Knowledge Representation and Visual Feedback Joaquim Fernando Silva, Francisco José Restivo Porto University PortugalAbstract— Assessment is essential for certification, regulation Our research is focused on how to use technology, forand feedback within the education process. Our research effort formative assessment, in formal learning contexts tofocus on the last two features, and tries to identify the strengths individually infer which knowledge each student is acquiringand weaknesses of the students, in order to adapt assessment and identify the learning strengths and weaknesses. Ouractivities for better learning results. hypothesis is that students desire immediate results during the learning process and quantitative measures to differentiateThe approach consists in designing a technological artefact for themselves. Also there is an appealing interest in visualformative assessment with visual feedback during the learning representations, in line with visual thinkers, who use theprocess, to meet students’ demands for personalised tests in order emotional and creative part of the brain to organize informationto effectively assess the learning outcomes. or thoughts in an intuitive manner [1].While many of the assessment tools are oriented to quantitatively This paper has three contributions for AAS. Firstly, itmeasure the student’s knowledge acquisition, the idea behind this presents a general architecture with interrelated is to use assessment as an analysis tool to measure, optimize Secondly, it provides a bottom-up knowledge representationthe learning process, personalise assessment and involve students, technique for teachers structure a domain knowledge. Thirdly,through an adaptive approach, where each student can visualize it presents a visual feedback solution for students’ enhancethe dynamic process of knowledge acquisition. learning. Keywords- Adaptive Assessment System, AAS Architecture, After we have introduced the problem, the remaining partKnowledge Representation, Visual Feedback. of this paper: section 2 describes several adaptive assessment systems analysed during the design of our technological I. INTRODUCTION solution; section 3 refers to our approach solution starting with Many students, who have repeated learning difficulties, a general architecture and the use case for the adaptivemake small gains in school performance, while their academic assessment system; section 4 refers to the proposed proof ofachievement gap widens year after year. There is a missed concept; and section 5 presents the paper conclusions and theguiding in the assessment role in education. Instead of ongoing work.identifying the literacy or numeracy skills, that should beworked out reflecting achievements, the effort seems to II. ADAPTIVE ASSESSMENT SYSTEMSserialize students. During the learning process, formative assessment gives Looking at the players, students usually are not motivated, continuous feedback about students’ current state of knowledgeor engaged in learning activities, and teachers aspire to find the acquisition. Our focus is on formative adaptive assessment.average student. In classes, with many students, individual There are different approaches for formative adaptiveteaching is unviable. Teachers claim they do not have enough assessment systems. Some authors associate adaptation withtime to deliver individual tests, correct them and identify the presentation, when the environment is adjusted with differentunderlying difficulties each student has. colour schemes, layouts or amounts of functionality, whereas The need for individualised learning with personalised tests others associate it with curriculum sequencing, when learningis a consequence of the raising differentiation in education. In is tailored to individual preferences, objectives, learning stylesfact, there is an increasing heterogeneity in students’ profiles or prior knowledge; and also associate it with problem-solving,which demands for adaptive actions in education. Formative when feedback is provided with hints or solutions during theassessment can achieve this when both players: teachers and problem solving process [2, 3]. Other authors even considerstudents infer which knowledge each student is acquiring and adaptive assessment a place to obtain any evidence aboutidentify their learning difficulties. Also Adaptive Assessment knowledge, skills, or attitudes. It is possible to use adaptiveSystems (AAS), a technological solution used for personalised processes in tests, when assessment is item based, or intests and evaluate the student’s progress, can assist teachers in questions experiences when assessment is the curriculum itself.formative assessment .
  2. 2. Several adaptive assessment systems are presented next, The iAdaptTest [8] is a desktop-based tool for adaptiveclassified as computer adaptive tests, or adaptive questions question based on IMS QTI, IMS LIP and XML Topic Mapstests. These systems are a reference for our solution proposal. standards. It has the possibility of reusability and interoperability of data.A. Computer Adaptive Tests Computer adaptive assessment test uses engines to select The LEO [9] is a software program that provides a concept-questions according to the students’ abilities, eliminating map with extra features to design a curriculum. It has a graphquestions that are too easy or too difficult. The questions can like structure with two types of nodes: instructional andbe multiple choice or true and false or even matching questions explanation nodes that explain the topics.and tests assess student’s strengths and weaknesses on an item- All these systems and considerations led us to design aby-item basis. With this assessment approach students proposal, presented next, for the initial problem of using aaccomplish excellent results where they are proficient and technological tool for formative assessment, tailored by theidentify the flaws in their learning. teacher and adapted to each student in a continuous and Usually these systems use a statistical model, mostly the interactive manner.Item Response Theory (IRT) or the Rasch Modelling. There III. SOLUTION PROPOSALare several adaptive systems in literature. Some systems areworth mentioning identifying its advantages and disadvantages A. General Architecture of an Adaptive Assessment Systemin order to help us designing an improved adaptive assessment Based on the analysis of several adaptive assessmentsystem. systems (AAS), and on our own findings, we propose an The SIETTE [4] is a web-based system that mainly uses architecture with three sections and six modules.multiple-choice questions. The question selection is based on a 1) Domain and Activities Modulefunction that estimates the probability of correctness to The Activities Module defines, classifies and stores theparticular questions based on a temporary learner model, available activities for the assessment tests, and the Domainleading to an estimation of the students’ level of knowledge. Module supports the concept structure for representing the The COMPASS [5] is also a web-based system that uses a domain knowledge to be assessed by the student. Basically itconcept map assessment tool for helping students understand, uses a hierarchical tree structure with the general assessmentas well as to support the assessment process. It provides outcomes in the parent node, and at the child nodes thedifferent informative and tutoring individualized feedback. concepts.Students reconsider their beliefs, reflecting on them, and 2) Pedagogic and Student Moduleredefine their knowledge structure. The Pedagogical Module is responsible for the instructional The PASS is a web-based assessment module, which can method used, representing several sequence of concepts and/orbe integrated into an Adaptive Educational Hypermedia activities for the assessment outcomes.System (AEHS). AEHS are learning environments which The Student Module stores information about the students’address personalization with learning models based on goals, profile and performance when doing the tests. It introduces apreferences and knowledge of each student, adapting learning dynamic approach in choosing the test items not only with thecontent to the needs of each student. The PASS system assessment outcomes, but also with the students’ previousestimates students’ performance based on multiple assessment assessment results. The adaptive process, finding whichoptions (pre-test, self-assessment and summative assessment). concepts have been accomplished and the ones which areIt re-estimates the difficulty level of each question at any time getting behind, uses an artificial neural network for calculatingand relates with the importance of the educational material the probability of success for each concept with chosenavailable [6]. activities. Also the ratio of right/wrong answers for a particularB. Adaptive Question Tests set of activities, within a concept is considered. The adaptive questions approach starts with the premise The adaptive assessment has two dimensions. A staticthat a dynamic sequence of questions depend on students’ dimension based on the structured domain knowledge and aresponses. It is supported with rules about the student given dynamic dimension which models the student behaviour whenresponses and overlay the student model to knowledge doing test assessments. Both dimensions complement eachrepresentation based on topics or concepts. In comparison with other.adaptive testing this approach presents more flexibility toteachers in order to include didactical and personal methods To model the student’s behaviour, the system takes intothrough the creation of appropriate rules. account personal characteristics like age, sex, learning style or even the past and present class that he or she belongs to. Also The CosyQTI [7] is a web-based tool for authoring and the reaction towards failure and subsequent commitment is alsoadaptive questions test based on IMS QTI, IMS LIP and IEEE important. Some students give up when facing obstacles whileLTSC PAPI standards. It has the advantages of using standard- others enrol more when find difficult questions. This issue iscomplaint and open information policy, which makes it related with the learning pattern each student has or theinteroperable and reusable in other learning environments. learning curve. For instance, the level of confidence is measured with the amount of questions which are postponed
  3. 3. and actually answered correctly, along with the questions the Figure 1 represents the general components of the AASstudent did not answered. artefact. 3) Feedback and Environment Module Module Teacher The Feedback Module is responsible for transmitting, usingtext and graphics, the test results and gives hints or clues tostudents for future actions. Finally the Environment Module isrelated with the assessment presentation which can be Webbased, desktop based, game based or mobile based. Activities Module Domain Module Pedagogical Module The assessment feedback, about the achieved results isessential for each student get conscious of his or herperformance. Also feedback can provide hints or advices tooutcome the identified difficulties. The dynamic assessment Environment Feedback Student Modulebehaviour of the student is also essential to understand the Module Modulelearning evolution and increase the interaction levels. A personalized feedback, a text and visual representationabout student’s performance, helps students and teachers tobetter understand the knowledge acquisition and change theinteractions in the classroom for effective learning. Student Our focus is on visual representations for personalizedfeedback assessment. Visual representations based for Figure 1. Components of a general AASrepresenting hierarchical or semi-structured data [10] tovisualize the students results [11] are not an easy business and B. Use Caseusually the approaches are oriented to a particular problem The In this section the essentials of the use case used with thetime dimension is also essential for students understand the solution proposal are presented. The AAS solution is plannedlearning progress, which is a dynamic and ongoing process. In to be used in a high school context with selected teachers andfact, learning is a starting process that never ends. students. The selected teachers, from different domains just create their classes. The students are the ones of the selected A visual representation to express the temporal dimension teachers and just choose their class.has several approaches that might be used for visualassessment feedback. During our research we analysed several Teachers can author the activities, import them or buildapproaches and decided to analyse our problem, with a them, can structure the concept layer of the domain with asystematic approach, to select the right visual tool with a graph tool and define the desired abilities. The activities arecategorization criteria of time, data and representation [12]. classified accordingly to the three lower Bloom’s Taxonomies [13] (remembering, understanding, applying). The cognitive We consider assessment, within the AAS system, a cyclic taxonomy of remembering should take place beforetime dimension and the students achieved results an abstract understanding and then applying the concept in new where there is no spatial mapping. The other three upper taxonomies (analysing, evaluating and The Feedback Module, uses both visual representations: a creating) should take place only after the first three taxonomiesstatic visualization for the knowledge domain and a dynamic are mastered. For that reason the teacher also plays antime-oriented visual representation for the student performance. important role in the adaptive process because he or she can use the feedback results for enhancing education in the Along with visualization, user interaction is also very classroom [14].important. The student has a global image of the assessmentfeedback, but those who wish to acquire insight with the The students have the opportunity to do online assessmentpicture can explore interactively, with pop-up texts appearing tests with feedback results. The feedback result is both aon mouse over the petals, with additional information. For Treemap visualization for all concepts under study and ainstances the name of the concepts or the abilities and some PeopleGarden visualization for the ongoing achievements.recommendations associated with the red concepts, in order to Several other services can be associated, but the essentialorient the student in future class assignments. services are defined in Figure 2 with a UML use case diagram. In order to modify the student behaviours the social IV. PROOF OF CONCEPTdimension is very important. The student can compare his or The concept is applied to a high school context. In thisher result with their peers. experimental work all students are connected with their The Environment Module includes several channels, from teachers, based on the class and school. The experiment will beweb based to mobile based, promoting ubiquity. Besides monitored with weekly meetings with teachers and students tomobile solutions are growing in importance with the identify the positive aspects and negative aspects of the system.appearance of Smartphone’s with capabilities more than Finally all the players will answer the online questionnaire: insufficient for enhancing learning. the end of system evaluation.
  4. 4. After the first experiment, the case study will continue till The feedback visualization of results underpins the researchthe end of the year, measuring the full impact of the tool in effort to reach higher levels of students’ commitment and weterms of interest, changing attitudes towards learning and are currently designing the visual feedback, using a visual timeimproving knowledge acquisition. based representation. Adaptive Assessment System ACKNOWLEDGMENTS * Create Class «extends» Class «uses» Choose Class * The authors would like to thank the LIACC Laboratory for -End37 «extends» -End40 excellent working conditions. The first author would like to * * * -End33 * -End34 -End35 -End36 -End28 -End32 -End30 * thank Portuguese FCT for the support provided through -End38 -End22 * -End24 -End26 * Choose Domain * -End39 * * * * scholarship SFRH/BD/36206/2007. * -End27 Teacher n -End21 Register * Student n REFERENCES * -End29 -End23 Profile * [1] E. Copperman, C. Beeri, and N. Ben-Zvi, "Visual modelling of learning * -End31 Global Reports processes," Innovations in Education and Teaching International, vol. 44, -End25 * +End2 pp. 257-272, 2007. Author Activities * «uses» -End8 [2] P. d. Bra, Adaptive hypermedia. In Handbook on Information -End4 -End3 +End1 Classify Activities «uses» * -End10 Technologies for Education and Training Berlin: Springer, 2008. -End5 * -End5 -End7 * -End5 ** * -End6 Structure Concepts «uses» On Line Assessment -End9 -End11 Student 1 -End12 [3] T. L. Kinshuk and A. Patel, User Adaptation in Supporting Exploration * Test * * * * -End6 «uses» * Student 2 * Tasks in Virtual Learning Environments. Netherlands: Springer, 2006. «uses» Teacher n * Define Rules «uses» Student n [4] R. Conejo, E. Guzmán, E. Millán, M. Trella, Pérez-De-La-Cruz, and R. J. -End6 * Define Abilities * L., "SIETTE: A Web- based Tool for Adaptive Testing," International -End18 * * * -End17 -End19 -End15 -End20 Journal of Artificial Intelligence in Education, vol. 14, pp. 29-61, 2004. * -End14 -End13 Feedback Results -End16 ** Student 1 * [5] E. Gouli, A. Gogoulou, K. Papanikolaou, and M. Grigoriadou, Student 2 "COMPASS: AN ADAPTIVE WEB-BASED CONCEPT MAP Teacher n Student n ASSESSMENT TOOL," presented at International Conference on Figure 2. Use Case for the Proposed AAS Concept Mapping, Pamplona Spain, 2004. [6] E. Gouli, K. Papanikolaou, and M. Grigoriadou, "Personalizing Also some interpellations were made to students, trying to Assessment in Adaptive Educational Hypermedia Systems," presented atdelve into their motivations. The system will be used by some Second International Conference on Adaptive Hypermedia and Adaptiveinvited teachers with their students. We have decided not to use Web-Based Systems, 2002.a test pilot approach, but rather verify the system impact in the [7] P. Lalos, S. Retalis, and Y. Psaromiligkos, "Creating personalised quizzesplayers. That is, we are planning to use it with several classes both to the learner and to the access device characteristics: the Case ofand compare it with results of previous years, and moreimportant, in the end of the case study a questionnaire will be CosyQTI," presented at Workshop on Authoring of Adaptive anddelivered. The questionnaire will give emphasis for the Adaptable Educational Hypermedia, 2005.qualitative approach decision, in order to validate our research [8] F. Lazarinis, S. Green, and E. Pearson, 257-278., "Focusing on contentthat using technology for formative assessment in formal reusability and interoperability in a personalized hypermedia assessmentlearning contexts, the players can infer which knowledge each tool," Multimedia Tools and Applications, vol. 47, pp. 257-278, 2009.student is acquiring and identify the learning strengths and [9] J. W. Coffey, "LEO: A Concept Map Based Course Visualization Toolweaknesses. for Instructors and Students," presented at Knowledge and Information Visualization, LNCS, 2005. V. CONCLUSIONS AND ONGOING WORK [10] M. Lanzenberger, J. Sampson, and M. Rester, "Ontology Visualization: The model presented, for an adaptive assessment system, Tools and Techniques for Visual Representation of Semi-Structuredstarts with the premise the teacher provides knowledge domain Meta-Data," Universal Computer Science, vol. 16, pp. 1036-1054, 2010.structure with abilities and concepts attached with several [11] P. Brusilovsky, I.-H. Hsiao, and Y. Folajimi, "QuizMap: Open Socialactivities. The adaptive assessment algorithm, based onartificial neural network (ANN), predicts the true knowledge Student Modeling and Adaptive Navigation Support with TreeMaps,"each student has, identifying the strengths and weaknesses, presented at Towards Ubiquitous Learning, 6th European Conference ofsuggesting the next activity. The system delivers a visual Technology Enhanced Learning, EC-TEL 2011, LNCS,, for each student, about the ongoing assessment. To [12] Wolfgang Aigner, Silvia Miksch, Wolfgang Muller, Heidrun Schumann,validate this approach solution a use case will be carried out in and Christian Tominski, "Visualizing Time-Oriented Data – A Systematica high school context in several knowledge domains. View," Computers and Graphic, vol. 31, 2007. The student, using this system, can cement knowledge, [13] L. W. Anderson, D. R. Krathwohl, and A taxonomy for learning, teachingdeepening the concepts acquisition or exploring knowledge. and assessing: A revision of Blooms Taxonomy of educational objectives:Also the teacher can introduce, in the system, at any time, new Longman, 2001.and different activities in order to reinforce one or both [14] M. Pohl, Learning to Think, Thinking to Learn: Models and Strategies toassessment outcomes. Develop a Classroom Culture of Thinking, 2000.