Computer Assisted Assessment Patterns for Different Target Users: the Case of Multimedia Questionnaires Patricia Santos Rodríguez, Davinia Hernández-Leo, Toni Navarrete and Josep Blat Pompeu Fabra University, Department of Information and Communication Technologies 5 th March,2009

Content Motivation Objectives State of the Art Patterns proposal Survey study Conclusions & Future work Patricia Santos – eLearning Patterns workshop,

Motivation

Objectives

State of the Art

Patterns proposal

Survey study

Conclusions & Future work

Motivation Patricia Santos - eLearning Patterns workshop, New questionItems  New forms of evaluate knowledge = Improvement of students’ skills. Patterns (capture best practices in a CAA context based on eQuestionnaires) Education (Teachers & students, Assessment, Competences, Methodologies, Context, Problems...) Technology (Educational Technology standards, eQuestionnaries,...) Assessment Specification (QTI) Multimedia Web 2.0 services + +

Content Motivation Objectives State of the Art Patterns proposal Survey study Conclusions & Future work Patricia Santos - eLearning Patterns workshop,

Motivation

Objectives

State of the Art

Patterns proposal

Survey study

Conclusions & Future work

Objectives Analyse the stages, context, problems and users involved in our test-based CAA case Analyse if the steps followed in this case can potentially represent good practice solutions for common problems Find solutions to bridge the gap between users with problems in this context Patricia Santos – elearning Patterns workshop,

Analyse the stages, context, problems and users involved in our test-based CAA case

Analyse if the steps followed in this case can potentially represent good practice solutions for common problems

Find solutions to bridge the gap between users with problems in this context

Content Motivation Objectives State of the Art Patterns proposal Survey study Conclusions & Future work Patricia Santos – eLearning Patterns workshop,

Motivation

Objectives

State of the Art

Patterns proposal

Survey study

Conclusions & Future work

CAA based on Multimedia eQuestionnaires CAA use computers for assessing student learning New types of questionItems are necessary for evaluating students’ competences Assessment specifications & multimedia Web 2.0 services Patricia Santos – eLearning Patterns workshop,

CAA use computers for assessing student learning

New types of questionItems are necessary for evaluating students’ competences

Assessment specifications & multimedia Web 2.0 services

IMS Question & Test Interoperability IMS QTI educational technology specification for assessment To represent an assessment scenario using questions (assessmentItem) and tests (assessmentTest) and their corresponding results reports QTI offers different types of questions: Simple Interactions Text-based Interactions Graphical Interactions Miscellaneous Interactions To interpret the specification we need an engine (e.g. APIS) The interoperability of this specification enables the easy communication with other services. Patricia Santos – eLearning Patterns workshop,

IMS QTI educational technology specification for assessment

To represent an assessment scenario using questions (assessmentItem) and tests (assessmentTest) and their corresponding results reports

QTI offers different types of questions:

Simple Interactions

Text-based Interactions

Graphical Interactions

Miscellaneous Interactions

To interpret the specification we need an engine (e.g. APIS)

The interoperability of this specification enables the easy communication with other services.

New questionItems Teachers need to evaluate specific student’s skills If user can interact in new ways  the user could be evaluated in new ways We study the approach of combining GoogleMaps using the specification IMS QTI The student now is directly related to a geographic map, and not just with a verbatim question, the teacher can increase their evaluation possibilities because now he can get results from the interaction made over the map Patricia Santos – eLearning Patterns workshop,

Teachers need to evaluate specific student’s skills

If user can interact in new ways  the user could be evaluated in new ways

We study the approach of combining GoogleMaps using the specification IMS QTI

The student now is directly related to a geographic map, and not just with a verbatim question, the teacher can increase their evaluation possibilities because now he can get results from the interaction made over the map

Multimedia Web 2.0 questionItems A flexible and powerful model for including a wide variety of user interactions of geographic type with maps to perform assessment making use of GoogleMaps Technical solution: Middleware + web-based editor Patricia Santos – eLearning Patterns workshop,

A flexible and powerful model for including a wide variety of user interactions of geographic type with maps to perform assessment making use of GoogleMaps

Technical solution: Middleware + web-based editor

Modelling eAssessment Process Model the CAA context: the users their roles their tasks different stages of the process Patricia Santos – eLearning Patterns workshop,

Model the CAA context:

the users

their roles

their tasks

different stages of the process

Patricia Santos – eLearning Patterns workshop,

Content Motivation Objectives State of the Art Patterns proposal Survey study Conclusions & Future work Patricia Santos – eLearning Patterns workshop,

Motivation

Objectives

State of the Art

Patterns proposal

Survey study

Conclusions & Future work

Patterns A pattern is designed to capture best practice in a specific domain Provides: organized information that explains a reusable sound solution to a commonly occurring contextualized problem Pattern main elements: Context Problem Solution Patricia Santos – eLearning Patterns workshop,

A pattern is designed to capture best practice in a specific domain

Provides: organized information that explains a reusable sound solution to a commonly occurring contextualized problem

Pattern main elements:

Context

Problem

Solution

Patterns Learning Patterns for educationalists Software Patterns for computer scientists Patricia Santos – eLearning Patterns workshop,

Network of connections among patterns from different target users Patricia Santos – eLearning Patterns workshop,

Pattern Language: adding new opportunities in CAA based on questionnaires Teachers’ Pattern “ Evaluation of Exploration Skills using eQuestionnaires ”: This pattern explains how to create new questionItems to evaluate the exploration skills of the students automatically. Specification Expert Designers’ Pattern “Modelling new questionItems using a Multimedia Web 2.0 service”: This pattern can be used as a bridge of communication between teachers and developers. Developers’ Pattern “ Implementation of Multimedia Web 2.0 questionItems using an Assessment Specification to design eQuestionnaires ”: It is created to provide a solution for developers who want to communicate an assessment engine with a multimedia Web 2.0 service. This communication will enable the integration of new questionItems in an eQuestionnaire. Patricia Santos – eLearning Patterns workshop,

Teachers’ Pattern “ Evaluation of Exploration Skills using eQuestionnaires ”: This pattern explains how to create new questionItems to evaluate the exploration skills of the students automatically.

Specification Expert Designers’ Pattern “Modelling new questionItems using a Multimedia Web 2.0 service”: This pattern can be used as a bridge of communication between teachers and developers.

Developers’ Pattern “ Implementation of Multimedia Web 2.0 questionItems using an Assessment Specification to design eQuestionnaires ”: It is created to provide a solution for developers who want to communicate an assessment engine with a multimedia Web 2.0 service. This communication will enable the integration of new questionItems in an eQuestionnaire.

Content Motivation Objectives State of the Art Patterns proposal Survey study Conclusions & Future work Patricia Santos – eLearning Patterns workshop,

Motivation

Objectives

State of the Art

Patterns proposal

Survey study

Conclusions & Future work

Survey Study Survey study: “ Providing a solution to bridge the gap between developers of assessment tools, specification expert designers and teachers ” Conclusions: The majority of the teachers value positively the fact of integrating “help information” in software tools to facilitate the preparation of assessment materials To develop an educational technology tool which uses an external Web 2.0 service, a developer must have knowledge about: eLearning specifications, eLearning tools and Web 2.0 services It is necessary to create well-structured information about best practices in developing, designing and using assessment tools Technicians believe that it is very useful to know teachers good practices Example: A comment of a developer was : “Knowing those best practices would greatly help me to guess how to apply a solution to my own context. Furthermore, I could even discover new ideas on eLearning/eAssessment that I didn’t think of” [Technicians’ Questionnaire]. Patricia Santos – eLearning Patterns workshop,

Survey study: “ Providing a solution to bridge the gap between developers of assessment tools, specification expert designers and teachers ”

Conclusions:

The majority of the teachers value positively the fact of integrating “help information” in software tools to facilitate the preparation of assessment materials

To develop an educational technology tool which uses an external Web 2.0 service, a developer must have knowledge about: eLearning specifications, eLearning tools and Web 2.0 services

It is necessary to create well-structured information about best practices in developing, designing and using assessment tools

Technicians believe that it is very useful to know teachers good practices

Example: A comment of a developer was : “Knowing those best practices would greatly help me to guess how to apply a solution to my own context. Furthermore, I could even discover new ideas on eLearning/eAssessment that I didn’t think of” [Technicians’ Questionnaire].

Content Motivation Objectives State of the Art Patterns proposal Survey study Conclusions & Future work Patricia Santos – eLearning Patterns workshop,

Motivation

Objectives

State of the Art

Patterns proposal

Survey study

Conclusions & Future work

Conclusions The target users of a PL can be different types of users from different types of communities. We propose a network of connections among patterns from different PLs. The network of connections creates a new PL, which solves the complex problem in the CAA case we are studying. Patterns can be a good solution to capture good practices on CAA and bridge the communication between users of different communities. Our collection of patterns enables the addition of new opportunities on CAA using eQuestionnaires. Patricia Santos – eLearning Patterns workshop,

The target users of a PL can be different types of users from different types of communities.

We propose a network of connections among patterns from different PLs.

The network of connections creates a new PL, which solves the complex problem in the CAA case we are studying.

Patterns can be a good solution to capture good practices on CAA and bridge the communication between users of different communities.

Our collection of patterns enables the addition of new opportunities on CAA using eQuestionnaires.

Future Work Include patterns in an eAssessment authoring tool Increase the PL “patterns for adding new opportunities on CAA using eQuestionnaires” This is an initial work, then it is necessary to do further evaluations, an evaluation of the proposed PL is planned for demonstrating that the solutions proposed in the set of patterns can be used in similar contexts and with similar problems Patricia Santos – eLearning Patterns workshop,

Include patterns in an eAssessment authoring tool

Increase the PL “patterns for adding new opportunities on CAA using eQuestionnaires”

This is an initial work, then it is necessary to do further evaluations, an evaluation of the proposed PL is planned for demonstrating that the solutions proposed in the set of patterns can be used in similar contexts and with similar problems

Questions? Patricia Santos – eLearning Patterns workshop,

Thank you! Patricia Santos – eLearning Patterns workshop,

More information: [email_address] http://pilot.tencompetence.org/QTI_GMaps/Demo_Exam_qti_gmaps.html Patricia Santos – eLearning Patterns workshop,