Assessment outcomes from the TENCompetence project

Assessment in TENCompetence: Sharing Assessment Process Information Yongwu Miao Educational Technology Expertise Centre Open University of The Netherlands CETIS Assessment SIG meeting Glasgow, Sep. 26, 2007

Overview
TENCompetence Project
A Standard-based and Service-oriented Approach to Support Assessment Process
Assessment Process Specification (APS)
Assessment Process Modelling Tool
Conclusions and Future Work

TENCompetence is a 4-year (Dec. 2005 - Nov. 2009) EU-funded Integrated IST-TEL project and fifteen partners are involved.
Project Aims :
to build the European Network for Lifelong Competence Development
How? By providing a technical and organizational Infrastructure
Who for? For use by any individual, group or organization
What for? To develop their competences

Assessment in TENCompetence (WP6.B)
- Supporting assessment that is integrated with learning and instruction (which steps are to be carried out and by whom), e.g., formative & summative assessment
- Supporting new forms of assessment, which are process-based and are normally involved with multi-role/users, e.g., peer assessment, 360 degree feedback, and portfolio assessment
Objectives of WP6.B
Develop a formal assessment specification and supporting tools that combines new assessment types and the ones included in the IMS QTI.

A combined use of IMS LD, IMS QTI, and assessment-specific services to model assessment processes
Using IMS LD to model multi-role/user-involved assessment processes
Using IMS QTI to model classical assessment tasks and using assessment-specific services to model special assessment tasks
Using IMS LD to model a learn-flow with complicated process controls and decision rules
Using IMS QTI, IMS LD, and assessment-specific services to model data-flow

learning design (or assessment design) unit of learning (or unit of assessment) role activity environment property component method condition expression action play act role-part learning support service learning object staff learner internal change-property notification show/hide assessee assessor assessment- specific service item activity-description resource QTI_type external assessment An extended IMS LD model assessment_item outcome item_body assessment_test interaction response response_processing A simplified IMS QTI model associate choice . . . . . .

Architecture of Run-time Environment
A Standard-based and Service-oriented Approach to Support Assessment Process Player Player Player internet Dispatcher/Service broker IMS LD engine IMS QTI engine ePortfolio editor Concept mapping Latent Semantic Analysis (LSA) . . . . A A A A A generic assessment service special assessment services UoL management and enactment service client client

An example of run-time environment
A Standard-based and Service-oriented Approach to Support Assessment Process SLeD SLeD SLeD internet CopperCore Service Integration (CCSI) CopperCore APIS A A generic assessment service UoL management and enactment service client client

A peer assessment example A Standard-based and Service-oriented Approach to Support Assessment Process design assignment writing article1 reviewing article2 improving article1 writing article2 reviewing article1 improving article2 final assessment2 final assessment1 learner1 learner2 tutor article2 feedback1 improved a2 paper1 article1 feedback2 improved a1 paper2 tutor’s feedback1 tutor’s feedback2 assessment form1 assessment form2

A screenshot of execution of the peer assessment example

Problem:
It is too difficult for practitioners to specify an assessment process model by using IMS LD, IMS QTI, and assessment-specific services.
Solution:
Developing an additional layer, assessment process specification (APS) used to describe assessment processes in an interoperable and abstract way. By using APS, every assessment practice can be represented in a design description (an abstract assessment plan), which captures underlying design ideas and principles. Thus, practitioners can share and exchange their assessment process models (assessment designs).
Assessment Process Specification

Conceptual Model role activity assessment design artefact rule certifier designer assessee assessor stage rating responding instructing editing portfolio constructing item performing input tool output assignment portfolio assessment test assessment item evidence assessment result expression action portfolio editor concept mapping QTI editor feedback simulator LSA QTI player giving feedback react on feedback preparation evaluation evi. collection processing teaching-learning add/remove role member trigger activity alloca. activity to role send email change property …… …… evi. provider consultant decision maker …… …… distribute artifacts present information ……

Assessment process model Programmed instruction model Peer assessment model <…> 360 degree feedback model Domain-specific models Assessment process specification Unit of assessment mapping Assessment Process Specification IMS LD + IMS QTI + Assessment-specific services

Assessment Process Modeling Tool
Aims:
Enabling skilled users to define their own assessment process models by providing basic modeling facilities
Enabling practitioners to understand, reuse, and customize pre-defined assessment process models by providing design guidance
Transforming assessment process models into executable models represented in IMS LD + IMS QTI + assessment-specific services
The tool is still in design and will be implemented in the next year. The design and use of the tool will be explained by using a mock-up of the UI

Four kinds of stages in a peer assessment process
In each stage, people with a certain role perform activities
In each activity, there may be input and output artefacts
Assessment Process Modeling Tool Design assessment Give feedback Do assignment React to feedback constructing assi. /designing instruction /designing criteria /setting time viewing /rating feedback /requiring more /improving responding /performing 0,1 1 1 1 0,1 0,1 1 commenting /rating /grading /elaborating 1 preceding preceding preceding 1 1, * 1 1 1, * 1 1, * 1, * designer candidate reviewer candidate assignment 1, * 1, * 1, * 1, * 1, * 1, * 1, * 1, * 0, * 1, * 1, * 1, * assignment outcome 1, * 1, * 1, * 1, * feedback 1, * 1, * 1, * 1, * improved assignment outcome 1, * 0, * 1, * 1, * assessment form 1, * 0, * 0, * 1, * performing performing performing performing producing used_in

Guide to make design decisions by showing guidance and consequence
Make design decisions by selecting default or pre-defined options
Customize a process model (for skilled users)
Assessment Process Modeling Tool define stages? define activities? define roles? include design stage? default default additional feedback cycles? how many? for each stage, define sub-roles? how many instances for each roles? what are relations between candidates and reviewers? will tutor be involved? for each stage, define activities? completion? assign roles to activities? define artifact-flow? default choose distribution patterns (1-1, 1-m, m-1, m-m, m-n, rotate, mutual , reciprocal , random, ...) define assignment, assessment form, calculation rules Anonymous/ confidential/ public customize default completion? synchronize activities? define services/tools?

File Edit View Tool Template Help Peer assessment Here shows the guidance to make design decisions State bar user’s decisions Specify role structure, artifacts, and services Show and edit peer assessment model artifact role service lane stage

File Edit View Tool Template Help Peer assessment How many stages in the peer assessment process? Please make a decision user’s decisions artifact role service lane stage Default (three stages) With design assessment stage With additional feedback stage More complicated setting

File Edit View Tool Template Help Peer assessment How many roles will be involved? Five stages have been defined artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 Default (two persons) With the involvement of tutor More complicated setting

File Edit View Tool Template Help Peer assessment Would you like to define role structure? Default role setting and with the involvement of a tutor artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R

File Edit View Tool Template Help Peer assessment Would you like to define a typical activity for each stage? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R Default setting More complicated setting

File Edit View Tool Template Help Peer assessment Would you like to define a typical activity for each stage? Nine activities are defined and roles are assigned artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R learner1 learner1 learner1 learner 2 learner 2 learner 2 learner1 learner2 tutor

File Edit View Tool Template Help Peer assessment Which stages will the tutor be involved? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R learner1 learner1 learner1 learner 2 learner 2 learner 2 learner1 learner2 Give feedback 1 Give feedback 2 More complicated setting tutor

File Edit View Tool Template Help Peer assessment Which stages will the tutor be involved? The tutor is assigned to work in the stage of “give feedback 2” artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor

File Edit View Tool Template Help Peer assessment Would you like to create artifacts ? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R Default setting More complicated setting learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor

File Edit View Tool Template Help Peer assessment Would you like to create typical artifacts ? artifacts are created artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1.1 A feedback1.2 A outcome1 A outcome2 A assignment1 A assignment2 A improved outcome1 A improved outcome2 A feedback2.1 A feedback2.2 A

File Edit View Tool Template Help Peer assessment Would you like to change the titles of artifacts? artifacts are created finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1.1 A feedback1.2 A outcome1 A outcome2 A assignment1 A assignment2 A improved outcome1 A improved outcome2 A feedback2.1 A feedback2.2 A

File Edit View Tool Template Help Peer assessment Would you like to change the titles of artifacts? The title of an artifact has been changed finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A

File Edit View Tool Template Help Peer assessment Would you like to define artifacts? An artifact has been created finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A learner 2 learner 2 Review Form How do you think about this article? accept without modification accept with minor/major modification reject Comments: assessment form QTI assessment editor assessment form (QTI item) A

File Edit View Tool Template Help Peer assessment Would you like to define artifacts? An artifact has been defined finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A

File Edit View Tool Template Help Peer assessment How will the artifacts be distributed? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A Reciprocally Mutually Randomly Default setting

File Edit View Tool Template Help Peer assessment How will the artifacts be distributed? An artifacts flow has been defined artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A

File Edit View Tool Template Help Peer assessment How will the artifacts be distributed? An artifact flow has been defined artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A

Conclusions:
IMS LD, IMS QTI, and assessment-specific services can be used complementarily to support various assessment processes
A higher level assessment process specification and corresponding supporting tools are expected to enable practitioners to share and exchange assessment process information
Conclusions and Future Work
Future work:
Developing the assessment process modeling tool
Integrating QTI editor and other assessment services
Developing mapping algorithm to transform APS to IMS LD + IMS QTI + assessment-specific services.

Thanks for your attention! Questions?
Contact information:
Project website: www.tencompetence.org
For details of associate partnerships: http://www.tencompetence.org/node/16
Contact c.kew@bolton.ac.uk for all other enquiries

Assessment outcomes from the TENCompetence project

by University of Strathclyde

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Assessment outcomes from the TENCompetence project Presentation Transcript