Assessment outcomes from the TENCompetence project
Upcoming SlideShare
Loading in...5
×
 

Assessment outcomes from the TENCompetence project

on

  • 2,159 views

Yongwu Miao, Open University of the Netherlands

Yongwu Miao, Open University of the Netherlands

Statistics

Views

Total Views
2,159
Views on SlideShare
2,159
Embed Views
0

Actions

Likes
0
Downloads
17
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Assessment outcomes from the TENCompetence project Assessment outcomes from the TENCompetence project Presentation Transcript

  • Assessment in TENCompetence: Sharing Assessment Process Information Yongwu Miao Educational Technology Expertise Centre Open University of The Netherlands CETIS Assessment SIG meeting Glasgow, Sep. 26, 2007
  • Overview
    • TENCompetence Project
    • A Standard-based and Service-oriented Approach to Support Assessment Process
    • Assessment Process Specification (APS)
    • Assessment Process Modelling Tool
    • Conclusions and Future Work
    • TENCompetence is a 4-year (Dec. 2005 - Nov. 2009) EU-funded Integrated IST-TEL project and fifteen partners are involved.
    • Project Aims :
    • to build the European Network for Lifelong Competence Development
    • How? By providing a technical and organizational Infrastructure
    • Who for? For use by any individual, group or organization
    • What for? To develop their competences
    TENCompetence Project
    • Assessment in TENCompetence (WP6.B)
    • - Supporting assessment that is integrated with learning and instruction (which steps are to be carried out and by whom), e.g., formative & summative assessment
    • - Supporting new forms of assessment, which are process-based and are normally involved with multi-role/users, e.g., peer assessment, 360 degree feedback, and portfolio assessment
    • Objectives of WP6.B
    • Develop a formal assessment specification and supporting tools that combines new assessment types and the ones included in the IMS QTI.
    TENCompetence Project
  • A Standard-based and Service-oriented Approach to Support Assessment Process
    • A combined use of IMS LD, IMS QTI, and assessment-specific services to model assessment processes
      • Using IMS LD to model multi-role/user-involved assessment processes
      • Using IMS QTI to model classical assessment tasks and using assessment-specific services to model special assessment tasks
      • Using IMS LD to model a learn-flow with complicated process controls and decision rules
      • Using IMS QTI, IMS LD, and assessment-specific services to model data-flow
  • learning design (or assessment design) unit of learning (or unit of assessment) role activity environment property component method condition expression action play act role-part learning support service learning object staff learner internal change-property notification show/hide assessee assessor assessment- specific service item activity-description resource QTI_type external assessment An extended IMS LD model assessment_item outcome item_body assessment_test interaction response response_processing A simplified IMS QTI model associate choice . . . . . .
    • Architecture of Run-time Environment
    A Standard-based and Service-oriented Approach to Support Assessment Process Player Player Player internet Dispatcher/Service broker IMS LD engine IMS QTI engine ePortfolio editor Concept mapping Latent Semantic Analysis (LSA) . . . . A A A A A generic assessment service special assessment services UoL management and enactment service client client
    • An example of run-time environment
    A Standard-based and Service-oriented Approach to Support Assessment Process SLeD SLeD SLeD internet CopperCore Service Integration (CCSI) CopperCore APIS A A generic assessment service UoL management and enactment service client client
  • A peer assessment example A Standard-based and Service-oriented Approach to Support Assessment Process design assignment writing article1 reviewing article2 improving article1 writing article2 reviewing article1 improving article2 final assessment2 final assessment1 learner1 learner2 tutor article2 feedback1 improved a2 paper1 article1 feedback2 improved a1 paper2 tutor’s feedback1 tutor’s feedback2 assessment form1 assessment form2
    • A screenshot of execution of the peer assessment example
    A Standard-based and Service-oriented Approach to Support Assessment Process
    • Problem:
    • It is too difficult for practitioners to specify an assessment process model by using IMS LD, IMS QTI, and assessment-specific services.
    • Solution:
    • Developing an additional layer, assessment process specification (APS) used to describe assessment processes in an interoperable and abstract way. By using APS, every assessment practice can be represented in a design description (an abstract assessment plan), which captures underlying design ideas and principles. Thus, practitioners can share and exchange their assessment process models (assessment designs).
    Assessment Process Specification
  • Conceptual Model role activity assessment design artefact rule certifier designer assessee assessor stage rating responding instructing editing portfolio constructing item performing input tool output assignment portfolio assessment test assessment item evidence assessment result expression action portfolio editor concept mapping QTI editor feedback simulator LSA QTI player giving feedback react on feedback preparation evaluation evi. collection processing teaching-learning add/remove role member trigger activity alloca. activity to role send email change property …… …… evi. provider consultant decision maker …… …… distribute artifacts present information ……
  • Assessment process model Programmed instruction model Peer assessment model <…> 360 degree feedback model Domain-specific models Assessment process specification Unit of assessment mapping Assessment Process Specification IMS LD + IMS QTI + Assessment-specific services
  • Assessment Process Modeling Tool
    • Aims:
    • Enabling skilled users to define their own assessment process models by providing basic modeling facilities
    • Enabling practitioners to understand, reuse, and customize pre-defined assessment process models by providing design guidance
    • Transforming assessment process models into executable models represented in IMS LD + IMS QTI + assessment-specific services
    • The tool is still in design and will be implemented in the next year. The design and use of the tool will be explained by using a mock-up of the UI
    • Four kinds of stages in a peer assessment process
    • In each stage, people with a certain role perform activities
    • In each activity, there may be input and output artefacts
    Assessment Process Modeling Tool Design assessment Give feedback Do assignment React to feedback constructing assi. /designing instruction /designing criteria /setting time viewing /rating feedback /requiring more /improving responding /performing 0,1 1 1 1 0,1 0,1 1 commenting /rating /grading /elaborating 1 preceding preceding preceding 1 1, * 1 1 1, * 1 1, * 1, * designer candidate reviewer candidate assign- ment 1, * 1, * 1, * 1, * 1, * 1, * 1, * 1, * 0, * 1, * 1, * 1, * assign- ment outcome 1, * 1, * 1, * 1, * feedback 1, * 1, * 1, * 1, * improved assignment outcome 1, * 0, * 1, * 1, * assessment form 1, * 0, * 0, * 1, * performing performing performing performing producing used_in
    • Guide to make design decisions by showing guidance and consequence
    • Make design decisions by selecting default or pre-defined options
    • Customize a process model (for skilled users)
    Assessment Process Modeling Tool define stages? define activities? define roles? include design stage? default default additional feedback cycles? how many? for each stage, define sub-roles? how many instances for each roles? what are relations between candidates and reviewers? will tutor be involved? for each stage, define activities? completion? assign roles to activities? define artifact-flow? default choose distribution patterns (1-1, 1-m, m-1, m-m, m-n, rotate, mutual , reciprocal , random, ...) define assignment, assessment form, calculation rules Anonymous/ confidential/ public customize default completion? synchronize activities? define services/tools?
  • File Edit View Tool Template Help Peer assessment Here shows the guidance to make design decisions State bar user’s decisions Specify role structure, artifacts, and services Show and edit peer assessment model artifact role service lane stage
  • File Edit View Tool Template Help Peer assessment How many stages in the peer assessment process? Please make a decision user’s decisions artifact role service lane stage Default (three stages) With design assessment stage With additional feedback stage More complicated setting
  • File Edit View Tool Template Help Peer assessment How many roles will be involved? Five stages have been defined artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 Default (two persons) With the involvement of tutor More complicated setting
  • File Edit View Tool Template Help Peer assessment Would you like to define role structure? Default role setting and with the involvement of a tutor artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R
  • File Edit View Tool Template Help Peer assessment Would you like to define a typical activity for each stage? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R Default setting More complicated setting
  • File Edit View Tool Template Help Peer assessment Would you like to define a typical activity for each stage? Nine activities are defined and roles are assigned artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R learner1 learner1 learner1 learner 2 learner 2 learner 2 learner1 learner2 tutor
  • File Edit View Tool Template Help Peer assessment Which stages will the tutor be involved? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R learner1 learner1 learner1 learner 2 learner 2 learner 2 learner1 learner2 Give feedback 1 Give feedback 2 More complicated setting tutor
  • File Edit View Tool Template Help Peer assessment Which stages will the tutor be involved? The tutor is assigned to work in the stage of “give feedback 2” artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor
  • File Edit View Tool Template Help Peer assessment Would you like to create artifacts ? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 reviewer R designer R candidate R learner1 R tutor R learner2 R learner1 R learner2 R tutor R Default setting More complicated setting learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor
  • File Edit View Tool Template Help Peer assessment Would you like to create typical artifacts ? artifacts are created artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1.1 A feedback1.2 A outcome1 A outcome2 A assignment1 A assignment2 A improved outcome1 A improved outcome2 A feedback2.1 A feedback2.2 A
  • File Edit View Tool Template Help Peer assessment Would you like to change the titles of artifacts? artifacts are created finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1.1 A feedback1.2 A outcome1 A outcome2 A assignment1 A assignment2 A improved outcome1 A improved outcome2 A feedback2.1 A feedback2.2 A
  • File Edit View Tool Template Help Peer assessment Would you like to change the titles of artifacts? The title of an artifact has been changed finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A
  • File Edit View Tool Template Help Peer assessment Would you like to define artifacts? An artifact has been created finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A learner 2 learner 2 Review Form How do you think about this article? accept without modification accept with minor/major modification reject Comments: assessment form QTI assessment editor assessment form (QTI item) A
  • File Edit View Tool Template Help Peer assessment Would you like to define artifacts? An artifact has been defined finished artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A
  • File Edit View Tool Template Help Peer assessment How will the artifacts be distributed? Please make a decision artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A Reciprocally Mutually Randomly Default setting
  • File Edit View Tool Template Help Peer assessment How will the artifacts be distributed? An artifacts flow has been defined artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A
  • File Edit View Tool Template Help Peer assessment How will the artifacts be distributed? An artifact flow has been defined artifact role service lane stage design assessment do assignment give feedback react to feedback give feedback 2 learner1 learner1 learner1 learner 2 learner 2 learner 2 tutor tutor tutor feedback1 A feedback2 A article1 A article2 A paper1 A paper2 A improved article1 A improved article2 A tutor’s feedback1 A tutor’s feedback2 A assessment form A
    • Conclusions:
    • IMS LD, IMS QTI, and assessment-specific services can be used complementarily to support various assessment processes
    • A higher level assessment process specification and corresponding supporting tools are expected to enable practitioners to share and exchange assessment process information
    Conclusions and Future Work
    • Future work:
    • Developing the assessment process modeling tool
    • Integrating QTI editor and other assessment services
    • Developing mapping algorithm to transform APS to IMS LD + IMS QTI + assessment-specific services.
  • Thanks for your attention! Questions?
    • Contact information:
    • Project website: www.tencompetence.org
    • For details of associate partnerships: http://www.tencompetence.org/node/16
    • Contact c.kew@bolton.ac.uk for all other enquiries