A different installation of online assessment environment and the consequence for its use  Different installation of QMP a...
Presentations structure  (11.15-11.55) 11.15 Introduction of myself 11.20 Background of Albeda College and Kingston Univer...
Albeda College  <ul><li>approximately 25,000 students </li></ul><ul><li>approximately 2,400 members of staff </li></ul><ul...
Kingston University <ul><li>approximately 20,000 students </li></ul><ul><li>approximately 2,100 members of staff </li></ul...
Differences in QMP installation and organisation of assessments A. Differences in installation <ul><li>License  </li></ul>...
Differences in QMP installation and organisation of assessments B. Organisation of assessment process <ul><li>Assessment d...
Differences in assessment design and pedagogy <ul><li>Quality assurance </li></ul><ul><li>Use of QMP report tools </li></u...
Future developments <ul><li>Validity of objective test </li></ul><ul><li>Pedagogical developments </li></ul><ul><li>E-Asse...
Experience  Acknowledgement <ul><li>Formative assessments </li></ul><ul><li>Hardware limitations </li></ul><ul><li>Quality...
Upcoming SlideShare
Loading in …5
×

van der Sluis, H. (2009) ’A different installation of online assessment environment and the consequence for its use’

722 views

Published on

van der Sluis, H. (2009) ’A different installation of online assessment environment and the consequence for its use’. Conference paper presented at: 4th Plymouth e-Learning Conference 2009, University of Plymouth, 23 & 24 April 2009
http://www2.plymouth.ac.uk/e-learning/

Abstract
In this paper we compare two installations of an online assessment environment at institutions with contrasting assessment histories.
Kingston University (UK) has a range of tools available which support e-assessments, and includes Blackboard, Respondus and Questionmark Perception (QMP). The choice to use QMP is based on individual needs. Albeda College (Netherlands) had a pen and paper assessment tradition at the end of a module and QMP convert this to an online practice. The uses in each case have important consequences for its sustainability and lessons can be learnt by comparing and contrasting these two approaches.
The department of Social Care at Albeda College makes use of QMP as a standalone application, without integration with a VLE, to deliver summative assessments, with a central team that handles both the publishing of the assessment and provides an explicit quality assurance. This represents a centralist approach in which a small group of administrators deliver and support course teams but put constraints on the formative use of objective testing.
Kingston University, on the other hand, makes use of the Blackboard Connector to integrate Questionmark Perception with the VLE. Every lecturer is effectively an administrator and able to deliver formative and summative assessment to students. The installation provides a more decentralised approach enabling a greater degree of pedagogical flexibility; it demands different training and support for lecturers and varying levels of responsibility during the delivery of assessments. Quality assurance is provided through existing protocols for all assessments.
The paper will draw on ongoing experiences and recent literature to discuss the benefits and drawbacks of these two approaches and the implications for administration, assessment delivery, support, organisation, question analysis and pedagogy.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
722
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 4th Plymouth e-Learning Conference 2009, University of Plymouth 23 &amp; 24 April 2009 http://www2.plymouth.ac.uk/e-learning/ Abstract In this paper we compare two installations of an online assessment environment at institutions with contrasting assessment histories. Kingston University (UK) has a range of tools available which support e-assessments, and includes Blackboard, Respondus and Questionmark Perception (QMP). The choice to use QMP is based on individual needs. Albeda College (Netherlands) had a pen and paper assessment tradition at the end of a module and QMP convert this to an online practice. The uses in each case have important consequences for its sustainability and lessons can be learnt by comparing and contrasting these two approaches. The department of Social Care at Albeda College makes use of QMP as a standalone application, without integration with a VLE, to deliver summative assessments, with a central team that handles both the publishing of the assessment and provides an explicit quality assurance. This represents a centralist approach in which a small group of administrators deliver and support course teams but put constraints on the formative use of objective testing. Kingston University, on the other hand, makes use of the Blackboard Connector to integrate Questionmark Perception with the VLE. Every lecturer is effectively an administrator and able to deliver formative and summative assessment to students. The installation provides a more decentralised approach enabling a greater degree of pedagogical flexibility; it demands different training and support for lecturers and varying levels of responsibility during the delivery of assessments. Quality assurance is provided through existing protocols for all assessments. The paper will draw on ongoing experiences and recent literature to discuss the benefits and drawbacks of these two approaches and the implications for administration, assessment delivery, support, organisation, question analysis and pedagogy.
  • About myself worked at Albeda College from 1999-2007 as Teacher, ICT co-ordinator, and team co-ordinator. Since 2007 employed at Kingston University as Academic E-learning Developer. 
  • Albeda College (Vocational Education), Rotterdam   Named after dr. Wil Albeda, Albeda College is a comprehensive school for intermediate vocational education (“MBO”) and adult education in Rotterdam. approximately 25,000 students approximately 2,400 members of staff approximately 350 educations and qualification around 50 locations, in and around Rotterdam 6 different divisions, divided in several schools Vocational education in the area of Education, Healthcare and Social Care, Service Provisioning, Food, Technology and Economics. Albeda College is the result of increase in scale and merge of several schools in the area
  • Kingston University, Kingston upon Thames London   Kingston University is the largest provider of higher education in south-west London   approximately 20,000 students approximately 2,100 members of staff 4 campuses, in and around Kingston upon Thames 7 faculties, divided into schools Offering a comprehensive range of courses in the areas of Art &amp; design, Social Sciences, Business and Law, Engineering, Health and Social care, Science. 110 year history, going back to 1899 when Kingston Technical Institute opened. Became an ‘new’ university in 1992
  • License Albeda College -School of ‘Welzijn’ (Social care) only -Organisation, delivery, support comes from bureau for examination Welzijn ( Examenbureau Welzijn ) -Technical support IT services   Kinston University -University wide -Support Academic Development Centre (ADC) -Technical support It services E-assessment tools Albeda College Non, except for formative test provided by publishers   Kingston University -Respondus, -Bb Test Manager, -Web applications like Hot Potatoes, eXe Installation Albeda College Standalone application -Databases SQL -QMP v3 nl (2006-2007) -QMP v4.3 nl (2006-2007)   Kingston University Integration with Blackboard using the Blackboard Connector -Databases Oracle -QMP v4.3 and Blackboard v7 (2007-2008) -QMP v4.4 and Blackboard v8 (2008-2009) Scheduling Albeda College Scheduling done for every term/assessment. Groups, names &amp; assessment provided by team coordinators. Scheduled by Examenbureau Welzijn   Kinston University Scheduling for assessment done automatically by Bb Connector using same name and group as Bb module   Delivery Albeda College Web link provided by invigilators Login and password delivered separately   Kingston University Delivered as a link in the Blackboard module Authentication done by Blackboard Practice/usage Albeda College Intensive use of QMP at the end of each term -Approximately 3800 summative assessment 2008 -Centrally organised by Examenbureau Welzijn -10 different locations -done under supervision and in computer labs   Kingston University All year round, with peaks end of term -Approximately 4200 formative and summative assessment 2008 -Organised by individual lectures -Different locations -Computer labs &amp; online access   Administration Albeda College QMP administrators bases at Examenbureau Welzijn Training of administrators done by STOAS (QMP distributor in the Netherlands)   Kingston University Every lecturer administrator of courses in Blackboard and administrator in QMP Training provided by members of ADC
  • Assessment design, publishing &amp; delivery Albeda College -Need for items investigated by members of Examenbureau Welzijn Items written by teachers -Reviewed by committee, -Published and distributed by Examenbureau Welzijn   Kingston university Lecturer write, publish and distribute assessment   Invigilation Albeda College Students supervised in computer labs by course team members   Kingston University Students supervised in computer labs by lecturer and/or professional invigilators, Unsupervised and/or offsite   Marking and review Albeda College Assessment will be reviewed by committee, discuss item, set waiting, and mark items for review.   Kinston University Lecturer reviews assessment, set waiting and reviews assessment
  • Quality assurance Albeda College Examenbureau welzijn is a central team provides: - explicit quality assurance -organisation for writing items -advise on assessments design -support teams with advice on organisation   Kingston university Lecturers -quality assurance is provided through existing protocols for all assessments. -advise on assessments provided by ADC -advise on organisation provided by ADC   Use of QMP report tools Albeda College -usable by scale -used for re-evaluation   Kingston University -hardly usable by shortage of scale -hardly re-evaluations of questions   Third party items/assessments Albeda College Sometimes items sheared or sold to other colleges   Kingston University -sometimes items provide by publishers as part of a book or learning materials -could make more use HEA Subject Network    Assessment mix Albeda College -module content is assessed with by objective test -reflective interview at workplaces   Kingston University Assessment mix is a decided by lecturers -objective test only -mix with subjective test Design of objective test Albeda College -mainly MCQ -summative, no feedback   Kingston University -mainly MCQ -occasionally other item types -formative with feedback -summative without feedback  
  • Validity of objective test Albeda College Ongoing discussion on value of using objective assessments   Kingston University Position on objective test based on individual taste and culture   Pedagogical developments Albeda College Chance in teaching and learning in vocational education forced by government. More emphasis on Competent Based Learning (partly comparable with Social Constructive model ) -less modular delivery of content and subjects -probably no longer or fewer objective test. -new assessments strategies with portfolio   Kingston University More emphasis on -social constructive model of teaching and learning -assessment mix with different tools -skill bases learning -PDP Personal Development Plan -live long learning   E-assessment development Albeda College -implementing a CMS Portal, Portaal, (provided by i-Engine) -e-portfolios   Kingston University Availability of SNS (ELGG) for e-portfolio Assessments with Wiki’s &amp; Blog
  • Experience Formative assessments The installation at Kingston University using the QMP Blackboard Connector offers by more autonomy for lecturers because every lecture is an administrator. This enables lectures to offer students formative assessments. At Albeda College because of the installation which limits the possible administrators and the central organisation of assessments makes it difficult for individual teachers to use QMP for formative assessments.   Hardware limitations At Albeda College hardware puts constrains on assessment delivery. Not all buildings have sufficient computer labs to assess students and most computer labs are too small to host all students in once. Rotation of available places in computer labs put stress on students because they have to wait for available places and the rotations is noisy in the rooms. Secondly because of this limitation a pen-and-paper practice is still required, what results in extra work for Examination Department, this neutralize the staff time return of marking.   Quality insurance QMP offers more advanced analysis of assessment than for example the Blackboard test manager. Albeda College can fully use them because of the student scale. Lecturers as Kinston hardly use the Enterprise report tools to review assessment by lack of numbers for the statistics analysis and unawareness of the tools.   Value added It is a steep learning curve for lectures at Kingston University to master all features of QMP. It takes considerable time compared to the Blackboard Test Manager to use QMP. Secondly most lectures will almost never fully use the advanced features like the report tools Secondly, although QMP offers a couple of questions types more than other tools, most assessments are designed mainly with MCQ and other items that are available in the Blackboard Test manager for example.
  • van der Sluis, H. (2009) ’A different installation of online assessment environment and the consequence for its use’

    1. 1. A different installation of online assessment environment and the consequence for its use Different installation of QMP at Albeda College and Kingston University At the 4th Plymouth e-Learning Conference 2009, University of Plymouth Hendrik van der Sluis [email_address]
    2. 2. Presentations structure (11.15-11.55) 11.15 Introduction of myself 11.20 Background of Albeda College and Kingston University 11.25 Differences in QMP installation and organisation of assessing A. Difference in installation and organisation B. Assessment process 11.30 Differences in assessment design and pedagogy 11.35 Further developments 11.40-11.55 Experience/Questions
    3. 3. Albeda College <ul><li>approximately 25,000 students </li></ul><ul><li>approximately 2,400 members of staff </li></ul><ul><li>approximately 350 educations and qualification </li></ul><ul><li>around 50 locations, in and around Rotterdam </li></ul><ul><li>6 different divisions, divided in several schools </li></ul>
    4. 4. Kingston University <ul><li>approximately 20,000 students </li></ul><ul><li>approximately 2,100 members of staff </li></ul><ul><li>4 campuses, in and around Kingston upon Thames </li></ul><ul><li>7 faculties, divided into schools </li></ul>
    5. 5. Differences in QMP installation and organisation of assessments A. Differences in installation <ul><li>License </li></ul><ul><li>E-assessment tools </li></ul><ul><li>Installation </li></ul><ul><li>Scheduling </li></ul><ul><li>Delivery </li></ul><ul><li>Practice/Usages </li></ul><ul><li>Administration </li></ul><ul><li>Training </li></ul>
    6. 6. Differences in QMP installation and organisation of assessments B. Organisation of assessment process <ul><li>Assessment design </li></ul><ul><li>Publishing </li></ul><ul><li>Delivery </li></ul><ul><li>Invigilation </li></ul><ul><li>Marking </li></ul><ul><li>Review </li></ul>
    7. 7. Differences in assessment design and pedagogy <ul><li>Quality assurance </li></ul><ul><li>Use of QMP report tools </li></ul><ul><li>Third party items/assessments </li></ul><ul><li>Assessment mix </li></ul><ul><li>Design of objective test </li></ul>
    8. 8. Future developments <ul><li>Validity of objective test </li></ul><ul><li>Pedagogical developments </li></ul><ul><li>E-Assessment development </li></ul>
    9. 9. Experience Acknowledgement <ul><li>Formative assessments </li></ul><ul><li>Hardware limitations </li></ul><ul><li>Quality insurance </li></ul><ul><li>Value added </li></ul><ul><li>I gratefully acknowledge the support of: </li></ul><ul><li>Dr T. Linsey (Kingston University), </li></ul><ul><li>J. Berkhout (Albeda College). </li></ul>

    ×