Tracking and Assessing Vocational Qualifications

Uploaded on

Presentation at e-vocational 2013 Birmingham. Discussion of the use of assessment and tracking in a vocational context. HND assessment at a distance. Global assessment.

Presentation at e-vocational 2013 Birmingham. Discussion of the use of assessment and tracking in a vocational context. HND assessment at a distance. Global assessment.

More in: Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Tracking and Assessing Vocational Qualifications John Gordon Opus Learning 21st November 2013
  • 2. Opus Learning Ltd, Edinburgh • We develop online open courses to SQA HND standards – Direct to students who are enrolled with Opus – A white label service for students enrolled with other providers. • We provide the learning environment, and deliver the learning content and assessments.
  • 3. History of Opus • A spinout from • Developing and delivering Masters programmes for over 17 years
  • 4. History of Opus • We deliver highly interactive courses, delivered via an online platform such as Moodle. • With tools to manage and monitor learning with or without tutor support.
  • 5. Structure of our programmes • No longer standalone University Masters Modules • Now we build an integrated framework of Units for an SQA HND • Within a framework of commercial discipline.
  • 6. Today In this afternoon’s talk, I hope to indicate how we have: – Benefited by using technology to support and manage vocational learning; – Managed downwards the risks and costs of using technology.
  • 7. The problems Online Learning does not change the problems of assessing learning – Tracking learning – Authentication – Identifying problems in learning – Recognising need for intervention
  • 8. How do students interact when learning Student/Student Student Student/Tutor Student/Content Student/Environment Content Tutor Tutor/Content Tutor/Tutor Content/Content Content/Environment Tutor/Environment Environment Based on diagram in Anderson 2003 Environment/Environment
  • 9. Interactions and Monitoring Type of Interaction Human to human Student to Learning Platform Student to content Student and formative assessments Students and personal reflections Tracking Function Within Forums Via a ticket system for formal student to tutor interactions Via built in tracking tools on Platform Our in-built tracking and monitoring tool
  • 10. Interacting with Content Objects Student interacts with content Learner interacting with content, leavin g traces Makes notes, responds to quiz, send messages, etc Store in MLE Capture notes Print off record of learning The Digital Workbook
  • 11. The Digital Workbook is • A portfolio embedded in the content • A dashboard for learning • Rich seam of data and behaviours for mining • Ongoing formative assessment, and potential summative assessment • A pdf for the student
  • 12. Assessment In the SQA environment All units are stand alone, they can be delivered in a very flexible manner, but there are very clear pointers to how assessment should be carried out. It is no longer appropriate to base the assessment of competence on a final 3 hour written examination.
  • 13. Candidate Requests Assessment Instrument from VLE Assessment sent to Candidate – time limited PDF Assessment and Verification what happens Assessment pdf Candidate responds to Assessment Candidate Attempt Assessor reviews Assessment Attempt Makes comments Verifier consultation yes Satisfactory No Sign assessment record detailing Satisfactory Sign assessment record detailing re-assessment No Resubmit? yes Send assessment record to candidate – complete record in VLE/SQA Connect Candidate Assessment Record Verifier consultation Assessor sends comments to candidate Assessor comments Candidate attempt Candidate Resubmits
  • 14. Case study HND Business Unit by Unit Assessment In a typical instance of the HND Business each student has to: – Sit 6 formal examinations – Deliver 6 documented portfolio exercises , and – 22 other assessments - reports with a total word count of 34,500. A university Dip HE might only have 12 modules with 12 final examinations or courseworks
  • 15. Managing Assessment Load Integration of Assessment In classroom, all students work in synch possibilities of integrating assessment changing assessments due to the dynamics of teaching. The assessor and IV can • consider integration of assessment • develop a more holistic view of assessment.
  • 16. Assessment Integration Scenario From the classroom Assessment Units Nature of integration Simulated exercise Communication A meeting could be held to discuss financial performance of a business. The production of an accounting statement using, for example, spreadsheets could form the basis for an agenda item. Additional comments The emphasis is upon the development of skills that Business are applicable to a business Accounting environment. As a result, it is important to provide IT Applications candidates with the Software 1 opportunity to generate evidence through simulated IT in Business: business activity where Spreadsheets possible. For example, candidate may participate in a business meeting, they may produce a profit and loss account for a business, they may produce business documentation to support business One instrument providing a framework for the assessment of 4 units
  • 17. Open Distance Learning
  • 18. The Tensions in Technology Based Open and Distance Learning • Who is actually at the end of the screen? • What is the level of Granularity for assessment? – We want freedom for student to learn what they want when they want – We want to integrate assessment across a group of units • What is the cohort size? – We want individualised learning – We want efficiency in the use of tutor support
  • 19. Who is at the end of the screen? Opus Policy is: Digital Authentication + Physical Authentication
  • 20. Digital Authentication
  • 21. The Physical Presence Requirements of a Programme In the HND there are various points where physical presence is required by candidates. Closed book instrument of assessment Proctored examination, eg graded unit Presentation by candidate, eg in a unit using powerpoint Presence at meeting, eg unit on communications
  • 22. The Opus Authentication Table Action Point Action Sign on of candidate Assign secure login Username and to candidate to password wrong VLE, assign to tutor Invigilator/proctor Photo-id missing checks photo-id Photo-id not accepted Physical presence point Digital Check via workbook Digital Check via VLE Tutor monitors Candidate progress via Digital workbook Tutor monitors Candidate activity through activity logs Problem with ID Lack of progress Change of personality, or suspicion of personating Lack of activity Action on problem with id Re-issue username and password Do not proceed with assessment Report to Head of Centre, who will decide whether to proceed with assessment Discuss with candidate Report to Head of Centre. Discuss with candidate
  • 23. Managing Learning with Technology • At Opus we track student behaviour. • We use 2 levels of tracking – Moodle Based (the VLE) and – Content Based.
  • 24. Moodle Based Tracking • A good example: • • Nottingham University experiment tracking 800+ students
  • 25. Nottingham U. Sample Screen
  • 26. Tracking through the learning content • Opus content is massively connected to the students workbook • Every reflection, exercise, quiz, short response and so on are logged within the learning content and in the workbook • The tutor can view, track and compare workbooks
  • 27. DWB Examples – Source Page
  • 28. DWB Dashboard – Managing groups of students
  • 29. Some conclusions from tracking • Tracking appears to: – Provide the potential of personalised learning within mechanical presentation of study – Assist formative assessment in the content driving tracking and the experience of learning progress – Provide feedback on Tutor effectiveness and the effectiveness of the pedagogy underpinning the content
  • 30. The Commercial Imperative We wish to – drive down cost of assessment – Drive up the quality and efficiency of assessment within the context of a well resourced learner. We have to manage costs, and use of human resource
  • 31. Assessor/Tutor Activity Model in Distance Learning 1. Monitor VLE activity, identify any candidate with lack of progress and contact course leader for action. 2. Monitor Digital Workbook (if used) and identify any outliers in progress, contact course leader with details. 3. Respond to direct tickets from candidate, making use of FAQs for responses, if new response is required, copy response to course leader. 4. Responding to assessments; remediation advice
  • 32. Managing Resource • We can time and cost these elements • Times and costs vary with cohort size
  • 33. Timing Model for HND Business Assuming a cohort size of 20-30 students, averaged over the HND Assumptions Assessor time for single credit assessment Assessor time for double credit assessment Verifier Overhead Exam Marking Time Graded Unit Marking time minutes 22 26 7 45 Verifier minutes per script 90 Verifier minutes per project 20 40
  • 34. Timing Model vs Cohort size HND Business Timings Assumptions Students per assessor Average timing per unit for HND Business - minutes 1 5 10 15 20 93 85 77 70 64
  • 35. Learning Centre Cost Model • Based on the previous models – Time per tutor/unit/student can be estimated – Payment to Tutor/assessor is known – Viability of the Distance Course can be calculated. • We need to know these numbers and to watch them
  • 36. Potential Cost Model Possible costs/charges per student awarding body charges = £250; tutoring = £1000; tech + online content licence = £300; agent fees = £250 Centre overheads per student = £500 Total cost per student = £2300
  • 37. Potential Centre Surplus • In UK Charge to student £3500-£4500 • In India Charge to student £1500-£2500 We cannot serve one of largest markets in the world, with this model. We have to be able to manage costs.
  • 38. Learning from MOOCS • • • • Massive size of cohort Highly programmed week by week Clearly signposted weekly targets Using the student body as the main tutor toolset • Building a community of activists and lurkers • Managing and controlling costs • Generate revenue by conversions
  • 39. Learning from MOOCS • Do not do assessments – Until students join full programme • Keep the granularity of the course fairly small – not full degrees • Set up an honour system – make the students police themselves.
  • 40. Applying all these lessons • Manage the cohort size • Make sure investment reflects assessment and tutor support provided – the added value of a non-free course • Integrate assessments across units; manage the assessment overload • Make assessments social, encourage group help. Track this activity!
  • 41. Opus Strategy Small Private Open Courses • Organise cohorts in groups of 10s, not groups of 10k’s or 100k’s • Localise tutors and assessors • Build communities including assessors • Track activity and behaviour at keyboard level and at cognitive level • Record activity – perhaps award a badge for partially completed activity • Guide the learners towards completion, intervene as necessary
  • 42. Creating the Social Learner • Our DWB is part of a social learner strategy – Rich interaction spaces – Personal interaction with the content (DWB) – Interaction on specific issues (traversing the web) – Social learner (Forums, Facebook, etc.) • Manage forum and virtual space activity
  • 43. The Future • The DWB is sharable, including – with the Tutor – with other students, groups • In the future it will – support many content types – integrate with other portfolio components • The DWB will be our main assessment toolset • The DWB will be our main behaviour manager
  • 44. Conclusion • Do not just set exams and give badges – Track and Assess – Intervene and support – Audit and learn lessons • Build a community for the whole qualification • Record learning and make available for future generations • Continually improve learning content.
  • 45. The End John Gordon Opus Learning Thank You