Assessment and Credit in Online Open Qualifications


Published on

Presentation to eAssessment Tomorrow Conference January 2014, Edinburgh

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • ***
  • Assessment and Credit in Online Open Qualifications

    1. 1. Assessment and Credit in Online Open Vocational Qualifications John Gordon Opus Learning digital by default, digital by design 30th January 2014
    2. 2. Opus Learning Ltd, Edinburgh • We develop online open courses to SQA HND standards – A white label service for students enrolled with other providers. – Direct to students who are enrolled with Opus • We provide the learning environment, and deliver the learning content and assessments.
    3. 3. In this morning’s talk, I hope to indicate how we have: – Benefited by using technology to support and manage vocational learning; – Considered how to de-couple learning and assessment – Managed downwards the risks and costs of using technology.
    4. 4. Origin of Opus • A spinout from • Developing and delivering Masters programmes for over 17 years • • We deliver highly interactive courses, delivered via an online platform such as Moodle. With tools to manage and monitor learning with or without tutor support. •
    5. 5. Structure of our programmes • We build: – integrated frameworks of Units for SQA HNDs and University undergrad programmes – within a framework of commercial discipline. • Which are outward facing and globally oriented
    6. 6. The Main Issues of 2014 • • • • • • Learning and Assessment MOOCs and SPOCs Financing and selling Badges and Certificates Initial training and Life Long Learning RPL and de-coupled accredited assessment
    7. 7. The problems in an online context Online Learning does not change the problems of assessing learning – – – – Tracking learning Authentication Identifying problems in learning Recognising need for intervention MOOCs introduce new aspects – – – – – Learn without credit Gain credit after learning Building badges into credits Peer support Security issues
    8. 8. Interaction types Student/Student Student Student/Tutor Student/Content Student/Environment Content Tutor Tutor/Content Tutor/Tutor Content/Content Content/Environment Tutor/Environment Environment Based on diagram in Anderson 2003 Environment/Environment
    9. 9. Where do the major costs lie? Student/Student Student Support and Assessment Student/Content Student/Environment Content Review and Improvement Academic Activity Tutor Community Content/Content Student and Institution Management Content/Environment Environment Based on diagram in Anderson 2003 Environment/Environment
    10. 10. Interactions and Monitoring Type of Interaction Human to human Student to Learning Platform Student to content Student and formative assessments Students and personal reflections Tracking Function Within Forums Via a ticket system for formal student to tutor interactions Via built in tracking tools on Platform Our in-built tracking and monitoring tool
    11. 11. Interacting with Content Objects Student interacts with content Learner interacting with content, leavin g traces Makes notes, responds to quiz, send messages, etc Store in MLE Capture notes Print off record of learning The Digital Workbook
    12. 12. The Digital Workbook is • A portfolio embedded in the content • A dashboard for learning • Rich seam of data and behaviours for mining • Ongoing formative assessment, and potential summative assessment • A pdf for the student
    13. 13. MOOC Value Proposition • Education Access – On a massive, international scale. Currently, most students who are not enrolled college students. – May change as colleges integrate MOOC content to traditional teaching • Experimentation – New business model – New resources for learners – New learning technologies based on mining • Brand Extension – MOOCs can extend the institution’s reach and reputation internationally. – Particularly among elite research institutions, MOOCs have become a way of enhancing the institution’s brand and signalling innovation. – Successful professors can gain a global following, building their own reputation
    14. 14. How does this affect Learning and Assessment?
    15. 15. Assessment In the SQA environment All units are stand alone, they can be delivered in a very flexible manner, but there are very clear pointers to how assessment should be carried out. It is no longer appropriate to base the assessment of competence on a final 3 hour written examination.
    16. 16. Candidate Requests Assessment Instrument from VLE Assessment sent to Candidate – time limited PDF Assessment and Verification what happens Assessment pdf Candidate responds to Assessment Candidate Attempt Assessor reviews Assessment Attempt Makes comments Verifier consultation yes Satisfactory No Sign assessment record detailing Satisfactory Sign assessment record detailing re-assessment No Resubmit? yes Send assessment record to candidate – complete record in VLE/SQA Connect Candidate Assessment Record Verifier consultation Assessor sends comments to candidate Assessor comments Candidate attempt Candidate Resubmits
    17. 17. Open Distance Learning
    18. 18. The Tensions in Technology Based Open and Distance Learning • Who is actually at the end of the screen? • What is the level of Granularity for assessment? – We want freedom for student to learn what they want when they want – We want to integrate assessment across a group of units • What is the cohort size? – We want individualised learning – We want efficiency in the use of tutor support
    19. 19. Who is at the end of the screen? Opus Policy is: Digital Authentication + Physical Authentication
    20. 20. Managing Learning with Technology • At Opus we track student behaviour. • We use 2 levels of tracking – Moodle Based (the VLE) and – Content Based.
    21. 21. Tracking through the learning content • Opus content is massively connected to the students workbook • Every reflection, exercise, quiz, short response and so on are logged within the learning content and in the workbook • The tutor can view, track and compare workbooks
    22. 22. DWB Examples – Source Page
    23. 23. DWB Dashboard – Managing groups of students
    24. 24. Some conclusions from tracking • Tracking appears to: – Provide the potential of personalised learning within mechanical presentation of study – Assist formative assessment in the content driving tracking and the experience of learning progress – Provide feedback on Tutor effectiveness and the effectiveness of the pedagogy underpinning the content
    25. 25. The Commercial Imperative We wish to – drive down cost of assessment – Drive up the quality and efficiency of assessment within the context of a well resourced learner. We have to manage costs, and the use of human resource
    26. 26. Assessor/Tutor Activity Model in Distance Learning 1. Monitor VLE activity, identify any candidate with lack of progress and contact course leader for action. 2. Monitor Digital Workbook (if used) and identify any outliers in progress, contact course leader with details. 3. Respond to direct tickets from candidate, making use of FAQs for responses, if new response is required, copy response to course leader. 4. Responding to assessments; remediation advice
    27. 27. Assessor/Tutor Activity Model in MOOCs 1. Monitor VLE activity automatically. 2. Monitor Digital Workbook but ignore any outliers in progress, contact course leader with details. 3. Send any queries to student peer group for support 4. Responding to machine assessed assessments; only automatic remediation advice Take out the tutor!
    28. 28. Managing Resource • We can time and cost elements in traditional online courses • Times and costs vary with cohort size in traditional courses • For the MOOC approach, times and costs must be independent of cohort size.
    29. 29. Timing Model vs Cohort size HND Business Timings Assumptions Students per assessor Average timing per unit for HND Business - minutes 1 5 10 15 20 93 85 77 70 64
    30. 30. Learning Centre Cost Model SPOC • Based on the previous models – Time per tutor/unit/student can be estimated – Payment to Tutor/assessor is known – Viability of the Distance Course can be calculated. • We need to know these numbers and to watch them
    31. 31. Potential Cost Model Possible costs/charges per student awarding body charges = £250; tutoring = £1000; tech + online content licence = £300; agent fees = £250 Centre overheads per student = £500 Total cost per student = £2300
    32. 32. Potential Centre Surplus • In UK Charge to student £3500-£4500 • In India Charge to student £1500-£2500 We cannot serve one of largest markets in the world, with this model. We have to be able to manage costs.
    33. 33. MOOC Costs models • Charge to student – Zero • Cost for development of module not well reported – EdX charge $250k to produce a MOOC** – University of North Carolina quoted $150k for inhouse production* • Revenue shares between platform provider and university
    34. 34. Learning from MOOCS • • • • Massive size of cohort Highly programmed week by week Clearly signposted weekly targets Using the student body as the main tutor toolset • Building a community of activists and lurkers • Managing and controlling costs • Generate revenue by conversions?
    35. 35. Learning from MOOCS • Do not do assessments – Until students join full programme • Keep the granularity of the course fairly small – not full degrees • Set up an honour system – make the students police themselves and even support each others’ learning.
    36. 36. Applying all these lessons • Manage the cohort size • Make sure investment reflects assessment and tutor support provided – the added value of a non-free course • Make assessments social, encourage group help. Track this activity!
    37. 37. Opus Strategy Small Private Open Courses • Organise cohorts in groups of 10s, not groups of 10k’s or 100k’s • Localise tutors and assessors • Build communities including assessors • Track activity and behaviour at keyboard level and at cognitive level • Record activity – perhaps award a badge for partially completed activity • Guide the learners towards completion, intervene as necessary
    38. 38. Opus MOOC Experiment • From the 1st April 2014, there will at least 2 Opus Units following SQA curriculum – – – – – With no registration at SQA No access to Assessment Support Pack items With formative assessment – peer assessed With summative assessment with peer commenting Open badges for course completion/attainment • With the potential to register post mooc, and fast track to assessment.
    39. 39. The Opus Open Courses • Management: Developing Self Management Skills based on course H1F1 34 • Project Management: An Introduction based on course F1NH 34
    40. 40. Developing Self Management Skills Summative Assessment • Students are encouraged to provide evidence in a format which suits their particular situation. • It is particularly suited to an electronic portfolio method of assessment.
    41. 41. Project Management Summative Assessment • Outcomes can be assessed by the development of project management documentation, for a workplace project for which the candidate has responsibility
    42. 42. Our to do list • Design the courses • Design the peer assessment elements • Design the learning pathway (based on current open online pedagogy) • Design the Badge structure • Work with SQA to provide pathway to certification!!!!
    43. 43. The End John Gordon Opus Learning Thank You