2012 Computer-Assisted Assessment
•
1 like•1,323 views
My presentation at the 2012 CAA conference. An attempt to summarise how all the Moodle computer-marked assessment components fit together.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to 2012 Computer-Assisted Assessment
Similar to 2012 Computer-Assisted Assessment (20)
More from Tim Hunt
More from Tim Hunt (10)
Recently uploaded
Recently uploaded (20)
2012 Computer-Assisted Assessment
- 1. CMA in Moodle: Past, Present and Future Tim Hunt Leading Technical Developer The Open University
- 2. Overview of this talk This talk will focus on the ‘Present’ 1 Overview 2 Questions 3 Activities 4 Conclusions
- 3. What are we trying to achieve? A CMA system must address • pedagogic • administrative • technical considerations
- 4. Overview
- 6. Map of Moodle’s CMA tools
- 7. Questions
- 12. Administrative parts of Moodle
- 14. Plugins • The various conceptual parts of Moodle are made up of -plugins -core Moodle functionality • Plugins make it easy to add custom functionality • They break down the functionality into units that can be combined in different ways
- 15. Questions
- 16. Question behaviours • How the student interacts with a question • For example 1 The Student gives a response and clicks ‘Check’ 2 They immediately get told if they were right, with feedback 3 If they were wrong, the can click ‘Try again’ 4 If they get it right on a later try, they get less marks 5 After 3 incorrect tries, they are just told the answer • Some parts of this are configurable
- 17. Question behaviours (to date) • Interactive with multiple tries – as on last slide • Deferred feedback • Immediate feedback • Deferred feedback with CBM • Immediate feedback with CBM • Adaptive mode • Adaptive mode (no penalties) • Manually graded
- 18. Question types – run-time • I described the behaviour without giving any details of the question • Question types say what the question is -what is shown -how the student responds -how it is graded -how the feedback is generated
- 19. Question types – authoring • Moodle questions are authored using web forms • Teachers can define the question, within the template provided by the question type • Questions are stored in Moodle’s question bank -an area for each course / quiz -hierarchical categories in each area
- 20. Question types (to date) • All question types combine a few basic interactions -entering words -entering numbers / symbolic notations -selection -positioning things – points, molecules, parts of a diagram • Moodle now has question types for all of these separately -… but what about combinations?
- 21. Question import/export formats • A way to get questions -in to Moodle from other systems -out of Moodle • Also useful for bulk authoring
- 22. Question display options • Information should, or should not, be visible at given times -marks -right/wrong -different types of feedback • There are settings for this • and natural limits -e.g. no feedback until an answer is submitted
- 23. Putting questions together • In principle, any question can use with any behaviour • In practice, it is not that simple -e.g. essays can only be manually graded • Still, many interactive questions can be assembled
- 24. Activities
- 25. The quiz – set up • Sequence of questions to be answered -specific question or random selection -fixed order or shuffled -answered in sequence, or freely • Which students should have access • When and where attempts can be made
- 26. Quiz access rules All the different ways students may, or may not, be restricted from accessing the quiz • Open / close dates • Time limit • Number of attempts • IP address • Secure browser • Enforced delays between attempts • Acknowledge academic honesty statement first
- 27. The quiz – results • Question outcomes combined to give overall results • Results reported • Final grades sent to the grade-book
- 28. Quiz reports Allow the outcomes to be viewed, analysed and downloaded • Grades • Responses • Statistics – standard test psychometrics – CTT • Efficient manual grading
- 29. Lesson activity • Like a “choose your own adventure” book • Since Moodle 1.2 – not yet using the question bank/question engine
- 30. Other possibilities • Could be built in Moodle • Or IMS LTI could be used to connect to external tools
- 31. Summary
- 32. How does Moodle measure up? A CMA system must address • pedagogic • administrative • technical considerations
- 33. Pedagogy • Behaviours allow a range of learning tasks -Gibbs and Simpson conditions for feedback to promote learning → Interactive behaviour -Certainty Based Marking • Question types -designed to allow detailed feedback • Options for constructing quizzes • Quiz reports
- 34. Administration • Moodle provides a lot of this -courses, users, groups, authentication, enrolment -course as an sequence of activities -gradebook, event calendar, notifications • Question bank • Quiz set-up UI • Quiz access rules • Quiz reports
- 35. Technical design • Small core + plugins • Moodle as a framework -runs on Windows/Linux/Mac, Mysql/Postgres/... -concepts of courses, users, groups, uploaded files... • Simple but flexible database design for questions • Open source project -software as as a way to capture the learning of a community of practice
- 36. Where should Moodle do more? • More different activities using questions -lesson should use question bank / engine -question practice -question authoring / peer review as a learning task • Better facilities for collaborative authoring of questions -sharing between quizzes / courses / sites -versioning • More flexibility for building quizzes out of questions
- 37. Three things to remember • Question = behaviour + type • Plugins to organise software with lots of functionality • Moodle quiz comes from 10 years of community of practice -Future contributions to the open source project welcome Documentation http://docs.moodle.org/en/Quiz_module eAssessment at the OU with open source software http://v.gd/oueassessment
- 38. Map of Moodle’s CMA tools
Editor's Notes
- Before starting: Disable screen-saver. Check links on slide 5 and log in. Being a developer makes me a practitioner, rather than a theorist. I have a growing interest in what I can learn from the theory. Perhaps I have something to contribute too? As part of my role at the OU, I have been lead maintainer of the Moodle CMA tools since 2006.
- As I go, I will try to allude to relevant bits of the past. I will briefly say something about the future at the end.
- For now, this is just my assertion. You should judge as we go along. I will give summary of how Moodle achieves this at the end.
- First link is for OU staff only. The second link will work for anyone, but is using an old version of Moodle.
- This diagram summarises all the pieces of the jigsaw I hope to explain to you. In this overview, I will just give a quick summary. In the following sections, I will explain the pieces in more detail.
- Within the CMA tools, we can separate: The individual questions.
- The quiz activity.
- We can further subdivide questions into … The question engine, which deals with what happens at run-time, when a student attempts a question: How are questions displayed? How does the student enter their response? How and when is that graded? What feedback is given when?
- The question bank that lets teachers: Create and edit questions Preview the Sort them into categories Find them later Import and export them
- Once we have questions, we can combine them into quizzes. Educational activities that comprise a set of questions available to a given set of students with rules about who can attempt them when make the results available
- Once the quiz is complete, the grades are published to Moodle's gradebook. Grades can be imported from, and exported to, other systems. You can also do calculations and aggregation of grades.
- Of course, CMA is just one type of graded activity. As delegates of this conferences know, there are many type of CAA.
- Note how many other successful open source systems use plugins: E.g. Firefox add-ons. Hopefully, plugins are relatively orthogonal, allowing them to be combined freely.
- First let us think about one question at a time. Questions = Assessment items.
- Question behaviours are driven by pedagogic desires. The concept was formalised in Moodle 2.1 (June 2011) as part of the big question engine rewrite. But 'Adaptive mode' was added to Moodle 1.5 in 2005
- Don't really worry what these are. If you really want to know, preview a question in Moodle. It will let you try that question with all the behaviours.
- Again, the range of question types lets us achieve a range of pedagogic goals. Question types have been there from the start (Moodle 1.1, 2003).
- The authoring side involves a lot of administrative concerns. This is necessary to achieve the pedagogy at scale.
- No point listing the question types by name. They would not all fit on one slide anyway.
- Again, import/export is important for administration.
- The display options are not a type plug-in. Rather they are an aspect of the core question engine. This is back in the realm of pedagogic considerations.
- This is the point about the desirability of orthogonal plug-ins again. Lots of pedagogic possibilities.
- Now that we have a diverse range of individual questions, we want to put them together to form activities.
- Some issues pedagogic, others administrative.
- This type of plug-in encapsulates all the different types of restriction that Teachers, or perhaps institutions, wish to impose. Administrative – mostly. Latest type of functionality to be made a plug-in, in Moodle 2.2 (December 2011).
- Once students have attempted the quiz, you want to be able to do things with the outcomes. Important pedagogically and administratively.
- Grades report, a big table with one row per student, and one column per question, with numbers in. Responses, similar, but the response given, rather than the numerical mark. Statistics replaced the previous 'Item analysis' report in 2.0. The Manual grading report bends the definition of report.
- The fact that the Lesson uses its own question types, not the ones from the question bank, has been an anomaly since Moodle 1.6 (2006)
- The potential exists for many other types of activity built on the question bank and the question engine. IMS LTI allows the use of assessment tools outside Moodle. Interesting question whether it is right to build tools inside the VLE. But Moodle has been around for longer than LTI!
- This was the classification I proposed at the start.
- The main missing piece, pedagogically, is the lack of other activities built from questions.
- I don't see much acknowledgement of this in the literature. As a developer working at a large university, this consumes about half my time.
- Getting this right is important if your system must run at scale be reliable continue to evolve in response to user demands without becoming an unmaintainable mess The first and last points are probably the key ones here.
- The first one I have already mentioned several times. Versioning is required to promote sharing, if you look at the success of wikis – and open source software. Also, all of the authoring UI for teachers just needs to be made easier to use.
- That is all. If you just remember one thing, make it the first point: Behaviour and question type is a good, mostly orthogonal, decomposition of an assessment item. Time for your questions.
- .