Cmdn research symposium ajjawi final


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I would like to start by acknowledging the entire team who are involved in this project. This is a JISC 3 year funded project (£125, 000) to reengineer assessment feedback practices and look at the issue of feedback in distance learning. We are half way into the 3 year project.
  • We run a well known postgraduate cert, diploma and masters in med ed. It’s a fully distance learning programme that recently moved to an online platform. 3000students and no cohorts, rolling enrolments.
  • Effective feedback can lead to change in learners through affective processes of increased effort, motivation and engagement, or through cognitive processes including restructuring of knowledge and alternative strategies to understanding (Hattie and Timperley, 2007).This contributes to the low level of perceptions that students have about feedback and the feedback gap documented in the literature.
  • Underpinning theoretical approachDialogical process NOT a product to be delivered
  • 140 assignments assessedMain focus of feedback was content related (95%)Main level of feedback was on the task (72%)Positive and negative feedback was equal (51% vs. 49%)Confirmed the wide variety in quantity and nature of feedback provided by tutors
  • Modular approach to sequencing of assessmentExplicating feedback times in a student-tutor charterDevelopment of assessment rubrics for all summativeIncreased use of formative assessmentsFaculty development around feedback
  • Wiki journal acts as a repository of all the students’ work across the entire programme accessible to the student and tutors only.Main changes and how it links with dialogue
  • Cmdn research symposium ajjawi final

    1. 1. Dr Rola Ajjawi interACT: Interactive Assessment and Collaboration via TechnologyCentre for Medical Education, Tay Park House, 484 Perth Road, Dundee DD2 1LR, Scotland, UK
    2. 2. The Project Team Rola Ajjawi Karen Barton Grant Murray Susie SchofieldProject Director Project Officer Learning Technologist Project Manager Natalie Lafferty Sean McAleer David Walker e-learning Advisor Assessment Advisor Learning Technology Advisor
    3. 3.
    4. 4. A critique of monologicfeedback1-6• Lack of learner engagement with feedback• Lack of understanding of feedback (and acting on)• Transmitted feedback creates dependency on teacher• Not utilising self-evaluation or peer-feedback• High teacher effort — low efficiency• Lack of a shared context for assessment between teacher and learner• Reduced staff satisfaction as evidence of feedforward not seen
    5. 5. Theoretical shift in conceptionsof feedback that: is a dialogue not a product to be delivered (relational rather than transmissive)7 develops the students’ capacity to make evaluative judgements about their own and others’ work1,2 serves the function of progressively enabling students to better monitor, evaluate and regulate their own learning, independently of the teacher3
    6. 6. Aims & Research questionsAim We sought to re-engineer our assessment and feedback processes in line with good practice principles from the literature and the use of technologyResearch questions How can technology be used to promote feedback dialogue in online distance learning? What are tutors’ and students’ current conceptions of feedback and barriers to change? What is the impact of dialogic feedback processes on students’ self-assessment ability, on tutors’ feedback profiles and key stakeholder satisfaction?
    7. 7. Research approach Action research8 Data collection methods  Student and staff interviews  Questionnaire  Content analysis of cover pages and wikis  Quantitative data about engagement rates and time to progress through modules  Routinely collected data from module evaluations, external examiner reports  Feedback audit
    8. 8. PG Certificate in Medical Education Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 chapters chapters chapters chapters chapters chapters chapters chapters chapters chapters Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 Equivalent 2 chapters chapters chapters chapters chapters chapters chapters chapters chapters chapters Revised PG Certificate in Medical Education Formative assessmentChapters 2 4 6 8 10 Medium stakes assessment High stakes assessmentChapters 2 4 6 8 10Chapters 2 4 6 8 10 ©Mark Russell 2010Chapters 2 4 6 8 10 University of Hertfordshire ESCAPE project
    9. 9. Student engagement interACT launched 30th April 2012 100% completion of the cover page Wiki engagement varies from 65-20% 85.2%(n=46) of the students who responded to the survey thought that the instructions provided about the assignment submission process were clear
    10. 10. 40 35 Disagree/Strongly Disagree 30 Not sure 25Number of Students Agree/Strongly Agree 20 15 10 5 0 Valuable for my Valuable for my Promoted Promoted Self Beneficial to own learning own teaching dialogue about Evaluation future feedback assignment tasks
    11. 11. Majority agree the cover page isvaluable (CP) Ability to request feedback about specific issues Allows last review of work before submission and focuses the self assessment Gives you a chance to tell the tutor what your concerns about the assignment are up front‘ It is a chance to think critically about how you approached the assignment and the assignment assessment criteria There is an element of expectation that there will be some dialogue
    12. 12. Aspects I would like feedback on: Ideas of not preparing too rigidly in order to be flexible within sessions – practical advice would be welcomed! As the first essay I have written in nearly 20 years, I would like to know whether the standard overall was acceptable Please advise me how I can enter a specific page number in a reference when using Endnote Feedback on whether my peers have had similar thoughts for their own teaching, or other ideas that have been commonly developed would be beneficial in case I have not thought or considered them any part of it
    13. 13. How did previous feedbackinform this assignment It made me realise that instead of focusing on a single or a few key teaching principles, I focused on many of them without going into much detail Feedback that my writing style was agreeable was reassuring. I appreciated knowing my use of literature was valid so have tried to continue applying the literature to my work. I tried to be careful to define and reference jargon It was really helpful in writing present assignment
    14. 14. Which aspect(s) of your assignment would you specifically likefeedback on? Student comment (CP): I originally had a paragraph around learning principles and theories relating to small group teaching. I unfortunately had to delete it due to the word restriction. Is that something I should have included? Tutor feedback (CP): Would you if you were the new tutor have appreciated this? Student comment (wiki): I do think the tutor would have appreciated a small paragraph around educational principles and theories to help put small group teaching into context. I think that 5 pages though is plenty for the tutor to read, anymore than this and they may not have time to read it all. I have also learned how to feedback to the tutor based on the feedback given.
    15. 15. Which aspect(s) of your assignment would you specifically likefeedback on? Student comment: My choice of the Angoff method for this exam. I heavily considered the borderline group method, but decided against it as there was relatively little evidence for its use in exams with smaller examinee numbers such as this one. Thanks. Tutor feedback: Yes, when borderline group method is applied to an exam with a small group of candidates there is a risk of not finding any borderline candidates! However, if you use borderline regression method, in which marks of all candidates are considered for setting the pass mark, this risk can be overcome.
    16. 16. Conclusions & implications Creating assessment and feedback dialogue in online distance learning is possible Majority of students find it valuable for their learning and that it promotes self-evaluation and dialogue Feedback audit has been a useful tool for faculty development initiatives and measuring change in quantity and quality of feedback Challenges have included streamlining the process, dealing with tokenism, and improving the quality and timeliness of the feedback
    17. 17. Research &Development Continue to refine and streamline the process More work is needed to help students better understand the pedagogic rationale and how to optimise their use of feedback Embed into programmatic assessment approach through introduction of a personal development plan and patchwork/capstone assessment Future research needs to explore how the online environment mediates learning from feedback
    18. 18. Project website
    19. 19. So what did we do…
    20. 20. Contact DetailsRola AjjawiCentre for Medical EducationUniversity of DundeeEmail: @r_ajjawi BartonCentre for Medical EducationUniversity of DundeeEmail:
    21. 21. References1. Sadler, D. R. (2010) Beyond feedback: developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35:5, 535-550.2. Boud, D., & Associates. (2010). Assessment 2010: Seven propositions for assessment reform in higher education. Sydney: Australian Learning and Teaching Council3. Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218.4. Nicol, D. (2012). Assessment and feedback - in the hands of the student [Online]. JISC. Available: feedback%20-%20in%20the%20hands%20of%20the%20student [Accessed 01/02/12].5. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg (Eds) Innovative assessment in higher education. London, Routledge), 81-91.6. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational Research, 77, 81-112.7. Ajjawi, R. (2012). Going beyond ‘received and understood’ as a way of conceptualising feedback. Medical Education, 46(10), 1018-1019.8. Carr, W. & Kemmis, S. 1986. Becoming critical: education, knowledge, and action research, Lewes, Falmer Press.