Tracking student engagement in an e-learning Masters programme<br />Chris Blackmore & Prof DigbyTantam,<br />University of...
What do you think is the best way to track engagement on an e-learning module?<br />Well-being measures (pre- and post-mod...
The context<br />MSc in Psychotherapy Studies;<br />2 year PT e-learning programme;<br />NOT a clinical training course;<b...
Data<br />“Continuing Education in Psychotherapy” project funded by Leonard da Vinci fund ; <br />Data collected 2008-10 f...
New School of Psychotherapy and Counselling, UK;
University of Leuven, Belgium;
University of Bordeaux, France.</li></li></ul><li>Data set- insert earliercultural issues<br />
Partner 1<br />Mix of therapists (CPD) and lifelong learners;<br />5 or 10 week online modules;<br />5 tutors;<br />Pedago...
Partner 2<br />Psychotherapy trainees;<br />5 or 10 week online modules;<br />3 tutors; <br />Pedagogical approach- medium...
Partner 3<br />Established psychotherapists; <br />32 week cohort, in four 8 week-blocks;<br />Mix of e-learning and F2F m...
Partner 4<br />Medical undergraduates;<br />10 week online module; <br />3 tutors;<br />Pedagogical approach- low collabor...
Face-to-face comparison group<br />Psychotherapy trainees (via Partner 2);<br />Attending week-long intensive psychotherap...
What is student engagement?<br />“Student engagement is concerned with the interaction between the time, effort and other ...
Three dimensions<br />Behavioural engagement- Students who are behaviourally engaged would typically comply with behaviour...
Poles<br />Trowler, V. (2010)<br />
1. Well-being measures (pre- and post-module) <br />PHQ-9: the nine item depression scale of the Patient Health Questionna...
Which measure shall we complete as a group?<br />GAD-7: 7 items<br />SWLS: 5 items<br />5<br />
GAD-7<br />Over the last 2 weeks, how often have you been bothered by the following problems?<br />
Feeling nervous, anxious or on edge?<br />not at all<br />several days<br />more than half the days<br />nearly every day<...
Not being able to stop or control worrying?<br />not at all<br />several days<br />more than half the days<br />nearly eve...
Worrying too much about different things?<br />not at all<br />several days<br />more than half the days<br />nearly every...
Trouble relaxing?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = ...
Being so restless that it is hard to sit still?<br />not at all<br />several days<br />more than half the days<br />nearly...
Becoming easily annoyed or irritable?<br />not at all<br />several days<br />more than half the days<br />nearly every day...
Feeling afraid as if something awful might happen?<br />not at all<br />several days<br />more than half the days<br />nea...
SWLS<br />Below are five statements that you may agree or disagree with. Using the 1 - 7 scale below, indicate your agreem...
In most ways my life is close to my ideal<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree n...
The conditions of my life are excellent<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree nor...
I am satisfied with my life<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree nor disagree<br...
So far I have got the important things I want in life<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Nei...
If I could live my life over, I would change almost nothing<br />Strongly disagree<br />Disagree<br />Slightly disagree<br...
Data from the measures<br />PHQ-9: the nine item depression scale of the Patient Health Questionnaire<br />WEMWBS: The War...
Mean change scores<br />
Mean change scores<br />Yellow: p < .05;; red: p< .01<br />
Data from WB measures<br />After completing the “Existential & Human Issues” module, P1 students were more anxious, more a...
Data from WB measures<br />“Existential & Human Issues” module involves consideration of issues such as anxiety, death, me...
Pedagogy and satisfaction<br />The project partners recruited different students and used different pedagogical approaches...
1 - strongly disagree; 2 - disagree; 3 - neither agree nor disagree; 4 - agree; 5 – strongly agree<br />
Collaborative learning<br />Hypothesize that in terms of student satisfaction:<br />High-med collab e-learning<br />is bet...
2. Student activity<br />Data taken from Moodle database<br />NB- all interventions carry equal weight regardless of their...
3. Educational attainment measures (mid-module)<br />The e-learning programme was hosted via Moodle;<br />Students were as...
3. Mid-module measures of educational attainment<br />This provided feedback on student self-rated levels of:<br />relevan...
1- Almost never; 2- Seldom; 3- Sometimes ; 4- Often; 5- Almost always<br />
Collaborative learning<br />We hypothesize that in terms of student levels of educational attainment:<br />High-med collab...
4. Student mark<br />Available for P1 only- other institutions gave a pass/fail only;<br />P1 students assessed by combini...
Correlations- Kendall’s Tau<br />
Correlations- Kendall’s Tau<br />
Correlations- Kendall’s Tau<br />Yellow: p < .05;; red: p< .01<br />
Correlations<br />Using Kendall’s Tau <br />
Correlations<br />Using Kendall’s Tau <br />
Conclusion<br />Using the current data set, the lack of relationships between proposed methods of measuring engagement sug...
Conclusion<br />“Well-being” measures- <br />These give an indication of individual emotional engagement but they do not r...
Conclusion<br />Student activity-<br />This gives an indication of behaviouraland possibly cognitive engagement;<br />Ther...
Conclusion<br />Educational attainment measures- <br />These give an indication of emotional and cognitive engagement, bec...
Conclusion<br />Module mark <br />This gives an indication of cognitiveengagement;<br />This is not- in this instance- lin...
Conclusion<br />BUT this is largely dependent on:<br /> the assessment regime<br /> the connection between formative/ summ...
Conclusion<br />Another candidate<br />Linguistic analysis, using Pennebaker et al’s “Linguistic Inquiry and Word Count (L...
Conclusion<br />Qualitative analysis of a focus group on student engagement has provided epithets of engagement;<br />Thes...
Upcoming SlideShare
Loading in …5
×

HEA_Conference_Presentation_Blackmore_6.7.11

559 views

Published on

Published in: Education, Health & Medicine
  • Be the first to comment

  • Be the first to like this

HEA_Conference_Presentation_Blackmore_6.7.11

  1. 1. Tracking student engagement in an e-learning Masters programme<br />Chris Blackmore & Prof DigbyTantam,<br />University of Sheffield<br />HEA Annual Conference, 5-6 July 2011, Nottingham<br />
  2. 2. What do you think is the best way to track engagement on an e-learning module?<br />Well-being measures (pre- and post-module);<br />Student online module activity;<br />Educational attainment measures (mid-module);<br />Student end of module mark;<br />Something else.<br />5<br />
  3. 3. The context<br />MSc in Psychotherapy Studies;<br />2 year PT e-learning programme;<br />NOT a clinical training course;<br />Delivered via Moodle;<br />Combination of course materials (webpages), discussion forums and weekly chatrooms.<br />
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9. Data<br />“Continuing Education in Psychotherapy” project funded by Leonard da Vinci fund ; <br />Data collected 2008-10 from four partner institutions:<br /><ul><li>University of Sheffield, UK;
  10. 10. New School of Psychotherapy and Counselling, UK;
  11. 11. University of Leuven, Belgium;
  12. 12. University of Bordeaux, France.</li></li></ul><li>Data set- insert earliercultural issues<br />
  13. 13. Partner 1<br />Mix of therapists (CPD) and lifelong learners;<br />5 or 10 week online modules;<br />5 tutors;<br />Pedagogical approach- high collaborative e-learning.<br />
  14. 14. Partner 2<br />Psychotherapy trainees;<br />5 or 10 week online modules;<br />3 tutors; <br />Pedagogical approach- medium collaborative e-learning.<br />
  15. 15. Partner 3<br />Established psychotherapists; <br />32 week cohort, in four 8 week-blocks;<br />Mix of e-learning and F2F meetings;<br />3 tutors;<br />Pedagogical approach- medium collaborative e-learning.<br />
  16. 16. Partner 4<br />Medical undergraduates;<br />10 week online module; <br />3 tutors;<br />Pedagogical approach- low collaborative e-learning.<br />
  17. 17. Face-to-face comparison group<br />Psychotherapy trainees (via Partner 2);<br />Attending week-long intensive psychotherapy course;<br />Pedagogical approach- medium (?) collaborative F2F learning.<br />
  18. 18. What is student engagement?<br />“Student engagement is concerned with the interaction between the time, effort and other relevant resources invested by both students and their institutions intended to optimise the student experience and enhance the learning outcomes and development of students and the performance, and reputation of the institution.”<br />Trowler, V. (2010)<br />http://www.heacademy.ac.uk/resources/detail/studentengagement/Research_and_evidence_base_for_student_engagement<br />
  19. 19.
  20. 20. Three dimensions<br />Behavioural engagement- Students who are behaviourally engaged would typically comply with behavioural norms, such as attendance and involvement, and would demonstrate the absence of disruptive or negative behaviour.<br />Emotional engagement- Students who engage emotionally would experience affective reactions such as interest, enjoyment, or a sense of belonging.<br />Cognitive engagement- Cognitively engaged students would be invested in their learning, would seek to go beyond the requirements, and would relish challenge.<br />Trowler, V. (2010) from Fredricks, Blumenfeld and Paris (2004, 62-63), drawing on Bloom (1956).<br />
  21. 21. Poles<br />Trowler, V. (2010)<br />
  22. 22. 1. Well-being measures (pre- and post-module) <br />PHQ-9: the nine item depression scale of the Patient Health Questionnaire;<br />WEMWBS: The Warwick-Edinburgh Mental Well-being Scale;<br />GAD-7: the seven item scale of the Generalized Anxiety Disorder; <br />SIWB: Spirituality Index of Well-Being Scale;<br />SWLS: Satisfaction with Life Scale;<br />SAIL: Spiritual Attitude and Involvement List.<br />
  23. 23. Which measure shall we complete as a group?<br />GAD-7: 7 items<br />SWLS: 5 items<br />5<br />
  24. 24. GAD-7<br />Over the last 2 weeks, how often have you been bothered by the following problems?<br />
  25. 25. Feeling nervous, anxious or on edge?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = <br />
  26. 26. Not being able to stop or control worrying?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = <br />
  27. 27. Worrying too much about different things?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = <br />
  28. 28. Trouble relaxing?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = <br />
  29. 29. Being so restless that it is hard to sit still?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = <br />
  30. 30. Becoming easily annoyed or irritable?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = <br />
  31. 31. Feeling afraid as if something awful might happen?<br />not at all<br />several days<br />more than half the days<br />nearly every day<br />5<br />Mean = <br />
  32. 32.
  33. 33. SWLS<br />Below are five statements that you may agree or disagree with. Using the 1 - 7 scale below, indicate your agreement with each item by placing the appropriate number on the line preceding that item. Please be open and honest in your responding.<br />
  34. 34. In most ways my life is close to my ideal<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree nor disagree<br />Slightly agree<br />Agree<br />Strongly agree<br />5<br />Mean = <br />
  35. 35. The conditions of my life are excellent<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree nor disagree<br />Slightly agree<br />Agree<br />Strongly agree<br />5<br />Mean = <br />
  36. 36. I am satisfied with my life<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree nor disagree<br />Slightly agree<br />Agree<br />Strongly agree<br />5<br />Mean = <br />
  37. 37. So far I have got the important things I want in life<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree nor disagree<br />Slightly agree<br />Agree<br />Strongly agree<br />5<br />Mean = <br />
  38. 38. If I could live my life over, I would change almost nothing<br />Strongly disagree<br />Disagree<br />Slightly disagree<br />Neither agree nor disagree<br />Slightly agree<br />Agree<br />Strongly agree<br />5<br />Mean = <br />
  39. 39.
  40. 40. Data from the measures<br />PHQ-9: the nine item depression scale of the Patient Health Questionnaire<br />WEMWBS: The Warwick-Edinburgh Mental Well-being Scale<br />GAD-7: the Generalized Anxiety Disorder (7 questions)<br />SIWB: Spirituality Index of Well-Being Scale<br />SWLS: Satisfaction with Life Scale <br />SAIL: Spiritual Attitude and Involvement List<br />
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48. Mean change scores<br />
  49. 49. Mean change scores<br />Yellow: p < .05;; red: p< .01<br />
  50. 50. Data from WB measures<br />After completing the “Existential & Human Issues” module, P1 students were more anxious, more anxious & depressed, and expressed higher levels of spirituality <br />After completing the existential issues module, P3 students expressed higher levels of well-being<br />Other change scores not statistically significant<br />Individual scores did seem to pick up those struggling to engage in the modules, particularly on the WEMWBS and GAD-7 measures<br />
  51. 51. Data from WB measures<br />“Existential & Human Issues” module involves consideration of issues such as anxiety, death, meaning and freedom;<br />Students asked to relate theoretical content to personal experience;<br />Focus on anxiety during modules leads students to report experiencing more of it?<br />Increase in well-beingarrives from the satisfaction of learning with an online group, or the depth of experience?<br />
  52. 52. Pedagogy and satisfaction<br />The project partners recruited different students and used different pedagogical approaches;<br />As part of the post-module measure, they were asked four questions relating to satisfaction with the course.<br />
  53. 53. 1 - strongly disagree; 2 - disagree; 3 - neither agree nor disagree; 4 - agree; 5 – strongly agree<br />
  54. 54. Collaborative learning<br />Hypothesize that in terms of student satisfaction:<br />High-med collab e-learning<br />is better than<br />low collaborative e-learning<br />is better than<br />F2F learning (?med collab?)<br />
  55. 55. 2. Student activity<br />Data taken from Moodle database<br />NB- all interventions carry equal weight regardless of their nature or duration, e.g. logging on, posting to a forum, posting to a chatroom<br />
  56. 56.
  57. 57.
  58. 58. 3. Educational attainment measures (mid-module)<br />The e-learning programme was hosted via Moodle;<br />Students were asked to complete a 24-item Moodle questionnaire at the mid-point of each module.<br />
  59. 59. 3. Mid-module measures of educational attainment<br />This provided feedback on student self-rated levels of:<br />relevance, e.g. “my learning focuses on issues that interest me”; <br />reflective thinking, e.g. “I think critically about how I learn”;<br />interactivity, e.g. “other students respond to my ideas”;<br />tutor support, e.g. “the tutor models good discourse”;<br />peer support, e.g. “other students value my contribution”;<br />interpretation, e.g. “I make good sense of other students' messages”;<br />Students on a comparable face-to-face psychotherapy training also provided this data.<br />
  60. 60. 1- Almost never; 2- Seldom; 3- Sometimes ; 4- Often; 5- Almost always<br />
  61. 61. Collaborative learning<br />We hypothesize that in terms of student levels of educational attainment:<br />High-med collab e-learning<br />is better than<br />F2F learning (?med collab?)<br />is better than<br />low collaborative e-learning.<br />
  62. 62. 4. Student mark<br />Available for P1 only- other institutions gave a pass/fail only;<br />P1 students assessed by combining a 3,000 word assignment (75% of mark) and an assessment of their performance throughout the module (25% of mark);<br />This latter includes attendance and contribution in chatrooms and forums, plus scores in online MCQs.<br />
  63. 63. Correlations- Kendall’s Tau<br />
  64. 64. Correlations- Kendall’s Tau<br />
  65. 65. Correlations- Kendall’s Tau<br />Yellow: p < .05;; red: p< .01<br />
  66. 66. Correlations<br />Using Kendall’s Tau <br />
  67. 67. Correlations<br />Using Kendall’s Tau <br />
  68. 68. Conclusion<br />Using the current data set, the lack of relationships between proposed methods of measuring engagement suggests that none of the proposed methods are sufficient in and of themselves;<br />But we know that engagement is taking place!<br />Perhaps we can measure different aspects of engagement using different tools…<br />
  69. 69. Conclusion<br />“Well-being” measures- <br />These give an indication of individual emotional engagement but they do not relate to other potential measures of engagement such as student activity or module mark.<br />
  70. 70. Conclusion<br />Student activity-<br />This gives an indication of behaviouraland possibly cognitive engagement;<br />There is a strong relationship between student activity and self-rated levels of “Relevance”, “Interactivity” and “Peer support”; medium strength relationship with levels of “Reflective thinking”.<br />
  71. 71. Conclusion<br />Educational attainment measures- <br />These give an indication of emotional and cognitive engagement, because they capture cognitive activity and also relate to changes in anxiety and spirituality measures.<br />Also possible indication of behavioural engagement because they correlate well with student activity.<br />
  72. 72. Conclusion<br />Module mark <br />This gives an indication of cognitiveengagement;<br />This is not- in this instance- linked to other candidate markers of engagement.<br />
  73. 73. Conclusion<br />BUT this is largely dependent on:<br /> the assessment regime<br /> the connection between formative/ summative assessment and more general goals such as transformational/life-long learning and self-development<br />
  74. 74. Conclusion<br />Another candidate<br />Linguistic analysis, using Pennebaker et al’s “Linguistic Inquiry and Word Count (LIWC)”:<br /> Text analysis software program; <br /> Determines the degree any text uses positive or negative emotions, self-references, causal words, and 70 other language dimensions.<br />
  75. 75. Conclusion<br />Qualitative analysis of a focus group on student engagement has provided epithets of engagement;<br />These will be added to a LIWC dictionary and applied to the corpus of data;<br />Look for correlations with other measures, and perform ANOVA;<br />Work ongoing… <br />
  76. 76. Conclusion<br />
  77. 77. Further information<br />Today’s presentation- http://www.slideshare.net/cmblackmore/<br />University home page-<br />http://www.sheffield.ac.uk/scharr/sections/hsr/mh/cscr/staff/cb<br />

×