Appraising my Teaching Skills using MSF


Published on

Published in: Education, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Appraising my Teaching Skills using MSF

  1. 1. Appraising my ‘IMPROMPTU WARD-BASED Teaching’<br />Dr Colin Mitchell<br />MSc Geriatric Medicine<br />June 2009<br />
  2. 2. Choosing a Topic<br />I know some of my strengths as a teacher:<br />Can explain complex concepts<br />Keen to teach<br />And I’m aware of my weaknesses<br />Prone to long-windedness<br />Can over-elaborate<br />I already know to concentrate on the known weaknesses<br />But are there any issues that I’m not fully aware of?<br />
  3. 3. Is this right?<br />Previous 360o feedback<br />‘Can be intimidating’<br />Really?<br />Jars with my self-image<br />Could interfere with a conducive learning environment<br />
  4. 4. The MSF - Keeping it simple<br />Aim to focus on the learning environment<br />Is the environment conducive to learning?<br />Relaxed / Intimidating<br />Is some pressure a good thing?<br />Further focus on one skill: ward-based teaching (WBT)<br />Multiple exposures, multiple sources of feedback<br />Just F1s<br />
  5. 5. Assessing Educational Environment<br />DREEM - Dundee Ready Educational Environment Measure*<br />50 Questions – agree / disagree<br />Based on instruments in use for 40 years<br />Designed by 80+ experts using the Delphi process<br />Statistically validated to focus on reliable / discriminatory elements<br />Used worldwide to assess medical school teaching<br />(PHEEM)<br />*Roff et al, 1997. “Development and validation of the DREEM. Medical Teacher 19:4, 205-209<br />
  6. 6. Refining the Quantitative<br />Principles of choosing the items*:<br />Relevant<br />Clear, concise, focused and simple<br />Some balance of +ve and –ve (DREEM 82% +ve)<br />Chose 7 from DREEM (slightly adapted)<br />Added 4 of my own<br />Some concordance, not repetition<br />8 +ve, 3 -ve<br />*Adapted from Berk (2006), 13 Strategies to Measure College Teaching. Stylus: Virginia<br />
  7. 7. Choosing the scale<br />DREEM uses 5-point bipolar ‘intensity’ anchors:<br />Strongly Agree<br />Agree<br />Uncertain<br />Disagree<br />Strongly disagree<br />Simple but allows a breadth of response<br />Still sufficiently granular & simple<br />Balanced (✔) and has a midpoint (?✔)<br />
  8. 8. Defining the Qualitative <br />Opportunity to explore particular issues more fully<br />Correlate with quantitative themes<br />
  9. 9. Defining the Qualitative <br />Opportunity to explore particular issues more fully<br />Correlate with quantitative themes<br />
  10. 10. Making it anonymous - online<br />MSF in UK training – anonymous. In industry – 97% anonymous<br />“Maintaining the anonymity of the rater’s feedback is key to the process. Just as a tie to compensation can lead raters to soften their ratings, so can a lack of anonymity”*<br />“…when peer assessment is used in a high stakes setting, it results in inflated estimates of performance... There is no way to completely avoid this issue but it may help to ensure the anonymity of evaluators”**<br />Online forms preserve anonymity - This can be made clear in the MSF<br />Rogers E et al, 2002. “Improving the Payoff from 360-Degree Feedback”. Human Resource Development. 25: 44-54<br />**Norcini J, 2003. “Peer assessment of competence”. Medical Education. 37(6): 539-543<br />
  11. 11.
  12. 12. Validation<br />Cannot empirically validate without significant resources<br />Using 7 items from the DREEM – pre-validated<br />My own questions must be logically validated<br />Representative pilot<br />Sent to 5 SHOs. 4 replied with comments (unstructured)<br />Minor adjustments made to wording, stressing anonymity<br />
  13. 13. Results<br />
  14. 14. Rater Scores (% of Maximum)<br />
  15. 15. Question Scores (Fractions of 100%)<br />
  16. 16. Scoring<br />For negative aspects the score is reversed<br />Therefore: Higher score is a good thing<br />
  17. 17. Academic Perceptions<br />
  18. 18. Perception of the Teacher<br />
  19. 19. Teaching Atmosphere<br />
  20. 20. Free Responses (1)<br />What are the strengths of my ward-based teaching?<br />When a situation arises you take time to discuss the case (23/44*)<br />Enthusiastic and approachable… make[s] the learner think the problem through in a logical manner (30/40)<br />…instead of just giving an answer, you get us to think through the problem and help us work out the solution ourselves (24/44)<br />*Commenter’s overall score for quantitative elements shown in parentheses<br />
  21. 21. Free Responses (2)<br />What do you like LEAST about my style of ward-based teaching? (Only 5/11 replied)<br />Sometimes assume too much which can be uncomfortable (20/40)<br />Sometimes can go into a lot of depth for too long (26/44)<br />… on some occasions it would be easier to get an answer, especially when it gets busy, but I would not learn as well… (24/44)<br />Sometimes it becomes a little didactic (25/44)<br />
  22. 22. Free Responses (3)<br />How does the pressure of on-the-spot quizzing affect your learning (+vely or –vely)?<br />(In the US, “Pimping”generally viewed positively*)<br />8/11 were generally positive, 2 didn’t answer<br />Only negative response:<br />Too pressurizing and can be intimidating and lead to not wanting to ask for teaching (29/44)<br />*Wear D et al, 2005. “Pimping: perspectives of 4th year medical students”. Teaching and Learning in Medicine. 17(2) 184-191<br />
  23. 23. Free Responses (4)<br />Every teacher can improve his skills – how would you suggest I improve my ward-based teaching?<br />Be a bit more aware of when people have got lost in your explanations (20/40)<br />You are a good teacher and have a lot of knowledge, but could improve by talking more succinctly and not rambling (26/44)<br />There is very little to improve on, your teaching is excellent (25/44)<br />I think sometimes you loose [sic] the balance and focus too much on teaching rather than seeing the patients… and thus even though your teaching is good, you are not setting a good example for juniors (28/40)<br />
  24. 24. Analysis<br />
  25. 25. Robust?<br />Some basic statistical analysis of average total score:<br />Median = average (59%, around a 2.5)<br />95% CI: ±19%<br />Some observations can be made:<br />Lower scorers more likely to give a critique in the weaknesses question<br />Poor correlation between ridicule / intimidating / relaxed questions.<br />The lowest scorer commented: “the relaxed manner of teaching is definitely a good approach”<br />He/she also disagreed with the ‘relaxed atmosphere’ item (?)<br />
  26. 26. Findings – My Teaching<br />Quantitative results<br />Overall positive (59% compares with DREEM results)<br />I can be intimidating<br />Yet the atmosphere is generally relaxed (=3rd highest scoring item)<br />3 comments specifically mentioned friendly / approachable<br />Themes emerging from qualitative results<br />Rambling / long-windedness<br />Correlates with ‘teacher-centred’<br />Few negative comments about atmosphere<br />Only one comment directly mentioned ‘intimidating’ or synonyms.<br />Also note comment about assuming too much = ‘uncomfortable’ <br />This rater also agreed with both the intimidate and ridicule questions<br />
  27. 27. Findings – The Process<br />Getting good feedback is more difficult than it seems<br />Despite:<br />Limiting scope<br />Using validated questions<br />Piloting<br />Anonymous data entry<br />The results didn’t tell me much I didn’t already know<br />Partially corroborated ‘intimidating’ problem<br />Confirmed ‘long-winded’ problem<br />
  28. 28. assimilation<br />Borg Cube by Martin Teufel -<br />
  29. 29. How I see myself<br />Image from Top Gun - Paramount Pictures.<br />
  30. 30. How I see myself<br />
  31. 31. YOU BOY!<br />What I need to avoid:<br />Val-de-ree! <br />Val-de ha ha ha ha ha ha!<br />
  32. 32. Further Actions<br />I may not be intimidating to most…<br />But clearly a few may find me so<br />How do I know who they are? <br />What do I do about it?<br />Design my own feedback forms for my teaching sessions<br />Add demographics<br />Push for free-text responses<br />Don’t be afraid of constructive criticism<br />Criticism is worst when I don’t know how to fix the problem <br />
  33. 33. Questions?<br />Links:<br />Email:<br /><br />Twitter: @drcolinmitchell<br /><br />