Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluation techniques of teaching: focus groups and Nominal Group Technique


Published on

Slides accompanying a 90-min SEDA workshop on 15th Nov 2012.
Credit to Tunde Varga-Atkins, Jaye McIsaac and Ian Willis, University of Liverpool.

It is the first time we have introduced our new, combined approach for gathering student feedback on teaching. The method can also be used in other contexts such as curriculum review or development.

The Nominal Group Technique is akin to focus groups, but with more structure and an immediate, quantitative output. Our approach has been to combine two stages: stage 1, focus group, followed by stage 2: nominal group. We have found this an effective approach at the University of Liverpool.

Published in: Education
  • Be the first to comment

Evaluation techniques of teaching: focus groups and Nominal Group Technique

  1. 1. Evaluation techniques of teaching: focus groups and the Nominal Group Technique Tünde Varga-Atkins, eLearning Unit University of Liverpool 15 November 2012 SEDA Conference Aston Business School, Birmingham
  2. 2. OUTLINE• Introduction• Demonstration of the Nominal Group Technique• Benefits and disadvantages of NGT• A new, combined two-staged approach: Nominal Focus Group• Reflections in own context
  3. 3. OUTCOMES• Familiar with the NGT and its stages.• Aware of the benefits and potential challenges of NGT.• Contrast focus groups & NGT.• Consider a combined approach of FG&NGT.• Reflect on the evaluation technique in own context.
  4. 4. CONTEXT• Curriculum development• Student engagement• Our research-informed experiences as educational developers for last 4-5 years.• Hopefully a useful evaluation technique.• Works in other contexts: staff, or any setting requiring group-decision making.• Method selected to suit purpose!
  5. 5. LISTENING TO THE EXPERIENCE? survey face-to-face groups Wood-peckers? Wolves? Dinosaurs? “I found it extremely…. helpful to have not just an idea what is going on, but to hear what is said“with the questionnaire, by students.”(staff)you never know if [you]give the right questionout.”(staff)
  6. 6. LISTENING TO THE EXPERIENCE? survey face-to-face groups Wood-peckers? Wolves? Dinosaurs?….Please rate how usefulwas the wood-pecker‟ssong in relation to yourjourney:10-Very useful. 0-Not useful at all. Gary Robson - Flickr
  7. 7. PRIOR EXPERIENCES WITH NGT? face-to-face focus groups Nominal groups interviewsquantitative qualitative surveys Delphi technique at-a-distance
  8. 8. WHAT IS NOMINAL GROUP TECHNIQUE?• Structured group activity• One given topic• Facilitates group decision-making• Immediate results• Quantitative element• Reduces researcher/participant bias Delbecq & Van de Ven (1971)
  9. 9. NGT: STAGES Example question: „What would you change in your course?‟1) Individual 2) Clarification and 3) Ranking responsesresponses consolidation Duration = 1-1.5 hours „Nominally‟ group < individual
  10. 10. AND NOW: LET’S HAVE A GO... THE QUESTION IS: In your current role as educational developer what is one key challenging issue you are facing? [purpose: identifying top 3 key ones to tackle together] [normally we would ask participants to write 2-3 – but shortening the task here due to time constraints]
  11. 11. STAGE 1: CHALLENGING ISSUE(S) 1 2 3 .4 5 6
  12. 12. STAGE 2 CONSOLIDATION: CHALLENGING ISSUES Same answers 4 2 Same answers Same answer 3 6 5 1 .
  13. 13. STAGE 3 RANKING: YOUR TOP 3 KEY CHALLENGES that you want the group/SEDA etc. to tackle… Item no. Item description3 points 12 points 21 points 4 [normally top 5 but for brevity only doing 3]
  14. 14. THE TOP FIVE KEY CHALLENGES AS EDDEV-ER:1 1 2 2 3 3 4 4 5 5
  15. 15. SUMMARY & BRIEF QUESTIONS• Nominal Group Technique • 3 stages • 1-2 key questions explored. • Focus on individual work. • Group consensus. • Quantitative outcome. • Scalable: results from more groups can be integrated.
  16. 16. BENEFITS AND CHALLENGES• Paperchase exercise• 3 minutes for each flipchart: dis/advanta ges of NGT NGT vs focus NGT vs groups surveys
  17. 17. GUIDE TO NGT & PROJECT REPORT See References On , search for Nominal Group Technique
  18. 18. Context NGT is more useful for: NGT is less useful for:Research Evaluation and decision-making Researching generalpurpose learner experiencesTopic focus When you have one single topic When you have more to explore topics or a complex topic to exploreLikely “What changes would you make “What are yourresearch to your programme/curriculum?” experiences with yourquestions “What would help you improve programme so far?” the quality of feedback on this “What are your course?” experiences with the quality of feedback on this course?”Participants Participants with different power If power relations are not relations within the same group; an issue in the group. when consulting various stakeholders groups within same research (e.g. from students
  19. 19. A COMBINED, TWO-STAGED APPROACH: ‘NOMINAL FOCUS GROUP’ Why? What is it? Jaye McIsaac, Educational Development, University of Liverpool Video at:
  20. 20. A COMBINED, TWO-STAGED APPROACH: ‘NOMINAL FOCUS GROUP’Stage 1: focus group Stage 2: nominal, ranking „bit‟
  21. 21. STUDENT EVALUATION ONEVALUATION (SURVEY, N=13) This is the Worlde cloud of the student survey responses on the group process.
  22. 22. SURVEY: EFFECT OF SESSION & F2F“felt you could be honest “All of the students agreed andand discuss openly about appeared to be facing very similar your opinions” issues to myself in terms of feedback” “Was nice to see if theUniversity cared about the Visual attribution problems we are all of responses is for having.” illustrative purposes only (survey was anonymous)
  23. 23. VIEWS ON STAGE 1: FOCUS GROUP“can speak much more info “good to hear other students‟ than writing down on a opinions to help expand my own” post-it.” “able to agree/disagree with other people‟s “Discussing made me experiences and share remember problems in Visual attribution your own to enforce or previous years.” of responses is for illustrative refute their opinion.” purposes only (survey was anonymous)
  24. 24. VIEWS ON STAGE 2: ‘NOMINAL’ BIT “outlined the mainproblems with feedback “[gave] more time toand made it clear what is think about answers.” needed to improve.” “if individuals didn‟t contribute much in the open discussion, their views were still taken in to account Visual attribution [in stage 2]” of responses is for illustrative purposes only (survey was anonymous)
  25. 25. SURVEY: VIEWS ON COMBINATION “The open discussionallowed many ideas to beput forward, where as thesecond part of the session allowed a summary of all of the views that were discussed.” “The open discussion helped to get me thinking of my own experience of Visual attribution feedback, whilst writing my of responses is for opinion on the post-it note illustrative helped get my opinion purposes only (survey was across..” anonymous)
  26. 26. STAFF PERCEPTIONS ON THE TWO-STAGED ‘NOMINAL FOCUS GROUP’ “Yes the nominal group technique, in the end it brought everything together into a sharper point again. … It kind of made it easier for us to identify what the students thought was the most important thing...”
  27. 27. STAFF PERCEPTION ON THE ‘NOMINAL FOCUS GROUP’ “dealing with basically a bullet point, … you might get the meaning wrong. You might not understand, really what they meant. Whereas [the Focus Group the citations from students] explained a bit more of what they meant. ”
  28. 28. SUMMARY: FOCUS GROUP ONLYGroup energises Volume of issues?Group helps to formulate Overall feelings?ideas and feelings Issue bias?„others feel the same!‟, Participant bias?reassurance
  29. 29. SUMMARY: NOMINAL GROUP TECHNIQUE ONLYImmediate results If more questions?Quantitative ranking If experiences?indicates volume of issues Students warmed up?Overall impression
  30. 30. SUMMARY: ADVANTAGES OF 2-STAGED NOMINAL FOCUS GROUP Stage 1: FG Stage 2: NGTGroup energises Immediate resultsGroup helps to formulate Quantitative rankingideas and feelings indicates volume of issues„Others feel the same!‟ Overall impression
  31. 31. SMALL-GROUP DISCUSSION• What is the relevance of these techniques (NFG, NGT, FG) in your context? • What methods and techniques does your institution / department use for evaluation of teaching? • What works well? • What are the challenges? Opportunities? • Any relevance of these methods? • Or considerations for using these methods?
  32. 32. KEY MESSAGEWe have found the two-staged „Nominal Focus Group‟ to bean efficient and useful method for evaluation of teaching & curriculumalldevelopment. [It is a technique to add to your repertoire of evaluation methods. It may not suit contexts, and the full evaluation cycle is the most important including a feedback loop and action on results! ]
  33. 33. [OUTCOMES] YOU:• Are now familiar with the NGT and its stages.• Discussed benefits and potential challenges of NGT.• Contrasted focus groups & NGT.• Considered a combined approach of FG&NGT.• Reflected on methods in own context.
  34. 34. THANK YOUContact: Tünde Varga-Atkins #tundeva #elearninglpool‟s resources on for nominal group technique)
  35. 35. REFERENCES• Delbecq, A., Van de Ven, Andrew, & Gustafson, D. (1975). Group techniques for program planning  a : guide to nominal group and Delphi processes. Glenview Ill.: Scott Foresman.• Further references in: • Varga-Atkins, McIsaac et al (2011) Using the nominal group technique with clickers to research student experiences of e-learning: a project report [ ] • Varga-Atkins, McIsaac et al (2011) The Nominal Group Technique – a practical guide for facilitators []
  36. 36. HANDOUTS: STAGE 1 INDIVIDUAL RESPONSEParticipants FacilitatorParticipants enter their individual Ensures everyone works on theirresponse on a post-it note. own and writes clearly and legibly.Post-its are pinned on a Facilitator helps pin up responsesflipchart. and numbers each response so that they can be referred to later.Participants read out their own Facilitator , if needed, asks for aresponse. brief clarification on the item. The items are NOT discussed in detail in this stage.
  37. 37. HANDOUTS: STAGE 2 CONSOLIDATIONParticipants FacilitatorParticipants (Ps) find Facilitator prompts Ps to findsimilar/same items. similar items. Facilitator asks Ps to work together on merging items if they are the same.Ps discuss and agree on the Facilitator adds newlymerging of similar items (group formed/merged items as Psconsensus). discuss these. (and making sure items are not themed, but only similar items areParticipants do this until all items Facilitator makes sure each itemhave been grouped if relevant. is numbered. This is the longest stage.
  38. 38. HANDOUTS: STAGE 3 RANKINGParticipants FacilitatorParticipants are asked to Faciltiator hands out rankingchoose their top 3 (normally 5) sheet and explains the ranking.most important responses tothem. The order of importance isimportant.Participants rank the items ontheir ranking sheet. (or onflipchart is also possible.) 3 pointsgo to the most important one, 2points to the second mostimportant and 1 point to thethird most imp.Participants hand in their ranking Facilitator calculates rankingsheet. score.
  39. 39. RANKING: WHAT ARE THE 3 MOST IMPORTANT ITEMS TO YOU? Item no. Item description3 points2 points1 points