Enhancing Nominal Group Technique
        using technologies:
            a demonstration (ID275)

                   ALT-C
           11-13 September 2012

   Jaye McIsaac, Educational Development
      Tunde Varga-Atkins, eLearning Unit
            University of Liverpool




                                      ELESIG-funded project
OUTLINE

• Introduction and purpose
• What is the Nominal Group
  Technique?
• What is the problem?
• Technologies we tried
OUTCOMES

• To describe the Nominal Group
  Technique (NGT) as a student
  feedback & evaluation tool.
• To evaluate the use of learning
  technologies in the process of a face-
  to-face NGT session.
• To consider the value of using
  technologies in online settings using
  NGT.
WHAT IS
  NOMINAL GROUP TECHNIQUE?
• Structured group activity
• One given topic
• Facilitates group
  decision-making
• Immediate results
• Reduces researcher/participant
  bias
           - Delbecq & Van de Ven
(1971)
NGT: STAGES
  Example question: ‘What would you change in your
                      course?’




1) Individual     2) Clarification and   3) Ranking responses
responses         consolidation
STAGE 1: WHAT TO KEEP DOING
      (FEEDBACK TO STUDENTS)

               1               2               3
               Annotated       Model               Audio
               ,personalised   answers             feedbac
               feedback.                           k giving
                                                   extensive
                                                   commen
4
                   5
                                                   ts
Giving             Audio           6
model              feedback            Drafts that
answers for        for                 must be
essay titles       everythin           submitted
                   g!                  and marked
                                       prior final
                                       work
STAGE 2 : WHAT TO KEEP DOING
  (FEEDBACK TO STUDENTS)
                            Model answers
                                            4
                                2
                                        Giving
                                Model
                                        model
                                answers
                                        answers for
                                        essay titles
    Audio feedback
                                            Personalised
3   Audio        5
                              6
                                Drafts that feedback
    feedbac     Audio           must be       1
    k giving                    submitted     Annotated
                feedback
    extensive   for             and marked ,personalise
    commen                      prior final   feedback.
                everythin
    ts          g!              work
STAGE 3 : WHAT TO KEEP DOING
          (FEEDBACK)



           Item no.   Item description

5 points   1          Annotated, personalised
                      feedback
4 points   2          Giving model answers
3 points   4          Audio feedback
2 points   7          etc.
1 point    10         etc.
THE TOP FIVE QUALITIES OF GOOD
               FEEDBACK
1   Timing,
    quick


              Detailed,
              on how to
         2
              improve
                          Applicable
                          to future
                   3      work
                                       Personal

                              4                   Consistent
                                                  bw
                                              5   markers
‘PROBLEMS’




        Credit by Nathan T.
        Baker
TECHNOLOGIES WE TESTED




1) Individual   2) Clarification and   3) Ranking responses
responses       consolidation
TECHNOLOGIES WE TESTED
                      Polling:
                      GoogleModerator




mobile text-entry /
MCQ polling


                             online sticky-
                             note: linoit.com


clickers

                             Web-
                             conferencing
MOBILE TEXT ENTRY
(E.G. POLLEVERYWHERE.COM, TEXTWALL.CO.UK)




   1) Individual   2)              3) Ranking
   response        Consolidation
                   and
                   clarification

                                 
CLICKERS




1) Individual    2)              3) Ranking
response         Consolidation
                 and
                 clarification

                               
GUIDE TO NGT & PROJECT REPORT




                                           See References
          On slideshare.net , search for Nominal Group
          Technique
GOOGLE MODERATOR




  Live link:
GOOGLE MODERATOR




1) Individual   2)              3) Ranking
response        Consolidation
                and
                clarification

                              
WEB-CONFERENCING
  (ELLUMINATE)
WEB-CONFERENCING




1) Individual   2)              3) Ranking
response        Consolidation
                and
                clarification

                              
ONLINE STICKY-NOTES: E.G. LINOIT.COM
LINOIT.COM




1) Individual   2)              3) Ranking
response        Consolidation
                and
                clarification

                              
TECHNOLOGIES TRIED FOR
      NOMINAL GROUP
1) Individual   2)              3) Ranking
response        Consolidation
                and
                clarification

  




                                         so…
FOR NOW: KEEPING WITH




                    and…
UNINTENDED OUTCOME 1:
   ONLINE SETTINGS
UNINTENDED OUTCOME 2:
IN OTHER STUDENT FEEDBACK
         METHODS
UNINTENDED OUTCOME 3:
 PAPER AS TECHNOLOGY
      • ubiquitous
      • manipulable
      • overlay
      • multi-items
      • text entry
      • legibility
      • visibility

                        Image by Paul Duvall
THANK YOU


Contact
Jaye McIsaac
J.Mcisaac@liverpool.ac.uk

Tunde Varga-Atkins
tva@liverpool.ac.uk
REFERENCES
• Delbecq, A., Van de Ven, Andrew, & Gustafson, D. (1975).
  Group techniques for program planning  a guide to nominal
                                               :
  group and Delphi processes. Glenview Ill.: Scott Foresman.
• Luff et al (2007) Augmented Paper: Developing relationships
  between digital content and paper
• Oliver, M. (2012) Learning with technology as a coordinated
  sociomaterial practice: digital literacies as a site of
  praxiological study
• Varga-Atkins, McIsaac et al (2011) Using the nominal group
  technique with clickers to research student experiences of e-
  learning: a project report [http://slidesha.re/xQlBCg ]
• Varga-Atkins, McIsaac et al (2011) The Nominal Group
  Technique – a practical guide for facilitators
  [http://slidesha.re/AmYOgv]

Enhancing Nominal Group Technique using technologies: a demonstration

  • 1.
    Enhancing Nominal GroupTechnique using technologies: a demonstration (ID275) ALT-C 11-13 September 2012 Jaye McIsaac, Educational Development Tunde Varga-Atkins, eLearning Unit University of Liverpool ELESIG-funded project
  • 2.
    OUTLINE • Introduction andpurpose • What is the Nominal Group Technique? • What is the problem? • Technologies we tried
  • 3.
    OUTCOMES • To describethe Nominal Group Technique (NGT) as a student feedback & evaluation tool. • To evaluate the use of learning technologies in the process of a face- to-face NGT session. • To consider the value of using technologies in online settings using NGT.
  • 4.
    WHAT IS NOMINAL GROUP TECHNIQUE? • Structured group activity • One given topic • Facilitates group decision-making • Immediate results • Reduces researcher/participant bias - Delbecq & Van de Ven (1971)
  • 5.
    NGT: STAGES Example question: ‘What would you change in your course?’ 1) Individual 2) Clarification and 3) Ranking responses responses consolidation
  • 6.
    STAGE 1: WHATTO KEEP DOING (FEEDBACK TO STUDENTS) 1 2 3 Annotated Model Audio ,personalised answers feedbac feedback. k giving extensive commen 4 5 ts Giving Audio 6 model feedback Drafts that answers for for must be essay titles everythin submitted g! and marked prior final work
  • 7.
    STAGE 2 :WHAT TO KEEP DOING (FEEDBACK TO STUDENTS) Model answers 4 2 Giving Model model answers answers for essay titles Audio feedback Personalised 3 Audio 5 6 Drafts that feedback feedbac Audio must be 1 k giving submitted Annotated feedback extensive for and marked ,personalise commen prior final feedback. everythin ts g! work
  • 8.
    STAGE 3 :WHAT TO KEEP DOING (FEEDBACK) Item no. Item description 5 points 1 Annotated, personalised feedback 4 points 2 Giving model answers 3 points 4 Audio feedback 2 points 7 etc. 1 point 10 etc.
  • 9.
    THE TOP FIVEQUALITIES OF GOOD FEEDBACK 1 Timing, quick Detailed, on how to 2 improve Applicable to future 3 work Personal 4 Consistent bw 5 markers
  • 10.
    ‘PROBLEMS’ Credit by Nathan T. Baker
  • 11.
    TECHNOLOGIES WE TESTED 1)Individual 2) Clarification and 3) Ranking responses responses consolidation
  • 12.
    TECHNOLOGIES WE TESTED Polling: GoogleModerator mobile text-entry / MCQ polling online sticky- note: linoit.com clickers Web- conferencing
  • 14.
    MOBILE TEXT ENTRY (E.G.POLLEVERYWHERE.COM, TEXTWALL.CO.UK) 1) Individual 2) 3) Ranking response Consolidation and clarification   
  • 15.
    CLICKERS 1) Individual 2) 3) Ranking response Consolidation and clarification   
  • 16.
    GUIDE TO NGT& PROJECT REPORT See References On slideshare.net , search for Nominal Group Technique
  • 17.
  • 18.
    GOOGLE MODERATOR 1) Individual 2) 3) Ranking response Consolidation and clarification   
  • 19.
  • 20.
    WEB-CONFERENCING 1) Individual 2) 3) Ranking response Consolidation and clarification   
  • 21.
  • 22.
    LINOIT.COM 1) Individual 2) 3) Ranking response Consolidation and clarification   
  • 23.
    TECHNOLOGIES TRIED FOR NOMINAL GROUP 1) Individual 2) 3) Ranking response Consolidation and clarification    so…
  • 24.
    FOR NOW: KEEPINGWITH and…
  • 25.
    UNINTENDED OUTCOME 1: ONLINE SETTINGS
  • 26.
    UNINTENDED OUTCOME 2: INOTHER STUDENT FEEDBACK METHODS
  • 27.
    UNINTENDED OUTCOME 3: PAPER AS TECHNOLOGY • ubiquitous • manipulable • overlay • multi-items • text entry • legibility • visibility Image by Paul Duvall
  • 28.
  • 29.
    REFERENCES • Delbecq, A.,Van de Ven, Andrew, & Gustafson, D. (1975). Group techniques for program planning  a guide to nominal : group and Delphi processes. Glenview Ill.: Scott Foresman. • Luff et al (2007) Augmented Paper: Developing relationships between digital content and paper • Oliver, M. (2012) Learning with technology as a coordinated sociomaterial practice: digital literacies as a site of praxiological study • Varga-Atkins, McIsaac et al (2011) Using the nominal group technique with clickers to research student experiences of e- learning: a project report [http://slidesha.re/xQlBCg ] • Varga-Atkins, McIsaac et al (2011) The Nominal Group Technique – a practical guide for facilitators [http://slidesha.re/AmYOgv]

Editor's Notes

  • #19 Tunde
  • #23 Tunde
  • #24 Tunde & JayeWhat we found:That there is no one tool that enables/enhances all 3 stages of the NGT processReference UoL work in curriculum development/supporting learning & teaching with project and examples