This document provides an overview of consensus methods for determining agreement among experts or groups. It defines consensus methods, describes their key features like anonymity and iteration. It then explains two specific consensus methods - the Delphi technique and nominal group technique. The Delphi technique uses anonymous questionnaires and controlled feedback over multiple rounds to gather experts' judgments. The nominal group technique involves individuals privately generating ideas which are then discussed and ranked by the group. The document compares these methods and their characteristics like participant selection and data analysis.

1. CONSENSUS
METHODS
Upasana Sharma
PhD Scholar
ICMR-National Institute of Epidemiology
SCTIMST , Trivandrum

2. OUTLINE
1. Defining Consensus Methods
2. Application of Consensus Methods
3. Delphi Technique
4. Nominal Group Technique
5. Comparison of group decision making
processes
6. Conclusion
7. References
2
PhD Seminar-2

3. Defining Consensus Methods
 Aim of consensus methods is to determine the extent to
which experts or lay people agree about a given issue
 Means of dealing with conflicting scientific evidence
 Primarily concerned with deriving quantitative estimates
through qualitative approaches
 To assess the extent of agreement (consensus
measurement) and to resolve disagreement (consensus
development)
 Seeks to control the effects that can often bias the
process of expert discussion.
3
PhD Seminar-2

4. Features of Consensus Methods
Anonymity To avoid dominance; achieved by use of a
questionnaire in Delphi and private ranking in
nominal group
Iteration Processes occur in "rounds", allowing
individuals to change their opinions
Controlled feedback Showing the distribution of the group's
response (indicating to each individual their
own previous response in Delphi)
Statistical group
response
Expressing judgment using summary
measures of the full group response, giving
more information than just a consensus
statement
PhD Seminar-2
4

5. Consensus Methods-
Application
Delphi process, Nominal group
technique, Consensus development
conference : Consensus measurement
Nominal group technique, Consensus
development : Consensus
measurement & Consensus
development
5
PhD Seminar-2

6. The Delphi Technique
(History)
 Takes its name from the Delphic oracle's skills of
interpretation and foresight
 Referred to the principle of the oracle as non-falsifyable
prediction
 Developed at the beginning of the Cold War to forecast
the impact of technology on warfare.
 To cross the shortcomings of traditional forecasting
methods , the Delphi method was developed by Project
RAND during the 1950-1960s (1959)
6
PhD Seminar-2

7. Objectives of Delphi Method
 To determine or develop a range of possible
program alternatives;
 To explore or expose underlying assumptions or
information leading to different judgments;
 To seek out information which may generate a
consensus on the part of the respondent group;
 To correlate informed judgments on a topic
spanning a wide range of disciplines
 To educate the respondent group as to the
diverse and interrelated aspects of the topic
7
PhD Seminar-2

8. Characteristics of Delphi Method
 Delphi is an expert survey in two or more rounds
 Starting from the second round, a feedback is
given (about the results of previous rounds).
 The same experts assess the same matters once
more - influenced by the opinions of the other
experts.
 Delphi are judgment processes ,Persons involved
in it only give estimations
 Common surveys try to identify ‘what is’ whereas
Delphi technique attempts to address ‘ what
could/should be’
8
PhD Seminar-2

9. http://projectsmileireland.co
m
9

10. Expert panel in Delphi Method
 No set guidelines in literature
 Based on judgment and discretion of principal
investigator
 Leading figures in their field of expertise
 Experts from related field
 People who are accepted as experts by other
experts of that field (nomination process)
 10-15 Experts with homogenous background
 Upto 50 incase various reference groups
 Studies with no.of experts exceeding 50 have
been done
10
PhD Seminar-2

11. Time requirements in Delphi
method
 For administration of delphi study – 45 days
 Time between iterations – 2 weeks
11
PhD Seminar-2

12. Data analysis in Delphi method
 Decision rules must be established to assemble and
organize the judgments of experts
 One criterion recommends 80% of subjects’votes fall
within two categories on a 7 point scale (Ulschak,1983)
 Atleast 70% of the subjects need to rate 3 or higher on a
4 point likert scale and median should be 3.25 or higher
(Green,1982)
 Use of percentage measures is inadequate (Scheibe,
Skutsch, and Schofer ,1975)
 Measures of central tendency and levels of dispersion
 Literature strongly suggests median score based on
likert scale
 Use of ‘Mode’ has also been recommended (due to
clustering of results at 2 or more points )
12
PhD Seminar-2

13. Advantages of Delphi Method
 Multiple iterations (controlled feedback process)
 Subject anonymity
 Confidentiality of participants maintained by geographic
dispersion and electronic communication
 Focus is on ideas rather than individuals
 Ability to use statistical analysis technique – reduces
potential of group pressure for conformity
13
PhD Seminar-2

14. Disadvantages of Delphi
Method
 Judgments are those of the selective experts
and may not be representative
 Time consuming
 Expert biasing
 Requires adequate participant commitment
14
PhD Seminar-2

15. Delphi technique : Types
 Consensus delphi (Traditional delphi) : To gain consensus as the result of
the process
 Policy delphi (Modern delphi) : To bring into light new and fresh viewpoints,
knowledge and ideas and make other experts to react to them future states
of the topic under study
 Trend delphi : Starting point is a single trend which is presented as a
line/curve panelists are asked to continue the line/curve and then give
grounds to the course of development they drew by listing their
presumptions and those factors of uncertainty which might change the
line/curve In the end, each panelist is asked to vote about the different
alternative lines
 Real time delphi : Online questionnaire, does not include sequential rounds;
In the given time frame, the panelists can visit the questionnaire, read the
comments and discussion by the other panelists and then react and change
their opinion as often as they wish.
 Delphi survey : Quantitative, Large panels ,computer-based surveys, level of
15
PhD Seminar-2

16. Nominal Group Technique
PhD Seminar-2
16
 A structured process which taps the
experiences, skills, or feelings of participants
 Round-robin is a process for serially recording
ideas where each participant provides an idea
in turn
 A process in which the members work
independently but in each other’s presence
 No discussion occurs, although the leader may
ask for a show of hands on how many
participants had a similar idea
 Final judgments expressed by voting

17. Characteristics : Nominal Group
Technique
PhD Seminar-2
17
Developed in 1968 as
a participation
technique for social
planning by Delbecq
and Van de Ven
Structured variation
of focused group
discussion
Involves problem
identification, solution
generation, and
decision making
Used for primary data
collection, program
planning, evaluation
and exploratory
research
Discussion on one
given topic
Immediate results
Final outcome is a
list of ideas and a
rank-ordering of their
importance.
Reduces
researcher/participant
bias

18. The Nominal Group Technique –
Process
A small group of 5-9 people gather around a table. Facilitator
identifies judgment issue and gives participants procedural
instructions.
Participants write down all ideas that occur to them, keeping
their lists private at this point. Creativity is encouraged during
this phase.
Facilitator asks each participant to present ideas and writes
them on a blackboard or flipchart, continuing until all ideas have
been recorded.
Participants discuss each other’s ideas, clarifying, expanding,
and evaluating them as a group.
Participants rank ideas privately in their own personal order and
preference.
The idea that ranks highest among the participants is adopted
as the group’s judgment.
18
PhD Seminar-2

19. Participants and Facilitator in NGT
PhD Seminar-2
19
 The facilitator of the nominal group should not
bias the group toward his view
 He must control unwanted group behavior
without alienating people
 The facilitator should be an expert on the topic
for discussion, or a credible non-expert (Delbecq and
van de Ven 1971, Glaser 1980)
 The composition of the NG group may be
homogeneous or heterogeneous

20. Time requirement in NGT
PhD Seminar-2
20
 In a study quoted by Delbecq , et al. (1975),
the total administrative man-hours to prepare,
conduct, and follow through for one group
required an average of 4.4 hours for NGT
 No. of nominal group meetings to be held will
depend on the nature of the question and
accessibility to the key stakeholders best
suited to help address the problem

21. Other requirements in NGT
PhD Seminar-2
21
 A flip chart and marking pens are needed for
each group
 Index cards (or similar small cards),
approximately six per participant, facilitate the
voting process.
 The seating arrangement must allow all
members to easily focus on the ideas listed on
the flip chart
 Some means of displaying the completed
chart pages is necessary (e.g., tacks or
masking tape to afix them to the walls).

22. Analysis of NGT Data
PhD Seminar-2
22
Analysis of NGT data involves both qualitative and
quantitative procedures requiring four basic steps:
 Categorization of initial problem statements into
problem themes
 Regrouping of problem themes within conceptual
model to form major problem dimensions
 Calculation of a score or index reflecting the
importance of each problem theme
 Ranking of problem themes according to their
importance index

23. Advantages of NGT
PhD Seminar-2
23
Dominanc
e reduced
• Dominance by aggressive members is reduced since
each has an equal opportunity
Silent
generatio
n
• Silent generation of ideas minimizes the interruptions in
each person’s thought processes
• Time to record or think about as many ideas
Written
record
• Witten record increases the group’s ability to deal with a
large number of ideas. It also avoids the loss of ideas.

24. Advantages of NGT contd..
PhD Seminar-2
24
Prematur
e focus
avoided
• Avoids premature focusing on single idea
Clarificatio
n
• Opportunity to clarify or elaborate on ideas
• Recorded items checked for duplication/grouping
Inter&
Intra
group
difference
s
• Process of identifying and scoring problem themes
makes it possible to study both intra- and inter-
group differences

25. Disadvantages of NGT
PhD Seminar-2
25
1. Cross-fertilization of ideas is diminished due to the
structure imposed by the NGT
2. The NGT may reduce flexibility
3. The major disadvantage of NGT relates to sampling.
Because participants have to agree to come to a
central meeting location, attempts at probability
sampling are met by a serious level of non-response
4. Bringing group members together may be cost
prohibitive

26. Comparison of group decision making
processes
PhD Seminar-2
26
Attribute Decision making process
Delphi FGD Brainstormin
g
NGT
Face-to-face group meeting process No Yes Yes Yes
Generates a large number of ideas Yes May be May be Yes
Avoids focusing on a single train of
thought
Yes Yes No Yes
Encourages equal input from all
participants
Yes No No Yes
Highly structured process Yes May be No Yes
Meeting time usually 1-2 hours duration No Yes Yes Yes
Avoids ‘quick’ decision making Yes No No Yes
High degree of task completion Yes May be No Yes
Provision of immediate feedback No Maybe Maybe Yes
Measures the relative importance of Yes No No Yes

27. Conclusion
PhD Seminar-2
27
Consensus studies should focus on
carefully defined problems that can
be investigated in a timely and
economical way
Objective and skilled leaders shouild
administer the consensus process ,
participants should qualify for
selection because they are
representative of their profession,
advisable to include consumers
Decisions should be justified by
reliance on available empirically-
derived data as well as on
judgments, Level of consensus must
be defined in advance, Findings
should represent clear and specific
guides to action
Careful dissemination of the findings ,
Consensus process should be
summarized critically in a background
paper and reported to the participant
with estimations of their reliability,
Monitoring
Consensus
Method

28. Research studies using Consensus
Methods
PhD Seminar-2
28
 Ferri CP, Prince M, Brayne C, Brodaty H, Fratiglioni L, Ganguli M, Hall K,
Hasegawa K, Hendrie H, Huang Y, Jorm A. Global prevalence of dementia: a
Delphi consensus study. The lancet. 20 6 Jan 6;366(9503):2112-7.
 Turris SA, Steenkamp M, Lund A, Hutton A, Ranse J, Bowles R, Arbuthnott K,
Anikeeva O, Arbon P. International consensus on key concepts and data
definitions for mass-gathering health: process and progress. Prehospital and
disaster medicine. 2016 Apr 1;31(02):220-3..
 Corner J, Wright D, Hopkinson J, Gunaratnam Y, McDonald JW, Foster C. The
research priorities of patients attending UK cancer treatment centres: findings
from a modified nominal group study. British journal of cancer. 2007 Mar
26;96(6):875-81.
 Kaufmann CP, Stämpfli D, Hersberger KE, Lampert ML. Determination of risk
factors for drug-related problems: a multidisciplinary triangulation process. BMJ
open. 2015 Mar 1;5(3):e006376.
 Kleiner-Fisman G, Gryfe P, Naglie G. A patient-based needs assessment for
living well with Parkinson disease: implementation via nominal group technique.
Parkinson’s Disease. 2013 Feb 17;2013.
 Turner JP, Edwards S, Stanners M, Shakib S, Bell JS. What factors are
important for deprescribing in Australian long-term care facilities? Perspectives
of residents and health professionals. BMJ open. 2016 Mar 1;6(3):e009781.

29. References
PhD Seminar-2
29
1. Bourrée F, Michel P, Salmi LR. Consensus methods: review of original methods and their main
alternatives used in public health. Rev Epidemiol Sante Publique [Internet]. 2008 Dec [cited 2017
Mar 2];56(6):415–23. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3838316/
2. Ciałkowska M, Adamowski T, Piotrowski P, Kiejna A. [What is the Delphi method? Strengths and
shortcomings]. Psychiatr Pol. 2008 Feb;42(1):5–15. de Villiers MR, de Villiers PJT, Kent AP. The
Delphi technique in health sciences education research. Med Teach. 2005 Nov;27(7):639–43.
4. Falzarano M, Pinto Zipp G. Seeking consensus through the use of the Delphi technique in health
sciences research. J Allied Health. 2013;42(2):99–105.
5. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for
use. Am J Public Health. 1984 Sep;74(9):979–83.
6. Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: a research
tool for general practice? Fam Pract. 1993 Mar;10(1):76–81.
7. Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987 Nov;12(6):729–34.
8. Hutchings A, Raine R, Sanderson C, Black N. A comparison of formal consensus methods used for
developing clinical guidelines. Journal of Health Services Research & Policy [Internet]. 2006 Oct
1 [cited 2017 Mar 2];11(4):218–24. Available from:
http://dx.doi.org/10.1258/135581906778476553
9. Kennedy HP. Enhancing Delphi research: methods and results. J Adv Nurs. 2004 Mar;45(5):504–
11.
10. Pearson SD, Margolis CZ, Davis S, Schreier LK, Sokol HN, Gottlieb LK. Is consensus
reproducible? A study of an algorithmic guidelines development process. Med Care. 1995
Jun;33(6):643–60.
11. Peiró Moreno S, Portella Argelaguet E. [Consensus doesn’t always mean agreement: limitations
of consensus methods in health services]. Gac Sanit. 1993 Dec;7(39):294–300.

30. PhD Seminar-2
30
12.Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003 Feb;41(4):376–82
13. Toward JI, Ostwald SK. Exploring mental health service needs for the elderly: results of a modified Delphi study.
Community Ment Health J. 2002 Apr;38(2):141–9.
14. Uscher-Pines L, Babin SM, Farrell CL, Hsieh Y-H, Moskal MD, Gaydos CA, et al. Research priorities for syndromic
surveillance systems response: consensus development using nominal group technique. J Public Health Manag
Pract. 2010 Dec;16(6):529–34.
15. JohnBaker, KarinaLovell, NeilHarris. How expert are the experts? An exploration of the concept of “expert” within
Delphi panel techniques. Nurse Researcher [Internet]. 2006 Oct 1 [cited 2017 Mar 2];14(1):59–70. Available from:
http://journals.rcni.com/doi/abs/10.7748/nr2006.10.14.1.59.c6010
16. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public
Health [Internet]. 1984 Sep [cited 2017 Mar 3];74(9):979–83. Available from:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1651783/
17. Geist MR. Using the Delphi method to engage stakeholders: A comparison of two studies. Evaluation and Program
Planning [Internet]. 2010 May [cited 2017 Mar 2];33(2):147–54. Available from:
http://www.sciencedirect.com/science/article/pii/S0149718909000408
18. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ [Internet]. 1995 Aug 5
[cited 2017 Mar 2];311(7001):376–80. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2550437/
19. Kadam UT, Jordan K, Croft PR. A comparison of two consensus methods for classifying morbidities in a single
professional group showed the same outcomes. Journal of Clinical Epidemiology [Internet]. 2006 Nov 1 [cited
2017 Mar 2];59(11):1169–73. Available from: http://www.jclinepi.com/article/S0895-4356(06)00131-4/abstract
20. Minas H, Jorm AF. Where there is no evidence: use of expert consensus methods to fill the evidence gap in low-
income countries and cultural minorities. International journal of mental health systems [Internet]. 2010 [cited
2017 Mar 6];4(1):33. Available from: https://ijmhs.biomedcentral.com/articles/10.1186/1752-4458-4-33
21. Mokkink L, Terwee C, Knol D, Stratford P, Alonso J, Patrick D, et al. Protocol of the COSMIN study: COnsensus-
based Standards for the selection of health Measurement INstruments. BMC Med Res Methodol [Internet]. 2006
Jan 24 [cited 2017 Mar 2];6:2. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1368990/

31. PhD Seminar-2
31
22. Whitehead D. An international Delphi study examining health promotion and health education in nursing practice,
education and policy. Journal of Clinical Nursing [Internet]. 2008 Apr 1 [cited 2017 Mar 2];17(7):891–900.
Available from: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2702.2007.02079.x/abstract
23. Research priorities in health services in the National Health System. An expert consensus approach [Internet].
[cited 2017 Mar 2]. Available from: http://www.gacetasanitaria.org/es/linkresolver/prioridades-investigacion-
servicios-sanitarios-el/S0213911106715072/
24. What is the Delphi method? Strengths and shortcomings [Internet]. [cited 2017 Mar 2]. Available from:
https://www.researchgate.net/publication/5287982_What_is_the_Delphi_method_Strengths_and_shortcomings
25. Delp P, Thesen A, Motiwalla J, Seshardi N. Nominal group technique. Systems tools for project planning [Internet].
1977 [cited 2017 Mar 15];14–18. Available from:
http://www.aucd.net/docs/urc/Leadership_Institute/Subsequent%20Leadership%20Institute%20Materials/Nominal
%20Group%20Technique.pdf
26. Mac Giolla Phadraig C, Nunn J, Dougall A, O'Neill E, McLoughlin J, Guerin S (2014) What Should Dental Services
for People with Disabilities Be Like? Results of an Irish Delphi Panel Survey. PLoS ONE 9(11): e113393.
doi:10.1371/journal.pone.0113393

32. THANK YOU
32
PhD Seminar-2