Ted Gordon Senior Research Fellow The Millennium Project World Federation of United Nations Associations

First developed at RAND in the 1960’s A means of eliciting and combining expert judgments while avoiding the pitfalls of conference room confrontations. Used a sequential questionnaires, each building on the previous round. Required features: Participants guaranteed anonymity Feedback of reason for extreme opinions Has been used in thousands of studies

First developed at RAND in the 1960’s

A means of eliciting and combining expert judgments while avoiding the pitfalls of conference room confrontations.

Used a sequential questionnaires, each building on the previous round.

Required features:

Participants guaranteed anonymity

Feedback of reason for extreme opinions

Has been used in thousands of studies

From Gordon and Helmer, A Long Range Forecasting Study, RAND, 1964 Economically useful desalination of sea water 1970 1964-1980 Effective fertility control by oral contraceptive 1970 1970-1983 Development of new synthetic ultra light materials 1971 1970-1978 Automated language translators 1972 1968-1976 New organs through transplanting or prosthesis 1972 1968-1982 Reliable weather forecasts 1975 1972-1988 Operation of a central data storage facility 1980 1971-1991 Reformation of quantum relativity 1980 1975-1993 Implanted artificial organs made of plastic and electronic components 1981 1975-1988 Widespread and socially acceptable use of psychotropic drugs 1983 1980-2000 Creation of a form of artificial life (at least in the form of self replicating molecules) 1989 1979-2000 Economically useful exploitation of the ocean bottom 1980 1980-2000 Feasibility of limited weather control 1990 1987-2000 Commercial generation of protein for food 1990 1985-2003 Drug treatment for psychotic cases 1992 1983-2017 Biochemical general immunization 1994 1983-2000

Sociologist Economist Technologist Environment Person 1 XXX XXX Person 2 XXX Person 3 XXX Person 4 XXX Person 5 XXX XXX Person 6 XXX Person 7 XXX XXX Person 8 XXX TOTAL 3 2 2 3

First developed in a Defense Advanced Research Projects Agency (DARPA) study by Gordon and Pease (2004). The object was to invent a technique by which a commander could rapidly collect and synthesize opinions of field personnel, using Delphi, to help him reach a decision. Work was published in Technological Forecasting and Social Change . Since then, many improvements and adaptations.

First developed in a Defense Advanced Research Projects Agency (DARPA) study by Gordon and Pease (2004).

The object was to invent a technique by which a commander could rapidly collect and synthesize opinions of field personnel, using Delphi, to help him reach a decision.

Work was published in Technological Forecasting and Social Change .

Since then, many improvements and adaptations.

An on-screen form presents a series of questions to each participant, and for each: The average (or median) response of the group so far (and possibly the distribution of responses) The number of responses made so far A button that opens a window showing reasons that others have given for their responses. A button that opens a window that allows the respondent to type in justifications for their own answer. A space for the new numerical estimate, answering the question.

An on-screen form presents a series of questions to each participant, and for each:

The average (or median) response of the group so far (and possibly the distribution of responses)

The number of responses made so far

A button that opens a window showing reasons that others have given for their responses.

A button that opens a window that allows the respondent to type in justifications for their own answer.

A space for the new numerical estimate, answering the question.

There is only one round; no explicit second round is necessary as in a conventional Delphi. When a respondent revisits the study in a minute or a day, the original input form is presented with their prior answers in place and the new average or median of the group, and if desired, the distribution of responses. By then others may have contributed judgments, so the averages or medians may have changed since the original answers were provided.

There is only one round; no explicit second round is necessary as in a conventional Delphi.

When a respondent revisits the study in a minute or a day, the original input form is presented with their prior answers in place and the new average or median of the group, and if desired, the distribution of responses.

By then others may have contributed judgments, so the averages or medians may have changed since the original answers were provided.

If an answer is beyond a pre-specified distance from the average an indicator “flag” can be used to show the respondents the questions in which their answers differ significantly from the group. When the flag is “up” reasons become particularly important The respondents are not shown the group average until after they have inserted an answer and a selected number of responses have been received or a given time has elapse since the study has started.

If an answer is beyond a pre-specified distance from the average an indicator “flag” can be used to show the respondents the questions in which their answers differ significantly from the group.

When the flag is “up” reasons become particularly important

The respondents are not shown the group average until after they have inserted an answer and a selected number of responses have been received or a given time has elapse since the study has started.

The answers can be submitted cell by cell or as an entire document. To avoid crowding, the reasons can be submitted and appear on a separate page. A notation appears after the respondent has submitted an answer indicating that the response has been received and entered into the database. Notations also appear if individual responses are outside of predetermined plausible ranges.

The answers can be submitted cell by cell or as an entire document.

To avoid crowding, the reasons can be submitted and appear on a separate page.

A notation appears after the respondent has submitted an answer indicating that the response has been received and entered into the database.

Notations also appear if individual responses are outside of predetermined plausible ranges.

Extensive hyperlink reference material can be included to permit the respondents to retrieve data on line while completing the questionnaire. Reference material is included directly on the questionnaire Split samples can be used to reduce the effect of bias introduced by question placement on the questionnaire. A permanent record is maintained of responses including both quantitative and narrative responses.

Extensive hyperlink reference material can be included to permit the respondents to retrieve data on line while completing the questionnaire.

Reference material is included directly on the questionnaire

Split samples can be used to reduce the effect of bias introduced by question placement on the questionnaire.

A permanent record is maintained of responses including both quantitative and narrative responses.

Collection of judgments about when certain developments might occur, their importance, and impacts and reasons for extreme opinions Choosing from among alternative decisions based on clear cut criteria Providing information for and sharpening of scenarios Input/output and cross impact matrixes Environmental monitoring

Collection of judgments about when certain developments might occur, their importance, and impacts and reasons for extreme opinions

Choosing from among alternative decisions based on clear cut criteria

Providing information for and sharpening of scenarios

Input/output and cross impact matrixes

Environmental monitoring

Decision models: Real Time Decision Making. (DARPA) Choosing a Configuration. (auto mfg) Priority studies: Assessing Priorities of Proposed Millennium Project Study Topic, 2006 . Assessing Priorities of Proposed Millennium Project Study Topic. 2007 Assessing Priorities of Proposed Millennium Project Study Topic. 2008 SOFI studies: Variables for the Global State of the Future Index Variables for the Republic of Korea State of the Future Index Variables for the South Africa Global State of the Future Index

Decision models:

Real Time Decision Making. (DARPA)

Choosing a Configuration. (auto mfg)

Priority studies:

Assessing Priorities of Proposed Millennium Project Study Topic, 2006 .

Assessing Priorities of Proposed Millennium Project Study Topic. 2007

Assessing Priorities of Proposed Millennium Project Study Topic. 2008

SOFI studies:

Variables for the Global State of the Future Index

Variables for the Republic of Korea State of the Future Index

Variables for the South Africa Global State of the Future Index

Substantive studies: Future education and learning possibilities Future Energy Developments Studies in support of UNESCO: Judgments on acceptability of a Delphi process . Scenario Drivers. Report construction Resource allocation (WB subcontractor) Performance Biodiversity Climate Change

Substantive studies:

Future education and learning possibilities

Future Energy Developments

Studies in support of UNESCO:

Judgments on acceptability of a Delphi process .

Scenario Drivers.

Report construction

Resource allocation (WB subcontractor)

Performance

Biodiversity

Climate Change

Possible Development Likelihood by 2030 (%) 1. National programs for improving collective intelligence Some richer as well as lower income countries have (by this year of 2030) made improving collective intelligence a national goal; this includes improving individual as well as intelligence for their nations-as-whole. Click here to see references Likelihood by 2030 The average group answer: 61.7 Respondents: 195 Comments on this possibility (page 2) click here 2. Just in time knowledge and learning Rote learning has diminished in importance. With ubiquitous computing and education for life-long learning, 'just in time knowledge' has become the norm. Reasoning, problem solving, and learning strategies form the core focus of public educational systems. Click here to see references Likelihood by 2030 The average group answer: 71.1 Respondents: 185 Comments on this possibility (page 2) click here

1. What might encourage this possibility? What positive consequences might follow? Please enter positive developments and consequences. If you wish to change your prior input, type it here:

Average, median, standard deviation Distribution of opinions Summaries by region, occupation, gender Correlation of qualitative responses with quantitative responses (e.g. “those who answered with a low estimate, had these reasons…”)

Average, median, standard deviation

Distribution of opinions

Summaries by region, occupation, gender

Correlation of qualitative responses with quantitative responses (e.g. “those who answered with a low estimate, had these reasons…”)