Foundations of Delphi for Foresight and Group Communication


Published on

Published in: Education, Technology
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Foundations of Delphi for Foresight and Group Communication

  1. 1. © 2011 Turoff 11Foundations of DelphiforForesight and Group CommunicationsMurray Turoffturoff@njit.edu
  2. 2. © 2011 Turoff 2From: “The People, Yes” by Carl SandburgThe white man drew a small circle in the sandand told the red man, “This is what the Indian,knows,” and drawing a big circle around thesmall one, “This is what the white man knows.”The Indian took the stick and swept an immensering around both circles: This is where thewhite man and the red man know nothing.”(A poetic reason for Delphi)A seer upon perceiving a flood should bethe first to climb a tree – Kahlil Gibran(Criteria for Study Designers: belief in the results)
  3. 3. © 2011 Turoff 33Definition from 1975 Design of a group communication processstructured/tailored around the nature of theapplication and the nature of the group Original paper and pencil rounds Anyone can change their view Anonymity or pennames Scaling theory to promote understanding Voting to focus discussion Select “knowledgeable” people A round took a month – three to five rounds Respondents 15 to 500 Prediction, policy analysis, conditional forecasts,planning, significance of contributions, new productcharacteristics, etc, etc. (book has many examples)
  4. 4. © 2011 Turoff 4Some interesting forecasts 1953 IBM estimates market for only50 computers in U.S. with IBM 650 1969 GE management claims BASIC isuseless 1975 HP rejects idea of personalcomputer before there was one 1979 Tandy expects to sell only 4,000of the first real portable computer
  5. 5. © 2011 Turoff 5Single Subjective Estimates Personal judgment As good as data, models andinformation to back it up Expert Opinion As many examples of wrong ones asright ones by experts. Xerox, Video text, Picture Phone,X.400 How do you know which expert touse?
  6. 6. © 2011 Turoff 66Added definitions and analogs today A rose by any other name! Recommender systems Marketplace Systems Collaborative Systems Knowledge structures and systems Collaborative tagging and Folksonomies A beautifully simple Delphi process Structural Modeling (user created models) Computer Mediated Communications The design of inquiry systems by philosophy Scientific: Leibnitz, Locke, Kant, Hegel,Singer – C. West Churchman book Non Scientific! Heidigger (Negotiated reality,management and marketing) Virtual Organizations, teams, groups
  7. 7. © 2011 Turoff 7Online and Real Time Delphi or AsynchronousCollaborative Systems Today this is the future of the Method and itsEvolution Many examples already exist and someexperimental evaluation efforts have beendone In many of these Anonymity, Pen Names orSignatures by authors (when they choose) orby the sponsors is a dynamic option As usual in the computer field technologicaloptions are expanding faster than they canbe evaluated except by looking at the results. Understanding the successful paperapproachs is still a good guide to how suchsystems work
  8. 8. © 2011 Turoff 8Objectives of a Delphi Process Promote exploration of a complex problem Gathering ideas and information from theliterature and the respondents Provide an understanding for the group of theviewpoints of individuals, subgroups and thewhole group. Use of voting and scaling theory Expose and Explore disagreements Encourage focused discussion Encourage participants to focus on what theyknow well Allow participants to change their minds
  9. 9. © 2011 Turoff 9Paper and pencil vs. On line Dynamic Delphis Paper and Pencil Specific round structures usually 2 to 5 for paperand pencil exercises. Weeks between rounds Much use of paper, scissors, scotch tape, and Xeroxto eliminate duplications On Line Any individual may deal with the part of theproblem they prefer to work with at the moment Computer tracks all communications and knowswhat any participant has or has not seen orresponded to For Both Use of a knowledge structure to captureinformation and organize it to avoid informationoverload
  10. 10. © 2011 Turoff 1010Delphi does not make decisions Delphi should never make a decision, it should beused to analyze a potential decision or action andprovide that to a single individual or role that isaccountable for the decision to be made. The decision maker for real time decision shouldalways be one of the participants In Emergency management many different rolesmake decisions that can interfere with one anotherand an Emergency System must track the decisionprocess to insure everyone has access to the mostcurrent relevant information This type of system alerts people to conflicts andthose responsible for oversight. The problem solvingDelphi (later) is useful in that type of situation. It is unpredictable what Roles (active 24 hours 7 daysa week) are needed at a given time for a given problembut it is always a group that is needed. The person in the role changes dynamically
  11. 11. © 2011 Turoff 1111Characteristics of Delphi I A content structure appropriate to the problem beingaddressed Automatic organization of content contributions Dynamic knowledge base creation Anonymity of voting and degrees of anonymity forcontributions Forced, voluntary, and pen names Some groups are told who other members are toconvince them of quality of group (peers) Anyone can change their original vote Primary goal of votes to expose different views andgenerate discussion of causes for that Ambiguities, uncertainties, misunderstandings, anddisagreements Need to eliminate ambiguities before other issues canbe resolved
  12. 12. © 2011 Turoff 1212Characteristics of Delphi II Classifying participants so that votes can be seenviewed by backgrounds of voters very important forlarge studies with very heterogeneous groups Use of voting and comments is to encourageparticipants to change their views based upon relevantcontributions of others. This is a different objective than surveys andassociated assumption about only measuring afixed state of mind (psychological characteristics)that do not change for an adult Not all survey guidelines and associated statisticsapply to a process intended to deal with complexproblems and causing people to rethink theirviewpoint Pennames extremely useful for allowing continuity ofthought across a set of comments and for execution ofrole playing learning games or story telling For example: comment on the following policyresolutions as the company auditor would
  13. 13. © 2011 Turoff 13Motivations for participant Want to exchange useful information with otherprofessionals about a complex problem It is a peer group and or people you don’t haveaccess to The facilitators have already gathered all the obviousmaterial to structure and seed the process You are not going to educate them and they willunderstand what you have to contribute The problem is important, part of the job, and/or doesnot have an obvious solution It can be a process to test ideas It is a consulting task that is paid for The sponsor will take the results seriously The results will be a useful document for a largeraudience
  14. 14. © 2011 Turoff 1414Ultimate Delphi Goal: Collective Intelligence Group problem solving result is better thanany single member could have done beforethe discussion process Possible with even paper and pencil Group can solve problems as fast as anysingle member considering the same relevantmaterials (asynchronous operation) Possible with computers and StructuredComputer Mediated Communications
  15. 15. © 2011 Turoff 1515Delphi Examples:These have been done very often and lendthemselves to online operation Trend Delphi: produces a forecast of a trend alongwith the mental model of the group making theextrapolation of the trend curve into the future Problem Solving Delphi: Collects solutions to theproblem which are rescaled to a group interval scalebased upon individuals ranking or paired comparisons.Use voting to focus discussion on items that need it. Policy Delphi: seeks policy resolutions and thestrongest pro and con evidence or arguments tosupport each policy resolution Cross Impact Modeling: Collaborative building of amodel of the future possible outcomes of a set ofunique events.
  16. 16. © 2011 Turoff 1616Creativity ExperimentOpen ended idea generation Distribution of total Raw, Unique, and Rare Ideas Online Discussion System alone and with DelphiVoting imposed x group size. Problem: product ideas for a pill sized data responder,ranked list report. Notes: Unique ideas (duplications removed), rare ideas(occurred in 3 or less groups); Statistically significanton per group and per person that Delphi leads to moreimprovisation or creativity. Cho, H. (2004) The effect of Delphi structure on smalland medium-sized asynchronous groups, Ph. D.Dissertation, January 2004, New Jersey Institute ofTechnology, Information Systems Department.
  17. 17. © 2011 Turoff 1717Number of Idea TypesStructure x Group Size44 groups (11 groups ineach condition)Small5-7personsMedium11-14personsDelphi:Raw ideas (duplicates) 157 247Unique ideas 67 111Rare ideas (g<=3)) 40 57Unstructured:Raw ideas (duplicates) 108 192Unique ideas 52 86Rare ideas (g<=3) 19 48Results for total ideas and ideas per person are both statisticallysignificant. No reduction in performance for individuals inmedium group. (open ended brain storming problem)
  18. 18. © 2011 Turoff 1818Easy Online Delphi Used for Experiment Bulletin Board System imposed structure bymonitor Good discussion thread structure Root items for main ideas Replies for discussions of each idea Allowed anonymous or real signatures Web survey system where monitor collectedand put the ideas for a vote and providedsummaries online Once a week for three weeks Non structured condition Same system but no imposed structure bymonitor and no surveys done
  19. 19. © 2011 Turoff 19Trend Forecasting Delphi Round One Provide one or more historical curves ofinterest (measures of success or failure,sales, enrollments, profit) Ask for: Future projection of a historical curve 3-5 Assumptions used to draw the curve Uncertainties that would cause you tochange the curve you drew Assumptions & uncertainties: Modifications, competitors actions,government policies, economicconditions, investments, etc.
  20. 20. © 2011 Turoff 20Forecasting Delphi Round Two Turn ALL into potential assumptions One persons assumption is another’suncertainty Ask for vote on validity: Certain, Likely, Maybe, Unlikely,False Show limits of 50 % spread of drawnprojections Spread increases with time into thefuture
  21. 21. © 2011 Turoff 211CERTAIN (Average of 1 to 1.5)•Low risk of being wrong; Decision based upon this will not be wrongbecause of this fact’; Most inferences drawn from this will be true.2RELIABLE (Average of 1.6 to 2.5)•Some risk of being wrong; Willingness to make a decision based uponthis; Assuming this to be true but recognizing some chance of error;Some incorrect inferences can be drawn.3NOT DETERMINABLE: (at this time) (Average of 2.6 to 3.5)•The information or knowledge to evaluate the, validity of this assertionis not available to anyone --expert or decision maker.4RISKY (Average of 3.6 to 4.5)•Substantial risk of being wrong; Not willing to make a decision basedupon this alone; Many incorrect inferences can be drawn; The converse,if it exists, is possibly RELIABLE.
  22. 22. © 2011 Turoff 225UNRELIABLE (Average of 4.6 to 5)•Great risk of being wrong; Worthless as a decision basis; Theconverse, if it exists, is possibly CERTAIN.6NOT PERTINENT/RELEVANT (Used to eliminate someassumptions from exercise) Even if the assertion is CERTAIN orUNRELIABLE it has no significance for the basic- issue; It cannotaffect the variable under question an observable amount.blankNO JUDGMENT•No knowledge to judge this item, but the appropriate individual(expert, decision maker) should he able to provide an evaluation Iwould respect.Validity and/or Confidence Scale
  23. 23. © 2011 Turoff 23Forecasting Delphi Round Three Show ordered assumptions fromcompletely true to completely false Ask for modifications to originalprojection Focus on most important “Maybe”range Ask for significance of impact of assumptionscontrollable or those not controllable Determine actions organization can take toforce outcome of controllable events Determine measures/signals of other externalassumptions becoming true
  24. 24. © 2011 Turoff 24Forecasting Delphi Round Four and “maybe”Five May not be same group, usually moredecision makers in this part of the process Determine desirability and feasibility ofactions Determine usability of measures or ways tobring about better intelligence Round five for significant disagreements Next phase Management group to do allocation ofresources to influence or measure thingsthat can change forecast
  25. 25. © 2011 Turoff 25You have to have a design for all of the process Request Curve Projection Assumptions and Uncertainties Turned into Potential Assumptions Requested Vote on Validity Provide List of Assumptions from True to False Request new curve estimate Extract Controllable events and Uncontrollable in theNot Determinable range Examine these events for influencing andmeasuring See Steel and Ferroalloy Delphi article from TheDelphi Method Book for examples
  26. 26. © 2011 Turoff 26Results of Trend Delphi A numeric projection and spread whichmight not differ from a regression analysis A “collaborative knowledge model” of thegroups reasons (assumptions) for theprojection A collaborative knowledge model of all thefactors that could change the forecast Three rounds A plan for: Trying to influence the outcome Monitoring for surprise changes fromexternal sources like competitor actions Two more rounds
  27. 27. © 2011 Turoff 27Modeling Delphi for the Steel Industry Flow diagram by three experts 45 flow of material legs between differentprocesses in the industry Only 15 legs reported in yearly data 40 other experts (planners) asked to fill in missingdata for prior year Not asked to do anything else 25 decided to modify the model because they didnot agree with it being the “best” model for them Modeling Delphis need more exploration Very risky to listen to only one text book, one expert,or one professor!!!! Review the literature always necessary beforedesign of a Delphi
  28. 28. © 2011 Turoff 28Policy Delphi I Major Policy Issue List obvious resolutions Vote on Desirability and Feasibility Allow more proposed Show results and disagreements Ask for pro, con, and neutral arguments onresolutions Vote on new resolutions Vote on arguments for Importance and Validity Continue rounds as long as vote changesare made and new resolutions emerge Agreement on underlying comments often leadto a synthesis or a better resolution Not necessary
  29. 29. © 2011 Turoff 29Policy Delphi ScalesForcing Specificity of Responses Policy Resolutions Very Desirable, Desirable, Undesirable,Very Undesirable Definitely Feasible, Possibly Feasible,Possibly Unfeasible, Definitely Unfeasible Comments on Resolutions: pro, con, neutral Very Important, Important, SlightlyImportant, Unimportant Certain, Reliable, Risky, Unreliable No Judgment always a choice on anything Easy to make dynamic
  30. 30. © 2011 Turoff 3030Why Delphi: Human Considerations Too many participants needed for face-to-face Three to five in any given type of expertise and/orexperiences to ensure all rationales are exposed Early experiments by Dalkey on rationales for specificsubjective With N types needed for a complex problem this is 3xN to 5xNparticipants required Different backgrounds that require elimination of ambiguityand misinterpretations as well as translations of concepts Severe disagreements to be mediated Freedom of expression and improvisation of ideas withoutloss of face One out of ten ideas valuable High status participants most concerned about this Equal Participation allowed Minimize group process losses An idea can be brought up at anytime
  31. 31. © 2011 Turoff 3131Critical Success Factors Composition and quality of the participants Results only as good as the people involved Compensation for Effort Communication with a peer group Letting participants nominate others helps (snowball) Will learn from others in effort Results will serve useful and important purpose Consulting pay if above not strong enough What is obvious already included, participants usedfor what is not obvious (no blank sheets of paper) A morphological structure to automatically organizeinput material Important functions and objectives: exploration,understanding one another, exposure of issues anduncertainties, examination of disagreements, andgeneration of agreements when possible
  32. 32. © 2011 Turoff 3232Why Delphi: operational challenges Subjective judgments required Models to support consistency of judgments byindividuals and group Building collective models (e.g. cross impact modeling,structural modeling) Producing collective Group views (e.g. Arrow’s paradox andscaling methods) Individual human biases (~10) Group process losses and gains (~20) Avoiding consensus (Asch & Hawthorne effects) pressure Participant effort and shadow time Asynchronous flexibility to use any time Language or cultural difficulties Virtual teams Detecting differences by backgrounds for feedback Multitasking, Cognitive Limitations, Information Overload
  33. 33. © 2011 Turoff 3333Delphi technology and methods Exploring Capturing individual knowledge Process of design, problem solving, derivation,knowledge structures Animation methods Promoting Understanding Forming a group synthesis Scaling methods Feedback to group Social judgment & voting theory Finalizing Viewpoints (no more changes) Evaluation by the group Collaborative processes
  34. 34. © 2011 Turoff 34Preventive Security Measures in ISManagement of IS Course (Graduate Students 50% working)Online system over three weeksPreventive Measures 72 Total Words (approx.) 25,000Comments on 433 Average comment size(words)480Pro comments 200 Contributors 20Con comments 104 Voters 27Neutral Comments 129 Contributions/person 23Modifications 25 Contributions/day 25Total of all items 530 Comments/Measure 6
  35. 35. © 2011 Turoff 35Top and bottom rated preventive measures
  36. 36. © 2011 Turoff 3636Rank Scale Disaster Damage Dimension20 20.00 Causalities and fatalities1918 18.00 Utilities Impact1716 16.0015.90Potential to spreadAbility of local response adequacy15 15.4315.4015.4015.38Loss of command and controlInfrastructure damageResources for aid or containmentTime needed to respond14 14.82 Duration of disaster13 13.09 Public reaction12 12.95 Geographic impact1110 10.07 Time to return to normal98 8.61 Chance of imminent reoccurrence7654 4.70 Financial Loss3210 0.01 Financial recovery costsFeedback: a Thurstone Scale forRelativeImportance of Measures ofDisaster ImpactGraduate Student class Delphi in anEmergency Management class withabout 30% with real experience inthe area.
  37. 37. © 2011 Turoff 3737Online Delphi’s TodayDynamic: entering main voting items, voting, making vote changes,and pro, con, neutral discussion comments, can all go on at once. Each individual can focus on what they want to System notifies user of changes Termination of vote changes is a sign of finalization of theresults People not confident about a given item can always not vote ordefer their vote to later People can propose rewording (Roberts Rules of Order) What took three months can happen in three weeks for complexproblems and large groups Straight forward problems (e.g. emergency allocation priorities) canbe done in real time by dispersed virtual groups of 5-20professionals. Typically voting anonymous, comments signed Voting really serves to focus discussion on differences of view Used as a collaborative learning tool in online class discussions System designed as a Social Decision Support System or DynamicDelphi
  38. 38. © 2011 Turoff 3838Approaches to the FutureGoals affect what questions to ask Automation Reduce people time and effort Eliminate jobs Productivity Increase quantity Increase quality Opportunity (preferred for Foresightstudies) Do new things Do things differently & Better
  39. 39. © 2011 Turoff 3939Approaches to the FutureDimensions of Group Communications Information Exchange Pooling of current data and information Cooperation Informing of plans and actions/efforts Coordination Joint planning of actions/efforts Collaboration Working together on the sameactions/effort Too often above words are used as if theywere all the same
  40. 40. © 2011 Turoff 40Possible Delphi Use in Planning(Continuous Organizational Collaborative Process) Problem Solving Delphi Pick key variables for analysis Trend Analysis Delphi Generate Key Potential Assumptions/Events Problem Solving Delphi Evaluate Important/Significant events Cross Impact and ISM Develop Integrated Model and Simplify Complexity Policy Delphi Expose and Investigate disagreements on Decisionand Policy options Start at the top again for unresolved situations
  41. 41. © 2011 Turoff 41End of First TalkQuestions or CommentsCarefullylooking atthe future!
  42. 42. © 2011 Turoff 4242Appendix Slides:Examples of general types of DelphiandDynamic Delphis These slides represent more detail onthe specific structures mentioned andalso provide a list of references forthose that want to dig deeper.
  43. 43. © 2011 Turoff 4343Examples of General Delphi DesignsandDynamic Delphis These have all been done with paper a number oftimes Some have been done partially or totally online or incombination They could all be done online in a mode where anyonecan work on any part at anytime Some of these still require a human monitor toperform certain functions and some can be fullyautomatic Users can: 1. Add new items or content 2. vote on any item, change their vote, and/orchoose not to vote, and/or vote later afterdiscussion 3. Suggest alternative wordings to existing item oradd a new version of an existing item.
  44. 44. © 2011 Turoff 4444System operations Allow lists to which options, solutions, goals, etc canbe added Allow voting scales for such lists Allow comments on list items Provide each user with new items of any type and alsoany whole discussion they want to see Provide vote summaries to indicated the relativestatus of voted items and highlight differences Determine any different vote patterns by participantswith different expertise and/or experience, and/orbackgrounds Show for any item Number voting Number of vote changes Number who might yet vote Number of discussion entries about an item
  45. 45. © 2011 Turoff 4545Integrated Collaborative Planning I1. Problem solving Delphi Process: Determine andconsider the variables that measure the future for anyorganizational objective and use the historical data toallow a projection of both a desirable and undesirablefuture that might define the range of uncertainty.2. Trend Analysis Delphi Process: Use a trend analysiscollaborative process to determine the events that willinfluence both the desirable and undesirableoutcomes. Events are any type of policies, actions,decisions, resource allocations etc that can influencethe future of the key variables.3. Problem Solving Delphi Process: Evaluate these optionsto reach agreements on which are the most importantfor influencing the future. Make sure to use abalanced set that reflects a rational assessment ofwhat makes the difference between a desirable and anundesirable future.
  46. 46. © 2011 Turoff 46Integrated Collaborative Planning II4. Develop a cross impact model for these unique eventsand use Interpretive Structural Modeling to take thesubjective judgments of those involved and determineout of what is potentially a very large event set themost consistent micro scenarios to reduce this largeevent set to a more manageable subset of miniscenarios.5. Use these mini scenarios and how they influence oneanother to actually build a working model that allowsindividuals test the significance of changes ininfluencing (general resource allocation) the outcomesthat these scenarios represent. This model can betreated as a game that professionals can use to testdifferent alternative decisions and resourceallocations.6. Use the Policy Delphi to actually resolvedisagreements or judgment uncertainties aboutpossible decision and policy options. This mightcause restarting the process for some aspect of theplanning problem
  47. 47. © 2011 Turoff 4747Trend DelphiSystem Functions Participants Responses Response System ActionsPresent a historical trend to beextrapolated by theparticipantsDraw a future curve orredraw a new one whena change has occurredin viewpoint.Present summary of 50%median and 0%, 25%,75% and 100%boundariesRequest assumptions anduncertainties used to makeabove estimateTurn all these into potentialassumptionsVote on validity scale foreach potentialassumption.Scale is from completelytrue to completely false.Reorder assumptions fromtrue to false.Focus on middle range(maybe) and ask whichcan be influenced ormeasured for occurrenceAssume these can reduce thefuture uncertainty in thecurveAsk for a redrawing of curveextrapolation based uponassumption list for eachtrend curve in the studySupply suggestions on howto influence or measurethe maybe assumptionscausing significantuncertainty in theprojected curve.Summarize importantfindings at any time:Trend, true and falseassumptions, assumptionsthat cause uncertainty,and their potentialactions, andmeasurements
  48. 48. © 2011 Turoff 4848Problem Solving DelphiSystem Functions Participants Responses Response System ActionsState the problem and requestsolution optionsProvide options to solvethe problemPresent options in order ofoccurrenceRequest paired comparisons tomeasure individualpreferences for optionsMake comparisons foroption pairs that aparticipant feelsconfident aboutjudging at any time.Use Thurstones law ofcomparative judgment(using incompleteinformation) to derive asingle group intervalscale.Calculate uncertainty due tothose who have not yetvoted with same type ofscale.Show interval scale.This indicates disagreementswhen two or more items areclose together. This alsoshows clustering. Ask forcomments about itemswhere people disagree withcurrent position of an item.Make comments aboutitems you want to seeothers change theirvotes about.Present discussions aboutitems for review. Asmore people vote orchange votes scales willreflect decreasinguncertainty and oftenmore separation betweenoptions.
  49. 49. © 2011 Turoff 4949Policy DelphiSystem Functions ParticipantsResponsesResponse SystemActionsState a policy issue to beexamined. Ask for specificpolicy solutionsAdd resolution options orspecific policiesRequest vote for Desirabilityand Feasibility scales ofeach solutionPlot two dimensionaldistribution of policyresolutionsExploring desirable butinfeasible solutions oftenimportantRequest commentsespecially about thoseshowing disagreementRequest comments aboutpolicy resolutions.Indicate if comment ispro, con, or neutral.Request vote on comments forimportance and validityIt might be consideredimportant because othersbelieve it to be trueA person may think acomment is importantbecause others think itis valid.Do same two dimensionalplots and summarizediscussions
  50. 50. © 2011 Turoff 5050Cross Impact Modeling ISystem Functions Participants Responses Response System ActionsUse problem solving Delphi toproduce a set of future uniqueevents focused on a givensituationEvaluate those events fortheir relativeimportance to thefuture objectiveguiding the choice ofeventsPlace the final most importantevents into a cross impactmodelAsk each individual to answer:What are the probabilities of eachevent occurring in some futuretime frame?Tell them for each event that theyshould assume it will or willnot occur and ask them toexpress any changes in theprobabilities of the otherevents due to that certainknowledge about the future.Show them the expectedoutcome of theirmodel which will havedifferences from theirpredictions. Allowthem to vary initialprobabilities to seehow the futurechanges.Allow them to go backand modify some oftheir estimatesCreate the cross impact modelusing the approach by Turoff(logistic, Fermi Diracequations). This provides ascale changes from nonlinearprobabilities (0 to 1) to alinear influence factorbetween each pair of events(plus to minus infinity).When participants are satisfiedwith their individual modelutilize the internal linearinfluencing factors (Cij) tocreate a group model.
  51. 51. © 2011 Turoff 5151Cross Impact Modeling IICreating ScenariosSystem Functions Participant Responses Response SystemActionsAnalyze the internal parametersto show people which of therelationships between whichevents show the mostdisagreement among thegroup.Ask for comments on thesecombinations form thosewho have inconsistent orextreme views.Show these comments andothers and allow thosewho wish to changesome of their originalestimations.Create a model ofinteracting scenarios byvoting on where to stopthe integration of theevents in process thatcan turn all the eventsinto one scenarioWhen no more changes arebeing made produce.Use InterpretiveStructural modeling togenerate a set of macroscenarios collectingindividual events thatare tightly coupled intoa set of scenarios thatinteract.Requires human monitor toknow when to triggerthe scenario creationpart
  52. 52. © 2011 Turoff 52521972 Paper ExampleThe Delphi Method BookEventnumberDescription1The U.S. gets in a trade war with one or more of its major trading partners (Japan, Canada, western Europeancountries).2Comprehensive Tax Revision S Enacted With Most Present exemptions And Exclusions Removed, But WithRates Lowered.3 Rigorous anti-pollution standards are adopted and strictly enforced for both air and water.4 The U.S. averages at least 4 percent per year growth rate of real GNP for the time frame.5 Defence spending declines steadily as a percent of the federal governments administrative budget.6The U.S. Experiences At Least One Major Recession (GNP Decline is greater than 5 Percent for a duration greaterthan 2 quarters) during the ten year period.7A federal income maintenance system (e.g., negative income tax) replaces essentially all current state and localwelfare programs.8 The oil import quota system is phased out and domestic oil prices allowed to fall to the world price.9 The U.S. agricultural price support system is dismantled.10A federal-state and local revenue-sharing program is adopted which allocates at least 5 percent of federal revenuesto state and local governments.Table 1 Events
  53. 53. © 2011 Turoff 5353Use of Interpretative Structural Modelingwith Cross Impact AnalysisEach cycle is a mini scenario or possible new single event
  54. 54. © 2011 Turoff 54© 2009 Turoff 54Cross Impact Additions Decide which events you can replacewith micro scenarios to reduce eventcomplexity. Add objective and goals as event sinks Add source events as investments Develop offense and defense events tocreate a game between two players ortwo teams Can model natural and man madedisasters for exploration and training
  55. 55. © 2011 Turoff 5555The following slides contain referencesfor those that want to learn more
  56. 56. © 2011 Turoff 5656Papers on my website 1 Complete copy of original reference book on the Delphi Method The Delphi Method: Techniques and Applications by HaroldLinstone and Murray Turoff, 1975. ( Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H., Yao, X.,(2004),Online Collaborative Learning Enhancement through theDelphi Method, Proceedings of the OZCHI 2004 Conference,November 22-24, University of Wollongong, Australia Murray Turoff, Starr Roxanne Hiltz, Michael Bieber, Ajaz Rana(1998), Collaborative Discourse Structures in Computer MediatedGroup Communications, 1998 Turoff, Murray, and Starr Roxanne Hiltz (1996), Computer BasedDelphi Processes, a version will appear as an INVITED BOOKCHAPTER for Michael Adler and Erio Ziglio, editors., GazingInto the Oracle: The Delphi Method and Its Application to SocialPolicy and Public Health, London, Kingsley Publishers (in press).
  57. 57. © 2011 Turoff 57Papers on my website 2 Turoff, Murray (1997) Alternative Futures for Distance Learning:The Force and the Darkside, The material in this paper wasutilized for an Invited Keynote Presentation at the UNESCO /OPEN UNIVERSITY International Colloquium, April 27-29:Virtual Learning Environments and the Role of the Teacher, OpenUniversity, Milton Keynes. It also forms the basis of a planned talkat the Third International ALN (Asynchronous LearningNetworks) meeting in NY City, October, 1997. (PresentationOverheads for ALN conference, October, 1997, NYU) Turoff, Murray (1995), Software Design and the Future of theVirtual Classroom, Journal of Information Technology forTeacher Education, Vol 4, No. 2, 1995 Turoff, Murray (1999), An End to Student Segregation: No MoreSeparation Between Distance Learning and Regular Courses. Asummary of the invited plenary for the Telelearning 99 meeting inMontreal, Canada, November, 1999. (Also: ppt presentation usedin talk.)
  58. 58. © 2011 Turoff 5858Other Papers 1 Turoff, M., Hiltz, S.R.: The Future of Professional Communities ofPractice. In: Weinhardt, C., Luckner, S., Stößer, J. (eds.) WeB2008. LNBIP, vol. 22, pp. 144-158. Springer-Verlag, BerlinHeidelberg (2009) Xiang, Y, Turoff, M., and Chumer, M. Designing a group supportSystem to Review and Practice Emergency Plans in Virtual Teams,Proceedings of the 6th International ISCRAM Conference,Gothenburg, Sweden, May 2009 ( White, Connie, Murray Turoff, Bartel Van de Walle, A DynamicDelphi Process Utilizing a Modified Thurstone Scaling Method:Collaborative Judgment in Emergency Response, Proceedings ofISCRAM 2007, 4th International Conference on InformationSystems for Crisis Response and Management, Delft, theNetherlands, May 13-16, Brussels University Press Plotnick, Linda, Elizabeth Avey Gomez, Connie White, FurtheringDevelopment of a Unified Emergency Scale Using Thurstones Lawof comparative Judgment: A progress Report, Proceedings ofISCRAM 2007, 4th International Conference on InformationSystems for Crisis Response and Management, Delft, theNetherlands, May 13-16, Brussels University Press
  59. 59. © 2011 Turoff 59Other papers 2 Turoff, Murray, Starr Roxanne Hiltz, Xiang Yao, Zheng Li,Yuanqiong Wang, and Hee-Kyung Cho, Online CollaborativeLearning Enhancement Through the Delphi Method, TurkishOnline Journal of Distance Education-TOJDE April 2006 ISSN1302-6488 Volume: 7 Number: 2 Article: 6, Publisher: AnadoluUniversity, Eskisehir, Turkey, Hee-Kyung Cho, Murray Turoff, and Starr Roxanne Hiltz, TheImpact of Delphi Communication on Small and Medium SizedAsynchronous Groups: Preliminary Results, HICSS 36, January2003, IEEE Computer Society Press. Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H. "The DelphiProcess as a Collaborative Learning Method." In (edited by J. C.Moore) Elements of Quality Online Education: Into theMainstream: Wisdom from the Sloan Consortium, 121-134.Needham, MA: Sloan-C, September 2004 Banuls, V., and Turoff, M., Scenario Construction via Cross-Impact, Draft under review 2009.
  60. 60. © 2011 Turoff 6060Other Papers 3 Cho, H.K. & Turoff, M., “Delphi Structure and GroupSize in Asynchronous Computer-MediatedCommunications,” Proceedings of the AmericasConference on Information Systems, Tampa, August2003. Wang, Y., Li, Z., Turoff, M. and Hiltz, S.R. (2003).Using a social decision support system toolkit toevaluate achieved course objectives. Proceedings of theAmericas Conference on Information Systems, Tampa,August. (Nominated as a “best paper.”) Turoff, Murray and S. R. Hiltz, (1995), ComputerBased Delphi Processes, in Michael Adler and ErioZiglio, editors., Gazing Into the Oracle: The DelphiMethod and Its Application to Social Policy and PublicHealth, London, Kingsley Publishers, pp. 56-88.
  61. 61. © 2011 Turoff 61Other papers 4 Worrell, W., Hiltz, S. R., Turoff, M. and Fjermestad, J. (1995) Anexperiment in collaborative learning using a game and a computer-mediated conference in accounting games. Proceedings of the 28thAnnual Hawaii International Conference on System Sciences, Vol.IV, pp. 63-71. Los Alamitos, CA: IEEE Computer Society Press,1995. Hsu, Enrico Y. P., Hiltz, S. R., and Turoff, Murray (1992).Computer-Mediated Conferencing System as Applied to a BusinessCurriculum: A Research Update. In V. S. Jacob and H. Pirkul,eds., The Impact of Information Technology on Business Schools:Research, Teaching and Administration, Proceedings of the 20thAnnual North American Conference of the International BusinessSchool Computer Users Group, pp. 214- 227. Awarded "BestPaper- Teaching.“ Hiltz, S.R. and Turoff, M., The Network Nation: HumanCommunication via Computer, 1978, revised edition reprinted1993 by MIT Press