Raford PhD defense final

Large-Scale Participatory
Futures Systems
A Comparative Study of Online
Scenario Planning Approaches

Candidate: Noah Raford
PhD Candidate, Urban Information Systems Group, City Design and
Development Group, Department of Urban Studies and Planning, MIT

Committee: Michael Flaxman (Chair)
Assistant Professor, Urban Information Systems Group, MIT

Joseph Ferreira
Professor of Urban Planning and Operations Research, Associate
Department Head and Head of Urban Information Systems Group,
MIT

Andres Sevtsuk
Lecturer, Department of Urban Studies and Planning, MIT

Outline
1. Introduction
2. Review & Synthesis of the Literature
3. Study Design & Methodology
4. Findings & Discussion
5. Conclusion

Noah Raford, PhD Defence, MIT DUSP, Large Scale Participatory Futures Systems: A Comparative Study of Online Scenario Planning Approaches 2

Introduction
Qualitative Scenario Planning

Humans have important shortcomings that Client defines key questions through initial Meetings,
ID Issues
limit our ability to make effective decisions conversations & meetings conversations

under conditions of dynamic uncertainty Generate
key themes
Expert interviews, brainstorm with client,
desktop research
F2F & phone
interviews
(Dorner, 1997)
ID driving Extract key themes, create trends and Group
workshop
“A disciplined methodology for imaging
forces timelines, key events

possible futures in which organizational Rank factors
Select key uncertainties and forces, list by
uncertainty / impact, predetermined drivers

decisions may be played out” (Shoemaker, Develop draft Create scenario snippets, draft systems

1995) scenario logic diagrams, mix and match trends, 2x2 grids

Create draft Integrate themes from draft scenarios, create Consultant

“Tools for foresight discussions... final scenarios headlines and scenario narratives report

whose purpose is not a prediction or a Finalise Get client feedback, refine, detail, elaborate Group
workshop
plan, but a change in the mindset of the
scenarios narrative to final form

Identify key strategic themes, reflect on
people who use them” (de Gues, 1997) Consider
implications
strategic questions in the context of each
scenario

Identify ID key indicators in each scenario for Consultant
indicators strategic concerns report


Introduction
Purported Bene ts

Reduced individual and group decision bias
Scenarios (Tetlock, 2006)

Increased
Enhanced awareness of environmental change,
learning future risks & opportunities (Weick, 1999)

Gain appreciation of different
More accurate
mental models stakeholders’ positions and attitudes
(Chermack, 2003)

Better Greater exibility and better
decisions
decision-making (Schwartz,
1997)

Improved
performance

(Chermack, 2003)


Introduction
Challenges to Scenario Planning in the Public Realm

Labor intensive & expensive

Bene ts poorly documented (no veri cation or reputation systems)

Limited participation (time, space & numbers)

Predominance of senior decision-making elite (participant bias)

Highly dependent on facilitator skills & consultant synthesis (facilitator & author
bias)


Introduction
Research Questions

Do web-based participatory approaches add value to the traditional scenario
planning process? If so, where and in what ways?

If not, where do they fall short, in what ways, and why?


Outline
1. Introduction
5. Conclusion


Literature Review & Synthesis

Planning Support Urban Planning
Role of the Future
Systems (PSS) & Policy Policy

ICT Platforms Scenario
& Web 2.0 Planning

?

Urban Planning & Public Policy

“The future orientation of planning is unique to the ﬁeld's identity... The very
substance of urban planning is founded in time'' (Myers and Kitsuse, 2000)

Four planning traditions (Freidman, 1987):
• Social Reform
• Policy Analysis (Simon, 1945; Forrester, 1968; Stokey and Zeckhauser, 1978)
• Social Learning (Majone, 1989; Scott, 1998; Schon, 1983)
• Social Mobilization (Davidoff, 1965; Forester, 1989; Castells, 1977; Healey, 1992; Innes, 1996)

Growing demand for public participation (Arnstein, 1969; Hulchanski, 1977; APA, 1990)

“Urban planning has retreated from strategic, future-oriented topics to become
absorbed in operational and managerial activities characterized by short time
horizons and value choices likely to be equally short-sighted and ad
hoc” (Coucelis, 2005)


ICT Platforms

Planning Support Systems (PSS) “Loosely coupled assemblages of computer-
based techniques”, forming a mixed toolbox of techniques to help decision-
makers in their daily tasks (Britton Harris, 1989; Brail and Klosterman, 2001; Batty, 2003)
• PPGIS (Warnecke, Beatie, & Lyday, 1998; Craig & Elwood, 1998; Geertman & Stillwell, 2003)
• Alternative Futures Analysis (Steinitz, 2003; Lagigno & Reed, 2003; Hopkins & Zapata, 2007)
• Participatory Agent Based Modeling (Bousquet & Le Page, 2004; Barnaud et al., 2007;
Castella et al, 2005)

“Modelling as negotiation” (Guhathakurta, 1993)

“Complicated, convoluted, time-consuming, and intimidating... that do not
achieve genuine participation in planning or other decisions” (Innes & Booher, 2004;
Cooke & Kotari, 2001)


Crowdsourcing & Web 2.0

Web 2.0 (O’Rielly; 2005; Anderson; 2007)
• Crowdsourcing (Howe, 2006)
• Collective Intelligence (Levy, 1994; Por, 2008; Malone et al., 2010)
• Human Computation (Quinn and Bederson, 2010; Sakamoto et al., 2010)

“The creation, aggregation and interpretation of strategically relevant
information for decision-making through distributed means” (Por, 2008)

Wikipedia, Innocentive, Threadless, CrowdFlower, IdeaScale, Reddit, etc.

Have been studied but rarely used as research instruments themselves (Malone,
2010)


Scenario Planning

Creative, narrative, group-based processes for engaging with uncertainty and
change (Wack, 1985; Van der Heijden, 1997)

• Double loop organizational learning (Argys & Schon; 1974)
• Constructivist & social learning theory (Piaget, 1977)
• Sensemaking & organizational awareness (Weick, 1979; Kleine,1999)
• Activity- & practice-based strategizing (Jarzabkowski, 2005; Orlikowski, 1992)
• Competitive advantages of perception management (Boyd, 1976)

Labor intensive & expensive, bene ts poorly documented (no veri cation or
reputation systems), limited participation (time, space & numbers),
predominance of senior decision-making elite (participant bias), dependent on
facilitator skills & consultant synthesis (facilitator & author bias)


Introduction
Contribution of This Study

1. Operational: Help to understand the role that online systems might play in
enhancing multi-stakeholder policy creation, speci cally in the context of the
challenges of future-focused, public planning initiatives

2. Methodological: Help to generate new analytical frameworks that can
improve our understanding of how such systems may be used for
measurement instruments and data analysis platforms


Outline
1. Introduction
5. Conclusion


Study Design & Methodology
Research Questions
• Do web-based participatory approaches add value to the traditional scenario
planning process? If so, where and in what ways?
• If not, where do they fall short, in what ways, and why?

Participation Interaction
• The number and type of • The number of variables and
participants involved, and in opinions incorporated?
what phases? • The mechanism of analysis,
• The geographic scope of ranking and clustering?
participation enabled? • The time spent on data
• The range of expert professional collection and analysis?
disciplines consulted? • The amount of user debate
and re ection?


An Exploratory Case Study Approach

A three-tiered, mixed method, case-study based approach, including:

• Informant interviews to identify key
Interviews
themes and constructs (n=46)

• Creation of two novel, prototypical
In-depth
data generation platforms and
cases
application on in-depth cases

• Pair-wise comparison of case
Base case
studies to a base case

• Evaluation of three additional Comparative
comparative examples from examples
secondary sources


Case 1: Futurescaper: The Impact of Climate Change
Impacts on the UK
186 drivers,
ranked, analyzed
and visualized as
system maps


Case 2: SenseMaker Scenarios: Future of Public Services
Under Financial Uncertainty
• 265 participants, micro-
scenarios

• Aggregated to three
sketch scenarios based
on pre-de ned
archetypes


Base Case: Future of a Northern Region in Spain
• Face-to-face scenario method

• Expert scenario consultancy

• 15 in-depth interviews

• Two day workshop, 20
participants

• Four regional scenarios


Comparative Examples

Institute for the Future’s Foresight
Engine
• 700 participants
• 81 countries
• 5,000 submissions in 24 hours

WikiStrat Collaborative Strategy
Platform
• 30 teams
• 13 countries
• ~35,000 words of high-quality content created in 4
weeks

The Future of Facebook Project
• 25 video interviews
• 109 Quora interactions
• ~50 Facebook participants


Data Constructs Measured

Participant Characteristics Interaction Characteristics
• Degree of public openness • Tasks performed

(including promotion & recruitment • Amount and types of input

efforts) considered
• Amount of preparation required • Amount and types of visualization

• The number of participants involved tools used
• Reasons for participation • Amount and types of analytical

• Degree of user anonymity tools used
• Type of participants involved • Amount and kinds of socialization

• Level of Education enabled
• Professional Experience • Amount and kinds of feedback

• Professional Discipline provided
• Age

• Geographic Origin • Supplementary interviews


Challenges

1) The relevant categories and variables for measurement were unknown in
advance

2) There was little empirical evidence for, or agreement on, the key outcome
variables for scenario planning

3) There were no standard measurement instruments or protocols available
that could be readily applied

Both dependent and independent variables were unknown and no standard
method of comparison could be established.

An exploratory, or “revelatory” case study design (Yin, 1994) was appropriate.


Outline
1. Introduction
5. Conclusion


Findings & Discussion
Finding 1: Greater Number and Diversity of Participants
Number of Participants
More participants were involved 700
700

525

350

265

175
Base Case (166)
SenseMaker
150
125
Foresight Engine
WikiStrat
FoFB 35
0


Number of Countries Represented
More participants were involved 90

82
From more diverse locations
68

45

30
23
Base Case
SenseMaker 18
Foresight Engine
WikiStrat
5
0



More participants were involved Base Case: ~20 different disciplines

Case 1: 35 different disciplines
From more diverse locations
Case 2: Signi cant experience
Wider range of experts &
professional disciplines

WikiStrat: Mixed teams of highly
trained inter-disciplinary contributors

Finding 2: Most Participation Was Light, Skewed Towards a
Few Heavy Users
Base Case: ~4.5 contributions per
user, more extensive involvement &
conversation through-out workshop

Case 1: ~ 1 contribution per user

# of
Case 2: ~ 1 contribution per user
Cards Forecastin #
Name Location Occupation Played g Points SI Awards
!"#$%&$"'(")*'+ ,-./-. 012.3*' 456 447857 8 9*2&*.:*'(
IFTF: ~6 contributions per user (1.5
';"/";& <"&12.($-.=+ ?-;#@$"$2-."A+C5 5DE6 7 F*G.;".
>? B2-A-(2&$ original contributions, 4.5 responses to
&*!'*$*.(2.**' ?-A-'"/- H.(2.**' ID4 47J7 7 9*2&*.:*'(
0H>+0"A3 others), 20% of users = 70% of content
;"$1#@.3 K-'$A"./=+LM N"$1*;"$2!2". I46 E84 7 N"!O'$1@'
9*2&*.:*'(
$1*#A-$P$12!3*.&B?=+?"."/" HA*!$'2!"A+
H.(2.**'
II5 5II 7
WikiStrat: Intensive contribution
through-out process, ~7,000 words
per team

Finding 3: Rapid Driver Generation & Exploration
Driver Generation:

Base Case: 80 hours + 120 minutes in workshop (5 hours per driver)
Case 1: ~5 minutes per driver
Case 2: ~10 minutes per driver
IFTF: ~90 seconds per driver

Clustering & Ranking:

Base Case: ~2 hours in workshop, “not enough time to discuss”
Case 1: Instantly sortable along number of dimensions
Case 2: Instantlysortable along number of dimensions
IFTF: N/A
WikiStrat: N/A
FoFB: Unknown, but “signi cant and more than we thought”


Finding 4: Most In uential at Early Stages
Detailed Case Comparative
Studies Examples

Increases the likelihood
Scenario
Planning Case 1: Case 2: Foresight WikiStrat Open
Futurescaper SenseMaker Engine Foresight
Steps
that a wide variety of
ID Issues forces and factors will
be included
Generate
key themes
Increases likelihood that
ID driving
forces
a diversity of
perspectives will be
Rank factors achieved

Develop draft
scenario logic
Implies that individual
Create draft
and group biases may
ﬁnal scenarios
be less dominant at the
Finalise
early drivers exploration
scenarios stage
Consider
implications

Identify
Strong scaling potential
indicators


Finding 5: “The Hourglass Effect”

Tension between structured / unstructured interfaces and analysis
approaches

Case 1: Highly structured interface, open-ended analysis
Case 2: Open-ended interface, highly structured analysis

More data = greater analytical burden

IFTF: Largest number of drivers and social interaction, but very dif cult to
make sense of
FoFB: “None of us had any idea it would take this long to complete.”

Trade off between ease of use & level of participation
IFTF: Simple, game-like engaging interface, very light analytic power
WikiStrat: High barrier of entry, rich analytic input and deep participation


Finding 5: “The Hourglass Effect”

Tension between structured / unstructured interfaces and analysis
approaches

Case 1: Highly structured interface, open-ended analysis
Case 2: Open-ended interface, highly structured analysis
“People enter these activities with little
More data = background experience. Part of your job is to
greater analytical burden
help model the thinking process that they
IFTF: Largest number of drivers and social interaction, but very dif cult to
should undergo.”
make sense of
FoFB: “None of us had any idea it would take this long to complete.”

Trade off between ease of use & level of participation
IFTF: Simple, game-like engaging interface, very light analytic power
WikiStrat: High barrier of entry, rich analytic input and deep participation


Finding 6: Role of Visuals & Multimedia

Increasing hydrological imbalance

Decreasing water quality Increasing malaria
Decreasing water availability

Increasing toxic algal blooms
Increasing hardships for women

Increasing improved water and sanitation

Increasing food prices

Increasing diarrhea

Increasing population displacement

Decreasing agricultural productivity

Decreasing crop yields
Increasing pollution

Decreasing sustainablilty of crop production
Decreasing deaths from cold temperatures

Increasing demand
Increasing migration

Increasing water shortages Increasing market

Increasing flooding Increasing air pollution

Increasing droughts
Increasing contamination of water supply


Finding 6: Role of Visuals & Multimedia Increasing uncertainty in food production

Decreasing agricultural productivity
Increasing potency of airborne diseases

Increasing droughts
Increasing food prices

Increasing diarrhea

Increasing hardships for women

Decreasing deaths from cold temperatures
Increasing malaria
Increasing air pollution

Decreasing water quality
Increasing hydrological imbalance

Increasing flooding

Increasing improved water and sanitation Decreasing water availability
Increasing toxic algal blooms Decreasing crop yields
Increasing water shortages
Increasing migration

Increasing contamination of water supply

Decreasing sustainablilty of crop production
Increasing population displacement

Increasing pollution

Increasing market
Increasing demand


Finding 6: Role of Visuals & Multimedia


Finding 7: Social Experience of Online Scenario Building
Base Case was far more effective
at producing active socialization
and interaction between
participants

“People need feedback in order to
stay involved. You can provide
automated feedback, but other
people are the best kind of
feedback you can possibly ask
for.”


Finding 7: Social Experience of Online Scenario Building
Base Case was far more effective
at producing active socialization
and interaction between
participants

“People need feedback in order to
stay involved. You can provide
automated feedback, but other
people are the best kind of
feedback you can possibly ask
for.”

Different kinds of experience were
possible with IFTF and WikiStrat
• Ranks and Roles
• “Coopetition”

Speculative Finding 1: Better Outcomes?

The evidence suggests that the use of such systems on their own will not
produce the desired outcome of the scenario process

Augment early-stages

• Transparency
• Speed
• Ef ciency
• Larger scale engagement

Suggests may be effective analytically, but is it psychologically? A hybrid
approach is worth exploring to get the full bene ts


Speculative Finding 2: Impact on Professional Standards

Greater transparency could facilitate reputation systems (eBay, Amazon)

“The futures profession is decentralized, eclectic and intellectually varied: there are
no schools that train its elite, few barriers to entry, no certi cation or regulatory
body.” (Pang, 2009)

Commoditize the scenarios market, split between “fast & cheap” or “slow &
bespoke”

Trade-off between quality (qualitative) aspects & quantity / speed


Speculative Finding 3: Impact on Scholarly Method

Continuous, self-re ective and emergent

Allow for user re ection on, and modi cation of, research constructs

“Moderators... sometimes have the feeling that they’re barely holding on
for dear life, because sometimes the carriage tries to run away without
them.”

Requires post-hoc and real-time evaluation, dif cult to determine what to study
in advance

Signi cantly enhanced potential for creativity, but signi cant challenges for
research design and rigor


Outline
1. Introduction
5. Conclusion


Conclusion
Contribution

1) Creating under-explored connections between urban planning, public
participation, online tools and scenario planning

2) The creation and evaluation of two unique online platforms for participatory
scenario planning in urban planning and public policy

2) The creation of an intellectual framework for measuring and evaluating their
role in the qualitative scenario planning process.


Conclusion
Limitations

Lack of a more rigorous experimental design, more controlled cases or a peer-
reviewed evaluation framework

Lack of a controlled, standardized recruitment process for participation

Differences in de nitions, processes and goals between cases and
comparative examples

Strongly dissenting views and participants self-selected out of being
interviewed, thereby biasing the results and discussion towards those available
and interested in the subject


Conclusion
Possible Evolution of These Approaches

Personal Futures Systems

Real-time Horizon Scanning & Scenario Generation Systems

Crowdsourced Think Tank Policy Review

Mass Media Speculation Engines


Conclusion
Areas for Future Research

Continue to develop more rigorous measures for evaluating the scenario
process and its outcomes

Conduct more controlled research on the impacts of speci c design and
interaction features

Explore the impact of various forms of socialization systems (chat,
commenting, voting, etc.) on the process and outcomes


Thank You

Noah Raford
PhD Candidate, Urban Information Systems Group, City Design and
Development Group, Department of Urban Studies and Planning, MIT

nraford@mit.edu

August 29, 2011

Questions?


Raford PhD defense final

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Viewers also liked

Viewers also liked (20)

Similar to Raford PhD defense final

Similar to Raford PhD defense final (20)

Recently uploaded

Recently uploaded (20)

Raford PhD defense final