Planning support systems aim to improve planning processes and outcomes, but have not been rigorously tested. This study conducted a controlled experiment using the Urban Strategy planning support system to assess its impacts. Students played the role of planners and used Urban Strategy or traditional methods to develop strategies. Results from usability indicators and observations of the planning process showed Urban Strategy improved the process quality. Metrics also indicated it enhanced outcome quality, though the study design has limitations around external and ecological validity that require further exploration of fundamental improvement mechanisms between participants and planning support systems.

1. (HOW) DO PLANNING SUPPORT
SYSTEMS IMPROVE PLANNING?
A controlled experiment

2. Urban environments – climate - mobility
NWO: CESAR PROGRAM
MARCO TE BRÖMMELSTROET UNIVERSITY OF AMSTERDAM
PETER PELZER UTRECHT UNIVERSITY
RALPH KLERKX TNO
SANDER SCHAMINÉE TNO

3. Research rationale
 PSS field is maturing
 ‘Knowledge technologies’
 Many claims & hypotheses
 Not much focus on testing
 Overview of PSS books:
 414 reported claims
 59 reported tests

4. Research rationale
 Implementation gap
 Especially in strategic planning
 Constructive learning requires:
 Consolidated aims
 Consolidated theory
 Consolidated methods
(Nijstad 2009 in: Te Brömmelstroet 2013)

5. Conceptual model

6. PSS: Urban Strategy
 Urban Strategy: developed by TNO since 2006
 Started as an environmental modeling tool
 Main features:
 Calculate and display effects interactively
 Centralized data storage
 Scenarios
 Communication between models

7. PSS: Urban Strategy
3D Database3D Database
IMB Communication Framework (Publish/Subscribe)
Traffic Noise Air Safety Energy Cost
1D
Viewer
2D (GIS)
Viewer
3D
Viewer
External
Data Link
External
Models
Offline
Models
Sensor
Link
Mobile
Client
Connect
PDA, SmartphonePDA, SmartphoneWMS, WFS,
NetCDF
Tablet, Surface TableTablet, Surface Table
OPS
Lotos
EUROS

8. PSS: Urban Strategy
Perform measures
in GIS environment

9. Experiment
Video of experiment
(5 min)

10. CLASSIC USABILITY INDICATORS
6,12 6,07
5,81 5,79 5,74 5,71 5,60 5,59 5,55 5,55
5,75
5,36 5,31 5,24
5,10 5,07
4,92
0,00
1,00
2,00
3,00
4,00
5,00
6,00
7,00
LESSTIM
E
DOM
ORE
SKETCHING
LESSEFFORT
DISPLAYED
EVALUATION
UNDERSTANDABLE
CREATING
COM
M
UNICATIVE
CREDIBLE
OVERALL
USERFRIENDLY
DETAIL
BROADNESS
TRANSPARANCY
INDICATORS
SPECIFIC

11. RESULTS ON PLANNING PROCESS QUALITY

12. RESULTS ON PLANNING OUTCOME QUALITY

13. Methodological reflection
Both observation and indicators show a positive effect
on planning (process). But:
 Low ecological validity
 combine with real world observations
 High measurement validity
 ‘Quality of planning’ concept under development
 External validity
 Is Urban Strategy a typical PSS (for strategy making)

14. Methodological reflection
Both observation and indicators show a positive effect
on planning (process). But:
 Role playing is problematic
 Students from different backgrounds
 Holding treatment constant is problematic
 More focus on fundamental mechanisms
 Final lesson: Potential improvements on organizing the exchange
of knowledge between participants and PSS

15. Thank you for your kind attention
E-mail: brommelstroet@uva.nl
Recent publications:
Te Brömmelstroet, M. (2012). "Transparency, flexibility, simplicity: From buzzwords towards
strategies for real PSS improvement." Computers, Environment and Urban Systems, 36(1),
96-104
Te Brömmelstroet M. (2013). “Performance of Planning Support Systems: What is it, and
how do we report on it?”. Computers, Environment and Urban Systems, in press