1.
CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
1
CESAR
WORKING
DOCUMENT
SERIES
Working
document
no.2
Erreichbarkeitsatlas
as
Planning
Support
System
Testing
the
performance
in
supporting
strategy
making
M.
te
Brömmelstroet
26
January
2011
This
working
document
series
is
a
joint
initiative
of
the
University
of
Amsterdam,
Utrecht
University,
Wageningen
University
and
Research
centre
and
TNO
The
research
that
is
presented
in
this
series
is
financed
by
the
NWO
program
on
Sustainable
Accessibility
of
the
Randstad:
http://www.nwo.nl/nwohome.nsf/pages/nwoa_79vlym_eng

2. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
2

3. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
3
TABLE
OF
CONTENT
1.
INTRODUCTION
...............................................................................................................
4
2.
ASSESSING
THE
PERFORMANCE
OF
A
PSS
.......................................................................
5
2.1
Dual
goals
of
PSS
..................................................................................................................
5
2.2
Operationalizing
PSS
goals
...................................................................................................
5
3.
SETUP
OF
THE
EXPERIMENT
............................................................................................
7
3.1
Intervention:
The
Erreichbarkeitsatlas
.................................................................................
7
3.2
Strategy
making
trial
with
students
......................................................................................
8
3.3
Testing
the
performance
......................................................................................................
8
APPENDIX
1:
PROCESS
QUESTIONNAIRE
..............................................................................
14
APPENDIX
2:
OUTCOME
PERFORMANCE
QUESTIONNAIRE
..................................................
17
APPENDIX
3:
INSTRUMENT
USABILITY
QUESTIONNAIRE
......................................................
18

4. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
4
1. INTRODUCTION
There
are
a
large
number
of
computer
based
systems
that
aim
to
support
integrated
land
use
and
transport
planning;
more
than
100
in
the
Netherlands
alone
(Al
2005).
These
so-­‐called
Planning
Support
Systems
(PSS)
have
been
developed
since
the
1970s
and
are
still
continuously
improved.
In
a
recent
survey
among
land
use
and
transport
planners
in
the
Netherlands
we
found
that,
just
as
in
almost
all
planning
fields,
these
instrument
fail
to
support
an
important
phase
of
planning
where
land
use
and
transport
should
come
together;
the
strategic
planning
phase
(Te
Brömmelstroet
2010).
Figure
1
lists
the
reasons
that
were
found
to
block
the
widespread
use
of
these
tools
in
daily
practice
of
integrated
land
use
and
transport
strategy
making.
Figure
1
Bottlenecks
for
PSS
in
integrated
land
use
and
transport
strategy
making
(%
of
respondents
(124))
that
think
it
is
a
(highly)
problematic.
Figure
1
shows
that
it
is
mainly
a
set
of
soft
characteristics
that
hamper
widespread
use
of
PSS.
Transparency,
low
communication
value,
user
friendliness
and
interactiveness
are
seen
as
(highly)
problematic
by
more
than
half
of
the
respondents.
There
is
no
shortage
of
ideas
to
bridge
what
has
been
coined
the
implementation
gap.
Some
of
these
focus
on
improving
PSS
software
by
adding
new
functionality:
PSS
that
are
more
integrated
(i.e.
What
If
developed
by
Klosterman
1999),
more
interactive
(i.e.
Urban
Strategy
developed
by
TNO
2011)
or
more
user-­‐friendly
(i.e.
UrbanSim
developed
by
Waddell
2002,
2011).
Others
focus
more
on
the
hardware,
such
as
Maptables
(see
Vonk
&
Ligtenberg
2009)
and
other
visual
gadgets.
Then
there
is
the
process-­‐oriented
line
that
focuses
on
bridging
the
human
gap
between
the
potential
end-­‐users
and
the
PSS
developers
with
more
participative,
iterative
PSS
development
structures
(i.e.
Lee
1973;
1994;
Te
Brömmelstroet
&
Schrijnen
2010;
Vonk
2006).
One
of
the
instruments
that
aim
to
follow
the
strategy
of
improving
the
software
is
the
Accessibility
Atlas
(Erreichbarkeits
Atlas)
that
has
recently
been
developed
by
the
Technical
University
of
Munich.
Here,
we
report
on
a
randomized
clinical
trial
that
was
set
up
in
2011
to
test
the
performance
of
this
innovative
instrument
in
supporting
strategy
making
processes.
To
do
so,
I
first
define
how
the
performance
of
a
PSS
can
be
measured
(section
2).
Then,
I
describe
the
methodological
choices
and
set
up
of
the
experiment
(section
3).
In
section
4,
the
results
are
presented
and
this
report
closes
by
discussing
the
findings
and
reflecting
on
its
meaning.

5. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
5
2. ASSESSING
THE
PERFORMANCE
OF
A
PSS
2.1 Dual
goals
of
PSS
Recently
we
have
proposed
an
integrated
performance
framework
that
allows
us
to
structurally
test
if
PSS,
or
specific
improvement
strategies
for
them,
are
effective
in
supporting
strategy
making
processes
(Te
Brömmelstroet,
2012).
This
framework
is
based
on
the
concept
of
dual
goals
of
PSS.
First,
many
PSS
explicitly
aim
to
improve
planning
processes,
e.g.
by
structuring
them
better
and/or
making
them
more
interactive,
integrative
and
participatory.
Next
to
that
PSS
aim
to
improve
the
outcomes
of
these
processes
(e.g.
strategies,
plans
and
projects),
e.g.
by
providing
relevant
content
(knowledge,
information)
and
facilitating
design-­‐analyse
loops.
In
strategic
planning
this
link
is
particularly
problematic.
Following
Couclelis,
who
addressed
land
use
planning,
we
can
assert
that
strategic
planning
“is
a
hopelessly
complex
human
endeavour.
It
involves
actions
taken
by
some
to
affect
the
use
of
land
controlled
by
others,
following
decisions
taken
by
third
parties
based
on
values
not
shared
by
all
concerned,
regarding
issues
no
one
fully
comprehends,
in
an
attempt
to
guide
events
and
processes
that
very
likely
will
not
unfold
in
the
time,
place,
and
manner
anticipated”
(Couclelis,
2005;
p.
1355).
Following
this,
it
is
often
said
that
strategy
making
problems
are
‘wicked’
(Rittel
and
Webber,
1984;
Christensen,
1985);
problems
on
which
there
is
no
consensus
and
for
which
there
are
no
clear-­‐cut
answers
or
solutions.
Pelzer
(2012)
refers
to
this
problem
as
a
double
complex
one:
the
subject
of
planning
is
complex,
as
is
the
process
of
planning
itself.
In
such
situations,
a
rigid
protocol
of
planning
steps
is
hardly
helpful.
It
is
the
development
of
the
capacity
to
deal
with
these
problems,
rather
than
final
solutions
that
is
the
general
aim
of
strategic
planning
(Healey,
2006).
Planners
here
(should)
aim
to
become
aware
and
learn
to
cope
with
complexity
and
the
“unknown
unknowns”
(Taleb,
2007)
instead
of
collecting
knowledge
to
reduce
it/them.
2.2 Operationalizing
PSS
goals
Both
goals
of
PSS
(i.e.
improving
process
and
improving
outcomes)
have
been
operationalized
into
a
multidimensional
framework.
Based
on
academic
literature
on
ideational
output
(e.g.
Dean
et
al.
2006)
the
quality
of
a
planning
outcome
can
be
rated
on
four
dimensions;
• novelty
(originality,
paradigm
relatedness),
• workability
(implementability,
acceptability),
• relevance
(applicability
effectiveness),
and
• specificity
(completeness,
Implicational
explicitness,
clarity).
For
a
planning
outcome
to
be
of
high
quality,
it
has
to
score
on
all
these
dimensions.
Based
on
academic
work
on
Group
Model
Building
(Rouwette
et
al.
2002)
we
operationalized
the
quality
of
the
process
in
nine
dimensions;
• reactions
(enthusiasm,
satisfaction,
credibility),
• insight
(in
problems,
in
others’
assumptions),
• commitment,
• behavioural
change,
• communication,
• shared
language,
• consensus
(on
problem,
goal,
strategies),
• cohesion
and
• efficiency.

6. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
6
Again,
this
means
that
for
a
planning
process
to
be
of
high
quality
it
needs
to
score
high
on
all
these
dimensions.
Table
1
Dimensions
of
outcome
Table
2
Dimensions
of
process
Dimension
Operationalization
1.
Novelty
1a.
Originality
1b.
Paradigm
relatedness
2.
Workability
2a.
Implementability
2b.
Acceptability
3.
Relevance
3a.
Applicability
3b.
Effectiveness
4.
Specificity
4a.
Completeness
4b.
Implicational
explicitness
4c.
Clarity
Dimension
Operationalization
5.
Reaction
5a.
Enthusiasm
5b.
Satisfaction
5c.
Credibility
6.
Insight
6a.
Insight
in
problem
6b.
Insight
in
assumptions
7.
Commitment
8.
Behaviour
9.
Communication
10.
Shared
language
11.
Consensus
11a.
On
problem
11b.
On
goals
11c.
On
strategies
12.
Cohesion
13.
Efficiency
gains

7. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
7
3. SETUP
OF
THE
EXPERIMENT
3.1 Intervention:
The
Erreichbarkeitsatlas
In
2011,
the
Faculty
for
Urban
form
and
mobility
of
the
Technical
University
of
Munich
developed
an
accessibility
atlas
for
the
Munich
Metropolitan
region.
This
instrument
allows
users
to
explore
the
opportunities
and
treats
for
this
growing
region
in
terms
of
potential
activities
to
be
reached
within
different
travel
times
by
different
modes
of
transport.
Examples
of
maps
that
can
be
explored,
and
that
can
be
interactively
developed
by
the
user
via
an
online
platform,
are
the
number
of
people
that
can
reach
intercity
train
stations
within
30
minutes
or
mapping
the
commuting
flows
compared
to
the
accessibility
quality
of
these
connections.
The
Erreichbarkeitsatlas
is
an
attempt
to
transferuse
the
potential
of
accessibility
as
a
professional
language
for
integrated
land
use
and
mobility
planning
issues
to
the
realm
of
urban
and
regional
planning.
Accessibility
concepts
are
increasingly
acknowledged
as
fundamental
to
understand
the
functioning
of
cities
and
urban
regions.
In
particular,
accessibility
instruments
are
able
to
provide
a
framework
for
understanding
the
reciprocal
relationships
between
land
use
and
mobility.
Figure
2
A
map
from
the
Erreichbarkeitsatlas:
commuting
flows
vs.
accessibility
quality
A
collection
of
these
maps
is
used
in
the
experiment
in
printed
form.
In
the
next
section,
I
will
describe
the
experiment
in
more
detail.

8. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
8
3.2 Strategy
making
trial
with
students
To
test
the
performance
of
this
instrument
in
supporting
strategy
making
a
randomized
controlled
trial
with
students
was
set
up.
A
total
of
34
voluntary
students
took
part
in
the
trial,
from
the
Master
Environmental
Engineering
and
from
the
Master
Transport
Engineering.
These
students
were
randomly
divided
in
a
control
group
(that
received
only
a
set
of
empty
maps
of
the
region)
and
a
treatment
group
(that
received
the
maps
of
the
Erreichbarkeitsatlas,
introduction
to
these
maps
and
support
by
one
of
the
developers).
These
groups
were
then
again
divided
in
random
groups
of
three
students
that
then
engaged
in
a
strategy
making
process
(of
60
minutes)
for
the
Munich
Metropolitan
region.
The
planning
problem
that
all
groups
received
on
paper
is
presented
in
the
box
below.
In
the
near
future,
the
Europäische
Metropolregion
München
is
expecting
a
strong
growth
in
both
economic
and
demographic
terms.
In
the
coming
20
years,
the
region
faces
considerable
spatial
and
mobility
challenges:
within
the
region,
a
total
of
60.000
new
working
places
(offices
and
industry),
60.000
new
houses
and
supporting
leisure
and
shopping
areas
have
to
be
allocated.
The
region
aspires
that
these
new
developments
are
primarily
located
in
places
with
high
public
transport
accessibility.
There,
sustainable
mobility
with
limited
negative
impacts
can
be
guaranteed.
You
will
form
a
strategic
planning
team
together
with
two
colleagues.
This
team
is
asked
to
develop
a
rough
spatial
strategy
for
this
allocation
challenge.
You
will
have
a
map
of
the
region
and
an
empty
sheet
to
your
disposal
on
which
you
can
formulate
a
strategy
in
text
and
in
geographical
drawings
(please
remember
to
include
a
legend).
In
total
you
will
have
a
maximum
of
60
minutes
to
perform
this.
In
this
competition
you
are
asked
to
find
good
locations
(on
the
level
of
municipalities)
and
use
the
complete
region
as
an
“empty
sheet”
where
all
locations
are
potentially
possible.
Don’t
worry
about
the
size,
form
of
these
locations.
Your
strategy
will
be
judged
by
external
experts
on
the
quality
of
the
final
result.
This
quality
will
be
judged
in
terms
of:
• Novelty;
• workability;
• relevance
and;
• specificity.
3.3 Testing
the
performance
After
60
minutes
all
groups
had
to
hand
in
their
strategy.
For
this,
they
could
use
a
map
with
an
attached
empty
sheet
to
write
down
their
ideas
and
argumentation.
These
strategies
were
then
our
input
for
the
performance
test.
This
test
was
done
in
three
steps.
First,
all
participants
were
asked
to
fill
in
a
questionnaire
related
to
the
quality
of
the
planning
process.
This
questionnaire
consisted
of
32
statements
that
each
related
to
the
dimensions
of
table
1.
For
each
of
these
statements,
the
participant
was
asked
to
rate
his/her
agreement
on
a
7-­‐point
Likert
scale.
This
questionnaire
is
attached
in
appendix
1.
The
outcomes
of
these
ratings
were
grouped
and
this
allowed
us
to
compared
the
process
dimensions
of
the
control
group
with
these
of
the
treatment
group.
The
resulting
strategies
were
rated
on
their
outcome
dimensions
as
listed
in
table
2.
This
was
extended
with
one
statement
on
the
amount
of
accessibility
logic
followed.
Again,
the
same
method
of
(28)
7-­‐point
Likert
scale
statements
was
used.
The
questionnaire
is
listed
in
appendix
2.

9. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
9
Each
strategy
was
rated
by
two
external
raters.
For
this,
two
PhD
students
that
are
specialized
in
integrated
land
use
and
transport
from
the
University
of
Amsterdam
were
asked.
The
third
and
last
measurement
questionnaire
was
focused
on
the
perceived
quality
of
the
Erreichbarkeitsatlas.
For
that,
again
statements
were
offered,
but
now
only
to
the
participants
of
the
treatment
group.
They
were
asked
to
respond
to
several
dimensions
of
the
instrument
that
are
generally
related
to
the
usability
of
such
instruments
(see
e.g.
figure
1).
This
questionnaire
is
attached
in
appendix
3.

10. CESAR
Working
Document
Series
no.
2
Erreichbarkeitsatlas
as
PSS
Page
10
4. RESULTS
OF
THE
EXPERIMENT
4.1 Effects
on
the
planning
process
The
average
scores
on
the
planning
process
dimensions
are
listed
in
table
3.
The
table
distinguishes
the
groups
that
did
not
receive
planning
support
from
the
Erreichbarkeitsatlas
from
the
groups
that
did.
On
the
bottom,
the
differences
between
the
two
are
presented
with
the
level
of
significance
(deviation
from
the
zero
hypothesis
that
there
is
no
effect).
Although
both
groups
score
above
average
(4)
on
all
dimensions,
all
dimensions
but
cohesion
(no
effect)
show
a
negative
effect
of
the
support
of
the
Erreichbarkeitsatlas.
These
negative
effects
are
significant
on
a
0.05
level
for
the
dimensions
(bold
for
grouped
dimensions)
satisfaction,
reaction,
insight
in
the
problem,
insight
in
assumptions,
insight,
communication,
shared
language,
consensus
on
problem,
consensus
on
goals,
consensus
and
efficiency.
4.2 Effects
on
the
planning
outcome
The
average
scores
on
the
planning
outcome
dimensions
are
listed
in
table
4.
The
table
is
set
up
similar
to
table
3.
Here,
most
dimensions
score
below
or
just
above
average
(4).
Statements
on
two
of
them
(paradigm
relatedness
and
Implementability)
were
not
filled
in
by
one
of
the
external
raters
due
to
a
lack
of
insight
in
them.
These
should
therefore
not
be
considered
as
valuable
scores.
The
effects
of
the
Erreichbarkeitsatlas
are
more
mixed
here.
Small
positive
scores
are
found
for
bold
for
grouped
dimensions)
accessibility
logic
(0.39),
applicability
(0.19),
completeness
and
specificity
(both
0.13).
None
of
these
effects
are
significant.
The
other
dimensions
all
score
negatively,
but
also
not
significantly
so.
4.3 Perceived
quality
of
the
instrument
The
usability
characteristics
of
the
Erreichbarkeitsatlas
are
very
positively
rated
by
the
participants
that
received
support
from
it,
see
table
5.
Especially,
the
participants
found
the
instrument
easy
to
understand
(5.32).
They
rated
the
ability
of
the
instrument
to
support
creating
(5.22)
and
evaluating
(5.17)
strategic
ideas
as
very
positive
as
well.
Important
factors
as
transparency
and
user-­‐friendliness
also
received
a
score
over
5
(on
a
scale
from
1to
7).
The
lowest
score
is
for
the
understandability
of
the
indicators
used
(still
4.58).