Cesar working document 2 erreichbarkeitsatlas experiment 2

210 views
152 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
210
On SlideShare
0
From Embeds
0
Number of Embeds
49
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Cesar working document 2 erreichbarkeitsatlas experiment 2

  1. 1.   CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS     Page  1       CESAR  WORKING  DOCUMENT  SERIES   Working  document  no.2                 Erreichbarkeitsatlas  as  Planning  Support  System   Testing  the  performance  in  supporting  strategy  making       M.  te  Brömmelstroet     26  January  2011             This  working  document  series  is  a  joint  initiative  of  the  University  of  Amsterdam,    Utrecht  University,  Wageningen  University  and   Research  centre  and  TNO               The  research  that  is  presented  in  this  series  is  financed  by  the  NWO  program  on  Sustainable  Accessibility  of  the  Randstad:   http://www.nwo.nl/nwohome.nsf/pages/nwoa_79vlym_eng      
  2. 2. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  2        
  3. 3. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  3     TABLE  OF  CONTENT   1.   INTRODUCTION  ...............................................................................................................  4   2.   ASSESSING  THE  PERFORMANCE  OF  A  PSS  .......................................................................  5   2.1   Dual  goals  of  PSS  ..................................................................................................................  5   2.2   Operationalizing  PSS  goals  ...................................................................................................  5   3.   SETUP  OF  THE  EXPERIMENT  ............................................................................................  7   3.1   Intervention:  The  Erreichbarkeitsatlas  .................................................................................  7   3.2   Strategy  making  trial  with  students  ......................................................................................  8   3.3   Testing  the  performance  ......................................................................................................  8   APPENDIX  1:  PROCESS  QUESTIONNAIRE  ..............................................................................  14   APPENDIX  2:  OUTCOME  PERFORMANCE  QUESTIONNAIRE  ..................................................  17   APPENDIX  3:  INSTRUMENT  USABILITY  QUESTIONNAIRE  ......................................................  18    
  4. 4. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  4     1. INTRODUCTION   There  are  a  large  number  of  computer  based  systems  that  aim  to  support  integrated  land  use  and   transport   planning;   more   than   100   in   the   Netherlands   alone   (Al   2005).   These   so-­‐called   Planning   Support  Systems  (PSS)  have  been  developed  since  the  1970s  and  are  still  continuously  improved.  In   a  recent  survey  among  land  use  and  transport  planners  in  the  Netherlands  we  found  that,  just  as  in   almost  all  planning  fields,  these  instrument  fail  to  support  an  important  phase  of  planning  where   land   use   and   transport   should   come   together;   the   strategic   planning   phase   (Te   Brömmelstroet   2010).  Figure  1  lists  the  reasons  that  were  found  to  block  the  widespread  use  of  these  tools  in  daily   practice  of  integrated  land  use  and  transport  strategy  making.       Figure  1        Bottlenecks  for  PSS  in  integrated  land  use  and  transport  strategy  making  (%  of  respondents  (124))  that  think  it   is  a  (highly)  problematic.         Figure  1  shows  that  it  is  mainly  a  set  of  soft  characteristics  that  hamper  widespread  use  of  PSS.   Transparency,  low  communication  value,  user  friendliness  and  interactiveness  are  seen  as  (highly)   problematic  by  more  than  half  of  the  respondents.  There  is  no  shortage  of  ideas  to  bridge  what  has   been  coined  the  implementation  gap.  Some  of  these  focus  on  improving  PSS  software  by  adding   new  functionality:  PSS  that  are  more  integrated  (i.e.  What  If  developed  by  Klosterman  1999),  more   interactive   (i.e.   Urban   Strategy   developed   by   TNO   2011)   or   more   user-­‐friendly   (i.e.   UrbanSim   developed  by  Waddell  2002,  2011).  Others  focus  more  on  the  hardware,  such  as  Maptables  (see   Vonk   &   Ligtenberg   2009)   and   other   visual   gadgets.   Then   there   is   the   process-­‐oriented   line   that   focuses  on  bridging  the  human  gap  between  the  potential  end-­‐users  and  the  PSS  developers  with   more  participative,  iterative  PSS  development  structures  (i.e.  Lee  1973;  1994;  Te  Brömmelstroet  &   Schrijnen  2010;  Vonk  2006).       One  of  the  instruments  that  aim  to  follow  the  strategy  of  improving  the  software  is  the  Accessibility   Atlas   (Erreichbarkeits   Atlas)   that   has   recently   been   developed   by   the   Technical   University   of   Munich.   Here,   we   report   on   a   randomized   clinical   trial   that   was   set   up   in   2011   to   test   the   performance  of  this  innovative  instrument  in  supporting  strategy  making  processes.  To  do  so,  I  first   define   how   the   performance   of   a   PSS   can   be   measured   (section   2).   Then,   I   describe   the   methodological   choices   and   set   up   of   the   experiment   (section   3).   In   section   4,   the   results   are   presented  and  this  report  closes  by  discussing  the  findings  and  reflecting  on  its  meaning.  
  5. 5. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  5     2. ASSESSING  THE  PERFORMANCE  OF  A  PSS   2.1 Dual  goals  of  PSS   Recently  we  have  proposed  an  integrated  performance  framework  that  allows  us  to  structurally   test  if  PSS,  or  specific  improvement  strategies  for  them,  are  effective  in  supporting  strategy  making   processes  (Te  Brömmelstroet,  2012).  This  framework  is  based  on  the  concept  of  dual  goals  of  PSS.   First,  many  PSS  explicitly  aim  to  improve  planning  processes,  e.g.  by  structuring  them  better  and/or   making  them  more  interactive,  integrative  and  participatory.  Next  to  that  PSS  aim  to  improve  the   outcomes  of  these  processes  (e.g.  strategies,  plans  and  projects),  e.g.  by  providing  relevant  content   (knowledge,   information)   and   facilitating   design-­‐analyse   loops.   In   strategic   planning   this   link   is   particularly  problematic.  Following  Couclelis,  who  addressed  land  use  planning,  we  can  assert  that   strategic  planning       “is  a  hopelessly  complex  human  endeavour.  It  involves  actions  taken  by  some  to  affect  the   use  of  land  controlled  by  others,  following  decisions  taken  by  third  parties  based  on  values   not  shared  by  all  concerned,  regarding  issues  no  one  fully  comprehends,  in  an  attempt  to   guide  events  and  processes  that  very  likely  will  not  unfold  in  the  time,  place,  and  manner   anticipated”  (Couclelis,  2005;  p.  1355).     Following  this,  it  is  often  said  that  strategy  making  problems  are  ‘wicked’  (Rittel  and  Webber,  1984;   Christensen,  1985);  problems  on  which  there  is  no  consensus  and  for  which  there  are  no  clear-­‐cut   answers  or  solutions.  Pelzer  (2012)  refers  to  this  problem  as  a  double  complex  one:  the  subject  of   planning   is   complex,   as   is   the   process   of   planning   itself.   In   such   situations,   a   rigid   protocol   of   planning  steps  is  hardly  helpful.  It  is  the  development  of  the  capacity  to  deal  with  these  problems,   rather   than   final   solutions   that   is   the   general   aim   of   strategic   planning   (Healey,   2006).   Planners   here   (should)   aim   to   become   aware   and   learn   to   cope   with   complexity   and   the   “unknown   unknowns”  (Taleb,  2007)  instead  of  collecting  knowledge  to  reduce  it/them.       2.2 Operationalizing  PSS  goals   Both  goals  of  PSS  (i.e.  improving  process  and  improving  outcomes)  have  been  operationalized  into   a  multidimensional  framework.  Based  on  academic  literature  on  ideational  output  (e.g.  Dean  et  al.   2006)  the  quality  of  a  planning  outcome  can  be  rated  on  four  dimensions;     • novelty  (originality,  paradigm  relatedness),     • workability  (implementability,  acceptability),     • relevance  (applicability  effectiveness),    and     • specificity  (completeness,  Implicational  explicitness,  clarity).       For  a  planning  outcome  to  be  of  high  quality,  it  has  to  score  on  all  these  dimensions.  Based  on   academic  work  on  Group  Model  Building  (Rouwette  et  al.  2002)  we  operationalized  the  quality  of   the  process  in  nine  dimensions;     • reactions  (enthusiasm,  satisfaction,  credibility),     • insight  (in  problems,  in  others’  assumptions),     • commitment,     • behavioural  change,     • communication,     • shared  language,     • consensus  (on  problem,  goal,  strategies),     • cohesion  and     • efficiency.      
  6. 6. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  6     Again,  this  means  that  for  a  planning  process  to  be  of  high  quality  it  needs  to  score  high  on  all  these   dimensions.     Table  1      Dimensions  of  outcome   Table  2      Dimensions  of  process     Dimension   Operationalization   1.  Novelty   1a.  Originality     1b.  Paradigm  relatedness   2.  Workability   2a.  Implementability     2b.  Acceptability   3.  Relevance   3a.  Applicability     3b.  Effectiveness   4.  Specificity   4a.  Completeness     4b.  Implicational  explicitness     4c.  Clarity         Dimension   Operationalization   5.     Reaction   5a.  Enthusiasm     5b.  Satisfaction     5c.  Credibility   6.     Insight   6a.  Insight  in  problem     6b.  Insight  in  assumptions   7.     Commitment     8.     Behaviour     9.     Communication     10.  Shared  language     11.    Consensus   11a.  On  problem     11b.  On  goals     11c.  On  strategies   12.    Cohesion     13.    Efficiency  gains    
  7. 7. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  7     3. SETUP  OF  THE  EXPERIMENT   3.1 Intervention:  The  Erreichbarkeitsatlas   In  2011,  the  Faculty  for  Urban  form  and  mobility  of  the  Technical  University  of  Munich  developed   an  accessibility  atlas  for  the  Munich  Metropolitan  region.  This  instrument  allows  users  to  explore   the  opportunities  and  treats  for  this  growing  region  in  terms  of  potential  activities  to  be  reached   within   different   travel   times   by   different   modes   of   transport.   Examples   of   maps   that   can   be   explored,   and   that   can   be   interactively   developed   by   the   user   via   an   online   platform,   are   the   number   of   people   that   can   reach   intercity   train   stations   within   30   minutes   or   mapping   the   commuting  flows  compared  to  the  accessibility  quality  of  these  connections.       The  Erreichbarkeitsatlas  is  an  attempt  to  transferuse  the  potential  of  accessibility  as  a  professional   language  for  integrated  land  use  and  mobility  planning  issues  to  the  realm  of  urban  and  regional   planning.  Accessibility  concepts  are  increasingly  acknowledged  as  fundamental  to  understand  the   functioning  of  cities  and  urban  regions.  In  particular,  accessibility  instruments  are  able  to  provide  a   framework  for  understanding  the  reciprocal  relationships  between  land  use  and  mobility.       Figure  2      A  map  from  the  Erreichbarkeitsatlas:  commuting  flows  vs.  accessibility  quality         A   collection   of   these   maps   is   used   in   the   experiment   in   printed   form.   In   the   next   section,   I   will   describe  the  experiment  in  more  detail.      
  8. 8. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  8     3.2 Strategy  making  trial  with  students   To  test  the  performance  of  this  instrument  in  supporting  strategy  making  a  randomized  controlled   trial   with   students   was   set   up.   A   total   of   34   voluntary   students   took   part   in   the   trial,   from   the   Master   Environmental   Engineering   and   from   the   Master   Transport   Engineering.   These   students   were  randomly  divided  in  a  control  group  (that  received  only  a  set  of  empty  maps  of  the  region)   and  a  treatment  group  (that  received  the  maps  of  the  Erreichbarkeitsatlas,  introduction  to  these   maps   and   support   by   one   of   the   developers).   These   groups   were   then   again   divided   in   random   groups  of  three  students  that  then  engaged  in  a  strategy  making  process  (of  60  minutes)  for  the   Munich  Metropolitan  region.  The  planning  problem  that  all  groups  received  on  paper  is  presented   in  the  box  below.       In  the  near  future,  the  Europäische  Metropolregion  München  is  expecting  a  strong  growth  in  both   economic  and  demographic  terms.  In  the  coming  20  years,  the  region  faces  considerable  spatial   and   mobility   challenges:   within   the   region,   a   total   of   60.000   new   working   places   (offices   and   industry),  60.000  new  houses  and  supporting  leisure  and  shopping  areas  have  to  be  allocated.  The   region   aspires   that   these   new   developments   are   primarily   located   in   places   with   high   public   transport   accessibility.   There,   sustainable   mobility   with   limited   negative   impacts   can   be   guaranteed.     You  will  form  a  strategic  planning  team  together  with  two  colleagues.  This  team  is  asked  to  develop   a   rough   spatial   strategy   for   this   allocation   challenge.   You   will   have   a   map   of   the   region   and   an   empty  sheet  to  your  disposal  on  which  you  can  formulate  a  strategy  in  text  and  in  geographical   drawings  (please  remember  to  include  a  legend).  In  total  you  will  have  a  maximum  of  60  minutes  to   perform   this.   In   this   competition   you   are   asked   to   find   good   locations   (on   the   level   of   municipalities)  and  use  the  complete  region  as  an  “empty  sheet”  where  all  locations  are  potentially   possible.  Don’t  worry  about  the  size,  form  of  these  locations.     Your  strategy  will  be  judged  by  external  experts  on  the  quality  of  the  final  result.  This  quality  will  be   judged  in  terms  of:     • Novelty;   • workability;     • relevance  and;     • specificity.       3.3 Testing  the  performance   After  60  minutes  all  groups  had  to  hand  in  their  strategy.  For  this,  they  could  use  a  map  with  an   attached  empty  sheet  to  write  down  their  ideas  and  argumentation.  These  strategies  were  then  our   input  for  the  performance  test.  This  test  was  done  in  three  steps.     First,   all   participants   were   asked   to   fill   in   a   questionnaire   related   to   the   quality   of   the   planning   process.  This  questionnaire  consisted  of  32  statements  that  each  related  to  the  dimensions  of  table   1.  For  each  of  these  statements,  the  participant  was  asked  to  rate  his/her  agreement  on  a  7-­‐point   Likert   scale.   This   questionnaire   is   attached   in   appendix   1.   The   outcomes   of   these   ratings   were   grouped  and  this  allowed  us  to  compared  the  process  dimensions  of  the  control  group  with  these   of  the  treatment  group.     The   resulting   strategies   were   rated   on   their   outcome   dimensions   as   listed   in   table   2.   This   was   extended   with   one   statement   on   the   amount   of   accessibility   logic   followed.     Again,   the   same   method  of  (28)  7-­‐point  Likert  scale  statements  was  used.  The  questionnaire  is  listed  in  appendix  2.  
  9. 9. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  9     Each  strategy  was  rated  by  two  external  raters.  For  this,  two  PhD  students  that  are  specialized  in   integrated  land  use  and  transport  from  the  University  of  Amsterdam  were  asked.       The   third   and   last   measurement   questionnaire   was   focused   on   the   perceived   quality   of   the   Erreichbarkeitsatlas.  For  that,  again  statements  were  offered,  but  now  only  to  the  participants  of   the  treatment  group.  They  were  asked  to  respond  to  several  dimensions  of  the  instrument  that  are   generally   related   to   the   usability   of   such   instruments   (see   e.g.   figure   1).   This   questionnaire   is   attached  in  appendix  3.    
  10. 10. CESAR  Working  Document  Series  no.  2     Erreichbarkeitsatlas  as  PSS   Page  10     4. RESULTS  OF  THE  EXPERIMENT     4.1 Effects  on  the  planning  process     The  average  scores  on  the  planning  process  dimensions  are  listed  in  table  3.  The  table  distinguishes   the  groups  that  did  not  receive  planning  support  from  the  Erreichbarkeitsatlas  from  the  groups  that   did.  On  the  bottom,  the  differences  between  the  two  are  presented  with  the  level  of  significance   (deviation   from   the   zero   hypothesis   that   there   is   no   effect).   Although   both   groups   score   above   average  (4)  on  all  dimensions,  all  dimensions  but  cohesion  (no  effect)  show  a  negative  effect  of  the   support  of  the  Erreichbarkeitsatlas.       These   negative   effects   are   significant   on   a   0.05   level   for   the   dimensions   (bold   for   grouped   dimensions)   satisfaction,   reaction,   insight   in   the   problem,   insight   in   assumptions,   insight,   communication,   shared   language,   consensus   on   problem,   consensus   on   goals,   consensus   and   efficiency.       4.2 Effects  on  the  planning  outcome     The  average  scores  on  the  planning  outcome  dimensions  are  listed  in  table  4.  The  table  is  set  up   similar  to  table  3.  Here,  most  dimensions  score  below  or  just  above  average  (4).  Statements  on  two   of   them   (paradigm   relatedness   and   Implementability)   were   not   filled   in   by   one   of   the   external   raters  due  to  a  lack  of  insight  in  them.  These  should  therefore  not  be  considered  as  valuable  scores.       The  effects  of  the  Erreichbarkeitsatlas  are  more  mixed  here.  Small  positive  scores  are  found  for   bold   for   grouped   dimensions)   accessibility   logic   (0.39),   applicability   (0.19),   completeness   and   specificity   (both   0.13).   None   of   these   effects   are   significant.   The   other   dimensions   all   score   negatively,  but  also  not  significantly  so.       4.3 Perceived  quality  of  the  instrument     The  usability  characteristics  of  the  Erreichbarkeitsatlas  are  very  positively  rated  by  the  participants   that  received  support  from  it,  see  table  5.  Especially,  the  participants  found  the  instrument  easy  to   understand   (5.32).   They   rated   the   ability   of   the   instrument   to   support   creating   (5.22)   and   evaluating   (5.17)   strategic   ideas   as   very   positive   as   well.   Important   factors   as   transparency   and   user-­‐friendliness  also  received  a  score  over  5  (on  a  scale  from  1to  7).  The  lowest  score  is  for  the   understandability  of  the  indicators  used  (still  4.58).  

×