Smart Commute Evaluation Tools, Techniques and Lessons Learned in Monitoring and Evaluating Workplace-based TDM Programs
Outline <ul><li>Background </li></ul><ul><li>Approach </li></ul><ul><li>Monitoring Tools </li></ul><ul><li>Evaluation </li...
At-a-Glance <ul><li>Build evaluation into your initial planning </li></ul><ul><li>Monitor activities, benefits and custome...
Background <ul><li>Smart Commute </li></ul><ul><ul><li>Focus on commuters and workplaces </li></ul></ul><ul><ul><li>Partne...
Background <ul><li>Metrolinx  (Greater Toronto Transportation Authority) </li></ul><ul><ul><li>Formed in 2007 </li></ul></...
Background <ul><li>Two-tiered program delivery </li></ul><ul><ul><li>Metrolinx </li></ul></ul><ul><ul><ul><li>Central coor...
Background <ul><li>2001: BCRTMA / NTV </li></ul><ul><li>2004: 404-7 </li></ul><ul><li>2005: </li></ul><ul><ul><li>Mississa...
Background <ul><li>Employer program </li></ul><ul><ul><li>Improve and encourage commuting options </li></ul></ul><ul><ul><...
Approach - Ideal Goals Objectives Regional Transportation Plan Municipal Plans Implementation Strategic Plan Monitoring Ev...
Approach - Actual Goals Objectives RTP Municipal Plans Implementation Strategic Plan Monitoring Evaluation Federal Funding...
Approach <ul><li>2004-2007 - Contribution Agreement </li></ul><ul><ul><li>Negotiated between funder and municipalities </l...
Approach <ul><li>Inputs / Activities </li></ul><ul><ul><li>Number of businesses </li></ul></ul><ul><ul><ul><li>Active </li...
Approach <ul><li>Outputs / Benefits / Results </li></ul><ul><ul><li>Brand awareness </li></ul></ul><ul><ul><ul><li>% of em...
Monitoring <ul><li>Levels of monitoring </li></ul><ul><ul><li>Regional </li></ul></ul><ul><ul><li>TMA </li></ul></ul><ul><...
Monitoring <ul><li>Regional </li></ul><ul><ul><li>Existing sources </li></ul></ul><ul><ul><ul><li>Statistics Canada </li><...
Monitoring <ul><li>Regional </li></ul><ul><ul><li>New sources </li></ul></ul><ul><ul><ul><li>Commuter Attitudes Survey </l...
Monitoring <ul><li>TMA </li></ul><ul><ul><li>Quarterly reports – tied to Metrolinx funding </li></ul></ul><ul><ul><li>Summ...
Monitoring <ul><li>Employer – Baseline Module </li></ul><ul><ul><li>Baseline employee survey </li></ul></ul><ul><ul><ul><l...
Monitoring <ul><li>Customer – Commuter </li></ul><ul><ul><li>Service-related </li></ul></ul><ul><ul><ul><li>Monthly Carpoo...
Monitoring 10% 28% 19.7% 19.93% Transit 75.91% 48% 64.7% 65.90% Drive Alone 0.7% TTS (2006) 0.89% 1% 0.97% Cycling Surveys...
Evaluation <ul><li>Benchmarks and Trends </li></ul><ul><ul><li>75 employee baseline surveys </li></ul></ul><ul><ul><li>Sta...
Evaluation <ul><li>External </li></ul><ul><ul><li>Employer surveys </li></ul></ul><ul><ul><ul><li>Biannual survey – progra...
Evaluation <ul><li>External Review </li></ul><ul><ul><li>Consultant evaluation of operational models </li></ul></ul><ul><l...
Results <ul><li>Almost 100 members and partners </li></ul><ul><ul><li>March 2007 – 50 employers </li></ul></ul><ul><ul><li...
Results <ul><li>Transportation indicators </li></ul><ul><ul><li>76.4 million vkt </li></ul></ul><ul><ul><li>1.3 million tr...
Results <ul><li>Primary Mode Shift </li></ul>2006 2005 2008 Bike Walk/Jog Car/vanpool Public Transit Drive alone
Results <ul><li>Environmental indicators </li></ul><ul><ul><li>17,500 tonnes of greenhouse gas emissions </li></ul></ul><u...
Results <ul><li>Communications and Outreach </li></ul><ul><ul><li>400+ stories </li></ul></ul><ul><ul><li>75+ million medi...
Challenges <ul><li>Pressures to implement right away </li></ul><ul><ul><li>Show action </li></ul></ul><ul><ul><li>Pent-up ...
Challenges <ul><li>Measurement and Evaluation </li></ul><ul><ul><li>Employer buy-in is difficult </li></ul></ul><ul><ul><u...
Challenges <ul><li>Defining success </li></ul><ul><ul><li>Setting targets without enough information </li></ul></ul><ul><u...
Lessons <ul><li>Measurement and Evaluation </li></ul><ul><ul><li>Some items simple, but time-consuming </li></ul></ul><ul>...
Lessons <ul><li>Measure and Evaluation </li></ul><ul><li>Access resources at your disposal for help </li></ul><ul><ul><li>...
Lessons <ul><li>Provide incentives for monitoring </li></ul><ul><ul><li>Tie completion to access </li></ul></ul><ul><ul><l...
Next Steps <ul><li>Continue and improve data collection </li></ul><ul><ul><li>Address data management and manipulation </l...
Questions? Ryan Lanyon Team Lead, Smart Commute Metrolinx [email_address] 416-874-5933
Upcoming SlideShare
Loading in...5
×

Smart Commute Evaluation: Tools, Techniques and Lessons Learned in Monitoring and Evaluating Workplace-based TDM Programs

885

Published on

Presented by: Ryan Lanyon, BA
Presented at: ACT Canada 2008 TDM Summit, Halifax, October 2008

Published in: Automotive
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
885
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Smart Commute Evaluation: Tools, Techniques and Lessons Learned in Monitoring and Evaluating Workplace-based TDM Programs

  1. 1. Smart Commute Evaluation Tools, Techniques and Lessons Learned in Monitoring and Evaluating Workplace-based TDM Programs
  2. 2. Outline <ul><li>Background </li></ul><ul><li>Approach </li></ul><ul><li>Monitoring Tools </li></ul><ul><li>Evaluation </li></ul><ul><li>Results </li></ul><ul><li>Challenges </li></ul><ul><li>Lessons </li></ul><ul><li>Next Steps </li></ul>Ryan Lanyon Team Lead, Smart Commute Metrolinx (GTTA) October 22, 2008
  3. 3. At-a-Glance <ul><li>Build evaluation into your initial planning </li></ul><ul><li>Monitor activities, benefits and customer service </li></ul><ul><li>Scrutinize and understand indicators </li></ul><ul><li>Learn from results and mistakes </li></ul><ul><li>Resources available at www.smartcommute.ca </li></ul><ul><li>Just do it! </li></ul>Smart Commute works with various stakeholders to relieve traffic congestion, improve air quality and health, and reduce emissions that cause climate change.
  4. 4. Background <ul><li>Smart Commute </li></ul><ul><ul><li>Focus on commuters and workplaces </li></ul></ul><ul><ul><li>Partnership with municipalities </li></ul></ul><ul><ul><li>Staged development </li></ul></ul><ul><ul><ul><li>2001: Pilot TMA - BCRTMA </li></ul></ul></ul><ul><ul><ul><li>2004: Pilot regional implementation - Municipalities </li></ul></ul></ul><ul><ul><ul><li>2008: On-going implementation – Metrolinx + Municipalities </li></ul></ul></ul>
  5. 5. Background <ul><li>Metrolinx (Greater Toronto Transportation Authority) </li></ul><ul><ul><li>Formed in 2007 </li></ul></ul><ul><ul><li>Coordinates transportation across GTHA </li></ul></ul><ul><ul><li>Regional Transportation Plan </li></ul></ul><ul><ul><li>6.13 million residents, 3 million commuters </li></ul></ul>
  6. 6. Background <ul><li>Two-tiered program delivery </li></ul><ul><ul><li>Metrolinx </li></ul></ul><ul><ul><ul><li>Central coordinating body </li></ul></ul></ul><ul><ul><ul><li>Centralized service operation e.g. Carpool Zone </li></ul></ul></ul><ul><ul><ul><li>TMA Toolkit </li></ul></ul></ul><ul><ul><ul><li>Funding agency – 50% </li></ul></ul></ul><ul><ul><li>Local Smart Commute / TMA </li></ul></ul><ul><ul><ul><li>Delivery agent </li></ul></ul></ul><ul><ul><ul><li>Municipality, NGO, Chamber of Commerce/Board of Trade </li></ul></ul></ul><ul><ul><ul><li>Find other funding – municipalities, grants, service fees </li></ul></ul></ul>
  7. 7. Background <ul><li>2001: BCRTMA / NTV </li></ul><ul><li>2004: 404-7 </li></ul><ul><li>2005: </li></ul><ul><ul><li>Mississauga </li></ul></ul><ul><ul><li>NE Toronto </li></ul></ul><ul><li>2006: </li></ul><ul><ul><li>Brampton-Caledon </li></ul></ul><ul><ul><li>Halton </li></ul></ul><ul><li>2007: </li></ul><ul><ul><li>Central York </li></ul></ul><ul><ul><li>Durham </li></ul></ul><ul><li>2008: Toronto-Central </li></ul><ul><li>2009: Airport? </li></ul>
  8. 8. Background <ul><li>Employer program </li></ul><ul><ul><li>Improve and encourage commuting options </li></ul></ul><ul><ul><li>Assessment of current conditions - Baseline </li></ul></ul><ul><ul><li>Development of a plan </li></ul></ul><ul><ul><li>Implementation – ‘Commuter benefits program’ </li></ul></ul><ul><ul><ul><li>Carpool Zone and ERH </li></ul></ul></ul><ul><ul><ul><li>Walking, cycling, transit, carpool, telework programs </li></ul></ul></ul><ul><ul><ul><li>Campaigns, contests and special events </li></ul></ul></ul><ul><ul><li>Follow-up assessment – Measured against baseline </li></ul></ul>
  9. 9. Approach - Ideal Goals Objectives Regional Transportation Plan Municipal Plans Implementation Strategic Plan Monitoring Evaluation
  10. 10. Approach - Actual Goals Objectives RTP Municipal Plans Implementation Strategic Plan Monitoring Evaluation Federal Funding Requirements Implementation Implementation
  11. 11. Approach <ul><li>2004-2007 - Contribution Agreement </li></ul><ul><ul><li>Negotiated between funder and municipalities </li></ul></ul><ul><li>2008 - Monitoring Framework </li></ul><ul><ul><li>Established by Smart Commute stakeholders </li></ul></ul><ul><ul><ul><li>TMAs and Smart Commute Association (implementers) </li></ul></ul></ul><ul><ul><ul><li>Municipalities and Metrolinx (funders) </li></ul></ul></ul><ul><ul><li>Multipurpose </li></ul></ul><ul><ul><ul><li>Measure activity levels for funders </li></ul></ul></ul><ul><ul><ul><li>Track impacts to justify funding </li></ul></ul></ul><ul><ul><ul><li>Collect statistics for marketing purposes </li></ul></ul></ul>
  12. 12. Approach <ul><li>Inputs / Activities </li></ul><ul><ul><li>Number of businesses </li></ul></ul><ul><ul><ul><li>Active </li></ul></ul></ul><ul><ul><ul><li>Engaged </li></ul></ul></ul><ul><ul><ul><li>Supporting </li></ul></ul></ul><ul><ul><li>Number of pamphlets distributed </li></ul></ul><ul><ul><li>Number of cycling programs </li></ul></ul><ul><ul><li>Number of media releases </li></ul></ul><ul><ul><li>Number of commuters reached </li></ul></ul>
  13. 13. Approach <ul><li>Outputs / Benefits / Results </li></ul><ul><ul><li>Brand awareness </li></ul></ul><ul><ul><ul><li>% of employees aware; change from baseline </li></ul></ul></ul><ul><ul><li>Commuter satisfaction </li></ul></ul><ul><ul><ul><li>Ratings of Smart Commute program </li></ul></ul></ul><ul><ul><li>Reductions </li></ul></ul><ul><ul><ul><li>GHG emissions </li></ul></ul></ul><ul><ul><ul><li>CAC emissions </li></ul></ul></ul><ul><ul><ul><li>Vkt and trips </li></ul></ul></ul><ul><ul><li>Commuter cost savings </li></ul></ul>
  14. 14. Monitoring <ul><li>Levels of monitoring </li></ul><ul><ul><li>Regional </li></ul></ul><ul><ul><li>TMA </li></ul></ul><ul><ul><li>Employer </li></ul></ul><ul><ul><li>Commuter / customer </li></ul></ul><ul><li>Types of monitoring </li></ul><ul><ul><li>Behaviour change </li></ul></ul><ul><ul><li>Attitudinal change </li></ul></ul><ul><ul><li>Customer service </li></ul></ul>
  15. 15. Monitoring <ul><li>Regional </li></ul><ul><ul><li>Existing sources </li></ul></ul><ul><ul><ul><li>Statistics Canada </li></ul></ul></ul><ul><ul><ul><li>Transportation Tomorrow Survey (TTS) </li></ul></ul></ul><ul><ul><ul><ul><li>Conducted by University of Toronto </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Detailed behaviour survey </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Conducted every five years </li></ul></ul></ul></ul><ul><ul><ul><li>Cordon Count Program </li></ul></ul></ul><ul><ul><ul><ul><li>Conducted by municipalities, compiled by University of Toronto </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Observed data </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Conducted every two to three years </li></ul></ul></ul></ul>
  16. 16. Monitoring <ul><li>Regional </li></ul><ul><ul><li>New sources </li></ul></ul><ul><ul><ul><li>Commuter Attitudes Survey </li></ul></ul></ul><ul><ul><ul><ul><li>Conducted by Metrolinx </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Detailed attitudinal questions </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Conducted every two years </li></ul></ul></ul></ul><ul><ul><ul><li>Carpool Zone </li></ul></ul></ul><ul><ul><ul><ul><li>Conducted by Metrolinx </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Monthly activity survey </li></ul></ul></ul></ul><ul><ul><ul><li>TMAs and Employers </li></ul></ul></ul><ul><ul><ul><ul><li>Compiled data from localized sources </li></ul></ul></ul></ul>
  17. 17. Monitoring <ul><li>TMA </li></ul><ul><ul><li>Quarterly reports – tied to Metrolinx funding </li></ul></ul><ul><ul><li>Summary of activities per employers </li></ul></ul><ul><ul><li>Communications and outreach activity </li></ul></ul><ul><ul><li>Qualitative results </li></ul></ul><ul><ul><li>Comparison of activity levels between TMAs </li></ul></ul>
  18. 18. Monitoring <ul><li>Employer – Baseline Module </li></ul><ul><ul><li>Baseline employee survey </li></ul></ul><ul><ul><ul><li>Standardized across all TMAs and employers </li></ul></ul></ul><ul><ul><ul><li>Census survey </li></ul></ul></ul><ul><ul><li>Site assessment </li></ul></ul><ul><ul><ul><li>Current conditions </li></ul></ul></ul><ul><ul><ul><li>Inventory of advantages and deficiencies </li></ul></ul></ul><ul><ul><li>Vehicle and occupancy count </li></ul></ul><ul><ul><ul><li>Discreet sites </li></ul></ul></ul><ul><ul><ul><li>1,000+ employees </li></ul></ul></ul>
  19. 19. Monitoring <ul><li>Customer – Commuter </li></ul><ul><ul><li>Service-related </li></ul></ul><ul><ul><ul><li>Monthly Carpool Zone microsurvey </li></ul></ul></ul><ul><ul><ul><li>Emergency Ride Home post-ride survey </li></ul></ul></ul><ul><ul><li>Employee surveys </li></ul></ul><ul><ul><li>Campaign-related data collection e.g. Car Free Day </li></ul></ul>
  20. 20. Monitoring 10% 28% 19.7% 19.93% Transit 75.91% 48% 64.7% 65.90% Drive Alone 0.7% TTS (2006) 0.89% 1% 0.97% Cycling Surveys (2001-08) CAS (2008) Statistics Canada (2006) Mode
  21. 21. Evaluation <ul><li>Benchmarks and Trends </li></ul><ul><ul><li>75 employee baseline surveys </li></ul></ul><ul><ul><li>Statistics Canada </li></ul></ul><ul><ul><li>Transportation Tomorrow Survey </li></ul></ul><ul><ul><li>Cordon Counts </li></ul></ul><ul><ul><li>Commuter Attitudes Survey </li></ul></ul>
  22. 22. Evaluation <ul><li>External </li></ul><ul><ul><li>Employer surveys </li></ul></ul><ul><ul><ul><li>Biannual survey – program satisfaction rating </li></ul></ul></ul><ul><ul><ul><li>Satisfaction with Smart Commute services </li></ul></ul></ul><ul><ul><ul><li>Conducted by third party through Metrolinx </li></ul></ul></ul><ul><ul><li>Commuter surveys </li></ul></ul><ul><ul><ul><li>Employee follow-up surveys – program satisfaction rating </li></ul></ul></ul><ul><ul><ul><li>Annual survey – Carpool Zone satisfaction </li></ul></ul></ul><ul><li>Internal </li></ul><ul><ul><li>Smart Commute Technical Committee </li></ul></ul>
  23. 23. Evaluation <ul><li>External Review </li></ul><ul><ul><li>Consultant evaluation of operational models </li></ul></ul><ul><li>Research Partnerships </li></ul><ul><ul><li>University of Toronto (Mississauga) </li></ul></ul><ul><ul><ul><li>Factors for successful carpool formation </li></ul></ul></ul><ul><ul><ul><li>Further research </li></ul></ul></ul><ul><ul><li>Ryerson University </li></ul></ul><ul><ul><ul><li>TMA Toolkit assessment </li></ul></ul></ul>
  24. 24. Results <ul><li>Almost 100 members and partners </li></ul><ul><ul><li>March 2007 – 50 employers </li></ul></ul><ul><ul><li>March 2008 – 75 employers </li></ul></ul><ul><ul><li>June 2008 – 89 employers </li></ul></ul>
  25. 25. Results <ul><li>Transportation indicators </li></ul><ul><ul><li>76.4 million vkt </li></ul></ul><ul><ul><li>1.3 million trips </li></ul></ul><ul><ul><li>Equivalent of 10,000 cars </li></ul></ul><ul><ul><li>Increase in carpooling </li></ul></ul><ul><ul><ul><li>7% to 12% to 13% </li></ul></ul></ul>Cars Off the Road
  26. 26. Results <ul><li>Primary Mode Shift </li></ul>2006 2005 2008 Bike Walk/Jog Car/vanpool Public Transit Drive alone
  27. 27. Results <ul><li>Environmental indicators </li></ul><ul><ul><li>17,500 tonnes of greenhouse gas emissions </li></ul></ul><ul><ul><ul><li>Enough to fill Rogers Centre almost six times </li></ul></ul></ul><ul><ul><li>100,000 kg of criteria air contaminants </li></ul></ul><ul><ul><ul><li>Equivalent amount of NOx to smoking 3 billion cigarettes </li></ul></ul></ul>
  28. 28. Results <ul><li>Communications and Outreach </li></ul><ul><ul><li>400+ stories </li></ul></ul><ul><ul><li>75+ million media impressions </li></ul></ul><ul><ul><li>Brand recognition </li></ul></ul><ul><ul><ul><li>Smart Commute ↑ </li></ul></ul></ul><ul><ul><ul><li>Carpool Zone ↓ </li></ul></ul></ul>Awareness (2008) Clean Air Commute Commuter Challenge Carpool Zone Website Carpool Week Smart Commute
  29. 29. Challenges <ul><li>Pressures to implement right away </li></ul><ul><ul><li>Show action </li></ul></ul><ul><ul><li>Pent-up demand </li></ul></ul><ul><ul><li>More interesting than planning or evaluating </li></ul></ul><ul><li>All the pieces not always in place </li></ul><ul><li>Monitoring not appreciated until results urgently needed </li></ul><ul><li>Objectivity </li></ul><ul><ul><li>Of course my ‘baby’ was successful! </li></ul></ul>
  30. 30. Challenges <ul><li>Measurement and Evaluation </li></ul><ul><ul><li>Employer buy-in is difficult </li></ul></ul><ul><ul><ul><li>Some see value in surveys, monitoring </li></ul></ul></ul><ul><ul><ul><li>Others feel employees are oversurveyed </li></ul></ul></ul><ul><ul><ul><li>Often considered an expensive waste of employee time </li></ul></ul></ul><ul><ul><li>Standardization required, but unenforceable </li></ul></ul><ul><ul><ul><li>Moving to centralized service provision </li></ul></ul></ul><ul><ul><ul><li>Standardization vs. continuous improvement </li></ul></ul></ul><ul><ul><li>Data comparison can be unequal </li></ul></ul><ul><ul><ul><li>Apples to apples and apples to oranges </li></ul></ul></ul>
  31. 31. Challenges <ul><li>Defining success </li></ul><ul><ul><li>Setting targets without enough information </li></ul></ul><ul><ul><ul><li>Are examples applicable? </li></ul></ul></ul><ul><ul><li>Are targets achievable with the resources alloted? </li></ul></ul><ul><ul><li>Can these targets be effectively monitored? </li></ul></ul><ul><ul><li>What happens if we don’t reach these targets? </li></ul></ul>
  32. 32. Lessons <ul><li>Measurement and Evaluation </li></ul><ul><ul><li>Some items simple, but time-consuming </li></ul></ul><ul><ul><ul><li>Do you really need the data? </li></ul></ul></ul><ul><ul><li>Project benefits less easy to isolate </li></ul></ul><ul><ul><ul><li>Double-counting also possible through employee surveys and Carpool Zone surveys </li></ul></ul></ul><ul><ul><ul><li>External factors – environmental concerns, gas prices </li></ul></ul></ul><ul><ul><li>One clear conclusion: an impact was made. </li></ul></ul><ul><ul><li>Time and consistency open up opportunities </li></ul></ul><ul><ul><ul><li>Benchmarking </li></ul></ul></ul>
  33. 33. Lessons <ul><li>Measure and Evaluation </li></ul><ul><li>Access resources at your disposal for help </li></ul><ul><ul><li>Universities </li></ul></ul><ul><ul><li>Municipalities </li></ul></ul><ul><ul><li>Provincial and Federal Governments </li></ul></ul><ul><li>Evaluate what you do and how you do it </li></ul><ul><li>Give an adjustment period to monitoring </li></ul><ul><ul><li>Test! Test! Test! </li></ul></ul><ul><ul><li>Allow partners time to understand and see value </li></ul></ul>
  34. 34. Lessons <ul><li>Provide incentives for monitoring </li></ul><ul><ul><li>Tie completion to access </li></ul></ul><ul><ul><li>Award and reward success </li></ul></ul><ul><li>Measurement and Evaluation </li></ul><ul><ul><li>Learn from mistakes – you will make them! </li></ul></ul><ul><ul><li>Learn from poor results </li></ul></ul><ul><ul><ul><li>Was the program effective? </li></ul></ul></ul><ul><ul><ul><li>Was the monitoring effective? </li></ul></ul></ul>
  35. 35. Next Steps <ul><li>Continue and improve data collection </li></ul><ul><ul><li>Address data management and manipulation </li></ul></ul><ul><li>Benchmarks and comparisons </li></ul><ul><li>Strategic planning </li></ul><ul><ul><li>Service evaluation </li></ul></ul><ul><ul><li>Targets </li></ul></ul><ul><li>Baseline module refinement </li></ul>
  36. 36. Questions? Ryan Lanyon Team Lead, Smart Commute Metrolinx [email_address] 416-874-5933

×