Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How good are our models?

174 views

Published on

Grant Smith & Dave Hunter

Published in: Education
  • Be the first to comment

  • Be the first to like this

How good are our models?

  1. 1. 17 August 2016 AITPM Conference Sydney 2016 How good are our models? Grant Smith and David Hunter
  2. 2. 17 August 2016 AITPM Conference Sydney 2016 Wollongong/Shellharbour
  3. 3. 17 August 2016 AITPM Conference Sydney 2016 Wollongong/Shellharbour 253,000 people 96,300 households 85,300 jobs
  4. 4. 17 August 2016 AITPM Conference Sydney 2016 Model History 1993- First model built using 1991 data No future years
  5. 5. 17 August 2016 AITPM Conference Sydney 2016 Model History 1998 – two models built using 1996 data One with 376 zones Future years of 2006 and 2016
  6. 6. 17 August 2016 AITPM Conference Sydney 2016 Model History And one at 1996 with 470 zones Future year of 2026
  7. 7. 17 August 2016 AITPM Conference Sydney 2016 Model History 2004 – model updated to 2001 Experiment with dynamic and four step models No future years
  8. 8. 17 August 2016 AITPM Conference Sydney 2016 Model History In 2010 the model updated to 2006 census Future years of 2011, 2021, 2026 and 2036
  9. 9. 17 August 2016 AITPM Conference Sydney 2016 Summary Five models  1991  1996 (376 zones)  1996 (470 zones)  2001  2006
  10. 10. 17 August 2016 AITPM Conference Sydney 2016 Differences Model Detail  Number of zones
  11. 11. 17 August 2016 AITPM Conference Sydney 2016 Differences Number of zones 0 100 200 300 400 500 600 1991 1996 1996 2001 2006 Zones
  12. 12. 17 August 2016 AITPM Conference Sydney 2016 Differences Model Detail  Number of zones  Number of links
  13. 13. 17 August 2016 AITPM Conference Sydney 2016 0 5000 10000 15000 20000 25000 1991 1996 1996 2001 2006 Links Differences Number of links
  14. 14. 17 August 2016 AITPM Conference Sydney 2016 10.0km Network 1991 7,839 links
  15. 15. 17 August 2016 AITPM Conference Sydney 2016 10.0km Network 1996 376 zone system 6,009 links
  16. 16. 17 August 2016 AITPM Conference Sydney 2016 Network 1996 470 zone system 12,569 links
  17. 17. 17 August 2016 AITPM Conference Sydney 2016 Network 2001 12,508 links
  18. 18. 17 August 2016 AITPM Conference Sydney 2016 Network 2006 19,506 links
  19. 19. 17 August 2016 AITPM Conference Sydney 2016 Differences Model technology 1991  Generation – 3x3 category model (employees and cars per HH)  Distribution – negative exponential of cost  Central area Parking model included  Assignment – intersection delays at the approach
  20. 20. 17 August 2016 AITPM Conference Sydney 2016 Differences Model technology 1996 (376 zone)  Generation – 3x3 category model (employees and cars per HH)  Distribution – negative exponential of cost  Central area Parking model included  Assignment – intersection delays at the approach
  21. 21. 17 August 2016 AITPM Conference Sydney 2016 Differences Model technology 1996 (470 zone)  Generation – 3x3 category model (employees and cars per HH)  Distribution – negative exponential of cost  Central area Parking model included  Assignment – intersection delays at turn movement level
  22. 22. 17 August 2016 AITPM Conference Sydney 2016 Differences Model technology 2001  Generation – 3x3 category model (employees and cars per HH)  Distribution – negative exponential of cost  Central area Parking model included  Assignment – intersection delays at turn movement level
  23. 23. 17 August 2016 AITPM Conference Sydney 2016 Differences Model technology 2006  Generation – 5x3 category model (persons and cars per HH)  Distribution – negative exponential of time  Central area Parking model included  Assignment – intersection delays at turn movement level - Multi class assignment
  24. 24. 17 August 2016 AITPM Conference Sydney 2016 Model Validation 0 5 10 15 20 25 30 35 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 Number of screenline comparisons 1996 (470)1 2006 GEH 1996 (376) 1991 2001
  25. 25. 17 August 2016 AITPM Conference Sydney 2016 South of Bulli Pass South of Mount Ousley South of Masters Road South of Northcliffe Road South of Dapto Escarpment Comparison screenlines
  26. 26. 17 August 2016 AITPM Conference Sydney 2016 2006 Forecast from the 1996 model  Linear interpolation between 1996 and 2016  470 zones in the 2006 model compared with 376  More detailed network (twice as many links)  Different generation and distribution models  Intersections modelled at movement rather than approach level in the 2006 model
  27. 27. 17 August 2016 AITPM Conference Sydney 2016 Comparison at 2006 2006 Comparisons 2006 Model 1996 Model Difference Households 92,609 101,875 +10% Jobs 77,696 82,092 +6% Trips 57,842 70,092 +21% Screenlines South Bulli Pass 4,926 5,678 +15.3% Escarpment 4,970 5,636 +13.4% South of Mt Ousley 11,059 10,081 +9.7% South of Masters 12,862 14,634 +13.8% South of Northcliffe 9,981 12,987 +30.1% South of Dapto 7,423 8,645 +16.4% Railway 35,143 40,736 +15.9%
  28. 28. 17 August 2016 AITPM Conference Sydney 2016 Two Models validated in 1996  Same land use data  470 zones compared with 376  More detailed network (twice as many links in the 470 zone model)  Intersections modelled at movement rather than approach level
  29. 29. 17 August 2016 AITPM Conference Sydney 2016 Difference between two models Both using 1996 land use 1996 Comparisons 470 zone model 376 zone model Difference Households 85,038 83,092 +2.3% Jobs 68,040 72,684 -11.2% Trips 55,784 56,889 -2.0% Trip length (minutes) 12.6mins 11.2mins +12.5% Screenlines South Bulli Pass 4,086 4,430 -8.4% Escarpment 4,037 4,175 -3.4% South of Mt Ousley 8,194 9,130 -11.4% Figtree South of Masters 11,374 11,439 -0.6% South of Northcliffe 9,789 9,332 +4.7% South of Dapto 7,159 6,761 +5.6% Railway 30,276 30,380 -0.3%
  30. 30. 17 August 2016 AITPM Conference Sydney 2016 Model Validation 0 5 10 15 20 25 30 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 Number of points 1996 (470)1 GEH 1996 (376)
  31. 31. 17 August 2016 AITPM Conference Sydney 2016 Common land use over the five models  2011 census data converted into the zone system used by each model
  32. 32. 17 August 2016 AITPM Conference Sydney 2016 Common land use over the five models  2011 census data converted into the zone system used by each model  Models run and converged using the software versions used for the original validation
  33. 33. 17 August 2016 AITPM Conference Sydney 2016 Five models using 2011 data 2011 land use applied to the various models 2006 model 2001 model 1996 470 zone model 1996 376 zone model 1991 model Households 96,354 96,354 96,354 96,144 94,953 Jobs 85,307 85,311 85,311 85,050 87,792 Cars 158,524 158,523 158,523 158,090 155,785 Trips 65,501 67,518 67,368 74,104 77,073 Trip length (minutes) 11.40 mins 13.95mins 13.81mins 11.64mins 11.66mins Screenlines South Bulli Pass 5,609 5,616 5,468 5,844 4,242 Escarpment 5,735 5,655 5,675 5,649 3,210 South of Mt Ousley 11,069 10,696 10,722 11,493 11,048 South of Masters 13,219 12,507 12,167 13,009 12,443 South of Northcliffe 10,358 11,006 10,985 11,454 10,746 South of Dapto 8,077 8,540 8,523 8,911 8,108 Railway 37,145 37,744 37,303 38,416 40,607
  34. 34. 17 August 2016 AITPM Conference Sydney 2016 Five models using 2011 data 2011 land use applied to the various models 2006 model 2001 model 1996 470 zone model 1996 376 zone model 1991 model Households 96,354 96,354 96,354 96,144 94,953 Jobs 85,307 85,311 85,311 85,050 87,792 Cars 158,524 158,523 158,523 158,090 155,785 Trips 65,501 67,518 67,368 74,104 77,073 Trip length (minutes) 11.40 mins 13.95mins 13.81mins 11.64mins 11.66mins Screenlines South Bulli Pass 5,609 5,616 5,468 5,844 4,242 Escarpment 5,735 5,655 5,675 5,649 3,210 South of Mt Ousley 11,069 10,696 10,722 11,493 11,048 South of Masters 13,219 12,507 12,167 13,009 12,443 South of Northcliffe 10,358 11,006 10,985 11,454 10,746 South of Dapto 8,077 8,540 8,523 8,911 8,108 Railway 37,145 37,744 37,303 38,416 40,607
  35. 35. 17 August 2016 AITPM Conference Sydney 2016 Comparison of screenlines South Bulli Pass 2011 forecast flows for each model relative to the 2006 assignment 2001 model 1996 470 zone model 1996 376 zone model 1991 model South Bulli Pass Assigned flow 5,616 5,468 5,844 4,242 Percent difference 0% -3% 4% -24% Correlation coefficient 1.00 1.00 1.00 1.00 % RMS 8.25 14.63 5.94 38.92 R2 1.00 1.00 1.00 1.00 GEH overall 0.1 1.9 3.1 19.5 % GEH <7.0 100% 75% 100% 25%
  36. 36. 17 August 2016 AITPM Conference Sydney 2016 Comparison of screenlines Escarpment 2011 forecast flows for each model relative to the 2006 assignment 2001 model 1996 470 zone model 1996 376 zone model 1991 model Escarpment Assigned flow 5,655 5,675 5,649 3,210 Percent difference -1% -1% -1% -44% Correlation coefficient 0.990 0.989 0.992 0.970 % RMS 20.84 22.35 18.0 87.96 R2 0.980 0.978 0.984 0.940 GEH overall 1.1 0.8 1.1 37.8 % GEH <7.0 66.7% 83% 75% 67%
  37. 37. 17 August 2016 AITPM Conference Sydney 2016 Comparison of screenlines South of Mt Ousley 2011 forecast flows for each model relative to the 2006 assignment 2001 model 1996 470 zone model 1996 376 zone model 1991 model South of Mt Ousley Assigned flow 10,696 10,722 11,493 11,048 Percent difference -3% -3% 4% 0% Correlation coefficient 0.974 0.995 0.939 0.838 % RMS 13.99 9.15 20.92 33.28 R2 0.950 0.991 0.882 0.703 GEH overall 3.1 3.3 4.0 0.2 % GEH <7.0 80% 50% 60% 30%
  38. 38. 17 August 2016 AITPM Conference Sydney 2016 Comparison of screenlines Figtree South of Masters 2011 forecast flows for each model relative to the 2006 assignment 2001 model 1996 470 zone model 1996 376 zone model 1991 model Figtree South of Masters Assigned flow 12,507 12,167 13,009 12,443 Percent difference 6% 3% 10% 6% Correlation coefficient 0.983 0.982 0.996 0.965 % RMS 23.09 20.46 26.08 30.63 R2 0.966 0.965 0.992 0.913 GEH overall 6.6 3.6 11.1 6.1 % GEH <7.0 40% 56% 44% 11%
  39. 39. 17 August 2016 AITPM Conference Sydney 2016 Comparison of screenlines South of Northcliffe 2011 forecast flows for each model relative to the 2006 assignment 2001 model 1996 470 zone model 1996 376 zone model 1991 model South of Northcliffe Assigned flow 11,006 10,985 11,454 10,746 Percent difference 6% 6% 11% 4% Correlation coefficient 1.00 1.00 0.989 0.988 % RMS 16.63 18.23 31.02 36.56 R2 0.999 1.00 0.979 0.975 GEH overall 6.3 6.1 10.5 3.8 % GEH <7.0 50% 50% 38% 63%
  40. 40. 17 August 2016 AITPM Conference Sydney 2016 Comparison of screenlines South of Dapto 2011 forecast flows for each model relative to the 2006 assignment 2001 model 1996 470 zone model 1996 376 zone model 1991 model South of Dapto Assigned flow 8,540 8,523 8,911 8,108 Percent difference 6% 6% 10% 0% Correlation coefficient 0.997 0.992 0.998 0.849 % RMS 14.67 17.59 19.38 43.6 R2 0.994 0.983 0.996 0.720 GEH overall 5.1 4.9 9.0 .3 % GEH <7.0 50% 88% 88% 50%
  41. 41. 17 August 2016 AITPM Conference Sydney 2016 Comparison of screenlines Railway 2011 forecast flows for each model relative to the 2006 assignment 2001 model 1996 470 zone model 1996 376 zone model 1991 model Railway Assigned flow 37,744 37,303 38,416 40,607 Percent difference 2% -1% 3% 9% Correlation coefficient 0.935 0.938 0.959 0.908 % RMS 43.97 42.22 34.98 51.61 R2 0.874 0.879 0.919 0.825 GEH overall 3.1 1.9 6.5 17.6 % GEH <7.0 54% 57% 53% 49%
  42. 42. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  Most models produced similar trip generation rates.
  43. 43. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  Most models produced similar trip generation rates.  As detail increased, validation statistics improved. The 2006 validation was extremely good
  44. 44. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  All household forecasts were high
  45. 45. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  All household forecasts were high  When the same land use is used, all the models are consistent at the overall screenline level
  46. 46. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  All household forecasts were high  When the same land use is used, all the models are consistent at the overall screenline level  The coarser models are not good at getting the distribution on roads within screenlines correct
  47. 47. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  The finer the zone system and the more detail that can be included in the network, the better the validation
  48. 48. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  The finer the zone system and the more detail that can be included in the network, the better the validation  Forecasts of travel are only as good as the land use data that is used as input.
  49. 49. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  The finer the zone system and the more detail that can be included in the network, the better the validation  Forecasts of travel are only as good as the land use data that is used as input.  If the land use data is accurate, then the forecasts of traffic flow will also be accurate
  50. 50. 17 August 2016 AITPM Conference Sydney 2016 Conclusions  The finer the zone system and the more detail that can be included in the network, the better the validation  Forecasts of travel are only as good as the land use data that is used as input.  If the land use data is accurate, then the forecasts of traffic flow will also be accurate  a high degree of confidence can be placed in the models as forecasting tools
  51. 51. 17 August 2016 AITPM Conference Sydney 2016 ….and finally  He who lives by a crystal ball rapidly learns to eat ground glass
  52. 52. 17 August 2016 AITPM Conference Sydney 2016 ….and finally  He who lives by a crystal ball rapidly learns to eat ground glass  When you make a prediction it will always be wrong – the only questions are when and by how much
  53. 53. 17 August 2016 AITPM Conference Sydney 2016 ….and finally  He who lives by a crystal ball rapidly learns to eat ground glass  When you make a prediction it will always be wrong – the only questions are when and by how much  When you are right with a projection – don’t let anyone forget it
  54. 54. 17 August 2016 AITPM Conference Sydney 2016 Acknowledgments  Wollongong City Council  Shellharbour City Council  Transport for New South Wales (Roads and Marine Services)
  55. 55. 17 August 2016 AITPM Conference Sydney 2016 Thank you

×