Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A Modeller’s Dilemma:
Overfitting or Underperforming?
Tracey Pershouse & Yun Bu (AECOM)
About this presentation
 Requirements for transport modelling
 Three case studies
o Trip generation
o Trip distribution
...
Requirements for Transport
Modelling
Requirements for Transport Modelling
 Replicating base year situations: In practice, it is used as
key measurement of mod...
Case Study – Trip Generation
Case Study – Trip Generation
 Alternative B: Introduce retail job density as an additional
explanatory factor for the num...
Validation Performance
Presentation Title Page 7August 17, 2016
The Alternative B produces considerably worse validation r...
Forecasting Performance
Page 8
Alternative A: growth in jobs and growth in trip attractions are similar at
all the shoppin...
Case Study – Trip Distribution
Case Study – Trip Distribution
August 17, 2016
Calibrate gravity model for HBW trip distribution:
 Alternative A:
- Use o...
Case Study – Trip Distribution
Presentation Title Page 11August 17, 2016
Calibrating gravity model for HBW trip distributi...
Base Year Performance – for model calibration
Page 12August 17, 2016
Observed number of trip productions and attractions a...
Base Year Performance – for model validation
Presentation Title Page 13August 17, 2016
Modelled number of trip productions...
Forecasting Performance
Presentation Title Page 14August 17, 2016
Modelled elasticity of changes in trip length in respons...
Case Study – Trip Distribution and
Mode Choice
Case Study – Trip Distribution and Mode Choice
Presentation Title Page 16August 17, 2016
 Alternative A: mode choice occu...
Case Study – Trip Distribution and Mode Choice
Presentation Title Page 17August 17, 2016
Develop destination and mode choi...
Validation Performance - Overall mode shares
Presentation Title Page 18August 17, 2016
Similar performance between the two...
Validation Performance – Mode share for SA2 to SA2
movements
Presentation Title Page 19August 17, 2016
Alternative A:
Validation Performance – Mode share for SA2 to SA2
movements
Presentation Title Page 20August 17, 2016
Alternative B – sim...
Forecasting performance
August 17, 2016 Page 21
Car GC = 30
mins
PT GC = 70
min
Car GC = 30
mins
PT GC = 70
min
D1
D2
Base...
Forecasting performance
August 17, 2016 Page 22
Car GC = 40
mins
PT GC = 70
min
Car GC = 30
mins
PT GC = 70
min
D1
D2
Scen...
Forecasting performance
August 17, 2016 Page 23
Car GC = 40
mins
PT GC = 70
min
Car GC = 30
mins
PT GC = 70
min
D1
D2
Scen...
Summary
Implications
 Overfitting: Use synthetic measures for fitting base year
outcomes to observed data, which may affect forec...
Thank You
Upcoming SlideShare
Loading in …5
×

A modeller’s dilemma: overfitting or underperforming

196 views

Published on

Tracey Pershouse & Yun Bu

Published in: Education
  • Be the first to comment

  • Be the first to like this

A modeller’s dilemma: overfitting or underperforming

  1. 1. A Modeller’s Dilemma: Overfitting or Underperforming? Tracey Pershouse & Yun Bu (AECOM)
  2. 2. About this presentation  Requirements for transport modelling  Three case studies o Trip generation o Trip distribution o Trip distribution and mode choice  Summary Page 2
  3. 3. Requirements for Transport Modelling
  4. 4. Requirements for Transport Modelling  Replicating base year situations: In practice, it is used as key measurement of model quality and is often the focus of model development  Forecasting future year implications: In practice, it receives less attention although forecasting is the ultimate use of models
  5. 5. Case Study – Trip Generation
  6. 6. Case Study – Trip Generation  Alternative B: Introduce retail job density as an additional explanatory factor for the number of attractions of shopping trips Presentation Title Page 6August 17, 2016 Modelling trip attraction to major shopping centres: HBS trip attraction = a * number of retail jobs + b * Ln(retail job density, in jobs / hectare)  Alternative A: Flags a number of shopping centers as ‘special generators’ and apply adjustment factors derived from observed data HBS trip attraction = [a * number of retail jobs] * adjustment factors
  7. 7. Validation Performance Presentation Title Page 7August 17, 2016 The Alternative B produces considerably worse validation results for the largest centres (A and B) but better outcomes for relatively smaller shopping centres (C to H).
  8. 8. Forecasting Performance Page 8 Alternative A: growth in jobs and growth in trip attractions are similar at all the shopping centres Alternative B, faster growth rate for trip attractions at the shopping centres with relatively lower density, and slower growth rate at D where the density reaches a very high level
  9. 9. Case Study – Trip Distribution
  10. 10. Case Study – Trip Distribution August 17, 2016 Calibrate gravity model for HBW trip distribution:  Alternative A: - Use observed trip productions and attractions as inputs for model calibration, even though it is modelled trip ends are that input to the gravity model for model application - Only include the OD movements with observed trips > 0, excluding the ODs with zero trips, even though some of the zeros may reflect travellers’ choice of not travelling
  11. 11. Case Study – Trip Distribution Presentation Title Page 11August 17, 2016 Calibrating gravity model for HBW trip distribution:  Alternative B: - Use modelled trip productions and attractions as inputs for calibrating the gravity model parameters - Include all the o-d pairs including those with zero trips. This requires aggregating the movements to sector (SA2) level to remove the zeros that are due to the small sample size in HTS
  12. 12. Base Year Performance – for model calibration Page 12August 17, 2016 Observed number of trip productions and attractions are used to produce the trip distribution curves
  13. 13. Base Year Performance – for model validation Presentation Title Page 13August 17, 2016 Modelled number of trip productions and attractions are used to produce the trip distribution curves
  14. 14. Forecasting Performance Presentation Title Page 14August 17, 2016 Modelled elasticity of changes in trip length in response to cost increase
  15. 15. Case Study – Trip Distribution and Mode Choice
  16. 16. Case Study – Trip Distribution and Mode Choice Presentation Title Page 16August 17, 2016  Alternative A: mode choice occurs after the trip distribution (or destination choice)  This ordering is computationally convenient: OD trip tables from distribution provide inputs to the mode choice  Arguably, this order also fits in a modelers’ experience of real life in so far as on many occasions travelers’ decide on their destinations first and then choose between transport modes Develop destination and mode choice model for HBW trips:
  17. 17. Case Study – Trip Distribution and Mode Choice Presentation Title Page 17August 17, 2016 Develop destination and mode choice model for HBW trips:  Alternative B: mode choice occurs before the trip distribution (or destination choice).  Less computationally straightforward: the generalized cost input for mode choice must be aggregated at trip production level, instead of at trip (origin-destination) level.  Due to the aggregation of cost inputs, the resultant base year validation outcomes tend to be less desirable than the destination choice first approach.
  18. 18. Validation Performance - Overall mode shares Presentation Title Page 18August 17, 2016 Similar performance between the two alternatives
  19. 19. Validation Performance – Mode share for SA2 to SA2 movements Presentation Title Page 19August 17, 2016 Alternative A:
  20. 20. Validation Performance – Mode share for SA2 to SA2 movements Presentation Title Page 20August 17, 2016 Alternative B – similar performance to Alternative A
  21. 21. Forecasting performance August 17, 2016 Page 21 Car GC = 30 mins PT GC = 70 min Car GC = 30 mins PT GC = 70 min D1 D2 Base case assumptions: 1000 trips from origin to destination D1, D2, via Car or PT Destination and Mode choice results in base case (for both alternative): D1, Car = 478 trips D1, PT = 22 trips D2, Car = 478 trips D2, PT = 22 trips
  22. 22. Forecasting performance August 17, 2016 Page 22 Car GC = 40 mins PT GC = 70 min Car GC = 30 mins PT GC = 70 min D1 D2 Scenario testing: Car travel cost to D1 increases from 30 to 40 minutes Destination and Mode choice results , Alternative A (double constrained) D1, Car = 432 trips (decrease from 478 in Base) D1, PT = 68 trips (increase from 22 trips in Base) D2, Car = 478 trips (unchanged from Base) D2, PT = 22 trips (unchanged from Base)
  23. 23. Forecasting performance August 17, 2016 Page 23 Car GC = 40 mins PT GC = 70 min Car GC = 30 mins PT GC = 70 min D1 D2 Scenario testing: Car travel cost to D1 increases from 30 to 40 minutes Destination and Mode choice results , Alternative B D1, Car = 394 trips (more significant decrease) D1, PT = 106 trips (more significant increase) D2, Car = 480 trips (nearly unchanged) D2, PT = 20 trips (nearly unchanged)
  24. 24. Summary
  25. 25. Implications  Overfitting: Use synthetic measures for fitting base year outcomes to observed data, which may affect forecasting performance adversely.  Underperforming: Accommodate underperformance (compared with overfitting option) in base year validation, in exchange of improved forecasting performance Another option: Common practice?
  26. 26. Thank You

×