Recommended
Recommended
More Related Content
Similar to AITPM Risk and Governance
Similar to AITPM Risk and Governance (20)
More from JumpingJaq
More from JumpingJaq (20)
Recently uploaded
Recently uploaded (20)
AITPM Risk and Governance
- 1. AITPM Technical Forum – Risk and Governance – August 2015 Stephen Luke
- 2. What am I covering? • Internal processes for risk and governance • Case Study : west coast intercity rail franchise • Emerging UK considerations : Aqua (Quality analysis for government)
- 3. Context
- 6. Mott MacDonald Credentials • We are at the forefront of developing guidance and methods. • We have developed industry standard software, for example for demand modelling (DIADEM) and for business case appraisal (TUBA, INCA, WITA and PAR). • We contribute to industry guidance, for example we have written the latest Highways England guidance on Appraisal of Technology Schemes.
- 7. Mott MacDonald Credentials • We have a wide range of application experience, from large scale strategic and multi-modal applications to detailed microsimulation models of traffic management schemes or pedestrian movements • Highway modelling - Norwich Ring Road, UK • PT modelling – Abu Dhabi Surface Transport masterplan, UAE • Demand modelling • Microsimulation – Calgary LRT, Adelaide • Pedestrian modelling – North West Rail link, Melbourne Metro • Junction modelling – Sydney LRT • Economic Appraisal – Transparent Economic Assessment • Freight modelling – Transnet, South Africa • Land use modelling • Aviation modelling – London airports • Rail Modelling – High Speed 2 • Research • Transport modelling for private financed projects
- 8. Wider Thoughts • It’s not about software! • Decision making without traditional transport models – other ways of quantifying policy or project impacts
- 9. Who is making the decisions? Do they really know what they want? When do they need an answer? How robust does the answer need to be? Do they understand the data available? Have they any confidence in model outputs? Have they any technical knowledge? Are they politically motivated? Have they pre-conceived answers?
- 10. Application of Transport Modelling • As a front-end service to help determine the policy or infrastructure solutions • Provide an evidence base to support the preparation of a scheme business case • Provide an evidence base for the scheme environmental appraisal • To support transport infrastructure management business, for example for Managing Agent Contractor contracts
- 12. Transport Modelling Practice • Delivery – Peer Assist throughout project delivery – Checking plans – Share lessons • Staff Development – Encourage writing papers, attending conferences, technical committees – Helping to meet career aspirations
- 13. Transport Modelling Community site • Contacts and expertise – Skills matrix • Best Practice • Links and resources
- 14. Best practice papers – Why do we need them? • Simple errors in models have stopped schemes progressing • Clients have lost confidence and the industry has recognised change is needed – HA best practice networks – TRAMPNET best practice network set up by PTRC • There are gaps in published guidance • Modelling is a risky business!
- 15. Best Practice Papers 1. Peer assist and checking requirements 2. Addressing risks and liabilities in receiving and delivering transport models and data 3. Model design and data requirements 4. Record keeping and version control 5. Spreadsheet management 6. Network building 7. Checking transport models 8. Matrix building 9. Matrix estimation 10. Calibration and validation 11. Forecasting 12. Microsimulation
- 16. Best Practice – Key general messages • We need to identify and mitigate risks - our processes are designed to help us do this • ALL transport modelling projects must be subject to peer assist • Apply standard QES procedures • Models should be checked, and the checks recorded so this can be demonstrated • We must all use best practice procedures (or have a good reason not to)
- 17. Best Practice – Peer Assist • The rationale behind the application of Peer Assist in transport modelling is to identify problems early, to have an explicit approach to the identification and mitigation of risk, and to make better use of the expertise available • PA at scoping stage to identify and cost a checking plan • Peer Reviews at key stages, at least at Tender and 1/3 and 2/3 through delivery
- 18. Best Practice – Spreadsheet management • Name to describe calculation • Keep old versions in superseded directory • Use Spreadsheet with QA front sheet for all calculations • Describe the calculations: the description should be of sufficient detail such that any other technical colleague can open up the spreadsheet and be able to understand how the calculation works, and how they might update it if any of the key inputs have changed.
- 19. Best Practice – Checking networks • Input data checks • • Source material • • New coding against scheme drawings or other data • Model run checks • Version of software • File names • Assignment warning messages • Assignment convergence • Demand model warning messages • Demand model convergence
- 20. Best Practice – Checking networks 2 • Output results checks – Network statistics – Changes in trip matrices – Journey times – Simulation junction performance – Routings – Assignments – Fixed matrix – TUBA
- 21. Best Practice – Matrix Building • Lots of existing data sources • Cleaning and verifying survey data • Building station matrices • Combining station matrices to get complete observed matrices • Synthetic matrix methods and infilling • Validation of matrices • Sense checking
- 22. Best Practice - Calibration and Validation • Check guidelines at beginning – including requirements for reporting • Lots of published guidance in DMRB and WebTAG and from TfL • Calibration – adjustments to model, but a large part is checking • Validation – comparison with independent data
- 23. Summary • Transport modelling is a risky business! • Protection by using best practice procedures • Keep up to date – Best practice papers – Transport modelling community • Plan checking as part of Peer Assist and scoping • Record checks
- 24. West Coast Intercity Rail Franchise https://www.gov.uk/government/publication s/report-of-the-laidlaw-inquiry http://www.nao.org.uk/report/lessons-from- cancelling-the-intercity-west-coast- franchise-competition/ • A very public failure • Technical errors with modelling • Risk and governance issues • Comprehensive documentation of lessons learnt in public domain Annual revenue >$1.5bn pa (2009-10)
- 25. Background • Nearly all UK intercity rail services are franchised • Initially 25 Train Operating Companies (TOC’s) • Net cost operation • TOC separate from operating Group • Initial franchises awarded solely on financial bids • GFC (and overbidding), franchises failed
- 26. Background • Macroeconomic risk? • Cap and collar – surpluses capped, losses collared • Subordinated Loan Facility (SLF) – Insurance • GDP Mechanism • The DfT Model developed for another purpose was used to calculate the SLF • Bids submitted in excel model form
- 27. Findings • DfT model did not account for inflation (assumed it did) • Technical confusion regarding elasticity factors to be used • Bids not properly quantify risk and understated SLF • DfT model not provided to bidders (lack of transparency) • Adjustments made by Govt. tender assessment team. Inconsistent, not clearly documented
- 28. In laymans terms • There was a lack of transparency. The Department did not give bidders enough information on which to base their bids. • The Department did not follow its own published guidance. • The amount of capital that the two final bidders were asked to put into their bids was understated and inconsistently determined. • The Department’s planning and preparation was inadequate. • Roles and responsibilities for the project were unclear and resources were stretched. • The Department’s governance lacked efficacy. • Quality assurance was inadequate – limited scope and late (audit afterwards!)
- 29. Lessons • Clarity of objectives helps decision makers to form appropriate judgements by being a touchstone to refer back to throughout the decision-making process. • Strong project and programme management brings together and coordinates the different streams of work, identifies interdependencies and the sequence of events – the critical path – a programme needs to follow. • Senior oversight acts as a sense check. • Effective engagement with stakeholders, such as suppliers, helps by contributing their knowledge, signalling problems and brings them into the process. • Internal and external assurance provides a sense check and can identify any areas of concern to management
- 30. Recommendations • Clarity of objectives – Apply project and programme management disciplines to forming policy. Set timetables, identify key tasks and their dependencies, identify a critical path for making policy changes and allocate clear roles and responsibilities to deliver individual elements and the policy as a whole. – Identify the technical tools and models it requires to implement policy before delivery commences. Develop, quality assure and test these processes before it moving on – Provide training to staff on any new tools or policies. Before projects enter operational stages, staff need training so that they understand objectives and how to apply processes and tools. • Project and programme management – Timetables for major projects and programmes so they are realistic. It should consider the ‘usual’ timescales for typical projects and programmes, identify novel factors that might impact on these and be cautious in shortening existing timetables. – Key decision points build in sufficient time to properly consider decisions, include contingency in case extra work is required, and consider other options if it cannot decide to proceed.
- 31. AQUA • The Aqua Book: • Guidance on producing quality analysis for government • https://www.gov.uk/government/publications/the-aqua-book- guidance-on-producing-quality-analysis-for-government
- 32. AQUA effective quality assurance is achieved by creating an environment that is conducive to quality assurance and by embedding appropriate and proportionate processes
- 33. AQUA • environment: creating the conditions in which quality assurance processes can operate effectively, facilitated by a culture that values quality assurance and welcomes effective challenge, a well understood chain of responsibility and sufficient time for quality assurance; and • process: establishing a clear process for every stage of the analytical life-cycle. This includes ensuring there is a shared understanding about the purpose and any limitations of the analysis
- 34. Model Definition
- 35. AQUA
- 36. AQUA
- 37. Summary : Analysis with RIGOUR • Repeatable • Independent • Grounded in reality • Objective • Have understood and managed uncertainty • Results should address the initial question robustly • Establish how much we can rely upon the analysis for a given problem.
- 39. … TO THIS
- 40. WHAT ABOUT THIS?
- 41. www.mottmac.com
Editor's Notes
- Understanding the future demand for road travel is essential to help shape the policies we implement and the investments we make and to ensure that the outcomes for people's lives and livelihoods are fully understood. These are issues that have important outcomes for people's lives and livelihoods and involve billions of pounds of taxpayers' mone Forecasts are not a target to be met nor do they define the level of road capacity required, but to develop the right strategy it is vital that we are able to understand how road traffic might change over time. This requires a robust forecasting approach that is based on the best available evidence of the underlying drivers of traffic demand, their relationship with changes in traffic and an approach that can model this appropriately.
- Often in our analytical work, we expect that quality will be assured by suitable processes alone, more and more arduous processes, an acknowledgement that an appropriate modelling/analytical environment, an ethos even, is also vital. The environment involves a culture where leaders value and recognise good quality assurance but also recognise the need for adequate capacity, specialist skills and sufficient time; and finally establish a clear chain of responsibility plus a route for challenge. The process side is more focused on those carrying out the analyses, and is about a systematic approach to make quality assurance logical, easy and comprehensive but also accessible, based on clear guidance and clear documentation.
- Often in our analytical work, we expect that quality will be assured by suitable processes alone, more and more arduous processes, an acknowledgement that an appropriate modelling/analytical environment, an ethos even, is also vital. The environment involves a culture where leaders value and recognise good quality assurance but also recognise the need for adequate capacity, specialist skills and sufficient time; and finally establish a clear chain of responsibility plus a route for challenge. The process side is more focused on those carrying out the analyses, and is about a systematic approach to make quality assurance logical, easy and comprehensive but also accessible, based on clear guidance and clear documentation.
- Often in our analytical work, we expect that quality will be assured by suitable processes alone, more and more arduous processes, an acknowledgement that an appropriate modelling/analytical environment, an ethos even, is also vital. The environment involves a culture where leaders value and recognise good quality assurance but also recognise the need for adequate capacity, specialist skills and sufficient time; and finally establish a clear chain of responsibility plus a route for challenge. The process side is more focused on those carrying out the analyses, and is about a systematic approach to make quality assurance logical, easy and comprehensive but also accessible, based on clear guidance and clear documentation.
- In particular, it is important to accept that uncertainty is inherent within the inputs and outputs of any piece of analysis. It is important to establish how much we can rely upon the analysis for a given problem.