The document proposes a new methodology to assess the impact of air navigation research, development, and innovation projects funded by the European Union. It aims to evaluate these projects' impact on EU policy objectives and air navigation system performance. The methodology would qualitatively relate changes in key performance indicators to funding levels for different policy objectives over time. It would also account for external events affecting aviation. An initial validation of the methodology using data from 1995 to 2016 found that EU funding positively contributes to policy goals, though goals are seldom fully achieved. External events can significantly impact air navigation system performance. The next steps are to further validate the methodology by applying it to more projects and considering implementation impacts.

1. VI International Conference on Dynamics Games and Science
XIX Jornadas Latinoamericanas de Teoría Económica
European Air Navigation projects: A new
assessment methodology proposal for projects
supported by European Union funds
José Antonio Calvo Fresno
Jesús Morcillo Bellido
Beatriz Rodrigo Moya
1

2. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 2
OBJECTIVES OF THE STUDY
Analyze current situation of air navigation RDI projects funded by EU, and
propose a new methodology to:
• Evaluate the impact of Research, Development and Innovation (RDI) and Implementation projects:
 In the field of air navigation.
 Executed with EU funds.
• Establish qualitative relations between:
 EU policy objectives in the field of air navigation.
 Performance of the European air navigation system.
 Use of EU funds for RDI and Implementation projects in the field of air navigation.
• Provide solutions to the issues present in current impact assessment methodologies.

3. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 3
BACKGROUND (I)
Cockpit 50 years ago Cockpit today
ATC position 50 ATC position
years ago today
0 35 71
245
700
900
1,760
2,340
0
500
1,000
1,500
2,000
2,500
FP1 FP2 FP3 FP4 FP5 FP6 FP7 H2020
MEUR
CFMU
European Air Traffic
Management System
1st capacity crisis (end of 80’s)
2nd capacity crisis (end of 90’s)
Single European Sky

4. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 4
BACKGROUND (II)
Initiatives supported by EU funds are subject to evaluation:
 Ex-ante:
 Business Impact Assessment (1986) limited to macroeconomic effects of new regulations
 Regulatory Impact Assessment (2001) limited to new regulations
 Integrated Impact Assessment (2003). Any type of proposal. Economic, social, and environmental
effects. Too complex.
 Ex-post:
 Limited to financial audits; or
 Limited to technical results of an isolated project. Contribution to policy objectives difficult to assess.

5. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 5
BACKGROUND (III)
Program Impact Analysis:
Assessing the technical results of the program + assessing the contribution of the RDI activity to policy
objectives. In general more beneficial, BUT some issues found:
 Low availability of quantifiable policy objectives (MEFISTO, 2010)
 Unclear correlation between RDI objectives and policy objectives (METRONOME, 2009)
 Antagonism of objectives, in the same transport mode (AGAPE, 2010), and between different transport modes (AIMS, 2009)
 Low availability of reliable data (AIMS), (METRONOME)
 High impact of external events in the achievement of objectives, at the short term (AGAPE) and at the long term (MEFISTO)
 Low connection between RDI and real market needs (AIMS), low level of implementation of RDI results (SITPRO Plus, 2010)
 Low level of use of RDI results for rulemaking (SITPRO Plus) and standardization (METRONOME)
 Technical needs not fully addressed at the level of individual project (SITRO Plus)

6. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 6
RESEARCH METHODOLOGY
1. Qualitative research
2. Pattern matching
3. Secondary data analysis
4. Direct research
• Validation with data from all RDI and Implementation air
navigation projects funded by the EU in the period 1995 to 2016.
• Key element: reliable sources of data.

7. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 7
RELIABLE SOURCES OF DATA
• Vision 2020
• SRA 2002
• SRA 2004
• SRIA 2012
• ATM Master Plan 2008
• ATM Master Plan 2012
• Regulations (Parliament,
• Council, Commission)
• Decisions
• Communications
• Reports
• Web pages
• Annual Safety
Reviews (2005 –
2017)
• Performance Review Reports
(2000 – 2016)
• SRC Doc2
• Anual Safety Reports (2003
– 2016)
• State of Global Aviation
Safety (2011 – 2013)
• Safety Reports (2013 –
2016)
Policy aspects:
System performance aspects:
Other sources of data:

8. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 8
PROPOSED IMPACT ASSESSMENT METHODOLOGY
Policy
documents
POLICY
OBJECTIVES
INDICATORS
Expected
values of the
indicators
Real values of
the indicators
Difference betwen real
and expected values (D) -
per year
Reports
PROJECTS
FUNDS
Funds assigned per
Objective (F) - per year
Qualitative relation:
yearly evolution of (D) and
yearly evolution of (F) per
policy objective
CONCLUSIONS
Qualitative
influence of
external
events

9. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 9
EU POLICY OBJECTIVES
On environmental efficiency matters:
ENV-No: reduction of perceived noise
ENV-Em: reduction of CO2 and NOx emissions
On societal matters:
SOC-Mar: impact of European aviation industry in the global
market
SOC-Reg: impact on the elaboration of rules and technical
specifications
On safety matters:
SAF-Gen: accidents/incidents on all types of operation
SAF-Com: accidents/incidents on commercial aviation
On efficacy matters:
CAP-Tr: increase of available capacity
CAP-De: decrease of number of flights
delayed
On economic efficiency matters:
ECO-Aer: reduction of costs of aircraft operations
ECO-Nav: reduction of costs of the air navigation services
On security matters:
SEC-Aer: reduction of unlawful acts involving airports or
aircraft
SEC-Nav: reduction of unlawful acts using air navigation
systems

10. To Aviation
September 11 terrorist attacks
Severe Acute Respiratory Syndrome
global financial crisis
Eyjafjallajökull eruption
economic recovery
Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 10
EXTERNAL EVENTS
To air navigation RDI and Implementation projects
1998 ------------ capacity crisis in Europe
2001 --------------------------------------------------------
2002 --------------------------------------------------------
2003 - new rules on occurrence reporting
2008 --------------------------------------------------------
2010 -------------- labour problems in ATC1
2011 --------------------------------------------------------
2013 -------------- labour problems in ATC
2015 --------------------------------------------------------
1ATC: Air Traffic Control

11. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 11
PRELIMINARY CONCLUSIONS (I)
On the development of the new impact assessment methodology:
• A new impact assessment methodology tailored to the air navigation field can be developed building on
the results of this study.
• This new methodology can be a useful tool for planners and decision makers to optimize the use of EU
funds.
• Databases, documents and reports from European organizations and institutions can be considered as
sources of reliable data.
• Events external to RDI and Implementation air navigation projects, or even external to aviation, can
significantly affect the performance of the air navigation system. These events have to be identified and
their effect assessed as far as possible.

12. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 12
PRELIMINARY CONCLUSIONS (II)
On the validation of the proposed methodology for the 1995 – 2016 period:
• The European air navigation system shows a progressive performance improvement consistent with EU
policy.
• BUT EU policy objectives are seldom reached.
• EU funded air navigation research projects contribute positively to the achievement of EU policy
objectives.
• This potential positive effect manifests after 4 or 5 years of the maximum increase of the use of EU funds.
• The time to elaborate regulations depends significantly of the political aspects of the negotiation between
the EC and the Member States. The time to elaborate technical specifications depends more on the results
of RDI projects.

13. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 13
NEXT STEPS
• To complete the validation of the proposed methodology through its application to a large number of RDI
air navigation projects, grouped by large programmes where applicable.
• To include in the validation implementation projects to take into consideration effect of research in real
operation.
• To identify and assess as far as possible the effects of all related external events

14. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 14
Alliance of Science Organisations in Germany. (2016). On the Interim Evaluation of Horizon 2020. 13 July 2016.
Barrot, J. (2005). SESAR: un programme au service de l’avenir du secteur aérien. Bruselas, 17 de noviembre de 2005.
Baum, C. F., Lööf, H., Nabavi, P., & Stephan, A. (2017). A new approach to estimation of the R&D–innovation–productivity relationship. Economics of Innovation and New Technology, 26(1-2).
Baxter, P., & Jack, S. (2008). Qualitative Case Study Methodology: Study Design and Implementation for Novice Researchers. The Qualitative Report’, 13(4):544-559.
Benito Ruiz de Villa, A. (2009). La demanda de transporte aéreo. Jornada sobre demanda de transporte aéreo INECO. Madrid, 22 de enero de 2009.
Ciocanel, A. (2016). An Assessment Model of the Impact of Innovation on Competitiveness Growth in Europe. National Strategies Observer, 1(2).
Coghlan D & Coughlan P. (2008). Action Learning and Action Research (ALAR): A Methodological Integration in Inter-Organizational Setting, Systemic Practice & Action Research, 21(21):97-104.
Crain, A. D. (2007). Ford, Carter, and Deregulation in the 1970s. Journal on Telecommunications and High Technology Law, 5(2), pp. 416-418, 424-430, 432-434.
Cranfield University. (2005). International General Aviation and Corporate Aviation Risk Assessment (IGA-CARA). Final Report. Issue 1.1. Department of Air Transport, 17 de junio de 2005.
Doganis, R. (1973). Air Transport. A case Study in International Regulation. Journal of Transport Economics and Policy, 11(2), págs. 109-133.
Hinrichs-Krapels, S., & Grant, J. (2016). Exploring the Effectiveness, Efficiency and Equity (3e’S) of Research and Research Impact Assessment. Palgrave Communications, 2.
La Paix, L., & López-Lambas, M. (2010). A multi-criteria method for evaluating European transport research projects. 12th WCTR. Lisboa 11-15 de Julio.
Luukkonen, T. (1998). The difficulties in assessing the impact of EU framework programmes. Research Policy, 27(6), 599-610.
Luukkonen, T. (2000). Additionality of EU framework programmes. (Elsevier Science B.V., Ed.) Research Policy, 29, 711–724.
Mankins, J. C. (1995). Technology Readiness Levels. A White Paper. NASA, 22 December 2004.
McInally, J. (2010). EUROCONTROL History Book.
Muscio, A. (2006). The European Added Value of Framework Programmes: Evidence from the UK. Economia, Societa' e Istituzioni, XVIII(3).
Pape, A. M., Wiegmann, D. A., & Shappell, S. (2001). Air Traffic Control (ATC) related accidents and incidents : a human factor analysis. 11th International Symposium on Aviation Psychology. Columbus, Ohio 5-8 de Marzo.
Renda, A. (2006). Impact assessment in the EU. The state of the art and the art of the state. Bruselas: Centre for European Policy Studies.
Smismans, S. (2015). Policy Evaluation in the EU: The Challenges of Linking Ex Ante and Ex Post Appraisal. European Journal of Risk Regulation, 6(1), 6-26.
Tammemons Bakker, J., Schmidt, T., McMillan, D., Cron, R., Wachenheim, M., Aguado, V., . . . Jankovec, O. (2007). Report of the High Level Group for the future European Aviation Regulatory Framework. A framework for
driving performance improvement. Bruselas, julio 2007.
Tuominen, A., Järvi, T., Hyytinen, K., Mitsakis, E., López-Lambas, M., La Paix, L., . . . Sitov, A. (2011). Evaluating the achievements and impacts of EC framework programme transport projects. European Transport Research
Review, 3(2), 59-74Yin, R: Case study research: Design and methods. 2nd ed. Sage Publishing, Thousand Oaks, CA (1994).
Yin, R (1994). Case study research: Design and methods. 2nd ed. Sage Publishing, Thousand Oaks, CA.
REFERENCES

15. Madrid, 8-11 Mayo 2018 DGS VI, JOLATE XIX 15
Thanks for you attention