Models of evaluation systems in Greece Dimitris DENIOZOS NATO Advanced Research Workshop “ Bulgarian integration into European and NATO policies – best practices 16 – 19 May 2005

Evaluation as part of the general concept of management: Planning Organising Directing Controlling Controlling Evaluation

Evaluation as part of the general concept of management:

Planning

Organising

Directing

Controlling

EVALUATION: Process of identifying the degree of achievement of the objectives of a programme/project, the relatedness of the implemented activities compared to the planned ones, the appropriateness of the implemented strategies compared to the goals, the relation of the final products/outcome to the resources used. Assessment: Process of identifying the socio-economic, institutional, cultural etc impact from the adoption of a technical system, the socio-cultural challenges, expectations and issues that the technical change may raise as a final outcome as well as a process for solving social and cultural problems Review: Process of identifying the degree that a programme or project conforms to valid and modern approaches of science and technology Valuation: process defining the characteristics of a technical system, its potential capabilities when in use, compared to other technical systems, its contribution to the technological development of an organisation, a network, a country Appraisal Audit Benchmarking … . Terminology – concept clarification

CRITERIA, METHOD(OLOGIE)S-TECHNIQUES, INDICATORS OF RTD EVALUATION TIMING AD HOC IMPACT EX POST AD ITINERE / MONITORING EX ANTE APPLIED, INDUSTRIAL ACADEMICS INDUSTRIAL R&D, INNOVATION R&D INDUSTRIAL R&D, INNOVATION R&D TECHNICAL CENTRES RES.INSTITUT./UNIVERSITIES INDIVIDUAL RESEARCHERS PROJECTS PROGRAMMES INSTITUTIONS/ INSTIT.UNITS TARGETS

METHOD(OLOGIE)S – TECHNIQUES Peer review “ Pure” peer opinion “ Documented” peer opinion, documentation provided by supporting administration Cost / benefit analysis Variations and mixed techniques

METHOD(OLOGIE)S – TECHNIQUES

Peer review

“ Pure” peer opinion

“ Documented” peer opinion, documentation provided by supporting administration

Cost / benefit analysis

Variations and mixed techniques

CRITERIA Scientific merit / value (promised-expected / performed) Relevance to public policies industrial, environmental social etc Economic/ financial performance Managerial merit

INDICATORS Bibliometric: publications/ researcher-year, citations/ publication-researcher, patents, patent applications Networking: co-publications, co-citations, co-patenting Financial diversification: income from non public funding sources/ institutional funding

PEER REVIEW - SUPPORTING ADMINISTRATION Selection of peer evaluators – Academics Dispatching application forms, evaluations forms, organising meetings Securing access to material infrastructure: site, access to internet, libraries Explicitating criteria and indicators – to avoid implementation of implicit criteria by the evaluators Eventually reporting on strengths and weaknesses Eventually ranking Securing confidence, secrecy

FORECASTING & ASSESSMENT To provide: Visibility, clarity of the objectives Additional information as to the expected impact Insight on Social relevance Insight on Economic relevance

FORECASTING & ASSESSMENT

To provide:

Visibility, clarity of the objectives

Additional information as to the expected impact

Insight on Social relevance

Insight on Economic relevance

The Greek case: gradual introduction of evaluation AD HOC IMPACT EX POST AD ITINERE / MONITORING EX ANTE APPLIED, INDUSTRIAL ACADEMICS INDUSTRIAL R&D, INNOVATION R&D INDUSTRIAL R&D, INNOVATION R&D TECHNICAL CENTRES RES.INSTITUT./UNIVERSITIES INDIVIDUAL RESEARCHERS PROJECTS PROGRAMMES INSTITUTIONS/ INSTIT.UNITS TARGETS

The Greek case: gradual introduction of evaluation AD HOC IMPACT EX POST AD ITINERE / MONITORING EX ANTE APPLIED, INDUSTRIAL ACADEMICS INDUSTRIAL R&D, INNOVATION R&D INDUSTRIAL R&D, INNOVATION R&D TECHNICAL CENTRES RES.INSTITUT./UNIVERSITIES INDIVIDUAL RESEARCHERS PROJECTS PROGRAMMES INSTITUTIONS/ INSTIT.UNITS TARGETS

The Greek case: gradual introduction of evaluation AD HOC IMPACT EX POST AD ITINERE / MONITORING EX ANTE APPLIED, INDUSTRIAL ACADEMICS INDUSTRIAL R&D, INNOVATION R&D INDUSTRIAL R&D, INNOVATION R&D TECHNICAL CENTRES RES.INSTITUT./UNIVERSITIES INDIVIDUAL RESEARCHERS PROJECTS PROGRAMMES INSTITUTIONS/ INSTIT.UNITS TARGETS

The Greek case: gradual introduction of evaluation AD HOC IMPACT EX POST AD ITINERE / MONITORING EX ANTE APPLIED, INDUSTRIAL ACADEMICS INDUSTRIAL R&D, INNOVATION R&D INDUSTRIAL R&D, INNOVATION R&D TECHNICAL CENTRES RES.INSTITUT./UNIVERSITIES INDIVIDUAL RESEARCHERS PROJECTS PROGRAMMES INSTITUTIONS/ INSTIT.UNITS TARGETS

The Greek case: gradual introduction of evaluation AD HOC IMPACT EX POST AD ITINERE / MONITORING EX ANTE APPLIED, INDUSTRIAL ACADEMICS INDUSTRIAL R&D, INNOVATION R&D INDUSTRIAL R&D, INNOVATION R&D TECHNICAL CENTRES RES.INSTITUT./UNIVERSITIES INDIVIDUAL RESEARCHERS PROJECTS PROGRAMMES INSTITUTIONS/ INSTIT.UNITS TARGETS

PROBLEMS STILL TO BE SOLVED … Link evaluation (control) to planning, organising, directing Impact assessment/evaluation on institutions and programmes Cost / benefit techniques to be introduced Forecasting and assessment to link planning to control Training evaluators and supporting administrations Training trainers! Familiarising decision makers with the tools!

PROBLEMS STILL TO BE SOLVED …

Link evaluation (control) to planning, organising, directing

Impact assessment/evaluation on institutions and programmes

Cost / benefit techniques to be introduced

Forecasting and assessment to link planning to control

Training evaluators and supporting administrations

Training trainers!

Familiarising decision makers with the tools!