This document summarizes discussions with research units and panel members about how a new evaluation system called Metodika was understood. Research units found the questions in Metodika too numerous and complex. Panel members noted some reluctance from research units to fully answer questions. The document then briefly compares Metodika to the evaluation system used in France called HCERES, which involves a detailed application process and site visits over a five year period and focuses on specific evaluation criteria and future strategy.

1. Engineering Sciences
F. Lapicque (for J. Seville)
IPN METODIKA
www.metodika.reformy-msmt.cz

2. Outline of the presentation
• 1- Metodika: how was it understood?
• 1.1. By surveyed Research units
• 1.2. By the panel members
Survey: 10 main questions
NB. When agreed
• 2.- A brief comparison with the system used in France for
evaluation…
1

3. 1- Metodika- How was it understood?
1.1 by the surveyed research units
Question 1- Overall point of view
• Too many questions and too little space for commenting on
strategy and future
• Perfectible formulation of questions
• Overlapping in questions/items
Point of view for the panel members
• Certain reluctancy from the RU’s in answering the items
• Combined questions on qualitative and quantitative aspects in the main part
of the application
2

4. 1- Metodika- How was it understood?
1.1 by the surveyed research units
3
Q2- Formulation of the five criteria
• Research excellence and research performance are not clear
o One is for tiptop
o the rest for a number of data (money, papers, citations etc…).
• Things are not clearly defined and formulated on the form.
• Do prefer the CAS evaluation
Q3- Evaluation of excellence: 1 or 2% output ?…
• Discussion on the figure : is it 2%? Or 3%?
• First define excellence. Often used but with other meanings and purposes
Q4- Bibliometric report
• A number of answers and comments but hardly understandable
• Personal comment : this strongly depends on areas and topics !

5. 1- Metodika- How was it understood?
1.1 by the surveyed research units
4
Q5- Compare Metodika to the former Coffee grinder
• Find Metodika interesting, also because of the presence of external experts.
However more complex.
• Question on their impartiality and experience in the job. How were they recruited?
• For others, Metodika is better, more complete than others, with various weights,
the values of them being sometimes not agreed.
Q6-Comments, suggestions to Metodika?
Does this question just not emphasise Q5!?
• Most answers: please make it simpler
• Please provide experts of sufficient quality in evaluation
• Would love that things better prepared and organised

6. 5
1- Metodika- How was it understood?
1.1 by the surveyed research units
5
Q7- Has the panel reached a correct view on ..?
• Well, it is often expressed that RU have the feeling that they have been under-
evaluated.
My point of view, yes, but 30% of the items were not really, correctly
filled!!).(For this reason, we mentioned “with potential to..”
• It is sometimes reproached that the report be too neutral, too formal, with just
a number of data, but without mention about evaluation of outstanding
research outputs..
Q8- Are Panel’s recommendations useful?
• To some extent but a number of things were already known
(OK, but they were poorly presented actually!).
• Often regret that means for improvements are not given by the experts
probably right but sometimes difficult for us, e.g. useless research ….

7. 1- Metodika- How was it understood?
1.1 by the surveyed research units
Q9- Estimation of time
• Very diverse answers, actually..:
• from altogether more than 40 man-days to … approx.. 40 hours.
Nevertheless, regardless of the selection of papers, data etc..
sometimes have been poorly documented
6
Personal comment
• Both interest and reluctancy of evaluated RU’s in considering the test
• Too bad that no visits could be organised in some areas

8. 1- Metodika- How was it understood?
1.2 by panel members
7
Q4- Do you consider visits important..?
Yes, for sure for both the visited units and the experts!
Q5- Are grading of outputs by independent reviewers useful?
• Yes absolutely. 70% of the reviews made seem really good
• However the selection of the output made by the RU may be questionable
Q6- Assessment of the RU: done on the output or on the lot?
« output » does not suffice. Would prefer to have both
Judgements made on bibliometrics
• Engineering sciences differ from other other disciplines for this point
• Within engineering sciences, paper production rate may differ quite a lot, so…

9. 1- Metodika- How was it understood?
1.1 by the surveyed research units
Q7- Do you consider bibliometric data important in your area
• Yes with limits and clever evaluation of things
• Avoid overpublication and citation hunters!
8
Q8- Do you consider a calibration exercise necessary and useful?
• Certainly
• Presumably too short, i.e. would have preferred to contribute in its design
Q9- Comments on the pilot testing
• Great commitment and enthusiasm of the organising team
• Unfortunately sometimes insufficient quality of the reports transmitted
but this might understood …
• Presumably lack of time for the organisers

10. 2- Comparison with the current
system in France. “HCERES”
9
Haut Commissariat à l’Evaluation de la Recherche et de l’Enseignement
Superieur
• formerly AERES… evaluating system in France for more than 10 years
• Evaluation made every five years (« plan quinquennal »)
• For Etablishments (Universities)
Research units or
Formation/diploma
Focus on evaluation of Research units (CNRS* and University)
UPR (CNRS Research Laboratory)
UMR (Mixed CNRS-Univ. Research units
EA (Research teams)
(*) or other public
research centers

11. 2- Comparison with the current
system in France. “HCERES”
10
The application. Three files
1- Evaluation form.
Short and sharp with general presentation of the lab, summary and balance per
groups/teams for the 5 years, Involvement in Education through research, Strategy.
2- Appendices
More detailed description with all data and quantitative information, outcomes,
contracts, organisational aspects (safety, staff, formation for staff members
3- Data on the present period and on the future
Topics, human resources, budget

12. 2- Comparison with the current
system in France. “HCERES”
11
Criteria for evaluation
• Production of knowledge and scientific level
• Attractivity and reputation in the community
• Interaction with societal, economic and cultural environment
• Organisation in the RU
• Commitment in formation by research
• Strategy and scientific prospects within the next 5 years
NB: Criteria do not correspond to parts in the application files

13. 2- Comparison with the current
system in France. “HCERES”
12
Time schedule
Example of LRGP (Chemical engineering in Nancy, Fr.)
Jul. 2015. Start in collecting references, contracts, PhD, and actualise
Jan. 2016. Balance and summary written by per groups.
General presentation of the lab
Mar-May 2016: Preparation of the appendices
June-Aug 2016: Exchanges with the University
Sept. 2016: Final version of the application
Oct 2016: Validation of the various documents. Transfer to reviewers
Dec. 2016: Two-day intensive visit, with large emphasis on lab organisation
and life, in addition to strategy and prospects

14. 2- Comparison with the current
system in France. “HCERES”
13
Evaluation, marks and levels
• A detailed report is prepared by the evaluation committee (6-10 persons)
with point per point assessment
• Evaluation made both for the whole lab (330 p.) or per groups (av. 65p.)
• Comments and suggestions for the next period
Formerly: A, B and C, with later A+, A, A- etc..
No more marks but significant impact on funding, reputation, success in
applications, recruitment, PhD grants etc.

15. Thank you for your attention!
www.metodika.reformy-msmt.cz