PROPOSAL EVALUATION: Guidance for evaluatorsfor Calls in FP7 (Framework Programme 7) Sergey Mikhalovsky, Nazarbayev University
My recent experience with research fundingCo-ordinator, Principal Investigator or Team leader:European Union: FP6-MATISS (2006-2010)FP7-MONACO-EXTRA (2008-2012)FP7-OncoNanoBBB (2012-2015)FP7-ABREM (2010-2014), FP7-FRESP (2009-2012)FP7-Greenland (2009-2012)Interreg IIIA Stent (2005-2008)Interreg IVA Flax (2009-2012)TEMPUS III and TEMPUS IV (2005-2008, 2009-2012)UK: Technology Strategy Board - FullFlush(2010-2013), Department of Health (2008-2013),Medical Research Council (2012-2015)
My experience with proposal evaluationFP5, FP6, and FP7 in nanotechnologies,environmental sciences and Marie CurieProgrammeErasmus Mundus I and II, Lead Expert in LifeSciencesTEMPUS II and III, INTAS, EPSRC (UK),BBSRC (UK), national programmes forRussia, Austria, Cyprus, France, USA andKazakhstan
FP7 Specific ProgrammesCo-operation – Collaborative ResearchEuropean Research CouncilMarie Curie ActionsCapacities – Research Capacity JRC EURATOM
What about Kazakhstan?Kazakhstan is on the list of InternationalCooperation Partner Countries (ICPC)And eligible for most FP7 projects!
Rules on submission & evaluation Basic principles ² Excellence. Projects selected for funding must demonstrate a high quality in the context of the topics and criteria set out in the calls. ² Transparency. Funding decisions must be based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals. ² Fairness and impartiality. All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.7
Rules on submission & evaluation Basic principles ² Confidentiality. All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence. ² Efficiency and speed. Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework. ² Ethical and security considerations: Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award8
The Three Main Reference Documents 1 - Rules on submission and evaluation q This is the common and official reference for FP7 rules for submission of proposals, and the related evaluation, selection and award procedures. 2 - Guide for applicants q The Guide for Applicants contains the essential information to guide proposers through the mechanics of preparing and submitting a proposal. q All proposals shall contain a Part A (administrative forms) and a Part B (proposal description). Indications about the content and issues to be addressed are described in the Guides for Applicants q Please make sure that you read the “Guide for applicants” that corresponds to the funding scheme for the topic.9
The Three Principal Reference Documents ² 3 - The work programme: three complementary documents: q Work Programme - General Introduction q 2012 Work Programme: includes the topic’s description and criteria against which the proposals will be assessed. q Cooperation" Work Programme - General annexes Annex 1: List of International Co-operation Partner Countries (ICPC) Annex 2: Eligibility and Evaluation Criteria for Proposals Annex 3: Forms of Grant and Maximum Reimbursement Rates for Projects Funded Annex 4: General activities10
Example: Joint call ENV-NMP Work Programme – Topic description²Nanotechnologies for water treatmentNanotechnology presents many benefits for environmental technology applications, such asremediation, treatment or sensor development and monitoring purposes. In the field of water,nanotechnology has the potential to contribute to long-term water quality, availability, andviability of water resources such as through advance filtration that enables sustainable waterreuse, recycling or desalination.The aim of this action is to support research and technological development in the fieldof water treatment by applying developed or adapted nano-engineered materials topromising separation, purification and/ or detoxification technologies.Proposals should focus on process intensification aiming at improving selectivity,robustness, stability and performance while reducing energy requirements and by-product generation.Specific monitoring issues, as well as safety, environmental and health aspects, should beincluded if directly associated with the new technological solution proposed.Priority will be given to novel ideas and emerging technologies promising major advances anda large potential impact in the long-term, including cost-effectiveness11
Joint call ENV-NMP Work Programme – Expected ImpactsDevelopment and uptake of innovative and cost-efficient water treatmenttechnologies benefiting from progress and advances made in nanosciences,materials and technologies.This initiative should deliver step-change advances in water treatmenttechnologies, including validation and verification of arising prospects interms of improving treatment performance and reducing energyrequirements.By fostering the knowledge base in this area, the projects addressing this topic willcontribute to strengthening European competitiveness in the water sector and theimplementation of the Environmental Technologies Action Plan and theNanotechnology Action Plan.12
Independent experts² Expert evaluators are at the heart of the FP7 system² Provide independent, impartial and objective advice to the Commission q they represent neither their employer, nor their country!² Significant funding decisions will be made on the basis of their advice q They can also add value to projects through your comments and suggestions² The integrity of the process is crucial q They should follow the Code of Conduct annexed to the appointment letter13
Who evaluates? ² Selected from a wide pool from a database, on the basis of keywords ² Minimum 3 evaluators per project ² Selected per call ² Replace about ¼ in any given area annually ² Sign confidentiality and conflict of interest declarations ² Names published after evaluation (though not at call or proposal level)FP7 – How to apply ² Target at least 40% female ² Mix of geographical location and background ² To register: https://cordis.europa.eu/emmfp7
Overview of the Evaluation Process ” ot e be ” m ay ote “re m m “re IndividualSubmission Consensus Panel Finalisation reading Evaluators Evaluators Evaluators Final rankin Full Proposal list Proposal Rejection lis Criteria Criteria Criteria forms Proposals in Eligibility suggested priority orderCOMMISSION COMMISSION Role of experts 15
Process For each proposal: “remote” May be “remote” and / or central Proposal X IER copy 1 expert 1 Consensus meeting Proposal X IER CR copy 2 expert 2 3 experts Proposal X copy 3 IER expert 3Note: There may be more than 3 evaluators IER=Individual evaluation report CR=Consensus Report 16
Process Ethical Issues q The Consortium is asked to submit drafts of Information Sheet and Consent Form but does not need to submit copies of legislation q Proposals should comply with fundamental ethical principles relevant security procedures q … or be excluded from the process q Check if the proposal has in fact ethical issues If yes: tick ethical issues box in the CR Prepare Ethical Issues Report17
Process Evaluating a proposal² Three guiding principles:² Objectivity q each proposal is evaluated as it is written² Accuracy q The judgment is made against the official evaluation criteria, and nothing else² Consistency q The same standard of judgment applies to each proposal18
Process The evaluation criteria ² Criteria adapted to each funding scheme and each thematic area q specified in the work programme ² Three main criteria: q S&T Quality (relevant to the topic of the call) Concept, objective, work-plan q Implementation Individual participants and consortium as a whole Allocation of resources q Impact Contribution to expected impacts listed in work programme Plans for dissemination/exploitation19
Process FP7 Evaluation Criteria Applicable to ALL funding schemes 1. S/T quality 2. Implementation 3. Impact (in relation to the topics addressed by the call)² Sound concept, ² Appropriateness of ² Contribution, at theand quality of the management European and / orobjectives structure and international level, to procedures the expected impacts ² Quality and relevant listed in the work programme under experience of the the relevant topic / individual participants activity20
Process FP7 Evaluation Criteria Collaborative projects 1. S/T quality 2. Implementation 3. Impact(relevant to the topicsaddressed by the call)² Progress beyond ² Quality of the ² Appropriatenessthe state-of-the-art consortium as a whole of measures for (incl. complementarity, the dissemination² Quality and balance) and/or exploitationeffectiveness of the of projects results,S/T methodology ² Appropriate allocation and managementand associated and justification of the of intellectualwork plan resources to be property. committed (budget, staff, equipment)21
Process Funding schemes ² Collaborative projects Support to research projects carried out by consortia with participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research. The size, scope and internal organisation of projects can vary from field to field and from topic to topic. Projects can range from small or medium-scale focused research actions to large-scale integrating projects for achieving a defined objective Projects may also be targeted to special groups such as SMEs, Specific International Co-operation Actions, etc.22
International cooperation² Some topics in work programme seek participation from third countries q International Cooperation Partner Countries (ICPC) q industrialised countries² research partners from ICPC may be financed23
Process Proposals that are only partly in scope² Note: The “S/T quality” of a proposal (first criterion) is evaluated to the extent that the content is relevant to the topic(s) addressed by the call q E.g. If a proposal is only marginally relevant, or if only one work package is relevant, the evaluator must downgrade the score – no matter how excellent is the science!² Relevance to the objectives of the call is also considered under “Impact” (third criterion) q In relation to the sub-criterion “contribution to expected impacts listed in the work programme” 24
Process Proposal scoring: Interpretation of the scores² 0: The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information² 1: Very poor. The criterion is addressed in a cursory and unsatisfactory manner.² 2: Poor. There are serious inherent weaknesses in relation to the criterion in question.² 3: Fair. While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting.² 4: Good. The proposal addresses the criterion well, although certain improvements are possible.² 5: Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.25
Process Proposal scoring ² Each criterion is scored 0-5 q Marks can go from 0 – 5 in steps of 0.5, i.e half-marks are allowed q Experts are encouraged to use the whole range q Scores must pass thresholds if a proposal is to be considered for funding ² Thresholds apply to individual criteria… q Threshold is 3 ² …and to the total score q higher than the sum of the individual thresholds q Threshold is 10 ² Note that to receive a mark of 5, a proposal does not have to be perfect. An excellent proposal can have minor shortcomings. ² When writing comments in the IERs and Consensus Report, the severity of any weakness should be clearly stated, i.e. are they minor, moderate or significant26
Process Proposal scoring ² Evaluate the proposal and conclude whether the proposal is q Excellent q Good q Fair q Poor q Very poor q Failing to address the criterion ² Score the proposal accordingly27
Process Commission Follow-up² Evaluation summary reports sent to applicants q “initial information letter” New for q Redress procedure FP7² Draw up final ranking lists² Information to the Programme Committee² Contract negotiation² Formal consultation of Programme Committee (when required)² Commission decisions² Survey of evaluators² Independent Observers’ reports28
Process Redress² Proposers can complain if they believe there have been shortcomings in the handling of their proposal, and that these shortcomings have jeopardised the outcome of the evaluation process.² The quality and consistency of the evaluation reports (ESRs), derived directly from the CRs, is paramount to minimise the redress procedures29
Tips on writing a successful proposal ² An interesting and innovative idea ² Strong consortium ² Experienced co-ordinator ² Complementarity of skills and expertise ² Perfect matching between the proposal and the call ² At least 6 months to prepare a proposal ² European added value ² Sustainability after the end of the projectFP7 – Tips ² Dissemination strategy
The EC Funding Process – A Rough Guide to : Why, What and How Sergey Mikhalovsky
Why?² Good Money – typically 200,000-300,000 € per participating organisation for 2-3-4 years² More if you are the Lead Partner – all the money is distributed through you, which counts for your organisation – good for you and for your organisation – typically 1-2-3 million € for 2-3-4 years² Good Value for Money – typically 60% overheads
Why? (continued)² Fun – experience of different countries, cuisines and cultures, meeting new people² Knowledge transfer – opportunity to work in different labs, access to unique equipment and instruments, methods and techniques² Human mobility – you visit others and others visit you to exchange knowledge and experience² Fair competition and useful feedback
Why? (cont.)² Networking – you build up your own network² There may be unique expertise somewhere in Europe unavailable in this country² Career enhancement² Publicity – EC loves it and gives you lots of opportunities to do that (even if you don’t want it...)² Recruitment of high quality researchers² And you may even get a result...
What?² Incredibly diverse formats and degrees of participation:(i) individual fellowships – No age limit, very well paid!(ii) participation as a team – you can be paid or you can employ someone(iii) co-ordinator – the same as (i) and (ii) plus management costs plus permanent headache.
What? (cont.)Funding is available for salaries, overheads, travel, consumables, management, equipment and organisation of networking events.
How? Regularly monitor the funding opportunities Most obvious source of funding for regular monitoring are: (i) FP7 - become a regular visitor on this site: http://ec.europa.eu/research/participants/ portal/page/home and become familiar with the structure of this programme. It is the main but not the only source of R&D funding in Europe37
How? (cont.)(i) Be flexible with your idea of participation – sometimes there may be a Call which matches exactly what you want to do but that is unlikely.(ii) Identify potential areas of interest by monitoring previous Calls and - important! – analysing success rate in previous Calls. If the success rate is less than 10% - don’t go for it, unless you are a winner in a National Lottery or somebody else is prepared to do the main job of writing for you.
How? (cont.)² Try to get information about forthcoming Calls – it is possible to do so by attending consultations in Brussels or elsewhere, contacting your NCP (National Contact Point), getting information from an insider, etc. The earlier you know the contents of the future Call, the better your chances for success. When the Call has been announced, it is too late! You will have 2-3 months to submit a proposal from the date when the Call opens.
How ? (cont.)² There are also bottom-up Calls – these are my favourite – you can write about anything you like and the date of the new Call announcement is usually known well ahead, so you have plenty of time to think and prepare for the Call.
How ? (cont.)² There are also bottom-up Calls – these are my favourite – you can write about anything you like and the date of the new Call announcement is usually known well ahead, so you have plenty of time to think and prepare for the Call.² Erasmus Mundus and TEMPUS programmes offer in essence bottom-up Calls in Research&Education.
What to start with?² Formulate the idea, which fits a particular Call (probably several ideas to have a choice)² Identify potential partners – do a lot of networking, and do it all the time regardless of Calls.² Learn the Brussels speak – it takes time, but once you understand it – your chances for success are much higher.
What to do next?² You have to understand how to write a proposal² Become an expert evaluator for FP7 https://cordis.europa.eu/emmfp7/ ?fuseaction=wel.welcomeIt will help you to understand the evaluation process and ultimately improve your proposal writing skills
What to do next?² Write a proposal – preferably not by yourself if it is your first one.² Write a proposal yourself if you have done this before² Seek advice and assistance of people with experience: colleagues who have got such grants, Research Office² Interact with other partners² Notify your line managers and finances about your intention to submit a proposal in advance
S&T Quality of the proposal² Concept and objectives. What are the main ideas that led you to propose this work?Provide sufficient elements on the planned S&T methodology and describe in detail the S&T objectives. Show how they relate to the topics addressed by the call, which you should explicitly identify. The objectives should be those achievable within the project, not through subsequent development. They should be stated in a measurable and verifiable form, including through the milestones and deliverables.45
S&T Quality (cont.)² Progress beyond the state-of-the-art.² Describe the state-of-the-art in the area concerned, and the advance that the proposed project would bring about. If applicable, refer to the results of any patent search you might have carried out.² S/T methodology and associated work plan. A detailed work plan should be presented, which should follow the logical phases of the implementation of the project.46
S&T Quality (cont.)² Describe the overall strategy of the work plan.² Show the timing of the different WPs and their components² Deliverables and milestones² Describe any significant risks, and associated contingency plans² Milestones are control points where decisions are needed with regard to the next stage of the project. For example, a milestone may occur when a major result has been achieved. Another example would be a point when the consortium must decide which of several technologies to adopt for further development. Intangible – something which cannot be touched.² Deliverables: report, prototype, material, etc. Tangible - something which can be touched.47
Implementation² Management structure and proceduresDescribe the organisational structure and decision-making mechanisms of the project.Show how they are matched to the complexity and scale of the project.Individual participants.provide a brief description of the legal entity, the main tasks they have been attributed, and the previous experience relevant to those tasks. Provide also a short profile of the staff members who will be undertaking the work.
Implementation² Consortium as a whole.Describe how the participants collectively constitute a consortium capable of achieving the project objectives, and how they are suited and are committed to the tasks assigned to them. Show the complementarity between participants. Explain how the composition of the consortium is well-balanced in relation to the objectives of the project.Resources to be committed.Show how the resources will be integrated in a coherent way, and show how the overall financial plan for the project is adequate. Infrastructure. 49
Impact What does your project bring to: ²Individuals involved in the project; ²Teams involved in the project; ²Consortium as a whole; ²Region, country as a whole; international dimension ²New IP – profit ²Socio-economic benefits, environmental significance, health ²Development, dissemination, use of results50
Impact (cont.)Dissemination and/or exploitation of project results,and management of IP.Describe the measures you propose for thedissemination and/or exploitation of project results, andhow these will increase the impact of the project. Youshould take into account a variety of communicationmeans and target groups as appropriate (e.g. policy-makers, interest groups, media and the public at large).With regard to the innovation dimension, whereappropriate, describe the potential areas and markets ofapplication of the project results and the potentialadvantages of the resulting technologies/ solutionscompared to those that are available today.51
Impact (cont).With regard to the innovation dimension, where appropriate,describe the measures you propose to increase the likelihoodof market uptake of project results, such as: verification,testing, and prototyping; supporting the development oftechnical standards; identifying and collaborating withpotential users; identifying potential partners and sources offinance for commercialisation.Describe also your plans for the management of knowledge(intellectual property) acquired in the course of the project.
Networking organisation of scientific, training, and managerial network meetings invitation of external experts for specialist inputs (for scientific, administrative, financial review and complementary skills training) (category F) visits, secondments between contractors/partners for transfer of knowledge collaboration with other funded projects in similar or complementary fields
Training² additional skills (i.e. project management, drafting of project proposals, language courses, ethics, IPR, CV writing, job search, interview skills, etc).² consider wider disciplines and all potential research environments (academic/industrial).² “hands on” experience. → future employment prospects
Dissemination attendance at international conferences and workshops organisation of final conference with wider participation or satellite workshop at major conference dissemination of research undertaken send documentation of success stories to Commission for further dissemination
EthicsDescribe any ethics issues that may arise in the project. Inparticular, you should explain the benefit and burden of theexperiments and the effects it may have on the researchsubjects.All funded research must comply with the relevant nationaland international ethics related rules and professional codesof conduct.Where necessary, the beneficiary shall provide a writtenconfirmation that (a) favourable opinion(s) of the relevantethics committee(s) has (have) been received and, ifapplicable, the regulatory approval(s) of the competentnational or local authority(ies), before beginning any researchrequiring such opinions or approvals.56
What (not) to expect?Expect:Your first attempt will be a failureYour second attempt may or may not be successfulOne day you will succeed!Do not expect:Somebody else will do it for you!
Case Study² Individual Incoming Fellowship (from 3rd country to UK) submitted in 2008 (unsuccessful) and 2009 (successful)² Criterion 1: Scientific QualityIssues to be addressed when assigning an overall mark for this criterion: Scientific/technological quality, including any inter- disciplinary and multidisciplinary aspects of the proposal;Research methodology; Originality and innovative nature of the project, and relationship to the state of the art of research in the field; Timeliness and relevance of the project; Host scientific expertise in the field; Quality of the group/researchers in charge² 2008: Solid phase hydrosilylation conditions are not defined.It is not clearly defined how the synthesized products would be characterized. The novel materials that are claimed to be the outcome of the proposed project are too vaguely defined. 58
Case Study (cont.) ² Criterion 2: Transfer of knowledge Issues to be addressed when assigning an overall mark for this criterion: Potential of transferring knowledge to European host and/or bring knowledge to Europe; Clarity and quality of the transfer of knowledge objectives. ² 2008: The knowledge that the applicant will bring to the host institute has not been sufficiently defined. ² 2009: no weaknesses!59
Case Study (cont.)² Criterion 3: Quality of the researcher² Issues to be addressed when assigning an overall mark for this criterion: Research experience; Research results including patents, publications, teaching etc.; Independent thinking and leadership qualities, and capacity to transfer knowledge; Match between the fellows profile and project² 2008: The fellow has a relatively modest publication activity. No publications in high impact journals have been presented. It is therefore difficult to judge the quality of the applicants research from the information provided. It is not clear from the project description what characterization techniques the fellow used in her previous investigations.² 2009: no weaknesses! 60
Case Study (cont.)² Criterion 4: Implementation² Issues to be addressed when assigning an overall mark for this criterion: Quality of infrastructure / facilities and inter- national collaborations of host; Practical arrangements for the implementation and management of the scientific project; Feasibility and credibility of the project, including work plan; Practical and administrative arrangements, and support for the hosting of the fellow.² 2008: Training on new research equipment has not been sufficiently discussed.² 2009: no weaknesses!61
Case Study (cont.)² Criterion 5: Impact² Issues to be addressed when assigning an overall mark for this criterion: Potential for creating long term collaborations and mutually beneficial co-operation between Europe and the third country; Contribution to European excellence and European Competitiveness; Contribution to the socio- economic development of the Developing Countries or emerging and transition economies by transfer of knowledge and human capacity building (where relevant); Benefit of the mobility to the European research area² 2008: In the case of negative results the contribution to European excellence and European competitiveness will be low. There is no risk analysis.² 2009: no weaknesses!62
Case Study (cont.)² Overall comments:² 2008: The overall impact of this project, if successful in all aspects, would be high, with pronounced contribution to the European excellence and European Competitiveness in an important scientific field. However, the associated risks have not been properly treated in the proposal.² 2009: no comments!63
Case Study (cont.)² Score 2008 2009S&T quality (3) 4 4.8Transfer of knowledge (0) 3.8 4.8Quality of the researcher (4) 3.5 4.9Implementation (0) 4.6 4.9Impact (0) 4.2 4.6Total: 78.6 96.0(in brackets – threshold mark) (failed) (awarded)64
Some observations from 2011 evaluation² The aim of this project is a development of fundamental scientific principles of the obtaining of nanocomposite materials based on metal nanoparticles stabilized with polymeric matrices, a study of their structure, mechanisms of formation and electrocatalytic activity in the processes of hydrogenation of organic compounds, as well as nanocomposites from liquid crystal compounds and carbon nanotubes and a study of their structural features and physicochemical properties.² Your comments, please!65
Analysis of the previous slide² Too verbous;² difficult to measure,² it is a mixture of aim and objectives,² unclear what is this for?Actually, I gave it 7 out of 9 in S&T criterion!66
Expected results² Not quantifiable parameters;² ‘Designed magnetic nanotechnology will enable a new way to solve many scientific- technical and medico-biological problems’.67
Expected resultsБудет разработана технология нанесения отдельных пленок Ag,Cu, также пленок содержащих оба этих металла, что расширитдиапазон практического применения бактерицидных изделий. Поэтим технологиям будут выданы исходные данныe по проектамполучения бактерицидных изделия бытового и медицинскогоназначения, а также документации по эксплутационным характе-ристикам этих изделий. Практическая значимость заключаетсятакже в том, что в предлагаемых технологиях используется отно-сительно дешевая стандартная аппаратура. Это будет способство-вать ее применению на предприятиях малого и среднего бизнеса.Научная значимость результатов исследования заключается втом, что для получения пленок используется низкотемпературнаяреакция восстановления в газовой фазе. Причем восстанавливае-мые соединения могут находиться как в растворе, так и в твердойфазе. Благодаря этому можно добиться образования пленки и навнешней поверхности изделия и на внутренних поверхностяхотдельных пор. Это позволяет добиться хорошей адгезииполучаемых пленок. Кроме того это создает предпосылки дляприменения этого процесса в других технологиях. 68
Expected results - commentsThe expected results are described vaguely;not clear what products will be developedand why the products developed will bebetter than those existing on the market.There are no target properties given for theproducts.Not a single quantitative parameter ismentioned!69
Research facilitiesПроект будет выполнен силами сотрудников с использованиемсобственного оборудования. Нормативно-методическая обеспе-ченность будет осуществляться отделом метрологии и стандар-тизации. Патентно-лицензионная сторона проекта будут обеспе-чиваться патентным отделом, который функционирует в составенаучно-исследовательского отдела, а также образовательно-ин-формационным отделом, подключенного к Интернету. Приборы,используемые в выполнении эксперимента, прошлиметрологичеc-кую проверку, аттестацию и аккредитацию.Исследования физико-химических свойств продуктов будутпроводиться в аккредитованном испытательном центре XXX.Имеются 3 специа-лизированные лаборатории площадью более200 м2 с подведен-ными коммуникациями. Лаборатории оснащеныосновным и вспо-могательным оборудованием для проведенияисследований. Опытно-промышленная база для выпуска опытнойпартии про-дукта может быть создана в проблемной лаборатории,материаль-но-техническая оснащенность и производственныеплощади которой позволяют разместить все технологическиелинии. 70
Research facilities - comments² The description of the available equipment is inadequate; some new equipment is planned for acquisition.² there is no description of the required manpower and skills.71
References1. Обзор рынка редкоземельных элементов в СНГ. - М.:Инфомайн. 2008. - 101 с.2. Михайличенко А.И., Михлин Е.Б., Патрикеев Ю.Б.Редкоземельные металлы. М.: Металлургия. 1987. 232 с.3. Серебренников В.В. Химия редкоземельныхэлементов. - Томск.: Томский государственныйуниверситет. 1959. Т. 1. 521 с; 1961. Т.2. 802 с.4. Химия редких элементов. Соединения редкоземель-ных элементов. Под ред. И.В. Тананаева. – М.: Наука,1983. – 392 с.5. Бюллетень иностранной коммерческой информации.2007. №1206. Информационные сообщения Гиредмет поматериалам Industrial Minerals. 2002. April. С. 52-61. 72
References - Comments² Very basic, mostly in Russian and most out-of-date. Only one reference to an original publication in a peer-reviewed journal. The more recent references are to commercial reports rather than scientific or patent literature.73
S&T common errors² Lack of details in describing methodology;² Generic words instead of concrete examples;74
Conclusions² There is always somewhere funding waiting for you and your idea(s).² You can do it² If not you – who else? ² Questions? ² GOOD LUCK!