The document summarizes the results of an evaluation of SciELO Brazil journals conducted by FAPESP, the São Paulo Research Foundation. FAPESP funded and evaluated over 200 SciELO journals across various subject areas. The evaluation involved journal editors submitting editorial development plans responding to criteria on editorial boards, reputation, visibility, and sustainability. Review panels then evaluated the journals, finding most to be of good or very good quality. Some preliminary conclusions found the evaluation challenging due to reviewer biases, and suggestions were made to improve certain journals. Overall the evaluation provided useful feedback to help strengthen the quality of Brazilian open access scholarly publishing.

1. Open Science:
quality control, transparency & ethic
Marie-Anne Van Sluys,
Full Professor – USP
Life Sciences Advisor Panel Member- FAPESP

2. Evaluation of SciELO-Brazil journals by
FAPESP: preliminary results
J. Roberto F. Arruda
1
Luiz Nunes Oliveira
2
Marie-Anne Van Sluys
3
1 Professor, FEM-Unicamp 2 Professor, IFSC-USP 3 Professor, IB-USP
Special Advisors of the Scientific Director
The São Paulo Research Foundation - FAPESP
209/10/2018 fapesp-short-pres-20150916.pptx; © C.H. Brito Cruz e Fapesp

3. • Created in 1962 by the State of São Paulo
• 1% of all state revenues
• Mission: “Support research in all fields of knowledge”
09/10/2018 3© C.H. Brito Cruz e Fapesp
*PPP: Purchasing Power Parity
Fapesp: São Paulo Research Foundation

4. 09/10/2018 4© C.H. Brito Cruz e Fapesp
Fapesp: São Paulo Research Foundation

5. Web of Science Documents
SciELO Brazi
l
China
USA
InCites: Open Access

6. Normalized Citation Impact
Brazil
China
USA
InCites: Open Access
SciELO

7. Evaluation: Life Sciences
09/10/2018 7© C.H. Brito Cruz e Fapesp

8. Evaluation: Engineering, Physics &
Earth Sciences
09/10/2018 8© C.H. Brito Cruz e Fapesp

9. Evaluation: Humanities & Social Sciences
09/10/2018 9© C.H. Brito Cruz e Fapesp

10. SciELO funding by Fapesp
09/10/2018 10© C.H. Brito Cruz e Fapesp
0.00
5,000,000.00
10,000,000.00
15,000,000.00
20,000,000.00
25,000,000.00
30,000,000.00
2003-2005 2006-2008 2008-2011 2011-2013 2013-2016 2016-2019
R$
Abel Laerte PackerRogerio Meneghini
Need to evaluate outcome

11. Fapesp evaluation of SciELO journals
09/10/2018 11© C.H. Brito Cruz e Fapesp
FAPESP & SciELO joint discussion on EDP
• Defining key questions to the Editor
• Significance & Sustainability
• Best practices
• Journal metrics evolution
• Future Goals up to 2021
• Instructions for Editorial Development Plan (EDP)

12. FAPESP & SciELO: joint discussion
09/10/2018 12© C.H. Brito Cruz e Fapesp
FAPESP & SciELO joint discussion on evaluation form
• A lot – Sufficiently – Reasonably – Little - Nothing
• Criterion 1. Editorial Board
• Criterion 2. Area/Field Significance
• Criterion 3. International Visibility
• Criterion 4. Institutional & Sustainability
• General Recommendation

13. Fapesp evaluation of SciELO journals
09/10/2018 13© C.H. Brito Cruz e Fapesp
• Message sent to the Editors (11 June 2018)
• EDP sent by July 10 postponed to July 15
• July-August evaluation by the area panels
?

14. Fapesp evaluation of SciELO journals
09/10/2018 14© C.H. Brito Cruz e Fapesp
Panel PDEs
Evaluated
Journals
EVALUATION Total
VG G R L N N/A
Agri/Vet 28 16 3 6 6 1 0 0 16
Bio 15 10 4 4 2 0 0 0 10
Health 71 18 7 6 7 3 1 0 24
Natural Sci 8 8 4 2 1 1 0 0 8
Eng 10 10 5 3 2 0 0 0 10
Human Sci 61 28 19 7 0 0 0 2 28
Social 30 7 3 2 1 0 0 1 7
Lang/Arts 10 0 0 0 0 0 0 0 0
Multi 1 0 0 0 0 0 0 0 0
TOTAL 234 103 45 30 19 5 1 3 103
0
5
10
15
20
25
30
35
40
45
50
VG G R L N N/A
General figures
44% 77% 6%

15. Fapesp: Evaluation of SciELO journals
09/10/2018 15© C.H. Brito Cruz e Fapesp
• Q1a: Editor(s) are internationally recognized in the field of
the journal?
• Q2a: Journal reputation (quality of the papers and review)
• Q2b: Would you recommend publishing in this journal?
• Q3c: Impact indicators are good compared to other
journals in this field?
• Q4h: Final recommendation about keeping the journal in
SciELO

16. Evaluation: Agriculture and veterinary
09/10/2018 16© C.H. Brito Cruz e Fapesp
Q4h SJR Q
Q4h Final recommendation

17. Evaluation: Life Sciences
09/10/2018 17© C.H. Brito Cruz e Fapesp
Q4h SJRQ
Q4h Final recommendation

18. Evaluation: Natural Sciences and
Engineering
09/10/2018 18© C.H. Brito Cruz e Fapesp
Q4h SJR Q
Q4h Final recommendation

19. Evaluation: Social and Human Sciences
09/10/2018 19© C.H. Brito Cruz e Fapesp
Q4h SJR Q
Q4h Final recommendation

20. Fapesp evaluation of SciELO journals
09/10/2018 20© C.H. Brito Cruz e Fapesp
• Very preliminary analysis and conclusions
• Difficult to obtain unbiased evaluation, as experts
are almost always involved with the journals (past
or present)
• Uneven quality of evaluations
• Interesting suggestions by many reviewers will be
helpful to journals
• What to do with journals that have not send PDE?

21. Fapesp evaluation of SciELO journals
09/10/2018 21© C.H. Brito Cruz e Fapesp
Contact information:
José Roberto F. Arruda <JRArruda@fapesp.br>
Luiz Nunes de Oliveira <LNunes@fapesp.br>
Marie-Anne Van Sluys <MSluys@fapesp.br>

22. 09/10/2018 22© C.H. Brito Cruz e Fapesp

23. Fapesp evaluation of SciELO journals
09/10/2018 23© C.H. Brito Cruz e Fapesp
• Instructions for EDP preparation (up to 5 pages)
– Editorial practices
– Institutionality
– Sustainability
– Singularity (relevance)
– Visibility and impact
– Technology
– Editorial goals
• Evaluation form (4 criteria)
– Criterion 1: Quality of the Editorial Board and reviewers
– Criterion 2: Reputation of the journal
– Criterion 3: International visibility and impact factors
– Criterion 4: Institutionality and sustainability

24. 09/10/2018 24© C.H. Brito Cruz e Fapesp
Fapesp: São Paulo Research Foundation

25. 200
2005 2010 2015
Examples: Revista Bras. Ciências Sociais

26. 500
2005 2010 2015
Examples: Rodriguésia

27. 500
2005 2010 2015
Examples: Scientia Agricola

28. 120
2011 2015
Examples: LA J. of Solids and Structures

29. © C.H. Brito Cruz e Fapesp 2909/10/2018
Fapesp: São Paulo Research Foundation