Speech by Dr. Tom Olijhoek Editor-in-Chief DOAJ

1. ASSESSING THE QUALITY OF
SCHOLARLY PUBLISHING
1 3 t h A D R I C O N F E R E N C E :
S U R A B A Y A , I N D O N E S I A
M A Y 3 - 4 , 2 0 1 7
T O M @ D O A J . O R G
E D I T O R - I N - C H I E F D O A J

2. You are free to:
Copy, share, adapt, or re-mix;
Blog, live-blog, or post video of;
This presentation. Provided that:
You attribute the work to its author and respect the rights
and licenses associated with its components.
Slide Concept by Cameron Neylon, who has waived all copyright and related or neighbouring rights. This slide only ccZero.
Social Media Icons adapted with permission from originals by Christopher Ross. Original images are available under GPL at;

3. • A d v a n t a g e s a n d o p p o r t u n i t i e s o f O p e n A c c e s s
• Q u a l i t y o f p u b l i s h i n g a n d q u a l i t y o f s c i e n c e
• W h a t i s t h e r o l e o f t h e D O A J ?
• T h e D O A J A m b a s s a d o r p r o g r a m m e
• O p e n a c c e s s p o l i c i e s
• P u b l i s h i n g c o s t s O p e n A c c e s s
• A s s e s s i n g R e s e a r c h Q u a l i t y
• P e e r e v i e w a n d c i t a t i o n a n a l y s i s r e v i e w e d
• P e e r r e v i e w f l a w s
• C i t a t i o n a n a l y s i s : d e m i s e o f t h e i m p a c t f a c t o r I F
• C i t a t i o n a n a l y s i s : o p e n c i t a t i o n s I 4 O C
• A s s e s s i n g I m p a c t / R e l e v a n c e b y a l t m e t r i c s : D O R A
• T h e ‘ ? p r o b l e m ? ’ o f q u e s t i o n a b l e p u b l i s h e r s
Presentation overview

4. O P E N A C C E S S . . . . .
• A R T I C L E S R E C E I V E M O R E C I T A T I O N S
• A R T I C L E S A R E M O R E R E A D
• R E A C H E S A B I G G E R A U D I E N C E
• I S L E S S P R O N E T O B I A S
• A V O I D S D U P L I C A T E S T U D I E S
• A R T I C L E S C A N B E M O R E E A S I L Y C H E C K E D
• D A T A L E S S P R O N E T O M A N I P U L A T I O N
• I S B E T T E R F O R A D V A N C I N G S C I E N C E
• L E A D S T O M O R E I N N O V A T I O N
• L E A D S T O B E T T E R E D U C A T I O N
Reasons for Open Access

5. • DOAJ Directory of Open Access Journals
O P E N A C C E S S J O U R N A L S O N L Y , n o r a n k i n g
• SCOPUS
O P E N A C C E S S & S U B S C R I P T I O N J O U R N A L S , r a n k i n g
• WEB OF SCIENCE
O P E N A C C E S S & S U B S C R I P T I O N J O U R N A L S , r a n k i n g
• NATIONAL LISTS
O P E N A C C E S S & S U B S C R I P T I O N J O U R N A L S
r a n k i n g g r o u p s : D a n i s h e x a m p l e
INDEXING SERVICES
for quality control

6. • DENMARK: BIBLIOMETRIC RESEARCH INDICATOR
National lists
for quality control
http://ufm.dk/en/research-and-innovation/statistics-and-
analyses/bibliometric-research-indicator/bfi-rules-and-
regulations?set_language=en&cl=en

7. • QUALITY OF PUBLISHING
• QUALITY OF THE SCIENCE
Assessing the Quality of a Journal

8. What is the DOAJ?
The only global index of
trusted, high quality, peer-
reviewed, open access
journals, from all academic
disciplines,curated by the
community
https://doaj.org/
FREE to register and FREE to use

9. • IS4OA: DOAJ; OPEN CITATION CORPUS
• SHERPA/ROMEO
• CROSSREF
• ORCID
• ARCHIVING : e.g. LOCKSS
• PUBLON (reviewer platform)
Needed Infrastructure Services

10. Recognition of the DOAJ?

11. Use of DOAJ Data
• EBSCO
• ROAD
• NATIONAL LISTS
• MANY OTHER
ORGANIZATIONS
ROAD contains DOAJ journals + hybrids +
repository material + blogs + …………

12. DOAJ STRUCTURE

13. • DOAJ CRITERIA FOR QUALITY OPEN
ACCESS PUBLISHING
•Principles of transparency and Best Practice in Scholarly Publishing
•(DOAJ, OASPA, COPE, WAME)
Assessing the Quality of Publishing

14. WHAT IS QUALITY OPEN
ACCESS?
The BOAI Definition
Open Access is:
a publishing system where all content is freely available without charge to the user
or his/her institution. Users are allowed to read, download, copy, distribute, print,
search, or link to the full texts of the articles, or use them for any other lawful
purpose, without asking prior permission from the publisher or the author.

15. Copyrights
• Clearly described on web site*
• Recommend: author retains copyright
• Recommend: no exclusive publishing rights
• Recommend: no transfer of commercial rights
https://doajournals.wordpress.com/2015/06/02/copyright-and-licensing-part-2/
Essential criteria

16. Content Licensing
• Clearly described on web site*
• Recommend:
• licensing terms on all articles,
all versions (html, pdf, xml etc.)
• embedded in article level metadata
• Creative Commons licensing
Essential criteria

17. What is a license?
• Licensing means to grant a third party (anyone else
except the right holder) the right to use a
copyright-protected work
• A license is a permission to use a work in specific
ways
• Licenses can only be granted by copyright holder
• Copyright holder can be author or publisher
COPYRIGHT AND
LICENCING

18. DOAJ IDRC Funded Ambassador
Programme 2016-2017
CHINA
RUSSIAN FED.
AFRICA
MIDDLE EAST
LATIN AMERICA
INDIA

19.  Africa SE, NW 4 ambassadors
 Russia 3 ambassadors
 China 3 ambassadors
 Middle East 1 ambassador
 Latin America 1 ambassador
 India 3 ambassadors
Planned:
 Japan
 Indonesia
Extended DOAJ Ambassador
Programme (FUNDING??)

20. Aims of the Ambassador
programme
• Promote Open Access worldwide
• Helping journals meet open access criteria
• Organize meetings and webinars for education
• Quantify status of open access in regions
• Mapping the open access landscape in regions
• Evaluate applications for inclusion in DOAJ
• More complete access to open access journals worldwide
• Provide help to researchers and institutes to recognize
questionable publishers

21. Global open access map
http://www.openaccessmap.org

22. DOAJ JOURNALS global distribution

23. DOAJ APPLICATIONS
TOP TEN COUNTRIES WITH HIGHEST NRS
OF APPLICATIONS (2014)
USAGE WEBSITE
Total API hits: 87,821,912
2017 (until 24 April 2017)
total: 30,334,012
2016 total: 57,487,900

24. Open Access policies worldwide
(government level)
• Europe: publicly funded research open access by 2020
• *China: no active government policy
• *Japan: no active government policy
• US : publicly funded research open access by 2020
• Chili: publicly funded research open access now
• *South Africa: no active government policy
• *North Africa: Algeria actively promoting open access
• India: no active government policy
• *Indonesia: plans for active promotion of open access
• *Russian Federation: no active government policy
* DOAJ links with government funded organizations

25. Open Access policies (societies)
Some researchers feel it is more important to publish an
article in one of these high IF journals than to publish an
article that will have a real importance for the long-term
development of science
Pre-publication reviewing is important to control the
quality of articles. However, open archives and
preprint repositories also have a valuable role in
allowing the rapid dissemination of scientific work and
encouraging large scale, post-publication peer review by
the entire community
We would like to see science
publishing move away from
large corporate interests and
a stronger involvement of
academies and learned
societies in order that any
surplus funds may be used for
the benefit of science. At the
same time, authors should
always retain their intellectual
property rights.

26.  EU: Open Access by 2020
Netherlands: gold open access by 2024
DOAJ list recognized
Austria: Science Fund supports DOAJ
 Nordic countries
list of accredited journals for government funded research
DOAJ list for open access journals
government support for DOAJ
 Algeria
government website for journal hosting
required listing in DOAJ
required open access (no set date)
 Indonesia
 Permenristekdikti No 20 in 2017
Minister speech:http://www.dikti.go.id/menristekdikti-dorong-dosen-tingkatkan-publikasi-ilmiah/
Open Access Government policies
Mentioning DOAJ

27. Open Access policy in EU
• 100 % Open Access
in 2020
• scholarly communication
• open access
• open data
• open science

28. Open Access policy in
Netherlands
 All publicly funded research open access 2020
 100% GOLD Open Access by 2024
 Funding decisions from Dutch Research Organization ZWO (major
funder) not dependent on where or how much scientists publish
(2016 Directive)
 Universities demand and negociate fair open access agreements
with major publishers
http://www.magazine-on-the-spot.nl/openaccess/

29. Open access journals in Indonesia
DOAJ application status
Indonesia total accepted rejected evaluation
reapps 74 57 12 5
New apps 1725 508 599 618

30. Cost of Publishing
source: VSNU the netherlands

31. Cost of Open AccessPublishing
• Situation is not transparant
• What does a publication cost? (< €2000)
• Open APC (info on basis of Freedom of Information Act)
https://treemaps.intact-project.org/

32. Cost of Open AccessPublishing
Source: vsnu the netherlands
• ’new’ Big Deals in the Netherlands (combining cost-free open
access publishing and subscription based reading of non-open
access titles) AMOUNTS PAID ?????
• Authors from Dutch universities can now publish open access in
>8000 journals without additional costs
Friday, December 23, 2016.
Springer and Dutch universities also
close a 100% open access deal for 2017.
Springer was late 2014 the first
publisher to conclude a contract for
100% open access with the Dutch
universities. It became a great
success.

33. Journal subscription costs before 2016
NL Universities
€ -
€ 2,000,000
€ 4,000,000
€ 6,000,000
€ 8,000,000
€ 10,000,000
€ 12,000,000
€ 14,000,000
€ 16,000,000
€ 18,000,000
2011
2012
2013
2014
2015
Source: http://vsnu.nl/nl_NL/f_c_kostenpublicaties.html

34. o T H E B A S I C S :
o PEER-REVIEW
o CITATION ANALYSIS
Assessing Quality of Research

35. Problems with Peer-review
Eisen JA, MacCallum CJ,
Neylon C (2013) Expert
Failure: Re-evaluating
Research Assessment. PLoS
Biol 11(10): e1001677.
doi:10.1371/journal.pbio.10
01677
Evaluations
Vary much
too much
Fake peer
review is a
growing
problem
Old boys
networks
Lack of
underlying
data

36. Solutions for better Peer-review
peer review platforms

37. o C I T A T I O N A N A L Y S I S I S C R I T I C A L L Y F L A W E D
o Citing does not necessarily mean having read, self-
citation, many articles instead of one, many authors per
article, author contribution difficult to assess, ghost
authors, journal impact factor not related to individual
article quality
Citations & Quality of Research
Wageningen UR Library

38. T H E S C I E N T I F I C Q U A L I T Y O F A N A R T I C L E C A N N O T B E
A S S E S S E D B Y C I T A T I O N S A L O N E
o T H E S C I E N T I F I C Q U A L I T Y O F A J O U R N A L C A N N O T B E
A S S E S S E D A T A L L B Y T H E A V E R A G E C I T A T I O N S C O R E S O F
A R T I C L E S ( T H E J O U R N A L I M P A C T F A C T O R J I F )
Citations & Quality of Research

39. Table 2: Percentage of papers published
in 2013-2014 with number of citations
below the value of the 2015 JIF.
Larivière et al. (2016)
eLife 8.3 71.2%
EMBO J. 9.6 66.9%
J. Informetrics 2.4 68.4%
Nature 38.1 74.8%
Nature Comm. 11.3 74.1%
PLOS Biol. 8.7 66.8%
PLOS Genet. 6.7 65.3%
PLOS ONE 3.1 72.2%
Proc. R. Soc. B 4.8 65.7%
Science 34.7 75.5%
Sci. Rep. 5.2 73.2%
Journal JIF
% citable items
below JIF
0
10
20
30
40
50
60
70
80
90
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
eLife
0
5
10
15
20
25
30
35
40
45
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
EMBO J.
0
10
20
30
40
50
60
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
J. Informetrics
0
10
20
30
40
50
60
70
80
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Nature
0
50
100
150
200
250
300
350
400
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Nature Comm.
0
5
10
15
20
25
30
35
40
45
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
PLOS Biol.
0
20
40
60
80
100
120
140
160
180
200
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
PLOS Genet.
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
PLOS ONE
0
20
40
60
80
100
120
140
160
180
200
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Proc. R. Soc. B
0
10
20
30
40
50
60
70
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Science
0
200
400
600
800
1,000
1,200
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Sci. Rep.
highly skewed distribution of citations
doi: http://dx.doi.org/10.1101/
062109

40. New Citation Based Metrics for Journals:
Journal CiteScore only deals with discipline citing bias

41. San Francisco Declaration on Research Impact
http://www.ascb.org/dora/
It doesn’t matter where
you publish or how much
you publish, research
assessment should not be
based on journal citation
analysis like JIF or number
of articles published
Signed by many organizations
and publishers most recently
Nature (april 2017)

42. • T R A C E T H E S O U R C E O F D A T A
• M A K E R E S E A R C H C O N N E C T I O N S / N E T W O R K S V I S I B L E
• F I N D K E Y P A P E R S F O R S P E C I F I C T O P I C S ( O P E N C I T A T I O N S
C O R P U S , P A R T O F I S 4 O A )
B U T :
• N O C O N C L U S I O N S O N I M P A C T F R O M A V E R A G E C I T A T I O N S C O R E S
• N O C O N C L U S I O N S O N Q U A L I T Y F R O M N U M B E R O F C I T A T I O N S
What can we do with Citations?
Blog.plos.org/biologue
1104029017

43. Initiative for Open Citations I4OC

44. ALTMETRICS
T A K I N G I N T O A C C O U N T T O T A L E X P O S U R E ,
D I S S E M I N A T I O N , C I T A T I O N S , S O C I A L M E D I A ,
C O N V E N T I O N A L M E D I A ( N E W S P A P E R S , T V , V I D E O )
Another kind of Quality
Article Level Impact Assessment
not quite the same as ’Quality’ rather relevance
Springer/nature Elsevier Wiley

45. • I N T E G R A T I O N O F A L T M E T R I C S C O R E S
• I N T E G R A T I O N O F O R C I D
• S E M I - A U T O M A T I C I N D E X U P D A T E S A N D S C R E E N I N G
• S E M A N T I C L I N K I N G O F S P E C I F I C C O N T E N T
• I M P R O V E D S E A R C H P O S S I B I L I T I E S
• W O R L D W I D E A M B A S S A D O R P R O G R A M M E S
DOAJ Future Plans

46. How DOAJ detects questionable journals
• Low publishing quality
Journal name, website, fees, peer review*, publisher, ownership, volume of
articles, advertisements, prominent soliciting for editors
• Low scientific quality
focus, format, self-citations, plagiarism*
• Malpractice
false claims, hidden costs, spamming authors, wrong information
* most often encountered problems
The ’problem’ of questionable journals
Is it a problem??

47. Questionable Publishing Global Perspective
• NOT ONLY IN OPEN ACCESS JOURNALS
• Not AS HIGH AS OFTEN REPORTED
according to a study* by Walt Crawford the number questionable publishers
in 2014 was about 3275 publishing about 121,000 articles and not 420,000
(Shen and Björk ** reported 8000 journals and 420,000 articles!!)
* http://walt.lishost.org/2015/11/ppppredatory-article-
counts-an-investigation-part-1/
**Shen and Björk. BMC Medicine201513:230

48. Questionable Publishing in Perspective
• Proportion of low quality journals is comparable
between open access and subscription publishing
but it looks worse because Open Access journals are
more visible
FACTS
• Not all subscription journals are in Scopus or WoS: only 20-30,000 of 100,000
(data Ulrich’s Web)
• Not all open access journals are in DOAJ : only 8 -10,000 of 30,000 ***
CONCLUSION Percentage of Quality Journals is comparable
*** Walt Crawford
http://citesandinsights.info/civ17i1.pdf

49. Q U E S T I O N A B L E P U B L I S H I N G I S A P R O B L E M
I T I S N O T A P R O B L E M U N I Q U E T O O P E N A C C E S S
I T I S M O R E E X P O S E D I N O P E N A C C E S S
SOLUTION FOR TH E P ROB LEM
• BETTER Q UALITY BY P UBLIS HING O P EN ACCES S
• S TANDARDIZED Q UALITY CO NTROL
• INDEXING O F Q UALITY JO UR NALS
• TRANSPARENCY ON CRITERIA USED
• R AIS ING AWARENESS WITH AUTHORS
(THINKCHECKSUBMIT )
Handling Questionable Publishers

50. COMMON SENSE

51. Thanks to :
Our ambassadors in various parts of the world,
all the Library Consortia, Universities and Publishers and our
Sponsors for the financial support to DOAJ!

52. Thank you for your attention!
tom@doaj.org
lars
lars@doaj.org

53. DOAJ depends entirely on donations
https://doaj.org/membership