Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Assessing the quality of scholarly publishing

670 views

Published on

Speech by Dr. Tom Olijhoek
Editor-in-Chief DOAJ

Published in: Education
  • Be the first to comment

  • Be the first to like this

Assessing the quality of scholarly publishing

  1. 1. ASSESSING THE QUALITY OF SCHOLARLY PUBLISHING 1 3 t h A D R I C O N F E R E N C E : S U R A B A Y A , I N D O N E S I A M A Y 3 - 4 , 2 0 1 7 T O M @ D O A J . O R G E D I T O R - I N - C H I E F D O A J
  2. 2. You are free to: Copy, share, adapt, or re-mix; Blog, live-blog, or post video of; This presentation. Provided that: You attribute the work to its author and respect the rights and licenses associated with its components. Slide Concept by Cameron Neylon, who has waived all copyright and related or neighbouring rights. This slide only ccZero. Social Media Icons adapted with permission from originals by Christopher Ross. Original images are available under GPL at;
  3. 3. • A d v a n t a g e s a n d o p p o r t u n i t i e s o f O p e n A c c e s s • Q u a l i t y o f p u b l i s h i n g a n d q u a l i t y o f s c i e n c e • W h a t i s t h e r o l e o f t h e D O A J ? • T h e D O A J A m b a s s a d o r p r o g r a m m e • O p e n a c c e s s p o l i c i e s • P u b l i s h i n g c o s t s O p e n A c c e s s • A s s e s s i n g R e s e a r c h Q u a l i t y • P e e r e v i e w a n d c i t a t i o n a n a l y s i s r e v i e w e d • P e e r r e v i e w f l a w s • C i t a t i o n a n a l y s i s : d e m i s e o f t h e i m p a c t f a c t o r I F • C i t a t i o n a n a l y s i s : o p e n c i t a t i o n s I 4 O C • A s s e s s i n g I m p a c t / R e l e v a n c e b y a l t m e t r i c s : D O R A • T h e ‘ ? p r o b l e m ? ’ o f q u e s t i o n a b l e p u b l i s h e r s Presentation overview
  4. 4. O P E N A C C E S S . . . . . • A R T I C L E S R E C E I V E M O R E C I T A T I O N S • A R T I C L E S A R E M O R E R E A D • R E A C H E S A B I G G E R A U D I E N C E • I S L E S S P R O N E T O B I A S • A V O I D S D U P L I C A T E S T U D I E S • A R T I C L E S C A N B E M O R E E A S I L Y C H E C K E D • D A T A L E S S P R O N E T O M A N I P U L A T I O N • I S B E T T E R F O R A D V A N C I N G S C I E N C E • L E A D S T O M O R E I N N O V A T I O N • L E A D S T O B E T T E R E D U C A T I O N Reasons for Open Access
  5. 5. • DOAJ Directory of Open Access Journals O P E N A C C E S S J O U R N A L S O N L Y , n o r a n k i n g • SCOPUS O P E N A C C E S S & S U B S C R I P T I O N J O U R N A L S , r a n k i n g • WEB OF SCIENCE O P E N A C C E S S & S U B S C R I P T I O N J O U R N A L S , r a n k i n g • NATIONAL LISTS O P E N A C C E S S & S U B S C R I P T I O N J O U R N A L S r a n k i n g g r o u p s : D a n i s h e x a m p l e INDEXING SERVICES for quality control
  6. 6. • DENMARK: BIBLIOMETRIC RESEARCH INDICATOR National lists for quality control http://ufm.dk/en/research-and-innovation/statistics-and- analyses/bibliometric-research-indicator/bfi-rules-and- regulations?set_language=en&cl=en
  7. 7. • QUALITY OF PUBLISHING • QUALITY OF THE SCIENCE Assessing the Quality of a Journal
  8. 8. What is the DOAJ? The only global index of trusted, high quality, peer- reviewed, open access journals, from all academic disciplines,curated by the community https://doaj.org/ FREE to register and FREE to use
  9. 9. • IS4OA: DOAJ; OPEN CITATION CORPUS • SHERPA/ROMEO • CROSSREF • ORCID • ARCHIVING : e.g. LOCKSS • PUBLON (reviewer platform) Needed Infrastructure Services
  10. 10. Recognition of the DOAJ?
  11. 11. Use of DOAJ Data • EBSCO • ROAD • NATIONAL LISTS • MANY OTHER ORGANIZATIONS ROAD contains DOAJ journals + hybrids + repository material + blogs + …………
  12. 12. DOAJ STRUCTURE
  13. 13. • DOAJ CRITERIA FOR QUALITY OPEN ACCESS PUBLISHING •Principles of transparency and Best Practice in Scholarly Publishing •(DOAJ, OASPA, COPE, WAME) Assessing the Quality of Publishing
  14. 14. WHAT IS QUALITY OPEN ACCESS? The BOAI Definition Open Access is: a publishing system where all content is freely available without charge to the user or his/her institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles, or use them for any other lawful purpose, without asking prior permission from the publisher or the author.
  15. 15. Copyrights • Clearly described on web site* • Recommend: author retains copyright • Recommend: no exclusive publishing rights • Recommend: no transfer of commercial rights https://doajournals.wordpress.com/2015/06/02/copyright-and-licensing-part-2/ Essential criteria
  16. 16. Content Licensing • Clearly described on web site* • Recommend: • licensing terms on all articles, all versions (html, pdf, xml etc.) • embedded in article level metadata • Creative Commons licensing Essential criteria
  17. 17. What is a license? • Licensing means to grant a third party (anyone else except the right holder) the right to use a copyright-protected work • A license is a permission to use a work in specific ways • Licenses can only be granted by copyright holder • Copyright holder can be author or publisher COPYRIGHT AND LICENCING
  18. 18. DOAJ IDRC Funded Ambassador Programme 2016-2017 CHINA RUSSIAN FED. AFRICA MIDDLE EAST LATIN AMERICA INDIA
  19. 19.  Africa SE, NW 4 ambassadors  Russia 3 ambassadors  China 3 ambassadors  Middle East 1 ambassador  Latin America 1 ambassador  India 3 ambassadors Planned:  Japan  Indonesia Extended DOAJ Ambassador Programme (FUNDING??)
  20. 20. Aims of the Ambassador programme • Promote Open Access worldwide • Helping journals meet open access criteria • Organize meetings and webinars for education • Quantify status of open access in regions • Mapping the open access landscape in regions • Evaluate applications for inclusion in DOAJ • More complete access to open access journals worldwide • Provide help to researchers and institutes to recognize questionable publishers
  21. 21. Global open access map http://www.openaccessmap.org
  22. 22. DOAJ JOURNALS global distribution
  23. 23. DOAJ APPLICATIONS TOP TEN COUNTRIES WITH HIGHEST NRS OF APPLICATIONS (2014) USAGE WEBSITE Total API hits: 87,821,912 2017 (until 24 April 2017) total: 30,334,012 2016 total: 57,487,900
  24. 24. Open Access policies worldwide (government level) • Europe: publicly funded research open access by 2020 • *China: no active government policy • *Japan: no active government policy • US : publicly funded research open access by 2020 • Chili: publicly funded research open access now • *South Africa: no active government policy • *North Africa: Algeria actively promoting open access • India: no active government policy • *Indonesia: plans for active promotion of open access • *Russian Federation: no active government policy * DOAJ links with government funded organizations
  25. 25. Open Access policies (societies) Some researchers feel it is more important to publish an article in one of these high IF journals than to publish an article that will have a real importance for the long-term development of science Pre-publication reviewing is important to control the quality of articles. However, open archives and preprint repositories also have a valuable role in allowing the rapid dissemination of scientific work and encouraging large scale, post-publication peer review by the entire community We would like to see science publishing move away from large corporate interests and a stronger involvement of academies and learned societies in order that any surplus funds may be used for the benefit of science. At the same time, authors should always retain their intellectual property rights.
  26. 26.  EU: Open Access by 2020 Netherlands: gold open access by 2024 DOAJ list recognized Austria: Science Fund supports DOAJ  Nordic countries list of accredited journals for government funded research DOAJ list for open access journals government support for DOAJ  Algeria government website for journal hosting required listing in DOAJ required open access (no set date)  Indonesia  Permenristekdikti No 20 in 2017 Minister speech:http://www.dikti.go.id/menristekdikti-dorong-dosen-tingkatkan-publikasi-ilmiah/ Open Access Government policies Mentioning DOAJ
  27. 27. Open Access policy in EU • 100 % Open Access in 2020 • scholarly communication • open access • open data • open science
  28. 28. Open Access policy in Netherlands  All publicly funded research open access 2020  100% GOLD Open Access by 2024  Funding decisions from Dutch Research Organization ZWO (major funder) not dependent on where or how much scientists publish (2016 Directive)  Universities demand and negociate fair open access agreements with major publishers http://www.magazine-on-the-spot.nl/openaccess/
  29. 29. Open access journals in Indonesia DOAJ application status Indonesia total accepted rejected evaluation reapps 74 57 12 5 New apps 1725 508 599 618
  30. 30. Cost of Publishing source: VSNU the netherlands
  31. 31. Cost of Open AccessPublishing • Situation is not transparant • What does a publication cost? (< €2000) • Open APC (info on basis of Freedom of Information Act) https://treemaps.intact-project.org/
  32. 32. Cost of Open AccessPublishing Source: vsnu the netherlands • ’new’ Big Deals in the Netherlands (combining cost-free open access publishing and subscription based reading of non-open access titles) AMOUNTS PAID ????? • Authors from Dutch universities can now publish open access in >8000 journals without additional costs Friday, December 23, 2016. Springer and Dutch universities also close a 100% open access deal for 2017. Springer was late 2014 the first publisher to conclude a contract for 100% open access with the Dutch universities. It became a great success.
  33. 33. Journal subscription costs before 2016 NL Universities € - € 2,000,000 € 4,000,000 € 6,000,000 € 8,000,000 € 10,000,000 € 12,000,000 € 14,000,000 € 16,000,000 € 18,000,000 2011 2012 2013 2014 2015 Source: http://vsnu.nl/nl_NL/f_c_kostenpublicaties.html
  34. 34. o T H E B A S I C S : o PEER-REVIEW o CITATION ANALYSIS Assessing Quality of Research
  35. 35. Problems with Peer-review Eisen JA, MacCallum CJ, Neylon C (2013) Expert Failure: Re-evaluating Research Assessment. PLoS Biol 11(10): e1001677. doi:10.1371/journal.pbio.10 01677 Evaluations Vary much too much Fake peer review is a growing problem Old boys networks Lack of underlying data
  36. 36. Solutions for better Peer-review peer review platforms
  37. 37. o C I T A T I O N A N A L Y S I S I S C R I T I C A L L Y F L A W E D o Citing does not necessarily mean having read, self- citation, many articles instead of one, many authors per article, author contribution difficult to assess, ghost authors, journal impact factor not related to individual article quality Citations & Quality of Research Wageningen UR Library
  38. 38. T H E S C I E N T I F I C Q U A L I T Y O F A N A R T I C L E C A N N O T B E A S S E S S E D B Y C I T A T I O N S A L O N E o T H E S C I E N T I F I C Q U A L I T Y O F A J O U R N A L C A N N O T B E A S S E S S E D A T A L L B Y T H E A V E R A G E C I T A T I O N S C O R E S O F A R T I C L E S ( T H E J O U R N A L I M P A C T F A C T O R J I F ) Citations & Quality of Research
  39. 39. Table 2: Percentage of papers published in 2013-2014 with number of citations below the value of the 2015 JIF. Larivière et al. (2016) eLife 8.3 71.2% EMBO J. 9.6 66.9% J. Informetrics 2.4 68.4% Nature 38.1 74.8% Nature Comm. 11.3 74.1% PLOS Biol. 8.7 66.8% PLOS Genet. 6.7 65.3% PLOS ONE 3.1 72.2% Proc. R. Soc. B 4.8 65.7% Science 34.7 75.5% Sci. Rep. 5.2 73.2% Journal JIF % citable items below JIF 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations eLife 0 5 10 15 20 25 30 35 40 45 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations EMBO J. 0 10 20 30 40 50 60 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations J. Informetrics 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Nature 0 50 100 150 200 250 300 350 400 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Nature Comm. 0 5 10 15 20 25 30 35 40 45 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations PLOS Biol. 0 20 40 60 80 100 120 140 160 180 200 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations PLOS Genet. 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations PLOS ONE 0 20 40 60 80 100 120 140 160 180 200 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Proc. R. Soc. B 0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Science 0 200 400 600 800 1,000 1,200 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Sci. Rep. highly skewed distribution of citations doi: http://dx.doi.org/10.1101/ 062109
  40. 40. New Citation Based Metrics for Journals: Journal CiteScore only deals with discipline citing bias
  41. 41. San Francisco Declaration on Research Impact http://www.ascb.org/dora/ It doesn’t matter where you publish or how much you publish, research assessment should not be based on journal citation analysis like JIF or number of articles published Signed by many organizations and publishers most recently Nature (april 2017)
  42. 42. • T R A C E T H E S O U R C E O F D A T A • M A K E R E S E A R C H C O N N E C T I O N S / N E T W O R K S V I S I B L E • F I N D K E Y P A P E R S F O R S P E C I F I C T O P I C S ( O P E N C I T A T I O N S C O R P U S , P A R T O F I S 4 O A ) B U T : • N O C O N C L U S I O N S O N I M P A C T F R O M A V E R A G E C I T A T I O N S C O R E S • N O C O N C L U S I O N S O N Q U A L I T Y F R O M N U M B E R O F C I T A T I O N S What can we do with Citations? Blog.plos.org/biologue 1104029017
  43. 43. Initiative for Open Citations I4OC
  44. 44. ALTMETRICS T A K I N G I N T O A C C O U N T T O T A L E X P O S U R E , D I S S E M I N A T I O N , C I T A T I O N S , S O C I A L M E D I A , C O N V E N T I O N A L M E D I A ( N E W S P A P E R S , T V , V I D E O ) Another kind of Quality Article Level Impact Assessment not quite the same as ’Quality’ rather relevance Springer/nature Elsevier Wiley
  45. 45. • I N T E G R A T I O N O F A L T M E T R I C S C O R E S • I N T E G R A T I O N O F O R C I D • S E M I - A U T O M A T I C I N D E X U P D A T E S A N D S C R E E N I N G • S E M A N T I C L I N K I N G O F S P E C I F I C C O N T E N T • I M P R O V E D S E A R C H P O S S I B I L I T I E S • W O R L D W I D E A M B A S S A D O R P R O G R A M M E S DOAJ Future Plans
  46. 46. How DOAJ detects questionable journals • Low publishing quality Journal name, website, fees, peer review*, publisher, ownership, volume of articles, advertisements, prominent soliciting for editors • Low scientific quality focus, format, self-citations, plagiarism* • Malpractice false claims, hidden costs, spamming authors, wrong information * most often encountered problems The ’problem’ of questionable journals Is it a problem??
  47. 47. Questionable Publishing Global Perspective • NOT ONLY IN OPEN ACCESS JOURNALS • Not AS HIGH AS OFTEN REPORTED according to a study* by Walt Crawford the number questionable publishers in 2014 was about 3275 publishing about 121,000 articles and not 420,000 (Shen and Björk ** reported 8000 journals and 420,000 articles!!) * http://walt.lishost.org/2015/11/ppppredatory-article- counts-an-investigation-part-1/ **Shen and Björk. BMC Medicine201513:230
  48. 48. Questionable Publishing in Perspective • Proportion of low quality journals is comparable between open access and subscription publishing but it looks worse because Open Access journals are more visible FACTS • Not all subscription journals are in Scopus or WoS: only 20-30,000 of 100,000 (data Ulrich’s Web) • Not all open access journals are in DOAJ : only 8 -10,000 of 30,000 *** CONCLUSION Percentage of Quality Journals is comparable *** Walt Crawford http://citesandinsights.info/civ17i1.pdf
  49. 49. Q U E S T I O N A B L E P U B L I S H I N G I S A P R O B L E M I T I S N O T A P R O B L E M U N I Q U E T O O P E N A C C E S S I T I S M O R E E X P O S E D I N O P E N A C C E S S SOLUTION FOR TH E P ROB LEM • BETTER Q UALITY BY P UBLIS HING O P EN ACCES S • S TANDARDIZED Q UALITY CO NTROL • INDEXING O F Q UALITY JO UR NALS • TRANSPARENCY ON CRITERIA USED • R AIS ING AWARENESS WITH AUTHORS (THINKCHECKSUBMIT ) Handling Questionable Publishers
  50. 50. COMMON SENSE
  51. 51. Thanks to : Our ambassadors in various parts of the world, all the Library Consortia, Universities and Publishers and our Sponsors for the financial support to DOAJ!
  52. 52. Thank you for your attention! tom@doaj.org lars lars@doaj.org
  53. 53. DOAJ depends entirely on donations https://doaj.org/membership

×