Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measures for success: tools for evaluating quality, from 35th IATUL Conference, Espoo, Finland June 2014


Published on

Presented by Dom Mitchell, Community Manager for DOAJ to 35th Conference of International Association of Scientific and Technological University Libraries (IATUL).

A presentation exploring how DOAJ is using crowdsourcing to evaluate the ~9700 journals currently in DOAJ. Using a network of volunteers, every journals will be reassessed and evaluated based on the new criteria.

This version contains a handful of extra slides that were originally removed due to time restrictions.

Published in: Education
  • Login to see the comments

Measures for success: tools for evaluating quality, from 35th IATUL Conference, Espoo, Finland June 2014

  1. 1. Measures for Success: Measures of Quality Dom Mitchell Community Manager IATUL Conference 2-5 June 2014 University of Espoo, Finland
  2. 2. 2 Overview  What is the DOAJ?  What is our aim?  What issues do we face as a community?  The DOAJ solution  New application form with extended criteria  How do tighter criteria help?  Volunteers via crowdsourcing  Conclude
  3. 3. 3 What is DOAJ?  A database listing high quality, peer- reviewed, open access journals  Journals from ALL disciplines & all languages  A hub for the collection & distribution of metadata to 3rd parties
  4. 4. 4 What is DOAJ?  Started at Lund University, Sweden, led by Lars Bjørnshauge  Today managed by Infrastructure Services for Open Access C.I.C. ()  Developed & hosted on standards-based, open- source software by Cottage Labs ()
  5. 5. 5 What is our aim?  To become THE white list of open-access journals  To be truly global, curating partnerships worldwide  To increase visibility and awareness of quality open access journals  online: in social media and online learning environments  offline: where study & research happens, the labs
  6. 6. 6 What is our aim?  To encourage awareness of open access & its issues  To talk more to the community by increasing the transparency of our own operations:  DOAJ News Service:  Public consultations  Social media
  7. 7. 7 What issues?  The power of the internet:  New audiences, growing audiences  Increase in the availability of and access to publishing technologies (e.g. Smashwords for books)  Greater need to quickly pinpoint quality literature  Less quality in the research and literature  Less funding, increased subscription prices, Big Deals
  8. 8. 8 What issues? Scams  Predatory publishers  make money from author publishing charges (APCs)  Fake Impact Factor services  make money from journals wanting to boast a high impact factor  Journal hijacking/'phishing'  make money by copying a journal web site/branding and collecting money through it
  9. 9. 9  October 2013  February 2014 What issues? Scams
  10. 10. 10  ”International” subscription-based publishing excludes major parts of the world  Traditional editorial boards who exist on traditional print journals What issues? Exclusive & Biased
  11. 11. 11  The Journal Impact Factor is dominant  Determines research funding and directs research policy  Fails to embrace the real impact on practioners, the public and on society  Is flawed & prone to manipulation [Brembs: - slide 48 onwards]  What counts is not WHAT you publish but WHERE you publish it! What issues? Impact Factor
  12. 12. 12 What issues? Not all bad!  Open Access is growing! Fast. Heather Morrison's 'Dramatic Growth of Open Access' series  Services already exist to help people quality research, e.g. PubMed Central  Libraries have their own services for education and outreach to students, faculty and staff  Peer to peer networks like Mendeley thrive  More discussion on social media around open access
  13. 13. 13  How can DOAJ help the community?  More transparency is needed in these areas:  The editorial process  Peer-review  Reuse and readers' rights  Author services  Archiving  Permanent identifiers  Discoverability DOAJ's Solution
  14. 14. 14  How do we encourage transparency?  Developed with  Public consultation period: Advisory Board and to establish key quality indicators  Old form: 6 questions. New form: 56! DOAJ's New Criteria
  15. 15. 15 1. Peer review process 2. Governing Body 3. Editorial team/contact 4. Author fees 5. Copyright 6. Identification of and dealing with allegations of research misconduct 7. Ownership and management 8. Web site 9. Name of journal 10.Conflicts of interest 11.Access 12.Revenue sources 13.Advertising 14.Publishing schedule 15.Archiving DOAJ's New Criteria
  16. 16. 16
  17. 17. 17 DOAJ's New criteria  New form structured focusses on 3 different themes:  Quality  Openness  The delivery or technical quality  Publishers have to provide much more information to be indexed
  18. 18. 18  Encourages widespread adoption  Promotes best practice  Transparency  Empowers the community  Tackles the problem of fake or low quality publishers etc  Increases discoverability and visibility How do Tighter Criteria Help?
  19. 19. 19  Transparency: makes it simpler for funders, universities, libraries and authors to determine whether a journal is of high quality  Enables the community to monitor compliance  Tackles the problem of fake or low quality publishers, content and business practices How do Tighter Criteria help?
  20. 20. 20 The long tail
  21. 21. 21  To motivate and encourage ALL OA-journals, regardless of size, to:  be more explicit on issues of editorial process  be more explicit on issues of rights and reuse  improve their level of “technical” quality to foster dissemination and discoverability (e.g. 64% DOAJ publishers have no permanent article Ids or don't know what one is) How do Tighter Criteria help?
  22. 22. 22  Respect different [publishing] cultures, traditions, languages, sizes and capabilities  Cannot exclude any journal but rather we will facilitate and assist small journals to have as much value and as much visibility as large journals. Delicate Balance
  23. 23. 23  The community are experts: tap that resource  Get the community directly involved with DOAJ  A call for volunteers in 2014 had a huge response  Applying a crowdsourcing model This calls for volunteers!
  24. 24. 24  Crowdsourcing: harnessing librarian power!  Organised in a network of Editors and Associate Editors  Grouped by language and/or specialty  Starting test pilot with Chinese, English & Spanish  Aim is to cover as many Crowdsourcing
  25. 25. 25  Associate Editor responsibilities include:  Processing journal applications  Translation work  Regular review of indexed journals  Handling questions or alerts from the Community  Already started translating application form into Chinese, Portuguese & Spanish Crowdsourcing
  26. 26. 26  Associate Editor is assigned an application to review, check, verify for accuracy  Once satisfied, the journal is flagged for acceptance  Acceptance is confirmed by Managing Editor, a two-step process ensuring objectivity and consistency Crowdsourcing
  27. 27. 27  We're confident that our new criteria will improve the transparency and credibility of OA-journals!  We will continue to contribute to the momentum of open access publishing by  carefully promoting standards, transparency and best practice  without losing the global view To conclude
  28. 28. Our ambition: DOAJ to be the white list! I.E. if a journal is in the DOAJ, it complies with high standards and is of good or high quality
  29. 29. 29 Upgrading the DOAJ is a big project! Please support us!
  30. 30. 30 Thanks to all the Library Consortia, Universities and Publishers and to our Sponsors for their financial support of DOAJ!
  31. 31. 31 Thank you!