This interdisciplinary theme of the Conference addresses two of the very serious and controversial challenges of modern-day research, namely dismissive reviews (unsupported declarations of scholars declaring no previous research exists on certain topics, despite evidence of the contrary) and „citation cartels” (groups of scholars who agree to cite each others’ work and none of the other available literature). These practices have a considerable impact in the quality and ethical aspects of research, and are also reflected in the replicability crisis.

1. Dismissive Reviews, Citation Cartels, and
the Replication Crisis
Richard P. Phelps

2. The Replication Crisis

3. … smaller the sample size.
… smaller the effect.
… greater the number of tested relationships.
… greater the flexibility in design, definitions, outcomes, and
analytical modes.
… greater the financial and other interests and prejudices.
… hotter the scientific field.
… greater the publicity.
According to John Ioannidis …
… research findings are less likely to be true, the …

4. Other researchers found “gaming the system” to be
widespread
“P-hacking”
Only requiring P < .05
Non-replicable findings
Many scholars refusing to share data sets
I add 2 more: Dismissive Reviews, Citation Cartels
Ioannidis exaggerated,
but much of what he wrote was true

5. 0
1
Nature/Science Economics Psychology
NOT Replicated Replicated
Proportion of studies replicated at p < .05,
in three fields of study, 2015 to 2018

7. Number of citations to replicated and not replicated studies, as of 2021

8. The growth of “Metascience” as a
field of study
Research Policy (journal), 1971
Scientometrics (journal), 1978
Cochrane Library, 1993
Metascience (journal), 1996
Campbell Collaboration, 2000
Research Ethics (journal), 2005
Centre for Journalology, 2005
Journal of Informetrics, 2007
Retraction Watch, 2010
PubPeer, 2012
Science Advances (journal), 2015
Research Waste/EQUATOR, 2015
Research Integrity and Peer Review (journal), 2016

9. Information proliferation
• Pro-Quest UMI dissertation publishing:
– 3 + 5 million dissertations and theses
– 250,000 new works each year
• STM reports:
– 34,000 journals
• +4%/year
– 10,000 publishers

10. Still more proliferation
As of 2021…
…3 million articles are
published in peer-
reviewed journals every
year.
More than 5 new articles
per minute.

11. The proliferation of researchers
• Residing in the United
States alone (2008):
– 2.5 million with doctoral
degrees
– 5.5 million with professional
degrees
– 14.9 million with masters’
degrees

12. Knowing ALL the
research literature on a
topic
• There is so much, is
anyone qualified to speak
for all of it?
• It is genuinely difficult to do
something new and unique

13. Knowledge is Unlimited?
• It may be, but there are limits to the
amount that we can use.
• So, we filter it.
Two ways to filter:
Summarize all of it
Accept only a certain amount, a
certain type,…or only from
certain people

14. The World Wide Web is an
information filter
20th Century:
Reference librarians provides wide range of information
21st Century:
Search Engine Optimization (SEO) narrows range

15. Definitions: Firstness Claims &
Dismissive Reviews
• With a firstness claim, a researcher insists
that s/he is the first to study a topic.
• With a dismissive literature review, a
researcher assures the reader that no one
else has conducted a study on a topic.

16. The Effect
of Firstness
Claims and
Dismissive
Reviews
The public is told that no other
research exists on a topic, ergo,
there is no reason to look for it.

17. How difficult is a literature review?
• Not analytically difficult
• But a genuinely thorough
review requires a
substantial amount of
time, and some money
• Generally, neither search
nor ethics are part of US
professors’ training.

18. Professional incentives to do a
thorough literature review
THERE ARE NONE?
• Scholars get little credit for a
thorough literature review,
much more for “original work”
• In “publish or perish”
environments, lit reviews are
impediments to progress

19. Why do a thorough lit review?
• huge burden in time and distraction
• little to no benefit professionally
• no punishment for not doing it

20. Dismissive Reviews and Citation Cartels in
Education Policy Research – A List
https://nonpartisaneducation.org/Review/Resources/DismissiveList.htm
A database of > 1,000 dismissive reviews from top
US scholars in education policy
3 types:
• Firstness Claims
• Dismissals
• Denigrations

21. Examples of Firstness Claims
“This report is unique.”
“This is the only study to …”
“We provide the first evidence …”
“We construct the first nationwide measures of …”
“Our study is the first to precisely estimate …”
“We are the first to isolate the impact of …”
“Ours is the first comprehensive analysis of …”
“To our knowledge, this has not been studied before …”

22. Examples of Dismissals
“There is very little research …”
“Surprisingly little is known about …”
“The debate is mostly supported by anecdotes.”
“Scant empirical evidence has been provided.”
“Studies of … are only relatively recent.”
“Knowledge of … has remained nonexistent.”
“There has been little systematic effort to study …”

23. Examples of Denigrations
“Previous work is very limited …”
“To provide more rigorous evidence, we …”
“Existing studies focus on a very small number of ….”
“[we] found the bulk of studies to be flawed …”
“… significant measurement problems in previous studies.”
“[earlier] studies were unable to control for ....”
“[earlier] studies had major deficiencies …”
“There are statistical shortcomings in … the studies.”

24. “absence of evidence is not
evidence of absence”
• Eyewitness fallacy – what you see represents all
• Composition fallacy – the whole must have a property
because its parts have the property.
• Amazing Familiarity – the speaker seems to have
information that there is no possible way for him to get
• Argumentum ad ignorantiam – a proposition is true if it
has not been proven false, or false if not proven true.
• Wishful Thinking – things are (or will be) some way
because that is how we wish them to be.

25. Raw Declaration
No evidence of a literature search provided
Dismissed Work Not Identified
Providing no help in finding it
Discouraging search for it
Avoiding debate
Key Characteristics of Dismissive Reviews

26.  “absence of research” ~33,000
 “absence of studies” ~26,000
 “this is the first study” ~1,610,000
 “little research” ~931,000
 “paucity of research” ~121,000
Search engine counts (Google
Scholar), 9.5.2023

27. Dismissive reviews worse than plagiarism
Misrepresent the work
of one person
(by plagiarizing)
• reward is small
(saves some work & time)
• risk is large
(could ruin one’s reputation
and career)
Misrepresent the work
of hundreds (in
dismissive reviews)
• reward is large
(for being first & unopposed)
• risk is nil

28. Ethics of
dismissive reviews
“Whatever you allow, you
encourage.”
–Michael Josephson

29. Definition: Citation Cartels
In a Citation Cartel, scholars…
… cite and reference each other and
… ignore, dismiss, or denigrate other
research.

30. Definition: “Sincere” Scholarship
Sincere scholars behave ethically:
Conduct forthright and thorough literature reviews,
Cite all other, relevant research regardless of:
their own personal opinions of the individual scholars
their own personal preferences for their research results.
They provide full and accurate citations of all that previous work,
so that readers will have no trouble finding it.
Focus is on accuracy and adding knowledge

31. Definition: “Strategic” Scholarship
Strategic scholars behave strategically:
Avoid literature searches and reviews
Either:
Declare ”firstness” – no previous research
Reference only work from within their cartel
Do not pass up opportunities to promote their work…
…even if they lack expertise on a topic, they will say
something
Focus is on advancing their careers

32. 0
10,000
20,000
30,000
40,000
50,000
0 5 10 15 20 25 30
Years
Strategic Scholars
Sincere Scholars
Comparing Strategic and Sincere Scholarship:
Citation accumulation over thirty years

33. How can they do this?
• No legal restrictions on education research quality –
• no Hippocratic Oath, no license or certification
• Rewards still lie within one’s primary field –
• economics for economists, psychology for
psychologists, etc.
• Scholars are rewarded for what they add; but they are
not punished for what they subtract
• Many who could “blow the whistle” are afraid, or they
might dislike the research literature and like dismissing it

34. Journalists help to suppress information
When they print one
researcher’s firstness
claim or dismissive
review,
they help to suppress
others’ work and
competing evidence

35. * saves time, avoids tedium of reading the research literature
* adds to own citation totals, or those of one’s citation cartel,
… while not adding to rivals’ citation totals
* gives readers no help in finding rival evidence (by not even
citing it)
* establishes (false) bona fides as an “expert” on the topic
* attention by allegedly being "first," "original," "a pioneer."
* increases the likelihood of press coverage for the same
reason.
* increases prospects for grant funding to "fill knowledge gaps."
Benefits accrue to…
…individual scholars
and small groups:

36. Costs accrue
to… society
• Society loses information; remaining information
is skewed in favor of the powerful
• Policy decisions are based on information that is
limited and skewed
• Government and foundations may pay again for
research that has already been done

37. Research most vulnerable to dismissal
• That done by those below the
“celebrity threshold”*
• Studies by civil servants
(government agencies do not
promote or defend their work)
• That done by the deceased
All become:
“Zombie Researchers”
* Researchers below the celebrity threshold lack the resources and media access to
successfully counter dismissals of their work – they can easily be ignored.

38. Paradox of
research
proliferation
As the amount of research grows…
…so does the amount declared nonexistent
…so does the incentive to dismiss it
…so does the opportunity to dismiss it

39. What Can be Done?

40. Transparency
Too easy to publish dismissive reviews:
• Find 1 editor and 1-2 reviewers among the
hundreds of thousands of journals available
• If first journal one tries objects, there are
many others.
• Just submit same paperwork to another
journal. A genuine lit review takes weeks
or months.
Preprints and PubPeer are the only current,
effective deterrents

41. Ban firstness claims
and dismissive reviews
Add ban to the ethics codes of…
…journalists
…foundation research funders
…government research funders
In most cases, editors, reviewers, & journalists
have neither the time nor the resources to verify

42. Real punishment for false firstness
claims and dismissive reviews
Make literature reviews optional
for getting funding, but…
…make their accuracy mandatory,
…suspend violators from any
further funding

43. Remove any literature review
obligation from research articles
• Removes temptation
• Most do more harm than
good anyway because
they are partial and
selective

44. Is meta-analysis the solution?
• Let meta-analysts do all literature
reviews
• Meta-analysis review model:
Identify where you have looked
before making summary claims

45. Merci !