Research Quality, Bibliometrics and the Republic of Science

1. Research Quality, Bibliometrics
and the Republic of Science
Michael A Peters
Presentation at the Symposium
Higher Education: Quality Assurance Widens Democracy or Not?
University of Stellenbosch
Thursday 10 August, 2006

2. Outline
1. Ideologies of Knowledge & Knowledge Cultures
(i) English Renaissance: Forbidden Knowledge – Marlow’s Dr Faustus
(ii) French Enlightenment – Encyclopedic Knowledge – Diderot’s L'Encyclopédie
(iii) Postmodern Knowledge Economy - Thomson’s ‘total information solution’
2. Byblos, Bibliographies & Bibliometrics
(i) Journals, Journology & The Origins of Peer Review
(ii) Bibliometrics and the Architecture of Global Science
(iii) Research Quality and the Development of National Systems
3. Peer Review, Bibliometrics & the Governance of Science
(i) Quality Assurance Replaces ‘Truth’ as Core Commitment of Post-normal
Science & the Case for ‘Extended Peer Review’
(ii) The Centrality of Peer Review to the Republic of Science and the Shift to
Bibliometrics
(iii) The Limitations of Citation Analysis

3. 1. Ideologies of Knowledge &
Knowledge Cultures
(i) English Renaissance: Forbidden knowledge
O lente, lente, currite noctis equi:
The stars move still, time runs, the clock will strike.
The devil will come, and Faustus must be damned.
O, I'll leap up to my God: who pulls me down?
See, see where Christ's blood streames in the
firmament;
One drop would save my soule, half a drop, ah, my
Christ!
I'll burn my books! Ah, Mephistophilis.
--Faustus
Christopher Marlowe, The Tragicall History of D. Faustus,
1589

4. The Tragicall History of D. Faustus
The play trades on the theme of knowledge in its
relation to sin, redemption and damnation at the heart of
Christianity’s understanding of the world and also the
conflict between medieval and renaissance values,
knowledge and power as corrupting forces, magic and
the supernatural, and the divided nature of man. The
story of Faustus then exemplifies the new spirit of
inquiry that characterizes Renaissance science and the
intellectual ambition for ultimate knowledge and the
power it brings against a specifically Christian ideology
that warns about the dangers of seeking forbidden
knowledge.

5. Forbidden Knowledge
• Marlowe was a child of the English Renaissance and the Reformation, which was
also that troubled period called by the great scholar Dame Frances Yates, "the false
dawn of the Enlightenment," which was doomed to suppression and delay. He
shared his birth year,1564, with Galileo (and with Shakespeare). It was a dangerous
time in which to express an eager interest in the new scientific discoveries that were
exciting the minds of intellectuals all over Europe.
• In England Sir Walter Raleigh and the young Earl of Northumberland, Henry Percy
(also born in 1564), led a group of intellectuals, a select band of advanced thinking
noblemen, courtiers and educated commoners, including mathematicians,
astronomers, voyagers who had explored the New World, geographers, philosophers
and poets. They formed an esoteric club nicknamed "The School of Night" which met
secretly to discuss this forbidden knowledge, always "behind closed doors." Marlowe
became a member of this close circle, who were called Free-Thinkers and were all
stigmatised as "Atheists" in order to blacken them in the eyes of the ignorant.
• Francis Bacon’s (1561-1626) utopian inductive philosophy of science (The
Advancement of Learning, Novum Organum, The New Atlantis) strongly influenced by
alchemy & magick but also based on observation and experiment, became the
guiding ethos of the Royal Society established in 1660 as ‘The Empire of Learning’ led
by Robert Boyle, Sir Christopher Wren, Sir Robert Moray. The Philosophical
Transactions of the Royal Society became the first academic journal, its first edition
edited by Henry Oldenburg in 1665. Peer review was introduced as a practice of the
journal thereafter.

6. (ii) Encyclopedic Knowledge
Nothing is more contrary to the progress of knowledge
than mystery.... If it happens that an invention favorable
to the progress of the arts and sciences comes to my
knowledge, I burn to divulge it; that is my mania. Born
communicative as much as it is possible for a man to be,
it is too bad that I was not born more inventive; I would
have told my ideas to the first comer. Had I but one
secret for all my stock in trade, it seems to me that if the
general good should require the publication of it, I
should prefer to die honestly on a street corner, my back
against a post, than let my fellow men suffer.
- Denis Diderot

7. Diderot’s L’Encyclopédie
Denis Diderot with Jean Le Rond d'Alembert was the
chief editor of L'Encyclopédie which was published
between 1751 and 1772 in 17 volumes of text and 11
volumes of engravings, representing perhaps one of the
greatest achievements of learning of the Enlightenment.
The Encyclopédie outlined the then present state of
knowledge about the sciences, arts, and crafts, and was
explicitly designed to make the knowledge possessed by
the few accessible to the many. Most of the 71, 818
articles in the Encyclopédie were written by Diderot and
d'Alembert but also included prominent thinkers of the
day including Rousseau, Voltaire, Montesquieu, Baron
d'Holbach, Necker, Turgot, and Buffon.

8. Encyclopedic Knowledge Diversity
The notion of the encyclopedia, originally from a Greek word
meaning literally ‘a general education,’ represents the attempt to
collect all of the world’s knowledge in a single system and arguably
dates from the 37 volumed work of Pliny’s Naturalis Historia in the
first century CE. Many other attempts at this total knowledge
system followed: Cassiodorus’ Institutiones (560 CE), the first
catholic encyclopedia that inspired St Isidore of Seville’s Etymologiae
(636); Bartholomeus de Glanvilla’s De proprietatibus rebus (1240),
probably the most widely read encyclopedia in the Middle Ages;
Abu Bakr al-Razi’s encyclopedia of science, Al-Kindi’s (801-873)
corpus of over 250 works in music, philosophy, astronomy,
medicine, mathematics and geography; the Yongle encyclopedia of
the Ming dynasty consisting in 11,000 handwritten volumes
completed in 1408.

9. History of the Encyclopedia
The history of the encyclopedia indicates
something about the organization of knowledge,
the history of science and technology, the
development of library science and information,
the place of reference works per se, and imbued
with a conscious ideology or philosophy,
invariably was tied to new approaches to
knowledge involving its democratization.
Diderot’s aim, for instance, was to ‘change the
common way of thinking’ through the
expansion of knowledge and the development of
critical modes of thought.

10. Encyclopaideia, Philosopedia, or
Macropedagogy?
Of all pedagogical systems and reference works -- such
as dictionaries, almanacs, gazetteers, atlases and
directories -- the encyclopedia was the only
one to aim at a total comprehensive and self-contained
system, although there was no one set of principles
guiding the method for classification of entries or
arrangement of contents. Classifications varied
considerably in the period before the alphabetical
arrangement of entries was introduced through a
standardization that came with printing, reflecting the
working epistemologies of the age and the evolution of
disciplines and the formation of knowledges.

11. Ideology of the Encyclopedia
Diderot and D’Alembert developed their underlying epistemology
governing their Encyclopédie under the influence of Locke and
Condillac: a rational and positivistic classification of existing
knowledge based on the assumption of the unity of theory and
praxis. Diderot planned to provide complete alphabetical
treatment of the whole field of human knowledge from the
standpoint of the "Enlightenment" and the contributors included
Montesquieu, Voltaire, Rousseau, Turgot and Condorcet. The 28
volumes of the Encyclopédie completed between 1751-72 were not
simply a repository of human knowledge but also a polemic from
the viewploint of the French Enlightenment and as such included a
strong emphasis on democracy and equality, and a kind of tolerance
of philosophical views that were essentially subversive of the
established order.
Peters & Ghirdaldelli (1999)
http://www.vusst.hr/ENCYCLOPAEDIA/main.htm

12. Internet as Encyclopedia
In a few short years, the Internet has gone from being a specialist
site for a few scientists and engineers to a place -- a soft and almost
infinitely flexible architecture -- which incorporates a staggering
variety of spaces: not just information exchange or reference banks
or dictionaries but a vast conglomeration of different spaces,
teeming with activity.
The Internet, perhaps like the encyclopedia of old, now speaks to
implicit knowledge formations, hybrid discourses, personal
homepages that approve the existential conditions for aesthetically
transforming oneself --chatgroups, bargain basements, advertising,
new businesses and the whole vista of bookish elements now
chaotically "shelved" alongside video clips, images and other non-
textual items. For the first time, the desire of the medieval or
Enlightenment encyclopedists of bringing together the entire scope
and corpus of human knowledge and information looks more than a
possibility rather than simple an historical reflection of the vanity of
a bygone age

13. 2. Byblos, bibliographies,
bibliometrics
Byblos - 37 kilometres north of Beirut – was the commercial center
of the papyrus trade; the Greek name for the Phoenician city Gebal
(earlier Gubla) so called because it was through this port that
Egyptian papyrus was imported into Greece. Byblos is also directly
associated with the history and diffusion of the Phoenician alphabet.
Bibliography (1) Publication in either printed or electronic format,
usually arranged by author, date or subject, which offers a listing of
works produced either by one author, in one subject, in one place, or
during one period. (2) A list of works consulted in the composition
of a book, article, or assignment.www.library.nuigalway.ie/help/jargon/
Bibliometrics is a type of research method used in library and
information science. It utilizes quantitative analysis and statistics to
describe patterns of publication within a given field or body of
literature. Researchers may use bibliometric methods of evaluation
to determine the influence of a single writer, for example, or to
describe the relationship between two or more writers or works.

14. Bibliometrics and the Changing
Architecture of World Science
• It is only since the 1960s with the development of research evaluation and
increasing sophistication of bibliometrics that it has been possible to map
the emerging economy of global science, at least on a comparative national
and continental basis. The Science Citation Index provides bibliographic
and citational information from 3,700 of the world’s scientific and technical
journals covering over one hundred disciplines.[1] The expanded index
available in an online version covers more than 5,800 journals. Comparable
‘products’ in the social sciences (SSCI) and humanities (A&HCI) cover,
respectively, bibliographic information from 1,700 journals in fifty
disciplines and 1,130 journals. Additionally, the Web of Science covers more
than 5,800 scientific journals.
• Thomson’s ‘scientific products’ provide ‘a total information solution is one
that enables users to effortlessly navigate between essential research
information sources including <Web of Knowledge.., full-text documents
hosted by primary publishers, a growing list of key databases, and other
options such as online public access catalogs’ (my emphasis).

15. Thomson Scientific – An
Information Utility
The SCI is owned by Thomson Scientific and together with SSCI, A&HCI,
and the Web of Science http://scientific.thomson.com/products/sci/.
Thomson Scientific is part of the Thomson Corporation which advertises
itself as ‘a leading global provider of integrated information-based
solutions to business and professional customers.’
It is one of the new leading information utility corporations with some $8.5
billion in revenues from legal & regulatory, learning, financial, and
scientific and healthcare global market groups. Beginning in newspapers in
Canada in the 1930s Thomson expanded rapidly into TV in the 1950s,
magazines and books in the 1960s, as well as travel, oil & gas, specialized
information services, textbooks and technical books, legal publishing, and
market data systems in subsequent decades, acquiring the Institute for
Scientific Information (which was established by Eugene Garfield in 1958 in
Philadelphia) in 1992. During the 1990s and 2000s Thomson acquired 30
major information-related companies.
Source: http://scientific.thomson.com/products/sci/.

16. Growth of Periodicals
Ulrich’s Directory of International Periodicals lists nearly 165,000
journals from 78,000 publishers. However, most of the significant
information produced each year appears in only a small portion of
them. This principle is referred to as Bradford’s Law. It states, in
essence, that an essential core of journals forms the literature basis
for all disciplines and that most of the important papers are
published in relatively few journals. Recent citation analyses have
shown that as few as 150 journals account for half of what is cited
and about one quarter of what is published. A core of
approximately 2,000 journals now account for about 85 percent of
the world’s published articles and 95 percent of the world’s cited
articles. Only 40 percent of all published papers receive more than
one citation. About 25 percent of all published papers will never be
cited and less than 17 percent are cited 17 or more times.
Hattendorf Westney (1998)

17. World Scientific Production
(1981-2000)
In 2000 the SCI included a total of 584,982 papers, representing a
57.5% increase from 1981, when 371,346 papers were published
worldwide. Authors with addresses in developed countries wrote
87.9% of the papers in 2000, a decrease from 93.6% in 1981.
Developing countries, on the other hand, saw a steady increase in
their share of scientific production: from 7.5% of world papers in
1981 to 17.1% in 2000< Since 1981 the world map of publications
changed significantly. North America lost the lead it had in 1996,
and in 2000 produced 36.8% of the world total, a decrease from
41.4% in 1981. The opposite trend can be found in the European
Union, which in 2000 published 40.2% of the world total, up from
32.8% in 1981. Japan went up from 6.9% to 10.7% in 2000.
Collectively this ‘triad’ has therefore maintained its dominance,
accounting for 81% of the world total of scientific publications in
2000, up from 72% in 1981.
UIS Bulletin on Science and Technology Statistics (UNESCO 2005)

18. The Evolution of Science Indicators
The evolution of science indicators only merged relatively recently
with research evaluation in the 1960s with the Frascati Manual
developed by the OECD in 1963 for standardizing the collection of
statistics on R & D, revised in 2002 to take account of the impact of
globalization and the importance of innovation as a key element in
the knowledge-based economy (OECD, 1963; 2002). As Leydesdorff
(2005) comments mission-driven research was largely a result of
WWII and it was ‘Vannevar Bush’s 1945 report to the US President
entitled The Endless Frontier contained a plea for a return to a liberal
organization of science’ based upon traditional peer review
mechanisms (p. 1510). Most national science agencies followed the
model of the US National Science foundation established in 1947
determining the eventual role that the federal government would
play in supporting fundamental research at universities. Of course
these developments were deeply implicated in Cold War politics
and the Sputnik shock of 1957, when the Soviet science model
seemed to produce better results.

19. The Emerging Political Economy of
Global Science
The countries occupying the top eight places in the science citation
rank order < produced about 84.5% of the top 1% most cited
publications between 1993 and 2001. The next nine countries
produced 13%, and the final group share 2.5%. There is a stark
disparity between the first and second divisions in the scientific
impact of nations. Moreover, although my analysis includes only 31
of the world’s 193 countries, these produce 97.5% of the world’s
most cited papers (p. 314).
The political implications of this last comparison are difficult to
exaggerate. South Africa, at 29th place in my rank ordering, is the
only African country on the list. The Islamic countries are only
represented by Iran at 30th, despite the high GDP of many of them
and the prominence of some individuals, such as Nobel
prizewinners Abdus Salam (physics, 1979) and Ahmed Zewail
(chemistry,1999) (p. 314).

20. Development of National Systems
of Research Quality
• UK’s RAE
• Australia’s RQF
• NZ’s PBRF
Peer review versus bibliometrics

21. 3. Peer Review, Bibliometrics & the Governance
of Science
The old ideal of science as autonomous and
independent of social conditions first gave way
to
‘Kuhn's disenchanted picture of 'normal' science
(Kuhn 1970). Finally, post-Feyerabend studies of
science re-examined the whole institution
without presupposing any privileged status in
relation to either virtue or natural knowledge’

22. Quality Assurance Replaces ‘Truth’ as Core Commitment of Post-
normal Science & the Case for ‘Extended Peer Review’
Quality assurance can thus be seen as a core commitment of post-
normal science, replacing ‘truth’ as science’s ultimate regulative
principle (Funtowicz and Ravetz 1992). Defined in terms of
uncertainties and decision-stakes, quality assurance encompasses
‘public interest’, ‘citizen’, and ‘vernacular’ sciences. In a period of
domination by globalized corporate science (Gibbons et al 1994),
this effort to make scientists accountable to interested groups
presents a coherent conceptual alternative for the survival of the
‘public knowledge’ tradition of science. Collegial peer review is
thereby transformed into review by an ‘extended peer community’.
This new form of quality-assurance will be given its formal
structure and routines by those who put it into practice.
Funtowicz (2001)

23. The Republican Model of Science
Governance
• ‘republicanism is ultimately a theory of
liberty’, an ‘explicit product of a legal
system’, that requires ‘significant civic
participation’ (Fuller,2002: 198)
• The challenge from Mode 2 knowledge
and a form of knowledge managerialism
now at the center of knowledge capitalism

24. Mode 2 Knowledge
• in Mode 1 problems are set and solved in a context governed by the (largely
academic) interests of a specific community. By contrast, in Mode 2
knowledge is produced in a context of application;
• Mode 1 is disciplinary while Mode 2 is transdisciplinary;
• Mode 1 is characterised by relative homogeneity of skills, Mode 2 by their
heterogeneity;
• in organisational terms, Mode 1 is hierarchical and, in academic life at least,
has tended to preserve its form, while in Mode 2 the preference is for flatter
hierarchies using organisational structures which are transient;
• in comparison with Mode 1, Mode 2 is more socially accountable and
reflexive.
• in comparison with Mode 1, Mode 2 involves a much expanded system of
quality control. Peer review still exists to be sure, but in Mode 2 it includes
a wider, more temporary and heterogeneous set of practitioners,
collaborating on a problem defined in a specific and localised context.
Gibbons et al (1994)

25. Research Quality in Mode 1
Quality in Mode 1 is determined essentially through the peer review
judgements about the contributions made by individuals. Control is
maintained by careful selection of those judged competent to act as
peers, which is in part determined by their previous contributions to
their discipline. In this way, the peer review process is one in which
quality and control mutually re-enforce one another. It has both
cognitive and social dimensions, in that there is professional control
over what problems and techniques are deemed important to work
on as well as who is qualified to pursue their solution. In
disciplinary science, peer review operates to channel individuals
towards work on problems judged to be central to the advance of
the discipline. These problems are defined largely in terms of
criteria which reflect the intellectual interests and pre-occupations of
the discipline and its gatekeepers

26. Research Quality in Mode 2
In Mode 2 additional criteria are added through the context of
application which now incorporates a diverse range of intellectual
interests as well as other social, economic or political ones. To the
criterion of intellectual interest and its interaction, further questions
are posed, ‘Will the solution, if found, be competitive in the market?
Will it be cost effective? Will it be socially acceptable?’ Mode 2
quality is determined by a wider set of criteria that reflects the
broadening social composition of the review system. This implies
that ‘good science’ becomes more difficult to determine. Since it is
no longer limited strictly to the judgements of disciplinary peers, the
fear is that control will be weaker and result in lower quality work.
Although the quality control process in Mode 2 is more broadly
based, it does not follow that because a wider range of expertise is
brought to bear on a problem that it will necessarily be of lower
quality. It is of a more composite, multi-dimensional kind (Gibbons,
1998: 9-10).

27. Accountability & Democracy
First, core concepts of democratic theory, such as
citizenship, deliberation, and accountability, cannot be
understood satisfactorily without taking on board the
politics of science and technology. Second, in all three
countries, policies for the life sciences have been
incorporated into "nation-building" projects that seek to
reimagine what the nation stands for. Third, political
culture influences democratic politics, and it works
through the institutionalized ways in which citizens
understand and evaluate public knowledge
(http://www.pupress.princeton.edu/titles/7958.html)

28. Jasanoff: The Cognitive Front
On the cognitive front, the shift is from a realist to a constructivist view of
knowledge. Years of work on the social construction of science and
technology, and the contingency of similarity and difference judgments,
have taught us to be skeptical of absolutist claims concerning objectivity
and progress. Scientific knowledge, it is now widely accepted, does not
simply accumulate, nor does technology invariably advance benign human
interests...
The methods with which policymakers carry on their business similarly
cannot be taken as neutral, but must be seen as the result of political
compromise and careful boundary maintenance, favoring some voices and
viewpoints at others’ expense. The criteria by which one measures policy
success or failure are likewise products of negotiation; in applying them,
one implicitly adopts contingent, locally specific standards of reliability and
validity. The special authority of scientific claims is in competition with
other representations of reality diffused through the global media, and
scientific expertise is subject to appropriation by multiple, diffracted social
identities and interests. Any attempt to compare the performance of
national policy systems today must take these complexities into account (p.
13).

29. Jasanoff: The Political Front
• On the political front, the shift is toward a fracturing of the
authority of nation-states, with consequent pressures to rethink the
forms of democratic governance. State sovereignty is eroding under
the onslaught of environmental change, financial and labor
mobility, increased communication, the global transfer of technical
skills and scientific knowledge, and the rise of transnational
organizations, multinational corporations, and social movements.
Supranational concerns, such as the demand for free trade or
globally sustainable development, are gaining political salience, but
they are at the same time encountering resistance from tendencies
toward greater local autonomy based on particularities of culture
and place. As a result, the ‚old‛ politics of modernity—with its core
values of rationality, objectivity, universalism, centralization, and
efficiency—is confronting, and possibly yielding to, a ‚new‛ politics
of pluralism, localism, irreducible ambiguity, and aestheticism in
matters of lifestyle and taste (p.14).

30. The Governance of Science
There are strong theoretical grounds for approaching this complex
question in a way that recognizes:
• the changing nature and status of science – including the history
and philosophy of science, and the relationship of science to politics
and technology;
• the changing architecture of world science and its place within an
emerging global knowledge economy and international knowledge
system;
• the changing relationship between natural and social science –
including the significance of the ‘science wars’;
• the disciplinary hegemony of neoclassical economics and associated
concepts in business and management sciences;
• the development of new information and communication
technologies that are changing the social ecology of academic
journals and academic publishing including the emergence of
bibliometrics and webometrics.

31. Limitations of Citation Analysis
Cultural and Language bias
• American and English-language bias
• Draws only on a small subset of the total number of
journals world wide
• Marginalizes local, indigenous and cultural knowledges
– all those that are not journalized and nominated by ISI
Discipline & Knowledge bias between Natural and Social
Sciences/Humanities
• Distributed knowledge system favors natural sciences
over social sciences and humanities (350,000 new science
citations weekly, 50,000 in social sciences and 15,000 in
humanities – Goodall, 2006)

32. Limitations (cont’)
Discipline & Knowledge bias (cont’)
• Ignores differences between knowledge production models and cycles of
natural sciences on the one hand and social sciences and humanities on
the other
• Differences in publication rhythms and lead times for article publication
with weekly journals in medicine and quarterlies, three or even two
journals publications in some humanities disciplines
• Differences in form of publication where natural science favors journal
articles and social sciences and humanities favor the monograph/book
(which is indicative of differences in research organization)
• Citation analysis does NOT take into account book citation or reviews, or
wider difference between research and scholarship
• Differences between natural and social sciences in co-authorship – up to
3 or 4 in the social sciences versus 10s and 20s in natural and life sciences
• Problems of comparisons across disciplines (lifetime citation scores differ
radically for top scientists in natural (15-20K) and social science (3-4K)
(Goodall, 2006)

33. Limitations (cont’)
Methodological Problems
• Authors receive citations for ‘bad’ as well as ‘good’ work
• Problems with accuracy, consistency and reliability in citation data collection
• Problems of self-citation
Political Problems
• Its original use designed to examine impact of scientific work not to measure quality; the
relationship between citation and quality is still not clear or well understood (Albach, 2006)
• Misuse of citations may encourage distortions in the international knowledge system favoring
developed over developing countries and influence the uneven distribution of research funds that
further consolidates western dominance
• Misuse of citations may encourage distortions in knowledge cultures at institutional and
departmental levels preventing openness, sharing, collegiality and trust and through excessive
competition promoting knowledge hoarding, secrecy and academic individualism.
Ownership and Governance problems
• Who owns and governs science? What is the relationship between ownership and governance?
How do existing arrangements square with the republic of science and with the enhancement of
public knowledge cultures? Who is the republic of science meant to serve? (These are political
questions that citation analysis as a set of techniques cannot answer but yet presuppose).

34. New Developments
• New information and communication technologies and the rapid changes that are
taking place in this area generates a range of new questions concerning the republic
of science, including not only those about the governance of open source, open
access, open course ware and free science (and the central question of what ‘freedom’
means here in relation to electronic academic publishing), but also a raft of new
characteristics of the Web 2.0 platform that is altering the digital learning and
research environments
• Web 2.0 as platform is said to operate differently from Web 1.0 and to be
distinguished by tagging not taxonomy, rich user experiences, user as contributor,
customer self-service, participation not publishing, radical trust (e.g. Wikipedia), and
decentralization.[1] Its oft cited ‘democratic’ feature where amateurs can surpass
professionals, ‘when they have the right kind of system to channel their efforts’, as
Paul Graham notes,[2] raises all sorts of issues and problems for universities, for the
republic of science, and for notions of research quality.
• These developments have the potential not only to change the communication of
science but also to radically alter its traditional institutions, its organization, its value
and the historical underlying ideologies of forbidden knowledge (Christian), of the
encyclopedic knowledge system (Englightenment secularism), and of the knowledge
economy (capitalist technoscience).