This document discusses support programs in Turkey that aim to increase the number of scientific publications. It analyzes Turkey's Support Program for International Scholarly Publications, which was previously based on journal impact factors (JIFs) but now uses other metrics like Article Influence Scores. The author examines the impact of recent changes to the program's algorithms in 2013-2014 compared to pre-2012. Through analyzing a stratified sample of journals, the purpose is to understand the motives behind changes and their effects on determining the monetary support provided by TUBITAK.

1. Support
Programs
to
Increase
the
Number
of
Scien6fic
Publica6ons
Using
Bibliometric
Measures:
The
Turkish
Case
Yaşar
Tonta
Hace%epe
University
Department
of
Informa5on
Management
06800
Beytepe,
Ankara,
Turkey
yunus.hace%epe.edu.tr/~tonta/tonta.html
yasartonta@gmail.com
@yasartonta
ISSI
2015,
June
29-­‐July
3,
2015,
Boğaziçi
University,
İstanbul,
Turkey

2. Plan
• Use
of
bibliometric
measures
in
research
evalua5on
(JIF,
h-­‐index,
Ar5cle
Influence
Score)
• TUBITAK’s
Support
Program
of
Interna5onal
Scholarly
Publica5ons
• Method
• Findings
• Discussion
and
conclusions

3. Turkey
in
Brief
• 185
universi5es
• Over
5M
students
• 151K
faculty
• 7K
academic
book
5tles
published
p.a.
• Close
to
1,700
academic
journals
published
• About
400K
papers
in
total
in
WoS-­‐indexed
journals
Source:
h%ps://ista5s5k.yok.gov.tr/;
h%p://tuik.gov.tr/PreTablo.do?alt_id=1086;
Kozak,
2014;
h%p://dergipark.ulakbim.gov.tr;
h%p://webofscience.com

4. Year
#
of
papers
1990
1181
1991
1404
1992
1748
1993
2287
1994
2933
1995
3423
1996
4408
1997
5183
1998
6058
1999
6708
2000
6984
2001
8428
2002
10819
2003
13206
2004
16358
2005
17653
2006
20114
2007
24175
2008
25128
2009
28525
2010
29480
2011
30425
2012
32841
2013
35717
2014
34231
0
5000
10000
15000
20000
25000
30000
35000
40000
1990
1995
2000
2005
2010
2015
Number
of
papers
with
Turkish
affilia5ons
Source:
h%p://webofscience.com,
26
April
2015

5. Research
evalua5on
• Peer
review
• Academic
tenure
and
performance
• Research
and
publica5on
support
– E.g.,Research
Excellence
Framework
(REF)
– Publica5on
support
(e.g.,
TUBITAK,
universi5es)

6. Impact-­‐factor
obsession
Source:
Hicks
et
al.,
2015,
p.
431

7. Journal
Impact
Factor
(JIF)
• Cita5on
indexes
• Web
of
Science
(WoS)
• Journal
Cita5on
Reports
(JCR)
• Journal
Impact
Factor
– Developed
to
help
librarians
in
collec5on
development
– Average
number
of
cita5ons
to
papers
published
in
a
journal
(#
of
cita5ons
/
#
of
citable
items)

8. Drawbacks
of
cita5on-­‐based
metrics
• Skewed
cita5on
distribu5ons
– 43%
of
11,500
journals
listed
in
JCR
2012
have
JIF’s
between
0
and
1
• Different
publica5on
and
cita5on
prac5ces
in
different
disciplines
– Top
Economics
paper
received
60
cita5ons
whereas
top
Science
paper
received
more
than
1,000
cita5ons
between
2008-­‐2012
(hence
JIFs
vary)
• Changing
publishers
policies
– Web
of
Science
indexed
c.
8,000
journals
in
2006
and
11,500
in
2012
• “Gaming”
and
manipula5on
• “Bean
coun5ng”
• Do
not
to
use
“journal-­‐based
metrics
.
.
.
as
a
surrogate
measure
of
the
quality
.
.
.”
(San
Francisco,
2012)
• But
.
.
.
the
Higher
Educa5on
Council
and
TUBITAK
are
using
it
Sources:
Kaynak:
Al
ve
Soydal,
2014;
Casadevall
ve
Fang,
2014;
Hicks,
2015;
Seglen,
1997;
Sgroi
ve
Oswald,
2013,
s.
F256;
Tonta,
2014a,
2014b,
2015;
LSE,
2011,
s.
26;
SCImago,
2015,
h%p://www.scimagojr.com

9. TUBITAK’s
Support
Program
of
Int’l
Scholarly
Papers
• 1993-­‐2012:
based
on
JIFs
taken
from
JCR
– Journals
classified
as
A,
B,
C
and
D
based
on
their
JIFs
• 2013:
based
on
JIF5
and
cited
half-­‐life
(which
has
more
to
do
with
the
obsolescence
of
the
paper
than
its
quality)
• 2014:
based
on
Ar5cle
Influence
Score
(AIS),
which
measures
the
average
impact
of
a
paper
(similar
to
Google’s
PageRank
algorithm)
and
is
highly
correlated
with
JIF
Kaynak:
Arendt,
2010;
Franceschet,
2010;
Tonta,
2014a,
2014b,
2015

10. TUBITAK’s
Support
Program
of
Int’l
Scholarly
Papers
2011
2012
2013
2014
8.01
8.15
10.06
11.5
11,342
11,708
10,653
11,530
11,721
12,053
11,199
12,449
Teşvik
Tutarı
(Milyon
TL)
Yayın
Sayısı
Araşyrmacı
Sayısı
Source:
Satoğlu,
2015
h%p://193.140.98.158/wp-­‐content/uploads/2015/03/ULAKBIM_EYES-­‐Acilis.pptx
Amount
of
support
(MTL)
#
of
papers
#
of
researchers

11. Purpose
• To
examine
the
impact
of
TUBITAK’s
most
recent
algorithmic
changes
(2013-­‐2014)
and
compare
them
with
that
of
pre-­‐2012
• To
understand
the
mo5ves
behind
changes
and
their
effects
on
journal
scores
determining
the
amount
of
TUBITAK’s
monetary
support

12. Method
• Stra5fied
sample
– from
TUBITAK’s
list
of
2012
journals
(N=11,562)
– Strata:
journals
grouped
under
A
(36%),
B
(21%),
C
(41%),
and
D
(2%)
– One
third
were
Social
Science
journals
• Sample
size:
2%
(n=232)

13. Popula5on
parameters
&
sample
sta5s5cs

14. Findings

15. Amount
of
support
(in
Turkish
Lira)

16. Sca%er
of
journals
by
amount
of
TUBITAK
support
(2012-­‐2014)
Pearson’s
r’s
• 2012-­‐2013
r
=
0.29
• 2012-­‐2014
r
=
0.23
• 2013-­‐2014
r
=
0.77

17. Correla5ons
• Group
A
journals
of
2012
received
less
support
in
2013
and
2014
• Out
of
84
A
journals
in
2012,
only
15
(18%)
maintained
their
top
posi5ons
in
2013
and
2014
(12
Science
and
3
Social
Science
journals)
• No
discernible
pa%erns
between
the
amount
of
support
in
2012
and
the
succeeding
years
– 2012-­‐2013
r
=
0.29
(p
=
.000)
– 2012-­‐2014
r
=
0.23
(p
=
.000)
• Moderate
correla5on
between
2013
and
2014
– 2013-­‐2014
r
=
0.77
(p
=
.000)

18. Comparison
of
2013
and
2014
journals
!
r
=
0.77

19. Rela5onship
between
journal
score
and
the
amount
of
support
in
2013

20. Rela5onship
between
journal
score
and
the
amount
of
support
in
2014

21. TUBITAK’s
support
threshold
is
low
• Almost
all
WoS
indexed
journals
supported
• One
third
barely
meet
the
minimum
criteria
• Support
to
more
than
3,000
journals
can
be
discon5nued
• 80%-­‐90%
of
journals
received
less
than
2,500TL
• Only
5%
of
journals
received
more
than
4,000TL
• Social
Science
journals
are
worse

22. Amount
of
TUBITAK
support
for
Science
vs.
Social
Science
journals
(2014)

23. Discussion
• Journals
that
performed
poorly
in
2012
did
so
in
succeeding
years,
too
• 2013
and
2014
algorithms
do
not
differ
much
– 2013:
avg=701.00TL,
median꞊564.00TL
– 2014:
avg=770.00TL,
median꞊577.00TL
• Because
both
JIFs
and
AISs
are
based
on
the
number
of
cita5ons
and
highly
correlated
• TUBITAK
nullified
its
earlier
decision
of
not
suppor5ng
Group
C
Science
journals,
although
a
few
Group
C
Science
journals
performed
differently
in
2013
and
2014
• TUBITAK’s
support
does
not
seem
to
encourage
authors
to
publish
in
more
pres5gious
journals

24. Conclusions
• JIF:
“fatal
a%rac5on”,
“poor
man’s
cita5on
analysis”
• bibliometric
performance
measures
alone
are
not
the
sole
criteria
for
research
assessment
and
.
.
.
they
“should
be
applied
only
as
a
collec6ve
group
(and
not
individually)”
(IEEE,
2013,
original
emphasis)
• JIF
should
not
supplant
peer
review
but
support
it
• Different
publica5on/cita5on
prac5ces
by
different
fields
should
be
taken
into
account
• Bibliometric
measures
should
not
be
used
to
measure
the
quality
of
papers,
researchers
and
ins6tu6ons
and
compare
them
with
each
other
Source:
Van
Raan,
2005;
IEEE,
2013;
Hicks
et
al.,
2015;
Marx
&
Bornmann,
2013;
h%p://am.ascb.org/dora/

25. Epilogue
“Not
everything
that
counts
can
be
counted,
and
not
everything
that
can
be
counted
counts.”
-­‐-­‐
William
Bruce
Cameron,
1963
“When
a
measure
becomes
a
target,
it
ceases
to
be
a
good
measure.”
-­‐-­‐
Charles
Goodhart,
1975
Kaynak:
h%p://quoteinves5gator.com/2010/05/26/everything-­‐counts-­‐einstein/;
h%p://en.wikipedia.org/wiki/Goodhart%27s_law

26. Sources
• Al,
U.
&
Soydal.
İ.
(2014).
Akademinin
ayf
dizinleri
ile
savaşı.
Hace>epe
Üniversitesi
Edebiyat
Fakültesi
Dergisi,
31(1):
23-­‐42.
h%p://yunus.hace%epe.edu.tr/~umutal/publica5ons/war.pdf
• Arendt,
J.
(2010).
Are
ar5cle
influence
scores
comparable
across
scien5fic
fields?
Issues
in
Science
and
Technology
Librarianship,
No.
60.
h%p://www.istl.org/10-­‐winter/refereed2.html
• Casadevall,
A.
&
Fang,
F.C.
(2014).
Causes
for
the
persistence
of
impact
factor
mania.
mBio,
5(2).
tarihinde
h%p://mbio.asm.org/content/5/2/e00064-­‐14.full.pdf
• Centre
for
Science
and
Technology
Studies.
(2007).
Scoping
study
on
the
use
of
bibliometric
analysis
to
measure
the
quality
of
research
in
UK
higher
educa5on
ins5tu5ons.
Report
to
HEFCE.
Leiden:
Centre
for
Science
and
Technology
Studies,
Leiden
University.
• Franceschet,
M.
(2010).
Journal
influence
factors.
Journal
of
Informetrics,
4(3),
239-­‐248.
Retrieved,
January
26,
2015,
from
h%ps://users.dimi.uniud.it/~massimo.franceschet/publica5ons/joi10b.pdf
• Hicks,
D.,
Wouters,
P.,
Waltman,
L.,
de
Rijcke,
S.
&
Rafols,
I.
(2015,
23
Nisan).
The
Leiden
Manifesto
for
research
metrics.
Nature,
520,
429-­‐431.
h%p://www.nature.com/polopoly_fs/1.17351!/menu/main/topColumns/topLe}Column/pdf/520429a.pdf
• IEEE.
(2013,
September
9).
Appropriate
use
of
bibliometric
indicators
for
the
assessment
of
journals,
research
proposals,
and
individuals.
Retrieved,
April
7,
2015,
from
h%p://www.ieee.org/publica5ons_standards/publica5ons/rights/ieee_bibliometric_statement_sept_2013.pdf
• Kozak,
N.
(2014).
Türkiye
akademik
dergiler
rehberi-­‐2014.
Ankara:
Detay
Yayıncılık.
• LSE
Public
Policy
Group.
(Nisan
2011).
Maximizing
the
impacts
of
your
research:
A
handbook
for
social
scien5sts.
h%p://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf
• Marx,
W.
&
Bornmann,
L.
(2013).
Journal
Impact
Factor:
“the
poor
man’s
cita5on
analysis”
and
alterna5ve
approaches.
European
Science
EdiPng,
39(2),
62-­‐63.
h%p://www.ease.org.uk/sites/default/files/aug13pageslowres.pdf.
• Satoğlu,
M.M.
(2015,
4
Mart).
Bilimsel
yayınlarda
kaliteye
ulaşmak:
Editör
ve
yazar
eği5m
seminerleri.
h%p://193.140.98.158/wp-­‐content/uploads/2015/03/ULAKBIM_EYES-­‐Acilis.pptx
• SCImago.
(2015).
SJR
—
SCImago
Journal
&
Country
Rank.
h%p://www.scimagojr.com.
• Seglen,
P.O.
(1997,
Şubat
15).
Why
the
impact
factor
of
journals
should
not
be
used
for
evalua5ng
research.
Bri5sh
Medical
Journal,
314(7079):498-­‐502.
h%p://www.dcscience.net/seglen97.pdf
• Sgroi,
D.
&
Oswald,
A.J.
(2013).
How
should
peer-­‐review
panels
behave?
The
Economic
Journal,
123(570),
F255–F278.
• Tonta,
Y.
(2014a).
Akademik
Performans,
Öğre5m
Üyeliğine
Yükseltme
ve
Yayın
Destekleme
Ölçütleriyle
İlgili
Bir
Değerlendirme
(yayımlanmamış
makale).
h%p://yunus.hace%epe.edu.tr/~tonta/yayinlar/tonta-­‐yukseltme-­‐kriterleri-­‐hakkinda-­‐degerlendirme-­‐11-­‐Temmuz-­‐2014.pdf.
• Tonta,
Y.
(2014b).
Use
and
Misuse
of
Bibliometric
Measures
for
Assessment
of
Academic
Performance,
Tenure
and
Publica5on
Support.
Metrics
2014:
Workshop
on
Informetric
and
Scientometric
Research
(SIG/MET).
77th
Annual
MeePng
of
the
AssociaPon
for
InformaPon
Science
and
Technology,
October
31-­‐November
5,
2014,
Sea>le,
WA.
Bildirinin
tam
metni:
h%p://bit.ly/1ur3cGN;
Slaytlar:
h%p://slidesha.re/10VuVTf
• Tonta,
Y.
(2015).
Support
Programs
to
Increase
the
Number
of
Scien5fic
Publica5ons
Using
Bibliometric
Measures:
The
Turkish
Case
(bildiri).
ISSI
2015,
June
29-­‐July
4,
2015,
İstanbul.
• Van
Raan,
A.F.J.
(2005).
Fatal
a%rac5on:
conceptual
and
methodological
problems
in
the
ranking
of
universi5es
by
bibliometric
methods.
Scientometrics,
62(1),
133–43.

27. Support
Programs
to
Increase
the
Number
of
Scien6fic
Publica6ons
Using
Bibliometric
Measures:
The
Turkish
Case
Yaşar
Tonta
Hace%epe
University
Department
of
Informa5on
Management
06800
Beytepe,
Ankara,
Turkey
yunus.hace%epe.edu.tr/~tonta/tonta.html
yasartonta@gmail.com
@yasartonta
ISSI
2015,
June
29-­‐July
3,
2015,
Boğaziçi
University,
İstanbul,
Turkey