EC3metrics ha estado presente por primera vez en 2014 en la European Summer School for Scientometrics, la escuela de verano internacional que ofrece formación bibliométrica especializada cada año a 50 alumnos llegados desde todo el mundo.Nuestro compañero Álvaro Cabezas participó en dicho foro con una intervención acerca de las ventajas y limitaciones de Google Scholar, en una sesión dedicada a la evaluación de las Ciencias Sociales y Humanidades junto a Henk Moed, Philip Purnell, y Juan Gorráiz. En su intervención, Álvaro revisó los distintos productos de índole bibliométrica de Google Scholar mostrando sus puntos fuertes y débiles. Animó a los asistentes a experimentar con estos productos, si bien siendo conscientes de las precauciones que hay que tomar al usarlos con fines evaluativos.

1. Google Scholar as a research and
evaluation toolevaluation tool
Álvaro Cabezas-Clavijo

2. INDEX
• Goals
• GS as a research tool
– Search engine
– Alerts
– Updates
– Library + Cite
• GS as a tool for evaluating research• GS as a tool for evaluating research
– GS Citations (or Profiles)
– GS Metrics
– Bibliometric analysis
• Data manipulation
• Comparison to WOS/ Scopus
• Conclusions

3. To show GS possibilities to do research
To show GS possibilities to evaluate research
To point out strengths and weaknesses of GS as a
source to perform bibliometric analysis
GOALS
To introduce Google Scholar as a valid source for
carrying out research assessments (with
caution!) in Social Sciences and Humanities (SSH)

4. GS AS A TOOL FOR RESEARCHERSGS AS A TOOL FOR RESEARCHERS

5. • GS is a widespread tool for doing research
• Comprehensive source of academic literature
• Widely known by scholars and students
• Ease of use (just like google)
GS AS A TOOL FOR RESEARCHERS
•
• Free and fast
• Links to primary source (when it is freely available on the
web or can be accessed from our institution)

6. • Non-selective source (covers all material which meet the
technical requirements)
• No restrictions on language
• No restrictions on document type
• No discipline/areas bias
GS AS A TOOL FOR RESEARCHERS (II)
• No discipline/areas bias
• Estimated size is 100 million documents100 million documents (Web of Science
50 million; Scopus 53 million)

7. GS AS A TOOL FOR RESEARCHERS (III)

8. • GS is not just an academic search engine but a
“family” of tools for doing and evaluating research
GS AS A TOOL FOR RESEARCHERS (IV)
ALERTS UPDATES
LIBRARY
+ CITE

9. • Set up by user just clicking on the “Create alert” link when
performing a search
• Delivered by 2-4 days if there is fresh info to show
• Authors or thematic searches can be configured (but not for
journals)
ALERTS
journals)
1 2
3

10. • Configure an alert for citations to particular papers or authors
(in this case, if they have already set up a profile)
ALERTS
Following
Christian’s new
papers or
citations

11. • Alerts about papers of our interest automatically generated
by Google
• You only need to set up a GS profile. Algorithm takes into
account:
– Who we cite in our papers
– Who cite us in their papers
UPDATES
– Who cite us in their papers
– Titles of our papers
– Co-authors
– Content
• A red bell indicates the number of new papers that might be
of your interest

12. UPDATESSome paper recommendations
Ok, I have published about
these topics, so I may be
interested in these papers
Ok, this paper cites me so it should
be related to my research

13. • Kind of reference manager
• Save references in one click
• Arrange references by using labels
• Edit references & export them (but one by one!)
• Can’t import references from other bibliographic software
LIBRARY + CITE
• Can’t import references from other bibliographic software
•Cite easily from GS
results: formats
MLA, APA, Chicago
•Get the formatted
reference and
copy/paste it into a
Bibliography.

14. • By default; 2 folders:
My citations (actually
My papers) and Cited
by me
• You can set up other
LIBRARY + CITE
• You can set up other
labels/folders and
save new references
from GS searches

15. GS as a tool for evaluating researchGS as a tool for evaluating research

16. • GS provides different tools which can help in
assessments of individuals, journals and more
GS AS A TOOL FOR EVALUATORS (II)
PROFILES METRICS
SOURCE FOR
BIBLIOMETRIC ANALYSES

17. • Individual Web page with publication list and basic
bibliometric indicators
• Papers are ranked (by default) according to their number
of citations
• Also links to co-authors (if they have set up a profile) and
PROFILES
• Also links to co-authors (if they have set up a profile) and
keywordskeywords Ranking of researchers
by keywords/areas are
generated through this
classification
TIP: there is always a field where you
can be the most cited author! ☺

18. • Easy to set up and edit
• Strongly recommended for young researchers as it can
increase their visibility
• You will get an alert (maximum 3 times a week) every time
you are cited
PROFILES
• 72,579 researchers with profile (March 2012)
(Estimates by Ortega & Aguillo, 2013)
• GS profiles-derived products
– Some Spanish institutions are ranking their researchers
according to GSC (Univ. Granada, CSIC,…) stimulus to set up
a profile?

21. COMMUNICATION: 886 RESEARCHERS

22. SCIENTOMETRICS: 265 RESEARCHERS

23. CSIC RANKING OF RESEARCHERS
ACCORDING TO GSC

24. • Easy to set up (just search for
your papers)
• Terrific tool for comparing
researchers within a field or
department
• Automatically updated basic
bibliometric data
PROFILES
• Data can be easily manipulated
(a researcher can self-claim
non-authored papers)
• Can stimulate vanity and ego
• Can generate no fair
comparisons (for example,bibliometric data
• No restrictions on source,
language or area
• Everyone can measure their
performance (and their
colleagues')
comparisons (for example,
researchers from different
areas in a single univ)
• Can generate unfair analysis by
non-bibliometric experts
STRENGTHS WEAKNESSES

25. • GS service devoted to measure journal’s impact according to their
h-index (est. April 2012)
• Rankings are provided by language (nine languages) areas (8) and
subcategories (313) (just for English-language journals) and
journal title’s words (i.e, oncology)
• Subject/area rankings are limited to top 20 journals and language
METRICS
• Subject/area rankings are limited to top 20 journals and language
rankings are limited to top 100 journals.
• Calculations take into account output for 5 years (currently
2008-2012) and citations received until july 2013. A new update is
expected to be released in July 2014.
• Journals with fewer than 100 papers published in 5 years or with
no citation (h-index=0) are excluded
• arXiv and other repositories’ collections are also included

28. • Free product to compare and rank
journals
• We can get impact information about
non-JCR journals and about national
and SSH publications
• Transparency: citations for every
paper that contributes to the h-index
can be checked
• Methodological inconsistencies such
as comparing journals from
different areas
• Lack of proper bibliographic control
(duplicates, “dirty” data,…)
• No selection criteria
• No action against data manipulation
METRICS
can be checked
• High correlation with JCR Impact
factor (0.82)
• Simple to use and understand by any
scholar
• Can be easily replicated
• No action against data manipulation
• No info about new updates (expected
to be updated yearly)
• Just top results are presented
• H index favours journals with high
production
STRENGTHS WEAKNESSES

29. • GS is estimated to contain about 100 million documents
(87% of scholarly papers on the web) Khabsa & Giles
(2014)
• So, it is the largest source for bibliometric analysis that we
can use
SOURCE FOR
BIBLIOMETRIC
ANALISYS
can use
• GS-based products can also be used to help with research
assessment
– Publish or Perish (software for bibliometric analysis)
– H-Index Scholar (ranking of spanish researchers)
– Google Chrome add-ons
– …

30. PUBLISH OR PERISH

31. H-INDEX SCHOLAR

32. GOOGLE CHROME ADD-ONS

33. • Many things to sort out (unknown coverage,
duplicates, “dirty” data) but GS can be useful for:
Non-covered areas by WoS…
Non-English research…
SOURCE FOR
BIBLIOMETRIC
ANALISYS
Non-English research…
…Which means basically Social Sciences and
Humanities
institutions and individuals which lack access to
WoS/Scopus

34. GS data can be easily manipulated
• Just need to upload an “academic” paper to a institutional
repository, university webpage or similar
• “Academic” means a paper with title, authors, abstract and of
course, bibliographic references
• GS will count the citations from these fake papers, modifying• GS will count the citations from these fake papers, modifying
indicators of all cited scholars and journals.
Delgado, Robinson &
Torres (2014)

36. • Small diferences between rankings generated by GS, WoS and
Scopus
• Intensive Care Medicine Spearman correlation of 0.99 for
journals and 0.93 for top researchers
Comparison to Wos / Scopus
JOURNALSJOURNALS
Cabezas & Delgado (2013)
• Spearman correlation for Communication Journals: 0.895
(GSM-WoS); 0.879 (GSM-Scopus)
• Spearman correlation for Economics & Business Journals:
0.718 (GS-WoS)
JOURNALS
Delgado & Repiso (2013)
Harzing & der Wal (2009)

37. • Easy to use, fast and free
• Largest source for assessment
purposes
• Provides information about areas
and materials not covered by
• Lack of bibliographic control
• Data can be manipulated
• No possibilities to massive info
download
• Time-consuming, data needs to
SOURCE FOR
BIBLIOMETRIC
ANALISYS
and materials not covered by
traditional bibliometric products
• Rankings are very similar to
those offered by WoS / Scopus
• Possibility to build new products
based on GS data
• Time-consuming, data needs to
be “cleaned”
• GS and their services could be
closed down in any moment
STRENGTHS WEAKNESSES

38. ConclusionsConclusions

39. • Bibliometricians need to find other products to complement
WoS / Scopus-based evaluations (GS, ERIH, national
products,…)
• What if subscriptions to traditional databases are cancelled?
– Spain paying 4-6 million Euros to Thomson Reuters yearly for
national access to WoS from public research organizations
CONCLUSIONS
– Spain paying 4-6 million Euros to Thomson Reuters yearly for
national access to WoS from public research organizations
• SSH deserve to be measured (prominent rankings such as
Shanghai or Leiden ignore Humanities and a huge part of
Social Sciences!)
• GS tools are far from perfect but provide us with some
valuable information

40. • Bibliometricians take usually the “easy way”; only measure
what can be easily measured with databases such as WoS or
Scopus
• A challenge for bibliometricians is to get the most out of the
existing tools to evaluate SSH (and to develop new ones!)
• GS can be used as a complementary or alternative source to
CONCLUSIONS
• GS can be used as a complementary or alternative source to
shed some light on SSH
• Use with extreme caution, especially when assessing
individuals
• Keep an eye on the future: Institutions might be demanding
GS-based analysis soon.

41. Further readingsFurther readings

42. • Delgado-López-Cózar, E., & Cabezas-Clavijo, Á. (2013). Ranking journals: could Google
Scholar Metrics be an alternative to Journal Citation Reports and Scimago Journal
Rank?. Learned publishing, 26(2), 101-113.
• Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google
Scholar Experiment: how to index false papers and manipulate bibliometric indicators.
Journal of the Association for Information Science and Technology, 65(3), 446-454.
• Harzing, A. W., & Van Der Wal, R. (2009). A Google Scholar h-index for journals: An
FURTHER READINGS
• Harzing, A. W., & Van Der Wal, R. (2009). A Google Scholar h-index for journals: An
alternative metric to measure journal impact in economics and business. Journal of
the American Society for Information Science and Technology, 60(1), 41-46.
• Khabsa, M., & Giles, C. L. (2014). The Number of Scholarly Documents on the Public
Web. PloS one, 9(5), e93949.
• Ortega, J. L., & Aguillo, I. F. (2013). Institutional and country collaboration in an online
service of scientific profiles: Google Scholar Citations. Journal of Informetrics, 7(2),
394-403.

43. • BLOG: Google Scholar Digest: analysis of papers dealing with GS
http://googlescholardigest.blogspot.com.es/
• TWITTER: Follow the #googlescholar hashtag
• SOFTWARE: Publish or Perish http://www.harzing.com/pop.htm
• PRODUCTS:
FURTHER MATERIALS
• PRODUCTS:
• GOOGLE SCHOLAR CITATIONS http://scholar.google.com/citations
• GOOGLE SCHOLAR METRICS
http://scholar.google.com/citations?view_op=top_venues&hl=en

44. Google Scholar as a research and
evaluation toolevaluation tool
Álvaro Cabezas-Clavijo
Thank you!