Presentation given in Paris, France on May 23 within a workshop organized by HCÉRES-OST.

1. The SSH conundrum
A matter of audiences?
Nicolas Robinson-Garcia

2. 1. Context
2. Framing the problem
3. Current applications
4. Evidences of misadjustment
5. Considerations
Agenda

3. SSH -> the source of bibliometricians and research
managers’ headaches:
•Tends to underperform in comparison with other
fields
•Less productive, less impact
•Perception on their purpose and usefulness is
influenced by limitations in its assessment
Context

4. These limitations are useful for some types of
arguments:
1. Limiting funding
2. Hiding mediocrity
Context

5. •No international core of SSH literature
•Multiple outputs
•Multiple purposes
•Lack of normalization in quality control
•Differing citation patterns
•SSH as a “basket concept”
Listing shortcomings

6. Quick revision
1. The four literatures of the Social Sciences (Hicks,
2005)
2. Audiences (Nederhof, 2006)
3. The rural vs. the urban (van Leeuwen et al., 2017
adapted from Becher & Trowler, 2002)
Framing the problem

7. The four literatures Hicks, 2005
International
journals
Books
National
literature
Non-scholarly
literature

8. The four literatures Hicks, 2005
International
journals
Books
National
literature
Non-scholarly
literature

9. The four literatures Hicks, 2005
•The argument is focused on scientists’ outputs
•Some overlap is acknowledged but it is presented as
problem of coverages
•It does not question the use of citations, at least not
explicitly

10. Audiences Nederhof, 2001
International
journals
Books
National
literature
Non-scholarly
literature

11. Audiences Nederhof, 2001
Three audiences
1. The global scientific communities
2. Local experts
3. The general public

12. The rural vs. the urban van Leeuwen et al. 2017
SSH vs. STEM
1. Publication culture
Differing outputs and references
2. Language of publication
Differing targeted outreach
3. Locality of topics
Differing targeted impact

13. Tweaks to correct misadjustments
•Addition of books and chapter to bibliometric
databases (e.g., Book Citation Index)
•Addition of new data sources (e.g., Google Books, national
databases)
•Addition of indicators (e.g., book reviews, library holdings,
altmetrics)
•Rankings (e.g., publishers’ prestige, citation impact)
Confronting the problem

14. Some examples from Spain
•Journals – Criteria based on inclusion in
international databases
•Citations remain as main proxy for impact (informally
others can be alleged)
•Publisher rankings – Elea is the expert!
•Other impact proxies for books – Book reviews,
translations
Confronting the problem

15. CASE I
Department A – Contemporary History
• Data source: Dialnet
• 93 Faculty
• 2,724 documents
Data kindly provided by Carlos B. Amat, derived from Cañibano et al. 2018
But still…

16. Outputs
But still…
Articles
Chapters
Books

17. Outreach (only journals)
International
Domestic
Regional
Institutional
Departmental
But still…

18. Outreach (only journals)
But still…
Publications in Scopus in WoS
Prof. A 288 2 9
Prof. B 207 2 14
Prof. C 124 0 0
Prof. D 100 4 0

19. CASE II
Monographs published by University B
• Data source: CRIS + PlumX + Web of Science
• Period 2010-2016
• 2,957 books
Torres-Salinas, Robinson-Garcia & Gorraiz, 2017
But still…

20. Impacts
+
Outputs
But still…

21. There are too many aspects to consider to be able to
address all of them with a single approach
•Audiences (geographical outreach, stakeholders)
•Outputs (books, journal articles, social media, press media)
•Impacts (educational, scientific, social)
But still…

22. Considerations
In a convoluted world, nothing lasts forever:
“library holdings seem to be the most promising proxy of scholarly
impact”
Torres-Salinas et al. 2018
For how long?
Will books continue being relevant?
Will they migrate to electronic format?

23. • Productivity scales as a means to determine what is good –
Benchmarking
• Greater capability on developing automatic methods –
Connecting APIs from different sources
• Great opportunities for developing national data sources
and unique instruments
• More creativity is necessary
• Sometimes less data is better than more data
• Micro analyses are context-driven
• Impact can be stakeholder-driven (analyze communities)
• Contextual factors may lead to identify spillovers
Considerations

24. One suggestion
Considerations
“The introduction of knowledge about the process into assessment
procedures will also help us to understand how (potential) social impact
is being achieved.”
Spaapen & Drooge, 2011
Researchers’ context as a proxy of engagement potential
Social networks of scientists as proxy for social outreach

25. Global
Local
Geographical distance
One suggestion
Considerations

26. One suggestion
Considerations
Public sector
NGO
Unknown
Academia
Private sector
Politicians
Institutional affiliation

27. •Evaluation vs. Monitoring
• Evaluation design must be driven by purpose and context
(better and more varied approaches and methods)
• Monitoring requires control and exhaustiveness
(better and more comprehensive sources)
•Combinations of creativity and empiricism
• Point scales must not be aleatory but evidence-based
• Translating similar methods from one field to another is not
enough
The end of universal approaches

28. Thank you!
Web: http://nrobinsongarcia.com
Email: elrobinster@gmail.com
Twitter: @nrobinsongarcia