The document discusses challenges with using publications and citations alone to understand the outcomes and impact of research funding. It notes the inherent time lag of 3-8 years for publications, and that the relationship between citations and socio-economic impact is uncertain. The document proposes looking at current funding inputs and outputs through tools that use machine learning to classify and map funding descriptions to standard taxonomies in order to gain insights into the current and future knowledge landscape without long time lags. It also suggests that altmetrics data on social sharing and researcher networks can provide early insights into impact. The overall ambition is to build a more complete network of funding, research outputs, and influential documents using machine learning to better understand research insights and impact more fully and faster

1. Work smart. Discover more.
Forward-looking analysis based on grants
data and machine learning-based research
classifications as an analytical tool
Michael Taylor
Head of Metrics Development

2. Work smart. Discover more.
Understanding the outcomes of policy
decisions through publication output and
citation is troublesome
- The link between publications as the output and the
source of funding is provided by the author or the
authors is not straightforward.
- Focusing on publications introduces an inherent time
lag of 3-8 years into any given analysis
- The relationship between citation and socio-economic
impact is not certain

3. Work smart. Discover more.
1) Focusing on publications introduces an
inherent time lag of 3-8 years into any
given analysis
- Research notwithstanding (and even hiring can be an
issue…), the publishing process can be very slow
- The stages of submission, peer-review, actual
publishing
- The lag before citations happen and are indexed
- What data can give us an insight into the current and
future knowledge landscape?

4. Work smart. Discover more.
If we can’t see the outcomes of research
funding, what can we see?
- One solution is to look at the current inputs
- Understanding what funding is being made
available now
- To understand what work is in focus elsewhere

5. Work smart. Discover more.
Building these forms of solution are not
trivial (Uberresearch.com)
- A stream of data with funding descriptions needs to be
collected
- The descriptions parsed and matched to standard
descriptions
- Analytical tools need to be built.

6. Work smart. Discover more.
Building these forms of solution are not
trivial (Altmetric.com)
- A stream of data with social links needs to be
collected
- The descriptions parsed and matched to standard
metadata
- Analytical tools need to be built.

7. Work smart. Discover more.
The stream of data…
- (typically) – non-standard, fast, massive, heuristic,
challenging to obtain
- For example, the UberResearch Dimensions funding
database details over $1T funds, of which $282B is
active now, 2.1M projects
- No global metadata standards (in contrast to
publishing standards)
- What are the implications for the analyst trying to
understand funding and research futures on the global
stage?

8. Work smart. Discover more.
Using machine-learning to map
taxonomies
- The UberResearch approach was to take multiple
funding agencies’ subject classifications
- To ‘learn’ about the content of them, and how they
map to their funding descriptions
- And then to apply the ‘lessons learnt’ to other funding
descriptions
- This allows an analyst to use their taxonomy to
understanding other peoples open descriptions
- From unstructured funding descriptions, machine-
learning enables structured analyses and insights

9. Work smart. Discover more.
2) The relationship between citation and
socio-economic impact is not certain
- The word ‘proxy’ is used a lot in scholarly metrics
- Journals are proxies for subject codes
- Journal metrics are proxies for quality and impact
- What does citation actually tell us?
- What does it miss?
- What other data is available that is closer to socio-
economic impact?

10. Work smart. Discover more.
Part of the solution to understanding
broader impact comes from altmetrics
- Saves on researcher networks gives us an early
insight into citation behaviour, and also an insight into
non-citing researchers’ intent
- Shares on social networks can give us an early insight
into social impact
- Not for ‘novelty papers’, but imagine an HIV
prevention NGO tweeting a link to primary research

11. Work smart. Discover more.
Other altmetrics are slower, but tell us
more about impact
- Mining influence and impact on research from grey
literature – policy papers, law, white papers – is
extremely difficult
- No standards, documents don’t last, very often it’s
“people not papers” that influence policy
- Finding how discourse around research spreads is
extremely difficult
- Defining research questions enables research
answers

12. Work smart. Discover more.
To summarise: our ambition…
- Is to build a more complete network of funding,
research outputs and data, of discourse and influential
documents that provides a fuller (and faster) view of
insight
- Machine-learning has been shown to create structure
and to enable sense from unstructured documents
- The hope is that similar techniques will enable to
create wider networks of influence than currently exist,
without having to wait for infrastructure and standards

13. Work smart. Discover more.
Thanks.
Any
questions?