Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Integrating research indicators for use in the repositories infrastructure
Similar to Integrating research indicators for use in the repositories infrastructure (20)
More from petrknoth
More from petrknoth (20)
Recently uploaded
Recently uploaded (20)
Integrating research indicators for use in the repositories infrastructure
- 1. Integrating research indicators for use in the repositories infrastructure Drahomira Herrmannova and Petr Knoth CORE Knowledge Media institute, The Open University United Kingdom
- 2. In this talk • Research indicators: what (and what not) for and why • Challenges in integrating indicators in the repositories infrastructure
- 3. In this talk • Research indicators: what (and what not) for and why • Challenges in integrating indicators in the repositories infrastructure
- 4. Problem • Biblio-, webo-, alt-metrics – controversial in research evaluation • But, they can be applied with measurable success in information retrieval and research analytics • Most repositories (and aggregators) do not make an effective use of these metrics yet
- 5. Freely available collections Citation data • Microsoft Academic Graph - free alternative to Scopus and WoS • Initiative for Open Citations (I4OC) Usage data • Altmetrics API • Mendeley API • IRUS • Others
- 6. Where can research indicators be applied 1. Enhanced information retrieval • Search • Recommender systems 2. Research analytics • Analysis of research trends • Identify areas of strength within institutions • Expert search • Analysis of research collaboration networks • Analysis of research argumentation
- 7. Result of not using indicators Repository and cross-repository information retrieval systems have poor performance (and no one wants to use them when in combination with metadata only indexing) Little or poor research analytics available
- 8. In this talk • Research indicators: what (and what not) for and why • Challenges in integrating indicators in the repositories infrastructure
- 9. Challenges in integrating these datasets with the repositories infrastructure? • Not a complete overlap to merge • DOI • Combination of fields • Size of the datasets • The process can be resource intensive/complex: • Beyond the ability of a typical repository • Role for aggregators
- 10. Challenges in integrating these datasets with the repositories infrastructure? • Indicators are changing all the time, but metadata and resources are not • Merge • Integrate (index)
- 11. Approaches to integration • Batch: Merge data and index once in a while • Continuous (streaming the changes): Integrate immediately as indicators change
- 12. Indexing No updates in indexing! • Update means delete and reinsert • Many reinserts cause the index to grow in size and become less optimal for retrieval => rebalance the index
- 13. Consequences Batch Continuous Always up to date Make efficient use of metrics in IR systems
- 14. The conflict Staying up to date vs System response
- 15. Is there a middle path? • Elasticsearch parent-child relationship • Children are stored as separate documents • Lightweight structure, can be recreated quickly • But … you still pay a price in performance
- 16. Conclusions • Research indicators currently widely (and wrongly) used in research evaluation have a significant potential in academic information retrieval and research analytics. • Technical challenges in integrating them: staying up to date vs system response • Parent-child indexing approaches offer a solution somewhere in the middle
Editor's Notes
- The current repository infrastructure, which consists of thousands of repositories, does not make effective use of research indicators largely exploited by commercial players in the area. Research indicators, including citation counts and Mendeley reader counts, enable the development and improvement of functionality researchers use on a daily basis. For example, they make it possible to increase the performance in information retrieval and recommendation tasks and serve as an enabler for the development of research analytics & metrics functionality, such as the analysis of research trends or collaboration networks. We believe that there is a strong case for making a better use of these indicators within the repositories infrastructure to improve the functionality of services users rely on.
- The current repository infrastructure, which consists of thousands of repositories, does not make effective use of research indicators largely exploited by commercial players in the area. Research indicators, including citation counts and Mendeley reader counts, enable the development and improvement of functionality researchers use on a daily basis. For example, they make it possible to increase the performance in information retrieval and recommendation tasks and serve as an enabler for the development of research analytics & metrics functionality, such as the analysis of research trends or collaboration networks. We believe that there is a strong case for making a better use of these indicators within the repositories infrastructure to improve the functionality of services users rely on.
- Using scientometrics in research evaluation is at least controversial However, we can
- The result of not using any of these metrics is that search within or across repositories is crap
- Enhanced information retrieval: Search: Research indicators can be used to provide improved search functionality by enabling search result ranking using publication popularity and utility. Recommender systems: Research indicators can be used to improve recommender systems in many ways, from ranking by publication popularity to recommending papers which were frequently mentioned together in the same context. Research analytics: 2 https://www.mendeley.com/ 3 https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/ 4 http://dev.mendeley.com/ 2 Analysis of research trends: Having access to research indicators enables the analysis of emergence and popularity of research trends over time. Analysis of research strengths within institutions: Research indicators can also provide useful when analysing the research outputs of universities, from productivity of different groups to analysing their popularity. Expert search Analysis of research collaboration networks: This can reveal many interesting patterns, for example which groups are well connected, which groups collaborate with researchers outside of their discipline, which groups are not aware of each other, etc. Analysis of research argumentation: Analysis of how do different sides of an argument evolve and gain on popularity, etc.
- The result of not using any of these metrics is that search within or across repositories is crap
- The current repository infrastructure, which consists of thousands of repositories, does not make effective use of research indicators largely exploited by commercial players in the area. Research indicators, including citation counts and Mendeley reader counts, enable the development and improvement of functionality researchers use on a daily basis. For example, they make it possible to increase the performance in information retrieval and recommendation tasks and serve as an enabler for the development of research analytics & metrics functionality, such as the analysis of research trends or collaboration networks. We believe that there is a strong case for making a better use of these indicators within the repositories infrastructure to improve the functionality of services users rely on.
- The current repository infrastructure, which consists of thousands of repositories, does not make effective use of research indicators largely exploited by commercial players in the area. Research indicators, including citation counts and Mendeley reader counts, enable the development and improvement of functionality researchers use on a daily basis. For example, they make it possible to increase the performance in information retrieval and recommendation tasks and serve as an enabler for the development of research analytics & metrics functionality, such as the analysis of research trends or collaboration networks. We believe that there is a strong case for making a better use of these indicators within the repositories infrastructure to improve the functionality of services users rely on.
- The current repository infrastructure, which consists of thousands of repositories, does not make effective use of research indicators largely exploited by commercial players in the area. Research indicators, including citation counts and Mendeley reader counts, enable the development and improvement of functionality researchers use on a daily basis. For example, they make it possible to increase the performance in information retrieval and recommendation tasks and serve as an enabler for the development of research analytics & metrics functionality, such as the analysis of research trends or collaboration networks. We believe that there is a strong case for making a better use of these indicators within the repositories infrastructure to improve the functionality of services users rely on.
- The current repository infrastructure, which consists of thousands of repositories, does not make effective use of research indicators largely exploited by commercial players in the area. Research indicators, including citation counts and Mendeley reader counts, enable the development and improvement of functionality researchers use on a daily basis. For example, they make it possible to increase the performance in information retrieval and recommendation tasks and serve as an enabler for the development of research analytics & metrics functionality, such as the analysis of research trends or collaboration networks. We believe that there is a strong case for making a better use of these indicators within the repositories infrastructure to improve the functionality of services users rely on.
- Update means delete and reinsert
- The current repository infrastructure, which consists of thousands of repositories, does not make effective use of research indicators largely exploited by commercial players in the area. Research indicators, including citation counts and Mendeley reader counts, enable the development and improvement of functionality researchers use on a daily basis. For example, they make it possible to increase the performance in information retrieval and recommendation tasks and serve as an enabler for the development of research analytics & metrics functionality, such as the analysis of research trends or collaboration networks. We believe that there is a strong case for making a better use of these indicators within the repositories infrastructure to improve the functionality of services users rely on.