SlideShare a Scribd company logo
1 of 27
Download to read offline
1/26
Semantometrics: Towards full text-based
research evaluation
Petr Knoth and Drahomira Herrmannova
Semantometrics.org
2/26
Towards full-text based research metrics:
Exploring semantometrics
13th June 2016: Announcement of
the report release:
https://scholarlyfutures.jiscinvolve.or
g/wp/2016/06/towards-full-text-
based-research-metrics-exploring-
semantometrics/
Report available at:
http://repository.jisc.ac.uk/6376/1/Ji
sc-semantometrics-experiments-
report-final.pdf
3/26
Current impact metrics
• Pros: simplicity
• Cons: insufficient evidence they capture quality and research
contribution, ad-hoc/established axiomatically
4/26
The crisis of research evaluation?
Figure: Rejection rates vs Journal Impact Factor (JIF) according to (da Silva, 2015).
5/26
Problems of current impact metrics
• Sentiment, semantics, context and motives [Nicolaisen, 2007]
• Popularity and size of research communities [Brumback, 2009;
Seglen, 1997]
• Time delay [Priem and Hemminger, 2010]
• Skewness of the distribution [Seglen, 1992]
• Differences between types of research papers [Seglen, 1997]
• Ability to game/manipulate citations [Arnold and Fowler,
2010; PLoS Medicine Editors, 2006]
6/26
Alternative metrics
• Alt-/Webo-metrics etc.
– Impact still dependent on the number of interactions in a
scholarly communication network (downloads, views,
readers, tweets, etc.)
7/26
Semantometrics
Contribution to the discipline assessed by using the article
manuscript
8/26
Many possibilities for semantometrics …
• Detecting good research practices were followed
(sound methodology, research data/code shared
…)
• Detecting paper type …
• Analysing citation contexts (tracking facts
propagation) …
• Detecting the sentiment of citations …
• Normalising by size of community that is likely to
read the research …
• Detecting good writing style …
9/26
Semantometrics – contribution metric
Hypothesis: Added value of publication p can be estimated
based on the semantic distance from the publications cited by p
to publications citing p.
Detailed explanation: http://semantometrics.org
10/26
Contribution metric
• Based on semantic distance between citing
and cited publications
– Cited publications – state-of-the-art in the domain
of the publication in question
– Citing publications – areas of application
11/26
Contribution metric – a practical example
• Below- and above-average publication
12/26
Experiment – dataset
• Obtained by merging three open datasets:
– Connecting Repositories (CORE) – OA publications,
metadata and full-texts
– Microsoft Academic Graph (MAG) – citation
network
– Mendeley – publication texts (abstracts) and
readership information
• Over 1.6 million CORE publications, over 12
million publications in total
13/26
Experiment – dataset statistics
Articles from CORE matched with MAG 1,655,835
Average number of received citations 16.09
Standard deviation 66.30
Max number of received citations 13,979
Average readership 15.94
Standard deviation 42.17
Max readership 15,193
Average contribution value 0.89
Standard deviation 0.0810
Total number of publications 12,075,238
14/26
Experiment – dataset statistics
• Citation and readership distribution
15/26
Experiments – dataset statistics
• Contribution distribution
16/26
Experiment – dataset statistics
• Relation between citations and readership
17/26
Experiment – results
• No direct correlation between contribution
measure and citations/readership
• When working with mean citation, readership
and contribution values a clear behavioral
trend emerges
18/26
Experiment – results
• Relation between mean contribution and
citations
19/26
Experiment – results
• Relation between mean contribution and
readership
20/26
Current impact metrics vs semantometrics
Unaffected by Current impact metrics Semantometrics
Citation sentiment, semantics, context,
motives
✗ ✔
Popularity & size of res. communities ✗ ✔
Time delay ✗ ✗/✔*
Skewness of the citation distribution ✗ ✔
Differences between types of res. papers ✗ ✔
Ability to game/manipulate the metrics ✗ ✗/✔**
* reduced to 1 citation
** assuming that self-citations are not taken into account
12
21/26
Metrics for evaluating article sets
• Encourage focusing on quality rather than
quantity
• Comparable regardless of discipline, seniority,
etc.
22/26
Evaluating research metrics
• Need for a data driven approach
– Ground truth
– Human judgments
– Many facets of performance (societal impact,
economical impact, rigour, originality/novelty)
23/26
WSDM Cup – work on new metrics
• The goal of the challenge is to assess the
query-independent importance of scholarly
articles, using data from the Microsoft
Academic Graph (>120M papers).
• Human judgements
• But no full text in MAG
24/26
Dataset for semantometrics research
By connecting MAG, CORE and Mendeley data, we
have a dataset to study semantometrics.
25/26
Conclusions
• Full-text necessary for research evaluation
• Semantometrics are a new class of methods.
• We are studying one semantometric method
to assess the research contribution
• Need for a data driven approach for evaluating
metrics
26/26
References
• Jeppe Nicolaisen. 2007. Citation Analysis. Annual Review of
Information Science and Technology, 41(1):609-641.
• Douglas N Arnold and Kristine K Fowler. 2010. Nefarious
numbers. Notices of the American Mathematical Society,
58(3):434-437.
• Roger A Brumback. 2009. Impact factor wars: Episode V -- The
Empire Strikes Back. Journal of child neurology, 24(3):260-2,
March.
• The PLoS Medicine Editors. 2006. The impact factor game.
PLoS medicine, 3(6), June.
27/26
References
• Jason Priem and Bradely M. Hemminger. 2010. Scientometrics
2.0: Toward new metrics of scholarly impact on the social
Web. First Monday, 15(7), July.
• Per Ottar Seglen. 1992. The Skewness of Science. Journal of
the American Society for Information Science, 43(9):628-638,
October.
• Per Ottar Seglen. 1997. Why the impact factor of journals
should not be used for evaluating research. BMJ: British
Medical Journal, 314(February):498-502.

More Related Content

What's hot

DORA and the reinvention of research assessment
DORA and the reinvention of research assessmentDORA and the reinvention of research assessment
DORA and the reinvention of research assessmentMark Patterson
 
Responsible metrics: One size doesn't fit all
Responsible metrics: One size doesn't fit allResponsible metrics: One size doesn't fit all
Responsible metrics: One size doesn't fit allLudo Waltman
 
TeachingWithData.org -- Faculty Presentation
TeachingWithData.org -- Faculty PresentationTeachingWithData.org -- Faculty Presentation
TeachingWithData.org -- Faculty PresentationICPSR
 
Data in The Classroom: It's Not Just for Nerds Anymore!
Data in The Classroom:  It's Not Just for Nerds Anymore!Data in The Classroom:  It's Not Just for Nerds Anymore!
Data in The Classroom: It's Not Just for Nerds Anymore!ICPSR
 
European Trends in the Use of Evaluations and Indicators for Performance-Base...
European Trends in the Use of Evaluations and Indicators for Performance-Base...European Trends in the Use of Evaluations and Indicators for Performance-Base...
European Trends in the Use of Evaluations and Indicators for Performance-Base...ThomsonReutersMktg
 
Ppt chalkidis gkioulekas.pptx
Ppt chalkidis gkioulekas.pptxPpt chalkidis gkioulekas.pptx
Ppt chalkidis gkioulekas.pptxIliadis Dimitrios
 
Prediction-Improving Early Warning Systems With Categorized Course Resource U...
Prediction-Improving Early Warning Systems With Categorized Course Resource U...Prediction-Improving Early Warning Systems With Categorized Course Resource U...
Prediction-Improving Early Warning Systems With Categorized Course Resource U...Beste Ulus
 
SES grad student presentation
SES grad student presentationSES grad student presentation
SES grad student presentationLynda Kellam
 
P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...
P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...
P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...gehirntumora
 
ICPSR Data Services
ICPSR Data ServicesICPSR Data Services
ICPSR Data ServicesICPSR
 
Using altmetrics in an academic library
Using altmetrics in an academic libraryUsing altmetrics in an academic library
Using altmetrics in an academic libraryLucy Lambe
 
Overview of data collection methods
Overview of data collection methodsOverview of data collection methods
Overview of data collection methodsUlster BOCES
 
Ipn metodika konference_pardubice_sivertsen
Ipn metodika konference_pardubice_sivertsenIpn metodika konference_pardubice_sivertsen
Ipn metodika konference_pardubice_sivertsenMEYS, MŠMT in Czech
 
JCDL2014_HAN_XU_pptx
JCDL2014_HAN_XU_pptxJCDL2014_HAN_XU_pptx
JCDL2014_HAN_XU_pptxHan Xu, PhD
 
Learning the ABCs of Tracking APCs
Learning the ABCs of Tracking APCsLearning the ABCs of Tracking APCs
Learning the ABCs of Tracking APCsErin Calhoun
 

What's hot (20)

DORA and the reinvention of research assessment
DORA and the reinvention of research assessmentDORA and the reinvention of research assessment
DORA and the reinvention of research assessment
 
Student Survey
Student SurveyStudent Survey
Student Survey
 
test
testtest
test
 
Responsible metrics: One size doesn't fit all
Responsible metrics: One size doesn't fit allResponsible metrics: One size doesn't fit all
Responsible metrics: One size doesn't fit all
 
TeachingWithData.org -- Faculty Presentation
TeachingWithData.org -- Faculty PresentationTeachingWithData.org -- Faculty Presentation
TeachingWithData.org -- Faculty Presentation
 
00952
0095200952
00952
 
Data in The Classroom: It's Not Just for Nerds Anymore!
Data in The Classroom:  It's Not Just for Nerds Anymore!Data in The Classroom:  It's Not Just for Nerds Anymore!
Data in The Classroom: It's Not Just for Nerds Anymore!
 
European Trends in the Use of Evaluations and Indicators for Performance-Base...
European Trends in the Use of Evaluations and Indicators for Performance-Base...European Trends in the Use of Evaluations and Indicators for Performance-Base...
European Trends in the Use of Evaluations and Indicators for Performance-Base...
 
Concurso
ConcursoConcurso
Concurso
 
Ppt chalkidis gkioulekas.pptx
Ppt chalkidis gkioulekas.pptxPpt chalkidis gkioulekas.pptx
Ppt chalkidis gkioulekas.pptx
 
Prediction-Improving Early Warning Systems With Categorized Course Resource U...
Prediction-Improving Early Warning Systems With Categorized Course Resource U...Prediction-Improving Early Warning Systems With Categorized Course Resource U...
Prediction-Improving Early Warning Systems With Categorized Course Resource U...
 
SES grad student presentation
SES grad student presentationSES grad student presentation
SES grad student presentation
 
P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...
P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...
P.D.F Essentials of Biostatistics in Public Health ^Full.Pages By Lisa M. Sul...
 
ICPSR Data Services
ICPSR Data ServicesICPSR Data Services
ICPSR Data Services
 
Using altmetrics in an academic library
Using altmetrics in an academic libraryUsing altmetrics in an academic library
Using altmetrics in an academic library
 
Overview of data collection methods
Overview of data collection methodsOverview of data collection methods
Overview of data collection methods
 
Adams, "Profiles, not Metrics; Why it is important to drill into the data tha...
Adams, "Profiles, not Metrics; Why it is important to drill into the data tha...Adams, "Profiles, not Metrics; Why it is important to drill into the data tha...
Adams, "Profiles, not Metrics; Why it is important to drill into the data tha...
 
Ipn metodika konference_pardubice_sivertsen
Ipn metodika konference_pardubice_sivertsenIpn metodika konference_pardubice_sivertsen
Ipn metodika konference_pardubice_sivertsen
 
JCDL2014_HAN_XU_pptx
JCDL2014_HAN_XU_pptxJCDL2014_HAN_XU_pptx
JCDL2014_HAN_XU_pptx
 
Learning the ABCs of Tracking APCs
Learning the ABCs of Tracking APCsLearning the ABCs of Tracking APCs
Learning the ABCs of Tracking APCs
 

Viewers also liked

Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...
Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...
Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...Nancy Pontika
 
Thomson Reuters - Análise 360º do panorama de crescimento do Open Access
Thomson Reuters - Análise 360º do panorama de crescimento do Open AccessThomson Reuters - Análise 360º do panorama de crescimento do Open Access
Thomson Reuters - Análise 360º do panorama de crescimento do Open Accessb-on
 
Secondary research referencing guide
Secondary research referencing guideSecondary research referencing guide
Secondary research referencing guideChloeKyri
 
Research impact metrics for librarians: calculation & context
Research impact metrics for librarians: calculation & contextResearch impact metrics for librarians: calculation & context
Research impact metrics for librarians: calculation & contextLibrary_Connect
 
Qr Codes - the Business Card of Tomorrow
Qr Codes - the Business Card of TomorrowQr Codes - the Business Card of Tomorrow
Qr Codes - the Business Card of TomorrowMartin Ebner
 
Talking Walls, Hidden Clues, QRCodes in the Wild
Talking Walls, Hidden Clues, QRCodes in the WildTalking Walls, Hidden Clues, QRCodes in the Wild
Talking Walls, Hidden Clues, QRCodes in the WildBonnie Roalsen
 
QR Codes - fad or fab - at Alt Becta Successful Deployment Workshop
QR Codes - fad or fab - at Alt Becta Successful Deployment WorkshopQR Codes - fad or fab - at Alt Becta Successful Deployment Workshop
QR Codes - fad or fab - at Alt Becta Successful Deployment Workshopandyramsden
 
Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...
Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...
Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...Guus van den Brekel
 
Tay- Starting research from outside the library homepage, an analysis of user...
Tay- Starting research from outside the library homepage, an analysis of user...Tay- Starting research from outside the library homepage, an analysis of user...
Tay- Starting research from outside the library homepage, an analysis of user...IFLA_InfolitRef
 
Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...
Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...
Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...OAccsd
 
State of Blockchain Q4 2016
State of Blockchain Q4 2016State of Blockchain Q4 2016
State of Blockchain Q4 2016CoinDesk
 
Windkraftanlage.
Windkraftanlage.Windkraftanlage.
Windkraftanlage.guest92a081
 
Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...
Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...
Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...OAccsd
 
Mes statistiques dans HAL
Mes statistiques dans HALMes statistiques dans HAL
Mes statistiques dans HALOAccsd
 
Déposer dans HAL mode d'emploi
Déposer dans HAL mode d'emploiDéposer dans HAL mode d'emploi
Déposer dans HAL mode d'emploiOAccsd
 
IdHAL et CV
IdHAL et CVIdHAL et CV
IdHAL et CVOAccsd
 
Rechercher dans HAL et exploiter les résultats
Rechercher dans HAL et exploiter les résultatsRechercher dans HAL et exploiter les résultats
Rechercher dans HAL et exploiter les résultatsOAccsd
 
Déposer une thèse dans TEL ou HAL
Déposer une thèse dans TEL ou HALDéposer une thèse dans TEL ou HAL
Déposer une thèse dans TEL ou HALOAccsd
 

Viewers also liked (19)

Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...
Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...
Implementation of the RIOXX Metadata Guidelines in the UK's repositories thro...
 
Thomson Reuters - Análise 360º do panorama de crescimento do Open Access
Thomson Reuters - Análise 360º do panorama de crescimento do Open AccessThomson Reuters - Análise 360º do panorama de crescimento do Open Access
Thomson Reuters - Análise 360º do panorama de crescimento do Open Access
 
Secondary research referencing guide
Secondary research referencing guideSecondary research referencing guide
Secondary research referencing guide
 
Research impact metrics for librarians: calculation & context
Research impact metrics for librarians: calculation & contextResearch impact metrics for librarians: calculation & context
Research impact metrics for librarians: calculation & context
 
Qr Codes - the Business Card of Tomorrow
Qr Codes - the Business Card of TomorrowQr Codes - the Business Card of Tomorrow
Qr Codes - the Business Card of Tomorrow
 
Talking Walls, Hidden Clues, QRCodes in the Wild
Talking Walls, Hidden Clues, QRCodes in the WildTalking Walls, Hidden Clues, QRCodes in the Wild
Talking Walls, Hidden Clues, QRCodes in the Wild
 
QR Codes - fad or fab - at Alt Becta Successful Deployment Workshop
QR Codes - fad or fab - at Alt Becta Successful Deployment WorkshopQR Codes - fad or fab - at Alt Becta Successful Deployment Workshop
QR Codes - fad or fab - at Alt Becta Successful Deployment Workshop
 
Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...
Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...
Do Libraries Meet Research 2.0 : collaborative tools and relevance for Resear...
 
Tay- Starting research from outside the library homepage, an analysis of user...
Tay- Starting research from outside the library homepage, an analysis of user...Tay- Starting research from outside the library homepage, an analysis of user...
Tay- Starting research from outside the library homepage, an analysis of user...
 
Research and Referencing
Research and ReferencingResearch and Referencing
Research and Referencing
 
Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...
Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...
Carrefour IST : Episciences.org, une nouvelle dimension pour les archives ouv...
 
State of Blockchain Q4 2016
State of Blockchain Q4 2016State of Blockchain Q4 2016
State of Blockchain Q4 2016
 
Windkraftanlage.
Windkraftanlage.Windkraftanlage.
Windkraftanlage.
 
Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...
Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...
Bib2Hal, import par lot dans HAL à partir d'un fichier BibTeX -formation CCSD...
 
Mes statistiques dans HAL
Mes statistiques dans HALMes statistiques dans HAL
Mes statistiques dans HAL
 
Déposer dans HAL mode d'emploi
Déposer dans HAL mode d'emploiDéposer dans HAL mode d'emploi
Déposer dans HAL mode d'emploi
 
IdHAL et CV
IdHAL et CVIdHAL et CV
IdHAL et CV
 
Rechercher dans HAL et exploiter les résultats
Rechercher dans HAL et exploiter les résultatsRechercher dans HAL et exploiter les résultats
Rechercher dans HAL et exploiter les résultats
 
Déposer une thèse dans TEL ou HAL
Déposer une thèse dans TEL ou HALDéposer une thèse dans TEL ou HAL
Déposer une thèse dans TEL ou HAL
 

Similar to Semantometrics: Towards Fulltext-based Research Evaluation

Classification of Researcher's Collaboration Patterns Towards Research Perfor...
Classification of Researcher's Collaboration Patterns Towards Research Perfor...Classification of Researcher's Collaboration Patterns Towards Research Perfor...
Classification of Researcher's Collaboration Patterns Towards Research Perfor...Nur Hazimah Khalid
 
Predicting student performance using aggregated data sources
Predicting student performance using aggregated data sourcesPredicting student performance using aggregated data sources
Predicting student performance using aggregated data sourcesOlugbenga Wilson Adejo
 
Data analytics to support awareness and recommendation
Data analytics to support awareness and recommendationData analytics to support awareness and recommendation
Data analytics to support awareness and recommendationKatrien Verbert
 
Journal Club Prophylactic Intra-abdominal Drainage.pptx
Journal Club Prophylactic Intra-abdominal Drainage.pptxJournal Club Prophylactic Intra-abdominal Drainage.pptx
Journal Club Prophylactic Intra-abdominal Drainage.pptxarundev1231
 
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...NAP Global Network
 
TeachingWithData.org ASA Presentation 2010
TeachingWithData.org ASA Presentation 2010TeachingWithData.org ASA Presentation 2010
TeachingWithData.org ASA Presentation 2010ICPSR
 
How to make impact with journal publications on Software Process ImprovementH...
How to make impact with journal publications on Software Process ImprovementH...How to make impact with journal publications on Software Process ImprovementH...
How to make impact with journal publications on Software Process ImprovementH...Torgeir Dingsøyr
 
Mixed research methodology.pptx
Mixed research methodology.pptxMixed research methodology.pptx
Mixed research methodology.pptxPuneethKumarGB
 
Looking to the future: closing the gaps in our assessment approach
Looking to the future: closing the gaps in our assessment approachLooking to the future: closing the gaps in our assessment approach
Looking to the future: closing the gaps in our assessment approachORCID, Inc
 
Scholarly Metrics in Specialized Settings
Scholarly Metrics in Specialized SettingsScholarly Metrics in Specialized Settings
Scholarly Metrics in Specialized SettingsElaine Lasda
 
An Introduction to Evaluation in Medical Visualization
An Introduction to Evaluation in Medical VisualizationAn Introduction to Evaluation in Medical Visualization
An Introduction to Evaluation in Medical VisualizationNoeska Smit
 
Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...
Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...
Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...Platforma Otwartej Nauki
 
Mouton Dfid Presentaion 14 September2009
Mouton Dfid Presentaion  14 September2009Mouton Dfid Presentaion  14 September2009
Mouton Dfid Presentaion 14 September2009CRUOnline
 
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...NAP Global Network
 
peer 1 Traditionally, there are three branches of methodology q.docx
peer 1 Traditionally, there are three branches of methodology q.docxpeer 1 Traditionally, there are three branches of methodology q.docx
peer 1 Traditionally, there are three branches of methodology q.docxbartholomeocoombs
 
Aligning Learning Analytics with Classroom Practices & Needs
Aligning Learning Analytics with Classroom Practices & NeedsAligning Learning Analytics with Classroom Practices & Needs
Aligning Learning Analytics with Classroom Practices & NeedsSimon Knight
 
Altmetrics : Rodrigo Costas Comesaña
Altmetrics : Rodrigo Costas ComesañaAltmetrics : Rodrigo Costas Comesaña
Altmetrics : Rodrigo Costas ComesañaUrfistpacac
 

Similar to Semantometrics: Towards Fulltext-based Research Evaluation (20)

Classification of Researcher's Collaboration Patterns Towards Research Perfor...
Classification of Researcher's Collaboration Patterns Towards Research Perfor...Classification of Researcher's Collaboration Patterns Towards Research Perfor...
Classification of Researcher's Collaboration Patterns Towards Research Perfor...
 
Predicting student performance using aggregated data sources
Predicting student performance using aggregated data sourcesPredicting student performance using aggregated data sources
Predicting student performance using aggregated data sources
 
Data analytics to support awareness and recommendation
Data analytics to support awareness and recommendationData analytics to support awareness and recommendation
Data analytics to support awareness and recommendation
 
mixed_methods.pdf
mixed_methods.pdfmixed_methods.pdf
mixed_methods.pdf
 
Journal Club Prophylactic Intra-abdominal Drainage.pptx
Journal Club Prophylactic Intra-abdominal Drainage.pptxJournal Club Prophylactic Intra-abdominal Drainage.pptx
Journal Club Prophylactic Intra-abdominal Drainage.pptx
 
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...
 
TeachingWithData.org ASA Presentation 2010
TeachingWithData.org ASA Presentation 2010TeachingWithData.org ASA Presentation 2010
TeachingWithData.org ASA Presentation 2010
 
How to make impact with journal publications on Software Process ImprovementH...
How to make impact with journal publications on Software Process ImprovementH...How to make impact with journal publications on Software Process ImprovementH...
How to make impact with journal publications on Software Process ImprovementH...
 
Mixed research methodology.pptx
Mixed research methodology.pptxMixed research methodology.pptx
Mixed research methodology.pptx
 
Looking to the future: closing the gaps in our assessment approach
Looking to the future: closing the gaps in our assessment approachLooking to the future: closing the gaps in our assessment approach
Looking to the future: closing the gaps in our assessment approach
 
Scholarly Metrics in Specialized Settings
Scholarly Metrics in Specialized SettingsScholarly Metrics in Specialized Settings
Scholarly Metrics in Specialized Settings
 
ALA 2016 NISO Altmetrics
ALA 2016 NISO AltmetricsALA 2016 NISO Altmetrics
ALA 2016 NISO Altmetrics
 
An Introduction to Evaluation in Medical Visualization
An Introduction to Evaluation in Medical VisualizationAn Introduction to Evaluation in Medical Visualization
An Introduction to Evaluation in Medical Visualization
 
Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...
Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...
Linking Heterogeneous Scholarly Data Sources in an Interoperable Setting: the...
 
Mouton Dfid Presentaion 14 September2009
Mouton Dfid Presentaion  14 September2009Mouton Dfid Presentaion  14 September2009
Mouton Dfid Presentaion 14 September2009
 
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...
 
Carpenter/Lagace: NISO Recommended Practices to Support Adoption of Altmetric...
Carpenter/Lagace: NISO Recommended Practices to Support Adoption of Altmetric...Carpenter/Lagace: NISO Recommended Practices to Support Adoption of Altmetric...
Carpenter/Lagace: NISO Recommended Practices to Support Adoption of Altmetric...
 
peer 1 Traditionally, there are three branches of methodology q.docx
peer 1 Traditionally, there are three branches of methodology q.docxpeer 1 Traditionally, there are three branches of methodology q.docx
peer 1 Traditionally, there are three branches of methodology q.docx
 
Aligning Learning Analytics with Classroom Practices & Needs
Aligning Learning Analytics with Classroom Practices & NeedsAligning Learning Analytics with Classroom Practices & Needs
Aligning Learning Analytics with Classroom Practices & Needs
 
Altmetrics : Rodrigo Costas Comesaña
Altmetrics : Rodrigo Costas ComesañaAltmetrics : Rodrigo Costas Comesaña
Altmetrics : Rodrigo Costas Comesaña
 

More from petrknoth

Qui Bono? Cumulative advantage in open access publishing
Qui Bono? Cumulative advantage in open access publishingQui Bono? Cumulative advantage in open access publishing
Qui Bono? Cumulative advantage in open access publishingpetrknoth
 
OAI Identifiers: Decentralised PIDs for Research Outputs in Repositories
OAI Identifiers: Decentralised PIDs for Research Outputs in RepositoriesOAI Identifiers: Decentralised PIDs for Research Outputs in Repositories
OAI Identifiers: Decentralised PIDs for Research Outputs in Repositoriespetrknoth
 
UKRI OA policy requirements for repositories and how to meet them
UKRI OA policy requirements for repositories and how to meet themUKRI OA policy requirements for repositories and how to meet them
UKRI OA policy requirements for repositories and how to meet thempetrknoth
 
Enabling Educators to Locate High-Quality Teaching Resources
Enabling Educators to LocateHigh-Quality Teaching ResourcesEnabling Educators to LocateHigh-Quality Teaching Resources
Enabling Educators to Locate High-Quality Teaching Resourcespetrknoth
 
Tracking compliance of the REF2021 policy with the CORE Repository Dashboard
Tracking compliance of the REF2021 policy with the CORE Repository DashboardTracking compliance of the REF2021 policy with the CORE Repository Dashboard
Tracking compliance of the REF2021 policy with the CORE Repository Dashboardpetrknoth
 
Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...petrknoth
 
CORE Analytics Dashboard
CORE Analytics DashboardCORE Analytics Dashboard
CORE Analytics Dashboardpetrknoth
 
Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...petrknoth
 
Analysing the performance of open access papers discovery tools
Analysing the performance of open access papers discovery toolsAnalysing the performance of open access papers discovery tools
Analysing the performance of open access papers discovery toolspetrknoth
 
Assessing Compliance with the UK REF 2021 Open Access Policy
Assessing Compliance with the UK REF 2021 Open Access PolicyAssessing Compliance with the UK REF 2021 Open Access Policy
Assessing Compliance with the UK REF 2021 Open Access Policypetrknoth
 
Data interoperability toolkit (OpenMinTeD)
Data interoperability toolkit (OpenMinTeD)Data interoperability toolkit (OpenMinTeD)
Data interoperability toolkit (OpenMinTeD)petrknoth
 
Integrating research indicators for use in the repositories infrastructure
Integrating research indicators for use in the repositories infrastructure Integrating research indicators for use in the repositories infrastructure
Integrating research indicators for use in the repositories infrastructure petrknoth
 
Towards effective research recommender systems for repositories
Towards effective research recommender systems for repositoriesTowards effective research recommender systems for repositories
Towards effective research recommender systems for repositoriespetrknoth
 
COAR Next Generation Repositories WG - Text mining and Recommender system sto...
COAR Next Generation Repositories WG - Text mining and Recommender system sto...COAR Next Generation Repositories WG - Text mining and Recommender system sto...
COAR Next Generation Repositories WG - Text mining and Recommender system sto...petrknoth
 
Seamless access to the world’s open access research papers via ResourceSync
Seamless access to the world’s open access research papers via ResourceSyncSeamless access to the world’s open access research papers via ResourceSync
Seamless access to the world’s open access research papers via ResourceSyncpetrknoth
 
Aggregating Research papers from Publishers' Systems to Support Text and Data...
Aggregating Research papers from Publishers' Systems to Support Text and Data...Aggregating Research papers from Publishers' Systems to Support Text and Data...
Aggregating Research papers from Publishers' Systems to Support Text and Data...petrknoth
 
My repository is being aggregated: a blessing or a curse?
My repository is being aggregated: a blessing or a curse?My repository is being aggregated: a blessing or a curse?
My repository is being aggregated: a blessing or a curse?petrknoth
 
FOSTER - Content Delivery (WP3)
FOSTER - Content Delivery (WP3)FOSTER - Content Delivery (WP3)
FOSTER - Content Delivery (WP3)petrknoth
 
Towards an Infrastructure for Mining Scientific Publications
Towards an Infrastructure for Mining Scientific PublicationsTowards an Infrastructure for Mining Scientific Publications
Towards an Infrastructure for Mining Scientific Publicationspetrknoth
 

More from petrknoth (20)

Qui Bono? Cumulative advantage in open access publishing
Qui Bono? Cumulative advantage in open access publishingQui Bono? Cumulative advantage in open access publishing
Qui Bono? Cumulative advantage in open access publishing
 
CORE APIv3
CORE APIv3CORE APIv3
CORE APIv3
 
OAI Identifiers: Decentralised PIDs for Research Outputs in Repositories
OAI Identifiers: Decentralised PIDs for Research Outputs in RepositoriesOAI Identifiers: Decentralised PIDs for Research Outputs in Repositories
OAI Identifiers: Decentralised PIDs for Research Outputs in Repositories
 
UKRI OA policy requirements for repositories and how to meet them
UKRI OA policy requirements for repositories and how to meet themUKRI OA policy requirements for repositories and how to meet them
UKRI OA policy requirements for repositories and how to meet them
 
Enabling Educators to Locate High-Quality Teaching Resources
Enabling Educators to LocateHigh-Quality Teaching ResourcesEnabling Educators to LocateHigh-Quality Teaching Resources
Enabling Educators to Locate High-Quality Teaching Resources
 
Tracking compliance of the REF2021 policy with the CORE Repository Dashboard
Tracking compliance of the REF2021 policy with the CORE Repository DashboardTracking compliance of the REF2021 policy with the CORE Repository Dashboard
Tracking compliance of the REF2021 policy with the CORE Repository Dashboard
 
Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...
 
CORE Analytics Dashboard
CORE Analytics DashboardCORE Analytics Dashboard
CORE Analytics Dashboard
 
Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...Better together: building services for public good on top of content from the...
Better together: building services for public good on top of content from the...
 
Analysing the performance of open access papers discovery tools
Analysing the performance of open access papers discovery toolsAnalysing the performance of open access papers discovery tools
Analysing the performance of open access papers discovery tools
 
Assessing Compliance with the UK REF 2021 Open Access Policy
Assessing Compliance with the UK REF 2021 Open Access PolicyAssessing Compliance with the UK REF 2021 Open Access Policy
Assessing Compliance with the UK REF 2021 Open Access Policy
 
Data interoperability toolkit (OpenMinTeD)
Data interoperability toolkit (OpenMinTeD)Data interoperability toolkit (OpenMinTeD)
Data interoperability toolkit (OpenMinTeD)
 
Integrating research indicators for use in the repositories infrastructure
Integrating research indicators for use in the repositories infrastructure Integrating research indicators for use in the repositories infrastructure
Integrating research indicators for use in the repositories infrastructure
 
Towards effective research recommender systems for repositories
Towards effective research recommender systems for repositoriesTowards effective research recommender systems for repositories
Towards effective research recommender systems for repositories
 
COAR Next Generation Repositories WG - Text mining and Recommender system sto...
COAR Next Generation Repositories WG - Text mining and Recommender system sto...COAR Next Generation Repositories WG - Text mining and Recommender system sto...
COAR Next Generation Repositories WG - Text mining and Recommender system sto...
 
Seamless access to the world’s open access research papers via ResourceSync
Seamless access to the world’s open access research papers via ResourceSyncSeamless access to the world’s open access research papers via ResourceSync
Seamless access to the world’s open access research papers via ResourceSync
 
Aggregating Research papers from Publishers' Systems to Support Text and Data...
Aggregating Research papers from Publishers' Systems to Support Text and Data...Aggregating Research papers from Publishers' Systems to Support Text and Data...
Aggregating Research papers from Publishers' Systems to Support Text and Data...
 
My repository is being aggregated: a blessing or a curse?
My repository is being aggregated: a blessing or a curse?My repository is being aggregated: a blessing or a curse?
My repository is being aggregated: a blessing or a curse?
 
FOSTER - Content Delivery (WP3)
FOSTER - Content Delivery (WP3)FOSTER - Content Delivery (WP3)
FOSTER - Content Delivery (WP3)
 
Towards an Infrastructure for Mining Scientific Publications
Towards an Infrastructure for Mining Scientific PublicationsTowards an Infrastructure for Mining Scientific Publications
Towards an Infrastructure for Mining Scientific Publications
 

Recently uploaded

Interpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTInterpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTAlexander F. Mayer
 
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPRPirithiRaju
 
Anatomy of Duodenum- detailed ppt presentation for science students
Anatomy of Duodenum- detailed ppt presentation for science studentsAnatomy of Duodenum- detailed ppt presentation for science students
Anatomy of Duodenum- detailed ppt presentation for science studentsgargipatil3210
 
Advances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of CancerAdvances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of CancerLuis Miguel Chong Chong
 
ISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.ppt
ISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.pptISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.ppt
ISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.pptKARTHIK REDDY C A
 
Mycobacterium Mycobacterium tuberculosis
Mycobacterium Mycobacterium tuberculosisMycobacterium Mycobacterium tuberculosis
Mycobacterium Mycobacterium tuberculosisKARTHIK REDDY C A
 
Total Legal: A “Joint” Journey into the Chemistry of Cannabinoids
Total Legal: A “Joint” Journey into the Chemistry of CannabinoidsTotal Legal: A “Joint” Journey into the Chemistry of Cannabinoids
Total Legal: A “Joint” Journey into the Chemistry of CannabinoidsMarkus Roggen
 
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasBACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasChayanika Das
 
Physical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptxPhysical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptxchetanvgh
 
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptxPharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptxeAuditor Audits & Inspections
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaDr.Mahmoud Abbas
 
Environmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptxEnvironmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptxpriyankatabhane
 
Remote patient monitoring :Health care transformation
Remote patient monitoring :Health care transformationRemote patient monitoring :Health care transformation
Remote patient monitoring :Health care transformationfahad Alotaibiu
 
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep LearningCombining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learningvschiavoni
 
Density and Buoyancy for Grade 8 Science Class
Density and Buoyancy for Grade 8 Science ClassDensity and Buoyancy for Grade 8 Science Class
Density and Buoyancy for Grade 8 Science Class276318345hao
 
chapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Pointchapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Pointchelseakland
 
FBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxFBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxPayal Shrivastava
 

Recently uploaded (20)

Interpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTInterpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWST
 
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
 
Anatomy of Duodenum- detailed ppt presentation for science students
Anatomy of Duodenum- detailed ppt presentation for science studentsAnatomy of Duodenum- detailed ppt presentation for science students
Anatomy of Duodenum- detailed ppt presentation for science students
 
Advances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of CancerAdvances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of Cancer
 
ISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.ppt
ISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.pptISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.ppt
ISOLATION OF ALGAE (INDOOR) AND PHOTOBIOREACTOR K R.ppt
 
Mycobacterium Mycobacterium tuberculosis
Mycobacterium Mycobacterium tuberculosisMycobacterium Mycobacterium tuberculosis
Mycobacterium Mycobacterium tuberculosis
 
Total Legal: A “Joint” Journey into the Chemistry of Cannabinoids
Total Legal: A “Joint” Journey into the Chemistry of CannabinoidsTotal Legal: A “Joint” Journey into the Chemistry of Cannabinoids
Total Legal: A “Joint” Journey into the Chemistry of Cannabinoids
 
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasBACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
 
Physical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptxPhysical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptx
 
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptxPharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
 
Environmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptxEnvironmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptx
 
Remote patient monitoring :Health care transformation
Remote patient monitoring :Health care transformationRemote patient monitoring :Health care transformation
Remote patient monitoring :Health care transformation
 
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep LearningCombining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
 
Density and Buoyancy for Grade 8 Science Class
Density and Buoyancy for Grade 8 Science ClassDensity and Buoyancy for Grade 8 Science Class
Density and Buoyancy for Grade 8 Science Class
 
TOPIC OF ELECTROMAGNETISM.PHYSICS NOTES FORM 4
TOPIC OF ELECTROMAGNETISM.PHYSICS NOTES FORM 4TOPIC OF ELECTROMAGNETISM.PHYSICS NOTES FORM 4
TOPIC OF ELECTROMAGNETISM.PHYSICS NOTES FORM 4
 
chapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Pointchapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Point
 
Biopreservation .pptx
Biopreservation                    .pptxBiopreservation                    .pptx
Biopreservation .pptx
 
Introduction Classification Of Alkaloids
Introduction Classification Of AlkaloidsIntroduction Classification Of Alkaloids
Introduction Classification Of Alkaloids
 
FBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxFBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptx
 

Semantometrics: Towards Fulltext-based Research Evaluation

  • 1. 1/26 Semantometrics: Towards full text-based research evaluation Petr Knoth and Drahomira Herrmannova Semantometrics.org
  • 2. 2/26 Towards full-text based research metrics: Exploring semantometrics 13th June 2016: Announcement of the report release: https://scholarlyfutures.jiscinvolve.or g/wp/2016/06/towards-full-text- based-research-metrics-exploring- semantometrics/ Report available at: http://repository.jisc.ac.uk/6376/1/Ji sc-semantometrics-experiments- report-final.pdf
  • 3. 3/26 Current impact metrics • Pros: simplicity • Cons: insufficient evidence they capture quality and research contribution, ad-hoc/established axiomatically
  • 4. 4/26 The crisis of research evaluation? Figure: Rejection rates vs Journal Impact Factor (JIF) according to (da Silva, 2015).
  • 5. 5/26 Problems of current impact metrics • Sentiment, semantics, context and motives [Nicolaisen, 2007] • Popularity and size of research communities [Brumback, 2009; Seglen, 1997] • Time delay [Priem and Hemminger, 2010] • Skewness of the distribution [Seglen, 1992] • Differences between types of research papers [Seglen, 1997] • Ability to game/manipulate citations [Arnold and Fowler, 2010; PLoS Medicine Editors, 2006]
  • 6. 6/26 Alternative metrics • Alt-/Webo-metrics etc. – Impact still dependent on the number of interactions in a scholarly communication network (downloads, views, readers, tweets, etc.)
  • 7. 7/26 Semantometrics Contribution to the discipline assessed by using the article manuscript
  • 8. 8/26 Many possibilities for semantometrics … • Detecting good research practices were followed (sound methodology, research data/code shared …) • Detecting paper type … • Analysing citation contexts (tracking facts propagation) … • Detecting the sentiment of citations … • Normalising by size of community that is likely to read the research … • Detecting good writing style …
  • 9. 9/26 Semantometrics – contribution metric Hypothesis: Added value of publication p can be estimated based on the semantic distance from the publications cited by p to publications citing p. Detailed explanation: http://semantometrics.org
  • 10. 10/26 Contribution metric • Based on semantic distance between citing and cited publications – Cited publications – state-of-the-art in the domain of the publication in question – Citing publications – areas of application
  • 11. 11/26 Contribution metric – a practical example • Below- and above-average publication
  • 12. 12/26 Experiment – dataset • Obtained by merging three open datasets: – Connecting Repositories (CORE) – OA publications, metadata and full-texts – Microsoft Academic Graph (MAG) – citation network – Mendeley – publication texts (abstracts) and readership information • Over 1.6 million CORE publications, over 12 million publications in total
  • 13. 13/26 Experiment – dataset statistics Articles from CORE matched with MAG 1,655,835 Average number of received citations 16.09 Standard deviation 66.30 Max number of received citations 13,979 Average readership 15.94 Standard deviation 42.17 Max readership 15,193 Average contribution value 0.89 Standard deviation 0.0810 Total number of publications 12,075,238
  • 14. 14/26 Experiment – dataset statistics • Citation and readership distribution
  • 15. 15/26 Experiments – dataset statistics • Contribution distribution
  • 16. 16/26 Experiment – dataset statistics • Relation between citations and readership
  • 17. 17/26 Experiment – results • No direct correlation between contribution measure and citations/readership • When working with mean citation, readership and contribution values a clear behavioral trend emerges
  • 18. 18/26 Experiment – results • Relation between mean contribution and citations
  • 19. 19/26 Experiment – results • Relation between mean contribution and readership
  • 20. 20/26 Current impact metrics vs semantometrics Unaffected by Current impact metrics Semantometrics Citation sentiment, semantics, context, motives ✗ ✔ Popularity & size of res. communities ✗ ✔ Time delay ✗ ✗/✔* Skewness of the citation distribution ✗ ✔ Differences between types of res. papers ✗ ✔ Ability to game/manipulate the metrics ✗ ✗/✔** * reduced to 1 citation ** assuming that self-citations are not taken into account 12
  • 21. 21/26 Metrics for evaluating article sets • Encourage focusing on quality rather than quantity • Comparable regardless of discipline, seniority, etc.
  • 22. 22/26 Evaluating research metrics • Need for a data driven approach – Ground truth – Human judgments – Many facets of performance (societal impact, economical impact, rigour, originality/novelty)
  • 23. 23/26 WSDM Cup – work on new metrics • The goal of the challenge is to assess the query-independent importance of scholarly articles, using data from the Microsoft Academic Graph (>120M papers). • Human judgements • But no full text in MAG
  • 24. 24/26 Dataset for semantometrics research By connecting MAG, CORE and Mendeley data, we have a dataset to study semantometrics.
  • 25. 25/26 Conclusions • Full-text necessary for research evaluation • Semantometrics are a new class of methods. • We are studying one semantometric method to assess the research contribution • Need for a data driven approach for evaluating metrics
  • 26. 26/26 References • Jeppe Nicolaisen. 2007. Citation Analysis. Annual Review of Information Science and Technology, 41(1):609-641. • Douglas N Arnold and Kristine K Fowler. 2010. Nefarious numbers. Notices of the American Mathematical Society, 58(3):434-437. • Roger A Brumback. 2009. Impact factor wars: Episode V -- The Empire Strikes Back. Journal of child neurology, 24(3):260-2, March. • The PLoS Medicine Editors. 2006. The impact factor game. PLoS medicine, 3(6), June.
  • 27. 27/26 References • Jason Priem and Bradely M. Hemminger. 2010. Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, 15(7), July. • Per Ottar Seglen. 1992. The Skewness of Science. Journal of the American Society for Information Science, 43(9):628-638, October. • Per Ottar Seglen. 1997. Why the impact factor of journals should not be used for evaluating research. BMJ: British Medical Journal, 314(February):498-502.

Editor's Notes

  1. In this presentation, I would like to talk about the problems of existing impact metrics. These problems led us to the development of a new approach, based on the processing of publication full-texts, for automatically assesing the research contribution.
  2. As you might know, there is a wide range of metrics that are currently used for evaluation of research. You can see them in the foloowing tag cloud. These metrics have one thing in common. They are all based on citations. This has both advantages and disdvanatages.They are … Let me say a few more things about the cons Evidence for citations comes externally. Does it mean that people who cite a paper need to read it? Not necessarily.
  3. As you might know, there is a wide range of metrics that are currently used for evaluation of research. You can see them in the foloowing tag cloud. These metrics have one thing in common. They are all based on citations. This has both advantages and disdvanatages.They are … Let me say a few more things about the cons Evidence for citations comes externally. Does it mean that people who cite a paper need to read it? Not necessarily.
  4. There is no correlation. Does it mean that citation counting does not work, does it mean that peer-review does not work? Probably a bit of both.
  5. There are a number of properties a good research evaluation metric should have. (Jennings, 2006) mentions the following: Reliability – Level of accuracy comparable or better than that of the current peer-review system. Digestibility – It must allow to make quick decisions about what to read. Economical and fast to produce – The production of the metric should be possible in reasonable time and shouldn’t be expensive or human labour intensive. Resistance to gaming – It shouldn’t be possible to improve the metric in any other way than by improving the research.
  6. There is now also a lot of interest in new metrics, which are referred to as Alt and Webometrics. However, these metrics are (in the same way as citation based metrics) still dependent on the number of interactions in the scholarly communication network. We are proposing a new class of evluation metrics => Semantometrics. These metrics are based on the assujmption that it is not possible to evaluate research impact without accessing the research outputs  . The problem with Altmetrics is that they measure popularity, the size of the community, etc, but not quality.
  7. It is an insult to researchers and to science that the current metrics do not consider the text of the manuscript in evaluation, but take external evidence into account only.
  8. This hypothesis is based on the process of how research builds on the existing knowledge in order to create new knowledge on which others can build. A publication, which in this way creates a \bridge" between what we already know and something new which will people develop based on this knowledge, brings a contribution to science. A publication has a high contribution if it creates a \long bridge" between more distant areas of science.
  9. Dasha and Petr have participated in the challenge which is part of the upcoming Web Search and Data Mining (WSDM) conference. The challenge, coorganised by Microsoft and Elsevier, was to assess the importance of scholarly articles, using data from Microsoft Academic Graph -- a large heterogeneous graph comprised of more than 120 million publications and the related authors, venues, organizations, and fields of study. Dasha and Petr (team called BletchleyPark) were the best out of 32 teams in the training round of the competition and after the validation round were invited to take part in the second phase of the challenge as one of the eight best of the 32 teams. They will also present their method at the workshop in San Francisco, California, in February. - See more at: http://kmi.open.ac.uk/news/article/18807#sthash.kn2oaULQ.dpuf
  10. The leads us to the conlusion that it is not possible to evaluate research impact without accessing the research outputs 