SlideShare a Scribd company logo
Do Citations and Readership
Predict Excellent Publications?
Dasha Herrmannova, The Open University, UK
Robert Patton, Oak Ridge National Laboratory, USA
Petr Knoth, The Open University, UK
Chris Stahl, Oak Ridge National Laboratory, USA
Research question
Q:Are current research evaluation metrics sufficient for identifying highly
influential papers?
Why care about metrics?
Research papers
Researchers
Funding agencies
Institutions
Who to fund?
Returns on investment?
Are we doing well?
What to subscribe to?
What to read?
Where to publish?
Collaborators? Citationanalysis
Altmetrics
Finding what works
• ML approach
• Evaluate all methods in terms of precision-recall/accuracy/…
• Requirement: ground truth
• Research evaluation
• No ground truth
• Authority often established axiomatically
• JIF, h-index, etc.
• Can we build a ground truth dataset?
Our understanding of "impact"
Low impact High impact
vs
Our understanding of "impact"
Low impact High impact
vs
Survey papers:
"A general view, examination or
description of someone or
something"
Seminal works:
"Strongly influencing later
developments"
Creating a dataset
• Online questionnaire
• Discipline?
• Reference to a survey paper
• Reference to a seminal paper
• Collected 314 papers
• Labels (seminal, survey)
• Title, authors, year of publication, abstract, DOI, ...
• Available online
• http://trueid.semantometrics.org
• Analysis
• Seminal papers on average 10 years older
• Seminal papers cited on average 5 times more
Do citations/readership predict excellent papers?
• Classify papers using citations and readership as features
• Model
• Select a threshold t
• If cit(d) ≥ t → label as seminal
• Else → label as survey
• Use threshold with best accuracy on the training set
• Leave-one-out cross-validation
• 3 experiments
• Aggregate
• Per discipline
• Per year
Results
Model Data Accuracy Upper bound
Baseline Citations 52.87% -
Readership 52.87% -
Aggregate Citations 63.06% 63.38%
Readership 42.68% 52.87%
Discipline based Citations 45.28% 68.11%
Readership 42.13% 62.60%
Year based Citations 55.23% 68.62%
Readership 51.05% 65.27%
Conclusion
• Both citations and readership provide an improvement over the baseline
• Neither of the two metrics is optimal
What next?
• Ideal dataset
• Multi-disciplinary
• Time span
• Publication types
• Peer review judgement
• Better metrics
• Citation context
• Analyzing content
Thank you!
Questions?
http://trueid.semantometrics.org

More Related Content

What's hot

Clinical Microbiology - searching for information
Clinical Microbiology - searching for informationClinical Microbiology - searching for information
Clinical Microbiology - searching for information
PaulaFunnell
 
PLoS Author Research 2009
PLoS Author Research 2009PLoS Author Research 2009
PLoS Author Research 2009
Liz Allen
 
Introduction to Systematic Review & Meta-Analysis
Introduction to Systematic Review & Meta-Analysis Introduction to Systematic Review & Meta-Analysis
Introduction to Systematic Review & Meta-Analysis
Hasanain Ghazi
 
Adam Etkin's Flash Presentation from STM Spring 2014
Adam Etkin's Flash Presentation from STM Spring 2014Adam Etkin's Flash Presentation from STM Spring 2014
Adam Etkin's Flash Presentation from STM Spring 2014
Adam Etkin
 
Probability and data 1w
Probability and data 1wProbability and data 1w
Probability and data 1w
KyoungilYoon
 
Research writing & Reference Management using Mendely
Research writing & Reference Management using  MendelyResearch writing & Reference Management using  Mendely
Research writing & Reference Management using Mendely
vijay kumar
 
Research Process Explained
Research Process ExplainedResearch Process Explained
Research Process Explained
360dissertations
 
research design
 research design research design
research design
kpgandhi
 
Allmetrics. Not Altmetrics.
Allmetrics. Not Altmetrics.Allmetrics. Not Altmetrics.
Feedback on the draft summary report
Feedback on the draft summary reportFeedback on the draft summary report
Feedback on the draft summary report
MEYS, MŠMT in Czech
 
Critical appraisal example systematic review and meta-analysis
Critical appraisal example  systematic review and meta-analysisCritical appraisal example  systematic review and meta-analysis
Critical appraisal example systematic review and meta-analysis
Nouran Hamza, MSc, PgDPH
 
Health Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature SearchingHealth Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature Searching
Jamie Halstead
 
Basics of Systematic Review and Meta-analysis: Part 1
Basics of Systematic Review and Meta-analysis: Part 1Basics of Systematic Review and Meta-analysis: Part 1
Basics of Systematic Review and Meta-analysis: Part 1
Rizwan S A
 
10. Have you finished your data collection?
10. Have you finished your data collection?10. Have you finished your data collection?
10. Have you finished your data collection?
DoctoralNet Limited
 
How to do qualitative research
How to do qualitative researchHow to do qualitative research
How to do qualitative research
Nagaland University
 
Basics of Systematic Review and Meta-analysis: Part 3
Basics of Systematic Review and Meta-analysis: Part 3Basics of Systematic Review and Meta-analysis: Part 3
Basics of Systematic Review and Meta-analysis: Part 3
Rizwan S A
 
Systematic review and meta analysis applications in medication safety 2
Systematic review and meta analysis applications in medication safety 2Systematic review and meta analysis applications in medication safety 2
Systematic review and meta analysis applications in medication safety 2
مركز البحوث الأقسام العلمية
 
Report writing
Report writingReport writing
Keck Year 2 Evidence Based Medicine - Systematic Reviews
Keck Year 2 Evidence Based Medicine - Systematic ReviewsKeck Year 2 Evidence Based Medicine - Systematic Reviews
Keck Year 2 Evidence Based Medicine - Systematic Reviews
lynnkysh
 
Keck Year 2 Evidence Based Medicine - Observational Studies and Trials
Keck Year 2 Evidence Based Medicine - Observational Studies and TrialsKeck Year 2 Evidence Based Medicine - Observational Studies and Trials
Keck Year 2 Evidence Based Medicine - Observational Studies and Trials
lynnkysh
 

What's hot (20)

Clinical Microbiology - searching for information
Clinical Microbiology - searching for informationClinical Microbiology - searching for information
Clinical Microbiology - searching for information
 
PLoS Author Research 2009
PLoS Author Research 2009PLoS Author Research 2009
PLoS Author Research 2009
 
Introduction to Systematic Review & Meta-Analysis
Introduction to Systematic Review & Meta-Analysis Introduction to Systematic Review & Meta-Analysis
Introduction to Systematic Review & Meta-Analysis
 
Adam Etkin's Flash Presentation from STM Spring 2014
Adam Etkin's Flash Presentation from STM Spring 2014Adam Etkin's Flash Presentation from STM Spring 2014
Adam Etkin's Flash Presentation from STM Spring 2014
 
Probability and data 1w
Probability and data 1wProbability and data 1w
Probability and data 1w
 
Research writing & Reference Management using Mendely
Research writing & Reference Management using  MendelyResearch writing & Reference Management using  Mendely
Research writing & Reference Management using Mendely
 
Research Process Explained
Research Process ExplainedResearch Process Explained
Research Process Explained
 
research design
 research design research design
research design
 
Allmetrics. Not Altmetrics.
Allmetrics. Not Altmetrics.Allmetrics. Not Altmetrics.
Allmetrics. Not Altmetrics.
 
Feedback on the draft summary report
Feedback on the draft summary reportFeedback on the draft summary report
Feedback on the draft summary report
 
Critical appraisal example systematic review and meta-analysis
Critical appraisal example  systematic review and meta-analysisCritical appraisal example  systematic review and meta-analysis
Critical appraisal example systematic review and meta-analysis
 
Health Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature SearchingHealth Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature Searching
 
Basics of Systematic Review and Meta-analysis: Part 1
Basics of Systematic Review and Meta-analysis: Part 1Basics of Systematic Review and Meta-analysis: Part 1
Basics of Systematic Review and Meta-analysis: Part 1
 
10. Have you finished your data collection?
10. Have you finished your data collection?10. Have you finished your data collection?
10. Have you finished your data collection?
 
How to do qualitative research
How to do qualitative researchHow to do qualitative research
How to do qualitative research
 
Basics of Systematic Review and Meta-analysis: Part 3
Basics of Systematic Review and Meta-analysis: Part 3Basics of Systematic Review and Meta-analysis: Part 3
Basics of Systematic Review and Meta-analysis: Part 3
 
Systematic review and meta analysis applications in medication safety 2
Systematic review and meta analysis applications in medication safety 2Systematic review and meta analysis applications in medication safety 2
Systematic review and meta analysis applications in medication safety 2
 
Report writing
Report writingReport writing
Report writing
 
Keck Year 2 Evidence Based Medicine - Systematic Reviews
Keck Year 2 Evidence Based Medicine - Systematic ReviewsKeck Year 2 Evidence Based Medicine - Systematic Reviews
Keck Year 2 Evidence Based Medicine - Systematic Reviews
 
Keck Year 2 Evidence Based Medicine - Observational Studies and Trials
Keck Year 2 Evidence Based Medicine - Observational Studies and TrialsKeck Year 2 Evidence Based Medicine - Observational Studies and Trials
Keck Year 2 Evidence Based Medicine - Observational Studies and Trials
 

Similar to Do Citations and Readership Predict Excellent Publications?

Writing a Scientific Article
Writing a Scientific ArticleWriting a Scientific Article
Writing a Scientific Article
Hythm Shibl
 
How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...
How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...
How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...
ilmideas
 
How to Design Research from Ilm Ideas on Slide Share
How to Design Research from Ilm Ideas on Slide Share How to Design Research from Ilm Ideas on Slide Share
How to Design Research from Ilm Ideas on Slide Share
ilmideas
 
Statistics for Librarians: How to Use and Evaluate Statistical Evidence
Statistics for Librarians: How to Use and Evaluate Statistical EvidenceStatistics for Librarians: How to Use and Evaluate Statistical Evidence
Statistics for Librarians: How to Use and Evaluate Statistical Evidence
John McDonald
 
Can We Fix Peer Review
Can We Fix Peer ReviewCan We Fix Peer Review
Can We Fix Peer Review
Micah Altman
 
Modesofinquiry
ModesofinquiryModesofinquiry
Modesofinquiry
Carla Piper
 
G2.suntasig.guallichico.maicol.alexander.english.project.design.docx
G2.suntasig.guallichico.maicol.alexander.english.project.design.docxG2.suntasig.guallichico.maicol.alexander.english.project.design.docx
G2.suntasig.guallichico.maicol.alexander.english.project.design.docx
Maicol Suntasig
 
Evaluating information
Evaluating informationEvaluating information
Evaluating information
Penelope Cole
 
Lecture 10.12.10
Lecture 10.12.10Lecture 10.12.10
Lecture 10.12.10
VMRoberts
 
Rm17 45 81-120
Rm17 45 81-120Rm17 45 81-120
Rm17 45 81-120
11class 12class
 
Evaluating published research-research publication
Evaluating published research-research publicationEvaluating published research-research publication
Evaluating published research-research publication
rehabonehealthcare
 
Literature Review.pptx
Literature Review.pptxLiterature Review.pptx
Literature Review.pptx
Abhishek Job
 
Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...
Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...
Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...
Daniel Katz
 
عزوز
عزوزعزوز
K-to-R Workshop: How to Structure the "Approach" Section (Part 1)
K-to-R Workshop: How to Structure the "Approach" Section (Part 1)K-to-R Workshop: How to Structure the "Approach" Section (Part 1)
K-to-R Workshop: How to Structure the "Approach" Section (Part 1)
UCLA CTSI
 
Research Awareness Programme-research & development
Research Awareness  Programme-research & developmentResearch Awareness  Programme-research & development
Research Awareness Programme-research & development
lochan100
 
Quality Research
Quality Research Quality Research
Quality Research
Sarang Bhola
 
Refining research question2010
Refining research question2010Refining research question2010
Refining research question2010
andycinek
 
Managing Quality In Qualitative Research
Managing Quality In Qualitative ResearchManaging Quality In Qualitative Research
Managing Quality In Qualitative Research
Mike Crabb
 
Beyond the Factor: Talking about Research Impact
Beyond the Factor: Talking about Research ImpactBeyond the Factor: Talking about Research Impact
Beyond the Factor: Talking about Research Impact
Claire Stewart
 

Similar to Do Citations and Readership Predict Excellent Publications? (20)

Writing a Scientific Article
Writing a Scientific ArticleWriting a Scientific Article
Writing a Scientific Article
 
How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...
How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...
How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide...
 
How to Design Research from Ilm Ideas on Slide Share
How to Design Research from Ilm Ideas on Slide Share How to Design Research from Ilm Ideas on Slide Share
How to Design Research from Ilm Ideas on Slide Share
 
Statistics for Librarians: How to Use and Evaluate Statistical Evidence
Statistics for Librarians: How to Use and Evaluate Statistical EvidenceStatistics for Librarians: How to Use and Evaluate Statistical Evidence
Statistics for Librarians: How to Use and Evaluate Statistical Evidence
 
Can We Fix Peer Review
Can We Fix Peer ReviewCan We Fix Peer Review
Can We Fix Peer Review
 
Modesofinquiry
ModesofinquiryModesofinquiry
Modesofinquiry
 
G2.suntasig.guallichico.maicol.alexander.english.project.design.docx
G2.suntasig.guallichico.maicol.alexander.english.project.design.docxG2.suntasig.guallichico.maicol.alexander.english.project.design.docx
G2.suntasig.guallichico.maicol.alexander.english.project.design.docx
 
Evaluating information
Evaluating informationEvaluating information
Evaluating information
 
Lecture 10.12.10
Lecture 10.12.10Lecture 10.12.10
Lecture 10.12.10
 
Rm17 45 81-120
Rm17 45 81-120Rm17 45 81-120
Rm17 45 81-120
 
Evaluating published research-research publication
Evaluating published research-research publicationEvaluating published research-research publication
Evaluating published research-research publication
 
Literature Review.pptx
Literature Review.pptxLiterature Review.pptx
Literature Review.pptx
 
Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...
Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...
Quantitative Methods for Lawyers - Class #2 - Research Design Part II + Intro...
 
عزوز
عزوزعزوز
عزوز
 
K-to-R Workshop: How to Structure the "Approach" Section (Part 1)
K-to-R Workshop: How to Structure the "Approach" Section (Part 1)K-to-R Workshop: How to Structure the "Approach" Section (Part 1)
K-to-R Workshop: How to Structure the "Approach" Section (Part 1)
 
Research Awareness Programme-research & development
Research Awareness  Programme-research & developmentResearch Awareness  Programme-research & development
Research Awareness Programme-research & development
 
Quality Research
Quality Research Quality Research
Quality Research
 
Refining research question2010
Refining research question2010Refining research question2010
Refining research question2010
 
Managing Quality In Qualitative Research
Managing Quality In Qualitative ResearchManaging Quality In Qualitative Research
Managing Quality In Qualitative Research
 
Beyond the Factor: Talking about Research Impact
Beyond the Factor: Talking about Research ImpactBeyond the Factor: Talking about Research Impact
Beyond the Factor: Talking about Research Impact
 

More from Dasha Herrmannova

Machine Learning for Data Extraction
Machine Learning for Data ExtractionMachine Learning for Data Extraction
Machine Learning for Data Extraction
Dasha Herrmannova
 
Do Authors Deposit on Time? Tracking Open Access Policy Compliance
Do Authors Deposit on Time? Tracking Open Access Policy ComplianceDo Authors Deposit on Time? Tracking Open Access Policy Compliance
Do Authors Deposit on Time? Tracking Open Access Policy Compliance
Dasha Herrmannova
 
Semantometrics: Text Analysis in Research Evaluation
Semantometrics: Text Analysis in Research Evaluation Semantometrics: Text Analysis in Research Evaluation
Semantometrics: Text Analysis in Research Evaluation
Dasha Herrmannova
 
An Analysis of the Microsoft Academic Graph
An Analysis of the Microsoft Academic GraphAn Analysis of the Microsoft Academic Graph
An Analysis of the Microsoft Academic Graph
Dasha Herrmannova
 
Visual Search for Supporting Content Exploration in Large Document Collections
Visual Search for Supporting Content Exploration in Large Document CollectionsVisual Search for Supporting Content Exploration in Large Document Collections
Visual Search for Supporting Content Exploration in Large Document Collections
Dasha Herrmannova
 
Unsupervised Identification of Study Descriptors in Toxicology Research: An E...
Unsupervised Identification of Study Descriptors in Toxicology Research: An E...Unsupervised Identification of Study Descriptors in Toxicology Research: An E...
Unsupervised Identification of Study Descriptors in Toxicology Research: An E...
Dasha Herrmannova
 
Simple Yet Effective Methods for Large-Scale Scholarly Publication Ranking
Simple Yet Effective Methods for Large-Scale Scholarly Publication RankingSimple Yet Effective Methods for Large-Scale Scholarly Publication Ranking
Simple Yet Effective Methods for Large-Scale Scholarly Publication Ranking
Dasha Herrmannova
 
Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...
Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...
Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...
Dasha Herrmannova
 
Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...
Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...
Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...
Dasha Herrmannova
 
Mining Research Publication Networks for Impact -- KMi Internal Seminar
Mining Research Publication Networks for Impact -- KMi Internal SeminarMining Research Publication Networks for Impact -- KMi Internal Seminar
Mining Research Publication Networks for Impact -- KMi Internal Seminar
Dasha Herrmannova
 

More from Dasha Herrmannova (10)

Machine Learning for Data Extraction
Machine Learning for Data ExtractionMachine Learning for Data Extraction
Machine Learning for Data Extraction
 
Do Authors Deposit on Time? Tracking Open Access Policy Compliance
Do Authors Deposit on Time? Tracking Open Access Policy ComplianceDo Authors Deposit on Time? Tracking Open Access Policy Compliance
Do Authors Deposit on Time? Tracking Open Access Policy Compliance
 
Semantometrics: Text Analysis in Research Evaluation
Semantometrics: Text Analysis in Research Evaluation Semantometrics: Text Analysis in Research Evaluation
Semantometrics: Text Analysis in Research Evaluation
 
An Analysis of the Microsoft Academic Graph
An Analysis of the Microsoft Academic GraphAn Analysis of the Microsoft Academic Graph
An Analysis of the Microsoft Academic Graph
 
Visual Search for Supporting Content Exploration in Large Document Collections
Visual Search for Supporting Content Exploration in Large Document CollectionsVisual Search for Supporting Content Exploration in Large Document Collections
Visual Search for Supporting Content Exploration in Large Document Collections
 
Unsupervised Identification of Study Descriptors in Toxicology Research: An E...
Unsupervised Identification of Study Descriptors in Toxicology Research: An E...Unsupervised Identification of Study Descriptors in Toxicology Research: An E...
Unsupervised Identification of Study Descriptors in Toxicology Research: An E...
 
Simple Yet Effective Methods for Large-Scale Scholarly Publication Ranking
Simple Yet Effective Methods for Large-Scale Scholarly Publication RankingSimple Yet Effective Methods for Large-Scale Scholarly Publication Ranking
Simple Yet Effective Methods for Large-Scale Scholarly Publication Ranking
 
Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...
Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...
Semantometrics in Coauthorship Networks: Fulltext-based Approach for Analysin...
 
Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...
Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...
Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing...
 
Mining Research Publication Networks for Impact -- KMi Internal Seminar
Mining Research Publication Networks for Impact -- KMi Internal SeminarMining Research Publication Networks for Impact -- KMi Internal Seminar
Mining Research Publication Networks for Impact -- KMi Internal Seminar
 

Recently uploaded

How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
DianaGray10
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
The Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - CoatueThe Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - Coatue
Razin Mustafiz
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
ZachWylie3
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
AmandaCheung15
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 
Accelerating Migrations = Recommendations
Accelerating Migrations = RecommendationsAccelerating Migrations = Recommendations
Accelerating Migrations = Recommendations
isBullShit
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
Zilliz
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Zilliz
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
SelfMade bd
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
Zilliz
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
SynapseIndia
 
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
OnBoard
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
KIRAN KV
 

Recently uploaded (20)

How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
The Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - CoatueThe Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - Coatue
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 
Accelerating Migrations = Recommendations
Accelerating Migrations = RecommendationsAccelerating Migrations = Recommendations
Accelerating Migrations = Recommendations
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
 
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
 

Do Citations and Readership Predict Excellent Publications?

  • 1. Do Citations and Readership Predict Excellent Publications? Dasha Herrmannova, The Open University, UK Robert Patton, Oak Ridge National Laboratory, USA Petr Knoth, The Open University, UK Chris Stahl, Oak Ridge National Laboratory, USA
  • 2. Research question Q:Are current research evaluation metrics sufficient for identifying highly influential papers?
  • 3. Why care about metrics? Research papers Researchers Funding agencies Institutions Who to fund? Returns on investment? Are we doing well? What to subscribe to? What to read? Where to publish? Collaborators? Citationanalysis Altmetrics
  • 4. Finding what works • ML approach • Evaluate all methods in terms of precision-recall/accuracy/… • Requirement: ground truth • Research evaluation • No ground truth • Authority often established axiomatically • JIF, h-index, etc. • Can we build a ground truth dataset?
  • 5. Our understanding of "impact" Low impact High impact vs
  • 6. Our understanding of "impact" Low impact High impact vs Survey papers: "A general view, examination or description of someone or something" Seminal works: "Strongly influencing later developments"
  • 7. Creating a dataset • Online questionnaire • Discipline? • Reference to a survey paper • Reference to a seminal paper • Collected 314 papers • Labels (seminal, survey) • Title, authors, year of publication, abstract, DOI, ... • Available online • http://trueid.semantometrics.org • Analysis • Seminal papers on average 10 years older • Seminal papers cited on average 5 times more
  • 8. Do citations/readership predict excellent papers? • Classify papers using citations and readership as features • Model • Select a threshold t • If cit(d) ≥ t → label as seminal • Else → label as survey • Use threshold with best accuracy on the training set • Leave-one-out cross-validation • 3 experiments • Aggregate • Per discipline • Per year
  • 9. Results Model Data Accuracy Upper bound Baseline Citations 52.87% - Readership 52.87% - Aggregate Citations 63.06% 63.38% Readership 42.68% 52.87% Discipline based Citations 45.28% 68.11% Readership 42.13% 62.60% Year based Citations 55.23% 68.62% Readership 51.05% 65.27%
  • 10. Conclusion • Both citations and readership provide an improvement over the baseline • Neither of the two metrics is optimal
  • 11. What next? • Ideal dataset • Multi-disciplinary • Time span • Publication types • Peer review judgement • Better metrics • Citation context • Analyzing content