Needed to: - Make sense of the available literature - Recognise the key important researchHow do you filter what you read?How do you filter the best material for your interests?
Not possible to use across disciplines
Fulltextdrivnebyl availableIs it possible to evaluate research based on what people say about it?
Mining ResearchPublication Networks forImpactDrahomira HerrmannovaKnowledge Media InstituteSupervisors: Dr Zdenek ZdrahalMr Petr Knoth04/06/2013 Drahomira Herrmannova firstname.lastname@example.org
The Title• Mining Research Publication Networks for Impact04/06/2013 Drahomira Herrmannova email@example.com
Background• How to evaluate quality of research?• It is highly relevant to• Researchers – what to read, career progression• Librarians – subscriptions• Universities and other institutions – grants, promotions,…• Publishers – promoting their journals04/06/2013 Drahomira Herrmannova firstname.lastname@example.org
Background• At what granularity do you want to evaluate thequality of research?• Papers• Journals• Researchers• Research groups / institutions• Metrics created for evaluating at one level ofgranularity are often used for evaluating otherlevels04/06/2013 Drahomira Herrmannova email@example.com
Current situation• Many current widely used measures depend oncitation analysis• Well known examples• Journal Impact Factor (journals)• Citation counts (individuals, papers)• H-Index (individuals)• Advantages• Simple and clear metrics• Well available04/06/2013 Drahomira Herrmannova firstname.lastname@example.org
Current situation04/06/2013 Drahomira Herrmannova email@example.com
Concerns• Not an accurate evidence of quality of a paper• Traditional methods often ignore• Reasons for citation• Co-authorship• Self-citation• Article type• Citation half-life• …04/06/2013 Drahomira Herrmannova firstname.lastname@example.org
Concerns• Many papers are never cited at all (Garfield, 2005)• “Citation half-life”• Can be up to several years long (Arnold, 2010)• “When metric becomes a target, it ceases to be agood metric” (Arnold, 2010)• Citation-based metrics are easily manipulated• Relationship between the paper and relatedliterature rests on the author04/06/2013 Drahomira Herrmannova email@example.com
Available data• Many possible types of data that can be used toevaluate research output• Citations• Usage• Blog coverage• Discussion threads• Social bookmarking• …04/06/2013 Drahomira Herrmannova firstname.lastname@example.org• Fulltext of publication• Expert reviewIs it possible to evaluateresearch based just on whatpeople say about it?
Methods• Metrics based on web data (altmetrics,webometrics)• File accesses and downloads• Mentions, comments and annotations• Citations and bookmarks• …• Methods based on analysing fulltext of papers• Semantic relatedness of papers• Context of citation04/06/2013 Drahomira Herrmannova email@example.com
Methods• Methods based on citations• Citation graph• Co-authorship network of researchers• Other methods• Using patent information• …04/06/2013 Drahomira Herrmannova firstname.lastname@example.org
Methods• Despite the number of possibilities still nostandard solution that would help up to addressthe issues• Bridge the time gap between publishing a paper andreceiving first citations• Explain the reason and meaning of quality of a paper• …04/06/2013 Drahomira Herrmannova email@example.com
Project topic• Design new methods for improving evaluation ofquality of research which will address thementioned challenges04/06/2013 Drahomira Herrmannova firstname.lastname@example.org
Questions and goals• Evaluate how the impact measures based oncitations represent the quality of a paper• Analyse which other factors, in addition tocitations, affect the quality of a publication04/06/2013 Drahomira Herrmannova email@example.com
Questions and goals• Design new methods for improving evaluation ofquality of research which will address thementioned challenges using• Fulltext of publications• Citation graph• Networks of researchers• Patent information• …04/06/2013 Drahomira Herrmannova firstname.lastname@example.org
Questions and goals• Analyse how these new methods can be used in other disciplines suchas• information retrieval• exploratory search• digital libraries• …• How they can be used to investigate trends in the development ofscientific disciplines• How they can be used by each group of people• Researchers• Librarians• Universities• …04/06/2013 Drahomira Herrmannova email@example.com
References• Douglas N Arnold and Kristine K Fowler. Nefariousnumbers. Notices of the American MathematicalSociety, 58(3):434-437, 2010.• Eugene Garfield. The agony and the ecstasy thehistory and meaning of the journal impact factor.pages 1-22, 2005.04/06/2013 Drahomira Herrmannova firstname.lastname@example.org