Practical Applications of                       Do-It-Yourself Citation Analysis                                        St...
Basic DIY Citation Analysis Method•     Select a target population, e.g.      –     journals in a sub-discipline      –   ...
Ranks of cited journals by single issues of Law and Human BehaviorTitle cited in LHB 2011                         frequenc...
rs correlations among issues of LHB                          2011 Feb Apr June Aug Oct                  Dec               ...
ROUGH rule of thumb for reliability of ranking by sample sizeSample                                  approximate          ...
Statistical analysis of this case suggests that so long as the sampled journals validlyrepresent the topic,a sample of•   ...
PRACTICAL APPLICATIONS OF DIY CITATION ANALYSISWhat’s the top journal on this topic?–     Compile cites from 10-20 article...
CITATION ANALYSIS: A SELECTIVE ANNOTATED BIBLIOGRAPHYCLASSICSS. C. Bradford, “Sources of Information on Specific Subjects,...
Maurice B. Line, "Changes in Rank Lists of Serials Over Time: Interlending versus Citation Data,"College & Research Librar...
Daniela Rosenstreich and Ben Wooliscroft, “Measuring the impact of accounting journals usingGoogle Scholar and the g-index...
Upcoming SlideShare
Loading in …5
×

Practical applications of citation analysis-handout

1,525 views
1,449 views

Published on

See the accompanying slideshow: http://www.slideshare.net/NASIG/blacknasig2012-practical-applications-of-citation-analysis-1

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,525
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Practical applications of citation analysis-handout

  1. 1. Practical Applications of Do-It-Yourself Citation Analysis Steve Black Serials and Reference Librarian The College of Saint Rose, Albany, NYJune 11, 2012Rationale for using citation analysis: The study of patterns and frequencies of citations is an objective,quantitative way to measure the impact of journals, authors, institutions, nations,etc. For librarians, citation analysis can help one identify journals for supportingresearch, building collections, or submitting papers for publication.Impact factor (simple version): cites to articles published in last 2 years ÷ number of articles published in last 2 yearsFull definition athttp://thomsonreuters.com/products_services/science/free/essays/impact_factor/Critiques of relying on impact factors include:(See bibliography) Data errors, both in the citations themselves and in how they are compiled by the citation indexes Matthew effect, coined by Robert Merton after the passage in Matthew (13:12), ―for unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath.‖ The more attention a work gets, the more attention it gets, skewing citations to papers that are notable for being noted, not necessarily because they’re the highest quality or the most important. Impact vs. quality vs. importance. Impact is how often cited. Quality is the caliber of the work—elegance of method, thoroughness of literature review, quality of writing, etc. Importance is contribution to a line of inquiry, regardless of how narrow. For various reasons high quality and very important papers may not be cited much. Global vs. local impact. Citation in publications may poorly correlate with the needs of local patrons. 1
  2. 2. Basic DIY Citation Analysis Method• Select a target population, e.g. – journals in a sub-discipline – group of researchers – subject(s) or keyword(s)• Select a sample that represents the target population• Compile works cited in the chosen sample• Sort and count the works citedCase study based on author’s ―Frequently Cited Journals in Forensic Psychology,‖Psychological Reports, v.110, no.1, (2012): 276-282.Tools used:• WorldCat (to identify most widely held journals)• PsycINFO via EBSCOhost (to identify works cited)• RefWorks (to organize works cited)• Excel (to count and rank cited journals)Notes on these tools: Our ability to use WorldCat holdings as a measure of importance may fade over time as fewer libraries subscribe to individual titles that appear as holdings in OCLC PsycINFO is particularly well suited to the task because all works cited are included in the database whether the article is in full text or not, and are in a format that is easily exported. If you do a similar project, seek out data sources that allow easy downloading of works cited. RefWorks is only one of many citation management tools that can be used for organizing citations. Excel has a COUNTIF function that could be used to count times cited, but it will undercount if there are any variations in spelling, punctuation, or spacing. It’s more accurate to count by highlighting the titles and noting the count of highlighted rows. 2
  3. 3. Ranks of cited journals by single issues of Law and Human BehaviorTitle cited in LHB 2011 frequency all Allrank FebRank AprilRank JunRank AugRank OctRank DecRankLaw and Human Behavior 154 1 1 1 1 1 1 1Criminal Justice and Behavior 45 2.5 9.5 6 2 2 9.5Journal of Applied Psychology 45 2.5 2 4.5 3 4.5 4Behavioral Sciences and the Law 40 4 3 6 4 2 16Journal of Personality and Social Psychology 35 5 16.5 7.5 3 8.5 9.5 11.5Psychological Assessment 32 6 3 13 8.5 11 4Applied Cognitive Psychology 29 7 4 17.5 13 12 13.5 2American Psychologist 26 8.5 16.5 9.5 8.5 9.5 8.5Psychiatric Services 26 8.5 2 16.5 6Journal of Abnormal Psychology 21 10.5 5 13 9.5 11.5Psychology, Public Policy, and Law 21 10.5 4.5 4.5 16Journal of Consulting and Clinical Psychology 18 12 8.5 5.5 21Personality and Individual Differences 15 13.5 4 16Psychological Bulletin 15 13.5 11.5 13 8.5Assessment 14 15.5 6.5 8.5 21Psychological Review 14 15.5 3 21Journal of Applied Social Psychology 13 18 16.5 17.5 10Journal of Experimental Psychology: Applied 13 18 16.5 8.5Psychological Science 13 18 16.5 11.5Psychology, Crime and Law 12 20 4International journal of law and psychiatry 11 21.5 16.5 13.5Legal and Criminological Psychology 11 21.5 17.5 13 16Tips on assigning ranks• In order to test reliability, the number of ranked items in each group must beequal. In this case December had only 22 titles with 2 or more times cited, so thetop 22 ranked journals in each group were compared.• Handle ties by averaging, e.g. tied for 2 & 3 are each ranked 2.5.• Assign bottom rank to titles with no match, e.g. if n=22, unmatched titles areranked 22.Formula for Spearman’s rho rank correlation:NOTE: This is the simple version of rsthat assumes no ties in ranks. See bibliography under―Technical‖ for sources of the more complex formula. In practice the two formulas return verysimilar resultseven with some ties between rankings. 3
  4. 4. rs correlations among issues of LHB 2011 Feb Apr June Aug Oct Dec 2011 1 Feb 0.24 1 Apr 0.47 -0.34 1 June 0.26 -0.05 -0.20 1 Aug 0.63 0.05 0.40 0.15 1 Oct 0.59 -0.05 0.46 -0.47 0.11 1 Dec 0.23 0.25 -0.66 -0.42 -0.03 -0.02 1Weak, scattered correlations indicate very low reliability of ranks at the singleissue level, which for this case n=250-350.rs correlation of LHB (2011) vs. ranking published in Psychological Reports LHB Psych 2011 Reports D D2title rank rankLaw and Human Behavior 1 1 0 0Criminal Justice and Behavior 2.5 3 -0.5 0.25Journal of Applied Psychology 2.5 5 -2.5 6.25Behavioral Sciences and the Law 4 2 2 4Journal of Personality and Social Psychology 5 6 -1 1Psychological Assessment 6 12 -6 36Applied Cognitive Psychology 7 9 -2 4American Psychologist 8.5 14 -5.5 30.25Psychiatric Services 8.5 20 -11.5 132.25Journal of Abnormal Psychology 10.5 22 -11.5 132.25Psychology, Public Policy, and Law 10.5 10 0.5 0.25Journal of Consulting and Clinical Psychology 12 16 -4 16Personality and Individual Differences 13.5 13 0.5 0.25Psychological Bulletin 13.5 22 -8.5 72.25Assessment 15.5 22 -6.5 42.25Psychological Review 15.5 22 -6.5 42.25Journal of Applied Social Psychology 18 21 -3 9Journal of Experimental Psychology: Applied 18 22 -4 16Psychological Science 18 22 -4 16Psychology, Crime and Law 20 11 9 81International journal of law and psychiatry 21.5 22 -0.5 0.25Legal and Criminological Psychology 21.5 15 6.5 42.25 sum of D squared 684rs=1-(6*684)/22*(222-1)= 1 – (4104/10626)= 1- .37rs= .63 4
  5. 5. ROUGH rule of thumb for reliability of ranking by sample sizeSample approximate expected correlation with population sample size (works cited)*one issue of journal n=250 very lowone volume of journal n=1500 moderateseveral volumes of 3-5 journals n>10,000 strong*Of course the number of works cited in a volume or issue will vary depending on the journalMeasure variation of individual titles’ frequency of citation withCoefficient of Variation = standard deviation ÷ mean Times Cited in 2011 Law and Human Behavior Coefficient of Feb Apr June Aug Oct Dec σX mean VariationLaw and Human 28 30 17 25 27 27 4.55 25.67 18%BehaviorJournal of Applied 13 9 0 8 8 7 4.23 7.50 56%PsychologyPsychology, Public 1 9 0 0 8 3 4.04 3.50 115%Policy, and Law In general, titles further down ranked lists have higher coefficients ofvariation. A higher coefficient of variation means less reliability of the ranking—it’s more likely that a different sample will result in a much different rank. 5
  6. 6. Statistical analysis of this case suggests that so long as the sampled journals validlyrepresent the topic,a sample of• n<1,000 works cited can indicate the top journal (if any)• n>1,000 can generate a rough indication of leading journals• n>10,000 can create a useful ranked listCAUTIONARY NOTE Even with n>10,000, different samples will yield different ranks, especiallyfurther down the list, so a journal’s rank must ALWAYS be taken as anapproximation of its ―true‖ ranking. Since rankings reflect the complex reasonsresearchers cite one thing or another, there can never be a static, definitive rankingfor journals in any topic. Similar statistical analysis I’ve done onjournals incommunication disorders suggests that a third of movement in rankings over timeis due to random variation.Tips for publication• Don’t be too parochial or narrow—editors must think their readers will be interested.• BUT choose something not already in Journal Citation Reports.• Be very thorough with your lit review before gathering data (search disciplinary databases as well as the library literature).• Gather citations and test sample before nailing down your method.• Take the time and energy to ensure sample is o representative o clearly defined o robust o valid on its face• Group data and run statistics by journal volume• Consider submitting to a journal in the topic area 6
  7. 7. PRACTICAL APPLICATIONS OF DIY CITATION ANALYSISWhat’s the top journal on this topic?– Compile cites from 10-20 articles on the topic to identify it (or findthere’s not one).What does my library need to support a proposed new major?– Compile cites from 1-2 volumes of 2-4 journals to flag titles notalready available.Which journals are REALLY used by that herd of cats in Dept. X?– Analyze publications & dissertations from last decade.OMG I’m up for tenure soon!– Identify a topic and do a thorough analysis of 2-3 volumes of 4-6journals; include reliability tests. 7
  8. 8. CITATION ANALYSIS: A SELECTIVE ANNOTATED BIBLIOGRAPHYCLASSICSS. C. Bradford, “Sources of Information on Specific Subjects,” Engineering 137 (1934): 85-86. This study of the concentration of citations to literature in applied geophysics and lubrication is the original source of Bradford’s Law of Distribution.Eugene Garfield, “Citation Analysis as a Tool in Journal Evaluation,” Science 178 (1972): 471-479. Garfield explains and argues for the Institute for Scientific Information’s Science Citation Index.Robert K. Merton, The Sociology of Science: Theoretical and Empirical Investigations (Chicago:University of Chicago Press, 1973). Includes the famous sociologist’s description of how the Matthew Effect impacts scientific output.OVERVIEWSLinda C. Smith, “Citation analysis,” Library Trends 30 (1981): 83-106. One of several important articles in a special issue of Library Trends devoted to bibliometricsin the early years of the “serials crisis,” when many libraries were first seriously confronted with having to decide which journals to cut.Thomas E. Nisonger, “Chapter 5, The Application of Citation Analysis to Serials CollectionManagement,” Management of Serials in Libraries (Englewood, CO: Libraries Unlimited, 1998):121-156. A well-organized presentation of major issues and a thorough bibliography.Thomas E. Nisonger, “Journals in the Core Collection: Definition, Identification, andApplications,” Serials Librarian 51, no. 3/4 (2007): 51-73. Summarizes ten methods for creating lists of core journals and discusses applications of core lists.CRITICAL ANALYSESMichael H. MacRoberts and Barbara R. MacRoberts, "Problems of Citation Analysis: A CriticalReview," Journal of the American Society for Information Science 40, no. 5 (1989): 342-349. A thorough and well organized critique with specific emphasis on Science Citation Index.Per O. Seglen, “Why the impact factor of journals should not be used for evaluating research,”British Medical Journal 314 (1997): 498-502. Concise summary of the problems associated with the use of journal impact factors. 8
  9. 9. Maurice B. Line, "Changes in Rank Lists of Serials Over Time: Interlending versus Citation Data,"College & Research Libraries 46, no. 1 (1985): 77-79.Robert N. Broadus, "A Proposed Method for Eliminating Titles from Periodical SubscriptionLists," College & Research Libraries 46, no. 1 (1985): 30-35.Maurice B. Line, "Use of Citation Data for Periodicals Control in Libraries: A Response toBroadus," College & Research Libraries 46, no. 1 (1985): 36-37. These 3 articles comprise a dialogue between Broadus and Line on the validity of citation data as a tool for collection development in an era of cost-cutting.R.E. Rice, Christine L. Borgman, Diane Bednarski, and P.J. Hart, “Journal-to-Journal CitationData: Issues of Validity and Reliability,” Scientometrics 15, no. 3 (1989): 257-282. Reviews issues of validity and reliability, discusses causes of measurement errors, and concludes with suggestions for how to reduce measurement errors.Ben R. Martin, "The Use of Multiple Indicators in the Assessment of Basic Research,"Scientometrics 36, no. 3 (1996): 343-362. Defines quality, importance, and impact and stresses the importance of respecting each.Gordon and Breach Science v. American Institute of Physics and American Physical Society,http://barschall.stanford.edu. This web site contains a thorough and well organized treatment of the Gordon and Breach case against Henry H. Barschall and the publishers of his studies. Barschall used citation counts and subscription costs to create rankings of physics journals. Gordon & Breach titles were shown to be among the poorest values in physics, and they sued the publisher for false advertising. The court records and related documents do an excellent job of presenting the issues surrounding applications of citation analysis.METHODSSteve Black, “Using Citation Analysis to Pursue a Core Collection of Journals for CommunicationDisorders,” Library Resources & Technical Services 45, no. 1 (2001): 3-9. Includes a basic method for do-it-yourself citation analysis.Steve Black, “How Much do Core Journals Change over a Decade?” Library Resources &Technical Services 56, no.2 (2012): 80-93. Describes methods for correlating ranked lists over time. Jeffery D. Kushkowski, Kristin H. Gerhard and Cynthia Dobson, “A Method for Building CoreJournal Lists in Interdisciplinary Subject Areas,” Journal of Documentation 54, no. 4 (1998):477-88. Describes a Simple Index Method for ranking journals based on results of subject or keyword searches in relevant databases. 9
  10. 10. Daniela Rosenstreich and Ben Wooliscroft, “Measuring the impact of accounting journals usingGoogle Scholar and the g-index,” British Accounting Review 41 (2009): 227-239. Valuable for its treatment of Google Scholar, a table summarizing common criticisms of citation-based journal rankings, and comparisons of ranking methods. (Also a good example of how an important paper can be published in an unexpected place!)Chris Piotrowski, “Top cited journals in forensic psychology: An analysis of the psychologicalliterature, “American Journal of Forensic Psychology 30, no. 2 (2012): 29-37. An example of using keyword searches to rank journals. Piotrowski’s method yields a very different ranked list from my list published in Psychological Reports.TECHNICALThomson Reuters, “The Thomson Reuters Impact Factor,”http://thomsonreuters.com/products_services/science/free/essays/impact_factor. An overview by Eugene Garfield that includes the formula for calculating impact factor, the rationale for using it, caveats and cautions.Stephen J. Bensman, "Probability Distributions in Library and Information Science: A Historicaland Practitioner Viewpoint,” Journal of the American Society for Information Science andTechnology 51, no. 9 (2000): 816-833. Argues that parametric statistics based on Poisson distribution are incapable of accurately modeling patterns of journal citations.Sidney Siegel, "Nonparametric Statistics," American Statistician 11, no. 3 (1957): 13-19. An authoritative, readable description of when and why to use various statistical methods including Spearman’s rho.Maurice Kendall and Jean Gibbons, Rank Correlation Methods. 5th ed. (New York: OxfordUniversity Press, 1990). Complete descriptions, formulas, and proofs of Spearman’s rho and Kendall’s tau (a calculation based simply on whether items go up or down in rank, disregarding the degree of change). Steve Black blacks@strose.edu (518) 458-5494 10

×