Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Practical Applications               of Do-it-Yourself              Citation Analysis                       Steve Black   ...
Citation analysis is     the study of     patterns and     frequencies of     citations
Why• Objective, quantitative measurement of  impact (relative frequency of citation) of  – journals  – authors  – institut...
Critiques of relying on impact factors     •   data errors     •   Matthew effect     •   gaming the system     •   impact...
Never forget,    & remind others when needed,impact factor is but one of many         considerations     for collection de...
Do-It-Yourself Citation Analysis  • tailored to specific needs not    covered elsewhere  • can be quick-and-    dirty, exh...
Basic DIY Approach• Select a target population  – journals in a sub-discipline (e.g. forensic psychology)  – researchers a...
Google Scholar & Zotero• Google Scholar is good for identifying articles  on a topic, BUT  – no authority control (which S...
Freeware can help compile . . .
. . .but Zotero’s not designed for DIY                citation analysis•   emphasis on full text searching•   no authority...
http://kimmonsdesign.com/node/24
Case study in DIY citation analysis:       forensic psychologyPublished as“Frequently Cited Journals in Forensic Psycholog...
Goal: create ranked list of most-cited   journals in forensic psychology• General Method  – identify leading journals spec...
Source Journals Sample• For published article:  – 2007, 2008, and 2009 volumes of 6 widely held    journals specific to fo...
Tools used• WorldCat (to identify most widely held journals)• PsycINFO via EBSCOhost (to identify works cited)      • RefW...
Retrieve records for articles insource journals, one issue at a time
Add cited references to folder
Export references . . .
. . . to RefWorks
RefWorks creates a temporary folder        of the downloads
Create a folder for each issue’s works cited (keep discrete units to allow statistical analysis of reliability)
Move the downloaded cites to the       appropriate folder
Export references in tab delimited             format
Save as a text document
Open the text file in Excel . . .
. . .by going through 3 steps to convert                 from text
Work with cited journal titles              shown as “Periodical Full”Note thatsometaggedreferencetypes areimprecise, don’...
Sort the title column alphabetically
For accurate counts, review titles and        remove discrepancies
Count citation frequency(choose a cutoff point equal to about 0.5% of sample)
Record frequency counts
Sort by largest to smallest   to create ranked list
Bradford distribution 1:n:n2:n3 roughly    holds in this case as 1:3:9:27...            Law and Human Behavior            ...
...even though LHB cites a mix offorensic and general psych journals  Law and Human Behavior                         154  ...
How should we take this neatly ranked   list of forensic psych journals?
It depends on the reliability of our            rankings• Measure reliability 2 ways:  – Macro: variability in ranked list...
Macro: How well do ranked lists correlate?   Feb 2011                              April 2011                             ...
Macro method:Test consistency (reliability) withSpearman’s rho rank correlation           coefficient
To calculate rs, first assign ranks• Choose a cutoff so number of ranks in each  group is equal.• Handle ties by averaging...
Ranks of journals cited in 2011 volume of                      Law and Human Behavior                                   (b...
Rank correlations among cites in singleissues of Law and Human Behavior are        weak and inconsistent            2011  ...
How does LHB (2011) correlate with       the published list?   "Frequently Cited Journals in Forensic Psychology," Psychol...
Spearman’s rho rank correlation for one volume     of LHB vs. the larger sample is rs=.63                                 ...
Sample size and expected reliabilitySample                   sample size     expected correlation                         ...
Micro Method• Measure coefficients of variation for individual  titles’ frequency of times cited            Coefficient of...
Examples for LHB 2011                                   Times Cited in 2011 Law and Human Behavior                        ...
Assuming the sampled journals    validly represent the topic,             a sample of• n<1,000 works cited can indicate th...
• Even with n>10,000, different samples will  yield different ranks, especially further down  the list,• So a journal’s ra...
Strengths of DIY Citation Analysis• Can target a precise area• Yields data not otherwise available• Objective, quantitativ...
Weaknesses of DIY Citation Analysis• Very time consuming to do well• A really robust ranking requires n>20,000, and  even ...
Tips for publication• Don’t be too parochial or narrow—editors must think their readers  will be interested.• BUT choose s...
Tips for Collection Development• Use citation analysis to assess impact of  specialized journals• Don’t be surprised to fi...
Practical Applications• What’s the top journal on this topic?   – Compile cites from 10-20 articles on the topic to identi...
Steve Blackblacks@strose.edu (518) 458-5494
Upcoming SlideShare
Loading in …5
×

Practical applications of citation analysis

1,818 views

Published on

Much of the demand today to add new journals comes from new multidisciplinary courses and programs. The usefulness of citation analysis as a tool for developing such collections is shown by the importance often placed on published impact factors. But not all libraries can afford access to the Web of Science’s Journal Citation Reports or Scopus, and many researchers have described weaknesses in those sources’ data. Google Scholar is helpful but it, too, has weaknesses. Critiques of these sources of impact factors will be briefly reviewed and a bibliography provided. But the most important reason to do an independent citation analysis is that impact factors reflect citations to journals from all disciplines. To determine the most-cited journals within a multidisciplinary field one should analyze only citations published within the topic area. Independently gathering citation data used to be a tedious process. But the inclusion of works cited in databases such as PsycINFO and ScienceDirect has made gathering data considerably easier, and citation management programs like RefWorks and Zotero speed the process of compiling and organizing citations. How to use these tools to relatively quickly assemble a sample of works cited in a specific topic area will be demonstrated. I’ll also discuss various methods of gathering a valid sample and issues to consider when determining sample size. I’ll conclude with a discussion of the strengths and weaknesses of using do-it-yourself citation analysis for collection development or supporting new program development and offer suggestions for anyone interested in doing a project for the purpose of publishing the results.
Presenter: Steve Black, College of Saint Rose

  • Be the first to comment

Practical applications of citation analysis

  1. 1. Practical Applications of Do-it-Yourself Citation Analysis Steve Black The College of Saint Rose June 10, 2012Accompanying handout: http://www.slideshare.net/NASIG/blackpractical-applications-of-citation-analysis
  2. 2. Citation analysis is the study of patterns and frequencies of citations
  3. 3. Why• Objective, quantitative measurement of impact (relative frequency of citation) of – journals – authors – institutions• Citation analysis can help answer “What are the best journals for ____?”
  4. 4. Critiques of relying on impact factors • data errors • Matthew effect • gaming the system • impact vs. quality vs. importance • global vs. local impact (See bibliography)
  5. 5. Never forget, & remind others when needed,impact factor is but one of many considerations for collection development, paper submissions, tenure & promotion.
  6. 6. Do-It-Yourself Citation Analysis • tailored to specific needs not covered elsewhere • can be quick-and- dirty, exhaustive, or in between • saves expense of Scopus or Web of Knowledge
  7. 7. Basic DIY Approach• Select a target population – journals in a sub-discipline (e.g. forensic psychology) – researchers at an institution, school, department, or even one faculty member (e.g. all the papers published in last 10 years by faculty of Dept. X of Big U.) – subjects or keywords (e.g. phrases on hot research topics)• Select a sample that represents the target population• Compile works cited in the chosen sample• Sort and count the works cited
  8. 8. Google Scholar & Zotero• Google Scholar is good for identifying articles on a topic, BUT – no authority control (which S Black?) – works cited displayed in many formats, if available at all
  9. 9. Freeware can help compile . . .
  10. 10. . . .but Zotero’s not designed for DIY citation analysis• emphasis on full text searching• no authority control• inconsistent data structure• extracting and downloading the right data to a spreadsheet not straightforward
  11. 11. http://kimmonsdesign.com/node/24
  12. 12. Case study in DIY citation analysis: forensic psychologyPublished as“Frequently Cited Journals in Forensic Psychology,”Psychological Reports, v.110, no.1, (2012), p.276-282.
  13. 13. Goal: create ranked list of most-cited journals in forensic psychology• General Method – identify leading journals specific to the field to use as sources of citations – gather citations from recent volumes of those journals – compile cites and count times cited – rank cited journals by frequency of citation
  14. 14. Source Journals Sample• For published article: – 2007, 2008, and 2009 volumes of 6 widely held journals specific to forensic psychology – n=16,518 works cited• For this presentation: – 2011 volume of Law and Human Behavior – n=1378 works cited
  15. 15. Tools used• WorldCat (to identify most widely held journals)• PsycINFO via EBSCOhost (to identify works cited) • RefWorks (to organize works cited) • Excel (to count and rank cited journals)
  16. 16. Retrieve records for articles insource journals, one issue at a time
  17. 17. Add cited references to folder
  18. 18. Export references . . .
  19. 19. . . . to RefWorks
  20. 20. RefWorks creates a temporary folder of the downloads
  21. 21. Create a folder for each issue’s works cited (keep discrete units to allow statistical analysis of reliability)
  22. 22. Move the downloaded cites to the appropriate folder
  23. 23. Export references in tab delimited format
  24. 24. Save as a text document
  25. 25. Open the text file in Excel . . .
  26. 26. . . .by going through 3 steps to convert from text
  27. 27. Work with cited journal titles shown as “Periodical Full”Note thatsometaggedreferencetypes areimprecise, don’tcount the“slop.”
  28. 28. Sort the title column alphabetically
  29. 29. For accurate counts, review titles and remove discrepancies
  30. 30. Count citation frequency(choose a cutoff point equal to about 0.5% of sample)
  31. 31. Record frequency counts
  32. 32. Sort by largest to smallest to create ranked list
  33. 33. Bradford distribution 1:n:n2:n3 roughly holds in this case as 1:3:9:27... Law and Human Behavior 154 Criminal Justice and Behavior 45 Journal of Applied Psychology 45 Behavioral Sciences and the Law 40 130 Journal of Personality and Social Psychology 35 Psychological Assessment 32 Applied Cognitive Psychology 29 American Psychologist 26 Psychiatric Services 26 Journal of Abnormal Psychology 21 Psychology, Public Policy, and Law 21 Personality and Individual Differences 15 Psychological Bulletin 15 220 Assessment 14 Psychological Review 14 Journal of Applied Social Psychology 13 Journal of Experimental Psychology: Applied 13 Psychological Science 13 Psychology, Crime and Law 12 International journal of law and psychiatry 11 Legal and Criminological Psychology 11 Criminology 10 Sex Roles 9 Aggressive Behavior 8 American Journal of Public Health 8 International Journal of Forensic Mental Health 8 International Journal of Offender Therapy and Comparative Criminology 8 Journal of Criminal Law and Criminology 7 Journal of personality assessment 7 Journal of the American Academy of Psychiatry and the Law 7 Aggression and Violent Behavior 6 American Journal of Psychiatry 6 Archives of General Psychiatry 6 Basic and Applied Social Psychology 6 British Journal of Psychiatry 6 Journal of personality disorders 6 Professional Psychology: Research and Practice 6 Virginia Law Review 6 Annual Review of Psychology 5 British Journal of Criminology 5 231
  34. 34. ...even though LHB cites a mix offorensic and general psych journals Law and Human Behavior 154 Criminal Justice and Behavior 45 Journal of Applied Psychology 45 Behavioral Sciences and the Law 40 130 Journal of Personality and Social Psychology 35 Psychological Assessment 32 Applied Cognitive Psychology 29 American Psychologist 26 Psychiatric Services 26 Journal of Abnormal Psychology 21 Psychology, Public Policy, and Law 21 Personality and Individual Differences 15 Psychological Bulletin 15 220
  35. 35. How should we take this neatly ranked list of forensic psych journals?
  36. 36. It depends on the reliability of our rankings• Measure reliability 2 ways: – Macro: variability in ranked lists – Micro: variability in individual titles’ times cited
  37. 37. Macro: How well do ranked lists correlate? Feb 2011 April 2011 June 2011Law and human behavior 28 Law and human behavior 30 Law and human behavior 17Journal of Applied Psychology 13 Psychiatric Services 14 Criminal Justice and Behavior 14Psychological assessment 11 Behavioral Sciences and the Law 12 Journal of personality and social psychology 11Applied Cognitive Psychology 7 Journal of Applied Psychology 9 Personality and Individual Differences 8Journal of abnormal psychology 6 Psychology, Public Policy, and Law 9 Behavioral Sciences and the Law 6Assessment 5 Criminal Justice and Behavior 7 Criminal Justice Policy Review 6Virginia Law Review 5 Journal of personality and social psychology 6 International Journal of Offender Therapy and Comparative Criminology 6Criminal Justice and Behavior 4 Sex Roles 6 Assessment 5Criminology 4 American Psychologist 5 Journal of consulting and clinical psychology 5Wisconsin Law Review 4 International Journal of Forensic Mental Health 5 Journal of Applied Social Psychology 4 ? Micro: How variable are frequencies of citation to LHB, J App Psych, etc.?
  38. 38. Macro method:Test consistency (reliability) withSpearman’s rho rank correlation coefficient
  39. 39. To calculate rs, first assign ranks• Choose a cutoff so number of ranks in each group is equal.• Handle ties by averaging, e.g. tied for 2 & 3 are each ranked 2.5.• Assign bottom rank to titles with no match, e.g. if n=22, unmatched titles are ranked 22.
  40. 40. Ranks of journals cited in 2011 volume of Law and Human Behavior (blanks will be filled with rank = 22)Title cited in LHB 2011 frequency all Allrank FebRank AprilRank JunRank AugRank OctRank DecRankLaw and Human Behavior 154 1 1 1 1 1 1 1Criminal Justice and Behavior 45 2.5 9.5 6 2 2 9.5Journal of Applied Psychology 45 2.5 2 4.5 3 4.5 4Behavioral Sciences and the Law 40 4 3 6 4 2 16Journal of Personality and Social Psychology 35 5 16.5 7.5 3 8.5 9.5 11.5Psychological Assessment 32 6 3 13 8.5 11 4Applied Cognitive Psychology 29 7 4 17.5 13 12 13.5 2American Psychologist 26 8.5 16.5 9.5 8.5 9.5 8.5Psychiatric Services 26 8.5 2 16.5 6Journal of Abnormal Psychology 21 10.5 5 13 9.5 11.5Psychology, Public Policy, and Law 21 10.5 4.5 4.5 16Journal of Consulting and Clinical Psychology 18 12 8.5 5.5 21Personality and Individual Differences 15 13.5 4 16Psychological Bulletin 15 13.5 11.5 13 8.5Assessment 14 15.5 6.5 8.5 21Psychological Review 14 15.5 3 21Journal of Applied Social Psychology 13 18 16.5 17.5 10Journal of Experimental Psychology: Applied 13 18 16.5 8.5Psychological Science 13 18 16.5 11.5Psychology, Crime and Law 12 20 4International journal of law and psychiatry 11 21.5 16.5 13.5Legal and Criminological Psychology 11 21.5 17.5 13 16
  41. 41. Rank correlations among cites in singleissues of Law and Human Behavior are weak and inconsistent 2011 Feb Apr June Aug Oct Dec 2011 1 Feb 0.24 1 Apr 0.47 -0.34 1 June 0.26 -0.05 -0.2 1 Aug 0.63 0.05 0.4 0.15 1 Oct 0.59 -0.05 0.46 -0.47 0.11 1 Dec 0.23 0.25 -0.66 -0.42 -0.03 -0.02 1
  42. 42. How does LHB (2011) correlate with the published list? "Frequently Cited Journals in Forensic Psychology," Psychological Reports v.110, no. 1, Feb. 2012: 276–282 Title Percent of citations times cited (n=16,518) rank Law and Human Behavior 5.71% 1243 1 Behavioral Sciences and the Law 2.46% 536 2 Criminal Justice and Behavior 1.79% 390 3 British Journal of Psychiatry 1.63% 354 4 Journal of Applied Psychology 1.62% 352 5 Journal of Personality and Social Psychology 1.50% 326 6 American Journal of Psychiatry 1.34% 291 7 Journal of Forensic Psychiatry and Psychology 1.30% 283 8 Applied Cognitive Psychology 1.20% 261 9 Psychology, Public Policy, and Law 1.15% 250 10 Psychology, Crime, and Law 1.08% 236 11 Psychological Assessment 1.03% 224 12 Personality and Individual Differences 1.01% 221 13 American Psychologist 0.99% 215 14 Legal and Criminological Psychology 0.96% 210 15 Journal of Consulting and Clinical Psychology 0.94% 204 16 Journal of the American Academy of Psychiatry and the Law 0.92% 201 17.5 Sexual Abuse: A Journal of Research and Treatment 0.92% 201 17.5 Archives of General Psychiatry 0.91% 198 19 Psychiatric Services 0.85% 185 20 Journal of Applied Social Psychology 0.84% 183 21 Psychological Bulletin 0.83% 180 22 Journal of Abnormal Psychology 0.70% 153 23
  43. 43. Spearman’s rho rank correlation for one volume of LHB vs. the larger sample is rs=.63 LHB Psych 2011 Reports D D2title rank rankLaw and Human Behavior 1 1 0 0Criminal Justice and Behavior 2.5 3 -0.5 0.25Journal of Applied Psychology 2.5 5 -2.5 6.25Behavioral Sciences and the Law 4 2 2 4Journal of Personality and Social Psychology 5 6 -1 1Psychological Assessment 6 12 -6 36Applied Cognitive Psychology 7 9 -2 4American Psychologist 8.5 14 -5.5 30.25 rs =1-(6*684)/22*(222-1)Psychiatric Services 8.5 20 -11.5 132.25 = 1 – (4104/10626)Journal of Abnormal Psychology 10.5 22 -11.5 132.25Psychology, Public Policy, and Law 10.5 10 0.5 0.25 = 1- .37Journal of Consulting and ClinicalPsychology 12 16 -4 16 rs = .63Personality and Individual Differences 13.5 13 0.5 0.25Psychological Bulletin 13.5 22 -8.5 72.25Assessment 15.5 22 -6.5 42.25Psychological Review 15.5 22 -6.5 42.25Journal of Applied Social Psychology 18 21 -3 9Journal of Experimental Psychology: Applied 18 22 -4 16Psychological Science 18 22 -4 16Psychology, Crime and Law 20 11 9 81International journal of law and psychiatry 21.5 22 -0.5 0.25Legal and Criminological Psychology 21.5 15 6.5 42.25 sum of D squared 684
  44. 44. Sample size and expected reliabilitySample sample size expected correlation (works cited) with populationone issue of journal n=250 very lowone volume of journal n=1500 moderateseveral volumes of 3-5 n>10,000 strongjournals
  45. 45. Micro Method• Measure coefficients of variation for individual titles’ frequency of times cited Coefficient of variation = standard deviation ÷ mean
  46. 46. Examples for LHB 2011 Times Cited in 2011 Law and Human Behavior Feb Apr June Aug Oct Dec σX mean Coeff. VariationLaw and Human Behavior 28 30 17 25 27 27 4.55 25.67 18%Journal of Applied Psychology 13 9 0 8 8 7 4.23 7.50 56%Psychology, Public Policy, and Law 1 9 0 0 8 3 4.04 3.50 115% Greater Coefficient of Variation = lower reliability • Top journal pretty stable • Titles lower down the ranked list are more variable
  47. 47. Assuming the sampled journals validly represent the topic, a sample of• n<1,000 works cited can indicate the top journal (or tell you if there is none)• n>1,000 can generate a rough indication of leading journals• n>10,000 can create a useful ranked list
  48. 48. • Even with n>10,000, different samples will yield different ranks, especially further down the list,• So a journal’s rank must ALWAYS be taken as an approximation of its “true” ranking.
  49. 49. Strengths of DIY Citation Analysis• Can target a precise area• Yields data not otherwise available• Objective, quantitative collection development tool• Results may be of fairly broad interest (publishable!)• If project is well organized, data can be gathered & sorted by research assistant(s)• Interdisciplinary--can team with faculty outside the library
  50. 50. Weaknesses of DIY Citation Analysis• Very time consuming to do well• A really robust ranking requires n>20,000, and even that’s not definitive• Works cited may not be readily available (few databases make it as easy as PsycINFO)• It’s recreating the (very expensive) wheel – better to use Scopus or Web of Knowledge if you can
  51. 51. Tips for publication• Don’t be too parochial or narrow—editors must think their readers will be interested.• BUT choose something not already in Journal Citation Reports• Be very thorough with your lit review before gathering data (search disciplinary databases as well as the library literature)• Gather citations and test sample before solidifying your method• Carefully select and describe the sample – representative – clearly defined – robust – good face validity• Group data and run statistics by journal volume• Consider submitting to a journal in the topic area
  52. 52. Tips for Collection Development• Use citation analysis to assess impact of specialized journals• Don’t be surprised to find surprises• Judge new titles by other criteria• Use ranked lists for assessment evidence
  53. 53. Practical Applications• What’s the top journal on this topic? – Compile cites from 10-20 articles on the topic to identify it (or find there’s not one)• What does the library need to support a proposed new major? – Compile cites from 1-2 volumes of 2-4 journals to flag titles not already available• What’s that herd of cats in Dept. X REALLY use? – Analyze publications & dissertations from last decade• OMG I’m up for tenure soon! – Identify a topic and do a thorough analysis of 2-3 volumes of 4-6 journals; include reliability tests
  54. 54. Steve Blackblacks@strose.edu (518) 458-5494

×