Pringle Research Eval
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Pringle Research Eval

on

  • 615 views

 

Statistics

Views

Total Views
615
Views on SlideShare
615
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Pringle Research Eval Presentation Transcript

  • 1. Thomson Reuters and Research Evaluation in the Humanities James Pringle, VP, Development Academic & Government, Thomson Reuters Scientific [email_address] September 19, 2008
  • 2. Citations in the 21 st Century
    • The “User-Generated Content” of formal scholarly communications
    • Fundamental to effective evaluation & management
    • Frequently mis-used and misunderstood
    • May have a place in humanities research evaluation
  • 3. Changing Scholarly Communications Value Chain
  • 4. A Contentious Issue
    • “… impact factors and citations are dominating minds, distorting behaviour and determining careers….
    • … The main villains are fashion, the management cult, and the politics of our time, all of which favour numerical evaluation of ‘performance’ and reward compliance.”
    • --Lawrence, P. A. (2007).
  • 5. The Management Challenge
    • “ Metrics democratization” leads to confusion
    • Reasonable efforts by university administrators to derive objective measures for promotion and tenure decisions free of bias.
    • Pressures on these administrators to upgrade the reputation and quality of their programs to compete in global academic markets.
    • Global competition in scholarship, leading to national policies designed to increase global competitiveness and demonstrate national achievement.
  • 6. Some Rules for Good Use of Citation Data
        • Consider whether available data can address question
        • Compare like with like
        • Use relative measures, not just absolute counts
        • Recognize skewed nature of citation data
        • Confirm data collected are relevant to question
        • Ask whether the results are reasonable
    ---Using Bibliometrics in Evaluating Research http://scientific.thomsonreuters.com/bibliometrics2/
  • 7. Institutional Research Evaluation Workflow
    • Supporting evaluation workflow:
      • Tools
      • Content
      • Workflow Services
  • 8. Government agencies/funding organizations Individuals Faculty, staff, students University Departments Institutional research, academic affairs, tech transfer, etc. We want to know how the funds that we have granted have been effective in promoting research. External Entities University Management Management, including committees, provost, vice provosts I’m responsible for seeing how we’re doing overall in performing in terms of bringing in research dollars in overall publications.” For those aspirational peers...I want to go in and evaluate the faculty and those departments to give me a direct comparison, then I get every single faculty member and run them through the citation index” “ We have built a research information system which is fully integrated with all of the systems—library, research office, grants, and the university -- it’s CV driven, and so every researcher in the institution can access a web based, online view and over time create a research profile for themselves containing up-to-date publications” “ Every author should have a unique identifier in order to retrieve unequivocally his/her articles . Citation Metrics in Research Evaluation
  • 9.
    • US, NSF: biennial Science & Engineering Indicators report (1974 - )
    • European Union: EC’s DG XII (Research Directorate)
    • UK: Office of Science & Technology; Higher Education Funding Council
    • Canada: NSERC, FRSQ (Quebec), Alberta Research Council
    • France: Min. de la Recherche, OST - Paris, CNRS
    • Germany: Max Planck Society, several gov’t labs, DKFZ, MDC
    • Italy: CRUI (University Rectors) MURST (Ministry of Research, CNR
    • Spain: CSIC (Spanish Science Agency), CIRIT (Catalonia)
    • Japan: National Institute of Informatics, Ministry of Education, Ministry of Economy, Trade & Industry
    • People’s Republic of China: Chinese Academy of Science
    • Korea: Korea Research Foundation, Korea Advanced Inst. Of S&T
    • Australia: Australian Academy of Science, gov’t lab CSIRO
    • Turkey: ULAKBIM
    • Others include: New Zealand, S. Africa, Portugal, Ireland, Switzerland, Austria, Poland, Czech Republic, Singapore, Malaysia, Thailand, Sweden, Norway, Denmark, Finland, Mexico, Brazil, Chile, Argentina, Uruguay, Russia… and more!
    Nations and Agencies using TS Evaluative Reports
  • 10. Citations and Evaluation in the Humanities
    • Medium of Scholarly Communication
      • Importance of monograph (e.g. Cronin & La Barre, 2004)
    • Disciplinary Role of Citations
      • Intellectual debt or literary artifact?
    • Usefulness of journal citations in humanities evaluation
      • High level of citations to non-journal literature (e.g. Larivere, 2006)
      • Useful in aggregate for specific disciplines? (e.g. Oppenheim, 2008)
      • Useful as indicators of non-journal literature? (e.g. Butler, 2006)
    • Characteristics of the ISI Citation Databases
      • Regional coverage issues (Archimbault, 2006; Nederhof, 2006)
      • Increasing Coverage in ISI Citation Databases
  • 11. Growth of Journal Coverage in WoS: 2002-2008 Web of Science 2002 2008 Growth Variance Total Journal Coverage 8,607 10,185 1,578 +18%
  • 12. Growth of Journal Coverage in WoS Focus: Regions Web of Science: Regions 2002 2008 Growth Variance Asia Pacific 497 767 270 +54% European Union 1684 2155 471 +28% Middle East/Africa 54 109 55 +102% Latin America 63 160 97 +154%
  • 13. Growth of Journals in Arts & Humanities 2002-2008 Overall Growth: Regional Growth: 2002 2008 Growth Variance A&H Coverage 1122 1238 116 10% Region 2002 2008 Growth Variance Asia Pacific 27 40 13 48% European Union 532 626 94 18% Latin America 7 16 9 129% Middle East/Africa 8 15 7 88% North America 548 541 -7 -1%
  • 14. Growth of Papers in Humanities Fields 1981-2007
  • 15. Comparison with European Reference Index for the Humanities (ERIH) Journal List European Reference Index for the Humanities (ERIH), http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html ERIH Category # ERIH Titles (Total) # Shared  w/ WoS Percent in WoS Anthropology, Evolutionary 36 27 75% Anthropology, Social 195 54 28% Archaeology 390 50 13% Art, Architectural & Design history 418 76 18% Classical Studies 208 41 20% Gender Studies 108 42 39% History 757 192 25% History & Philosophy of Science 146 55 38% Linguistics 542 149 27% Literature 728 215 30% Musicology 160 58 36% Pedagogical & Educational Research 434 151 35% Philosophy 246 89 36% Psychology 532 503 95% Religious Studies_Theology 305 58 19% Total 5205 1760 34%
  • 16. Comparison with ERIH “A” List ERIH Category ERIH "A List" Titles Included in WoS Percent of "A List" in WoS Anthropology, Evolutionary 8 8 100% Anthropology, Social 29 19 66% Archaeology 69 31 45% Art, Architectural & Design history 101 47 47% Classical Studies 47 16 34% Gender Studies 28 21 75% History 98 72 73% History & Philosophy of Science 25 20 80% Linguistics 83 55 66% Literature 170 93 55% Musicology 20 16 80% Pedagogical & Educational Research 73 50 68% Philosophy 23 19 83% Psychology 184 184 100% Religious Studies_Theology 53 20 38% Total 1011 671 66%
  • 17. Comparison with ERIH “B” List ERIH Category ERIH "B List" "B List" in WoS Percent of "B List" in WoS Anthropology, Evolutionary 20 17 85% Anthropology, Social 87 32 37% Archaeology 159 14 9% Art, Architectural & Design history 192 24 13% Classical Studies 86 20 23% Gender Studies 54 19 35% History 303 98 32% History & Philosophy of Science 80 31 39% Linguistics 195 69 35% Literature 343 104 30% Musicology 83 27 33% Pedagogical & Educational Research 274 97 35% Philosophy 104 42 40% Psychology 264 260 98% Religious Studies_Theology 168 35 21% Total 2412 889 37%
  • 18. References
    • Archambault, E., Vignola-Gagne, E., Côté, G., Larivière, Gingras, Y., “Benchmarking Scientific Output in the Social Sciences and Humanities: The Limits of Existing Databases,” Scientometrics , 68(3) 329-342, 2006.
    • Butler, L., Visser, M., “Extending Citation Analysis to Non-Source Items,” Scientometrics , 66(2): 327-343, 2006.
    • Cronin, B., and La Barre, K., “Mickey Mouse and Milton: Book Publishing in the Humanities”, Learned Publishing , 17(2): 85-98, 2004.
    • European Reference Index for the Humanities (ERIH), http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html
    • Larivière, V., Archambault, E., Gingras, Y, Vignola-Gagné, “The Place of Serials in Referencing Practices: Comparing Natural Sciences and Engineering with Social Sciences and Humanities,” Journal of the American Society for Information Science and Technology , 57(8): 997-1004, 2006.
    • Lawrence, P. A. (2007). "The Mismeasurement of Science." Current Biology 17 (15): R583-585.
    • Nederhof, A., “Bibliometric Monitoring of Research Performance in the Social Science and Humanities: A Review,” Scientometrics , 66(1) 81-100, 2006.
    • Oppenheim, C., Summers, M., “Citation Counts and the Research Assessment Exercise, Part VI: Unit of Assessment 67 (Music),” Information Research—An International Electronic Journal , 13(2): 342, June 2008.
    • Using Bibliometrics in Evaluating Research, http://scientific.thomsonreuters.com/bibliometrics2/
  • 19. Thank You!