Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluating information


Published on

Published in: Education, Technology
  • Be the first to comment

Evaluating information

  1. 1. PG6009 Unit 2: Using the Web Effectively & Evaluating Research Resources
  2. 2. Module Structure Unit 1: Research Resource Discovery Unit 2: Using the Web Effectively & Evaluating Research Resource Results Unit 3: Research – Keeping Up-to-date Unit 4:Managing Your Information Unit 6: Publishing / Disseminating Your Research Unit 5: Ethics in Using Research Information
  3. 3. Unit 2: Aim and Objectives • To provide an overview of web resources for research and networking. • To examine the importance of critical thinking skills and to consider the use of an analytical framework within which to evaluate the information you find.
  4. 4. Unit 2: Learning Outcomes On completion of this module the learner should be able to: •Apply best-practice techniques for identifying, locating and utilising key web resources in your research area. •Identify best-practices in effective networking and the effective use of web technologies. •Apply criteria for critical appraisal. •Use an analytical approach to evaluating information. •Contextualise information in your own research area.
  5. 5. Part 1: Using the Web More Effectively 1. Google & other search engines, portals/gateways, repositories 2. Web Technologies for Information Exchange - social bookmarking, academic networking, research profiles etc. Part 2: Evaluating Information 1. Importance of evaluation 2. Critical Analysis/Appraisal 3. Analytical framework 4. Citation analysis: an introduction Topics
  6. 6. Searching the Web More Effectively: Overview Web: Indexed Web 4.7 billion pages?? Google: uses index/database of web pages compiled from sites found by it’s spider programs. Full text of sites sent to Google index. ‘Pagerank’ retrieves based on word occurrence, proximity, location on page, links to the page, traffic etc. about 100-200 ‘ingredients’. For the most part a keyword search. See here for more Anyone can publish: quality control? Problem: finding relevant scholarly material Quantity and quality of information
  7. 7. Tips 1. Use ‘advanced search’ on Google and other search engines 2. Use Google Scholar, Google Books & ‘more’ 3. Try other search engines – not just Google 4. Don’t always use search engines, go straight to good portals and sites suitable for your research area 5. Use ‘social media’ to locate, bookmark and share information and to network 6. Always evaluate your findings for quality
  8. 8. 1. Use Advanced Search Techniques Won’t guarantee quality , but may help control quantity •Consider: word order, word choice, ‘stop words’ •Use Google advanced search, operators, domain search, phrase search etc. see: Inside search . Remember that Google personalises your search. See ‘Verbatim’ tool. See: Search the Web Effectively guide
  9. 9. 2. Use Google Scholar/Books & ‘more’ • Works better on-campus, or from home through the Databases A-Z (log in to your Library account) • If no full-text, check the journal portal and the Library catalogue as usual • See advanced search and email alerts on Google Scholar • See also: Microsoft Academic Search as alternative
  10. 10. Practice • Look for scholarly web material on: Alcohol and blood pressure
  11. 11. 3. Use Other Search Engines Don’t use same search engine for everything - try others (examples below). See here also: •General: Exalead, Ask Jeeves, Yahoo, Bing, Kartoo,, ZOO, DuckDuckGo, Yippy •Academic/educational: Microsoft Academic Search, iSeek, BASE
  12. 12. 4. Don’t Always Use Search Engines Save time by going straight to sites likely to be the most useful - gateways and portals can help: Look at UCC Library LIBGUIDES •Search catalogues from elsewhere: COPAC, British Library, Library of Congress, The European Library, Worldcat, World Digital Library, H-Net, Also: Hathi Trust Digital Library, Project Gutenberg, OAPEN, DOAB, Open Grey Open Repositories: CORA , RIAN, OAIster, Openaire, ETHOS, DART-Europe E-theses Portal, Irish Social Science Data Archive, Irish Qualitative Data Archive [Unit 6]
  13. 13. 5: Use the Social Web/Media Mostly free, easy to use, shared interests, interactive •Why use it as a researcher? 1. Find (& organise) information that you may not find through regular sources 2. Network with others who share an interest in your research area
  14. 14. Research Environment Who are ‘Researchers’? Masters, doctoral, contract researchers, early career researchers, established academic staff, senior researchers, experts New Review of Information Networking (2007) 13(2): 81-99 Important trends: • Multidisciplinary research: dispersed across disciplines, geographical and other boundaries (e.g. UCC interdisciplinary institutes) • Public & Private research: public-private partnership, knowledge transfer, technological transfer, business-led collaborative research, accountability. See: UCC Research Support Services • Global nature of research: what’s going on elsewhere? Partnership between universities: Asia, South America • Importance of sharing information and building effective networks
  15. 15. Networking (1.) Formal and Informal: at every stage of the scholarly communication process • Informal: colleagues, acquaintances, friends, family, social events etc. • Formal: professional networking groups, academic community, project teams, committees, training programmes, conferences, presentations etc. • Why network? For information gathering - your research area - your college/university/research community - your career and professional development Also: disseminating your ideas/research
  16. 16. Networking (2.) • Your network supports you: - Group projects - Staff/team conflicts - Peer issues - ‘Managing’ your boss/supervisor/director of studies • Sustaining your network: - Ongoing, strategic, practiced consistently - Information sharing must be mutually rewarding • Sharing Information: formally and informally
  17. 17. • Bookmarking: store bookmarks remotely; some allow you to share with others. Examples: General: Evernote, Diigo, Scoop-it, Instapaper , Delicious, Stache, Pocket Academic: Endnote Web, CiteUlike, Mendeley, Papers, Zotero • Feeds: [Unit 3] Examples: Feedly, Netvibes
  18. 18. • Social/Academic Networking & Profiles: Create a profile, join a network e.g. Facebook, Google+, LinkedIn,, ResearchGate, Methodspace, Academic Room, MLA Commons, Piirus Google Scholar Citations, ORCID, ResearcherID
  19. 19. • Blogs, Twitter [Unit 3] • Wikis: create/edit web content; many contributors, collaborative e.g. Wikipedia - Create: Wikispaces (basic is free), Mediawiki - Find: • Some other resources: - Dropbox, Google Drive,, OneDrive: upload, share - Slideshare: share presentations - Google groups: have discussions and upload documents etc.
  20. 20. Find/discover (searching, alerts, feeds): Research databases, Web, Google Scholar, Twitter, Blogs, Bookmarking sites, Academic networking sites Store temporarily in bookmarking site (perhaps) Store in Endnote, Zotero etc.
  21. 21. Practice Take a look at: • 101 Innovations in Scholarly Communication - the Changing Resea • How do you find researchers who want to collaborate? • ‘’ research networking site • ‘ResearchGate’ research networking site • Digital-identity-health-check-for-academics.pdf • • Feeling Better Connected’: Academics’ Use of Social Media
  22. 22. Part 2: Overview: Evaluating Information Sources • Critically important for 4th level research. Quality sources: fundamental to developing a review of the literature that can be used as a foundation on which new research can be built. • A critical approach helps determine relevance and value to your field. • Traditionally: journals, monographs, conference proceedings, primary sources. Now new information formats e.g. blogs, wikis, discussion lists, open access journals, open repositories, preprint repositories etc. • How can you decide if material is of sufficient quality, suitable for inclusion in your literature review?
  23. 23. Overview: Evaluating Information Sources Generally we think of: • Scholarly resources: aimed at those within the field; disseminate research within that discipline. Scholarly methods, make claims that are valid and trustworthy. • Popular resources: aimed at a wider public/mass audience; entertain, inform, promote viewpoints, sell products & services (vested interests). So at basic level, is it: • Factual, methodical, ‘scientific’, based on clearly referenced sources and documentation? OR • subjective, journalistic, personal accounts/impressions, opinion?
  24. 24. But: even material accepted for publication not always reliable: E.G. 1. Research of Andrew Wakefield (‘MMR controversy’) o The Lancet 02/28/98, Vol. 351 Issue 9103, p637 o The Lancet 06/03/04, Volume 363 Issue 9411, p750 o The Lancet 06/02/10, Volume 375 Issue9713, p445 See also: o The crash and burn of an autism guru o Booster shots o Why did the Lancet take so long?
  25. 25. E.G. 2. Dutch Social Psychologist Diederik Stapel Articles in: Science, European Journal of Social Psychology, Journal of Personality and Social Psychology, Journal of Experimental Social Psychology etc. •Tilburg University Press release •Dutch News article •Final Report on Stapel Also Blames Field As a Whole •Dutch Psychologist Diederik Stapel Relinquishes His Ph.D
  26. 26. More examples: • ‘Plastic Fantastic’ • ‘Elsevier published 6 fake journals’ • ‘Merchants of Doubt’ 174.95 ORES • Woo-Suk-Hwang • Joachim Boldt • Office of Research Integrity •
  27. 27. Critical Analysis/Appraisal • Focussed approach: process of reflective, reasonable, rational thinking to gather, interpret and evaluate information • Adopting an analytical and reflective mind-set at every stage of the research process will help you to - find the best sources - evaluate the information you find there - make decisions about your research methodology - discuss new findings - weigh-up evidence and form conclusions - make recommendations and contribute effectively to your field i.e. your literature review becomes a solid foundation on which you can build new research, ideas, theories etc.
  28. 28. An Analytical Framework An analytical approach to assessing quality/value of information: - Relevance - Authority - Method - Objectivity - Presentation - Currency of information Apply this checklist to all information sources that you use
  29. 29. 1. Relevance Why you wish to include a source; appropriate in the context of your research needs? Screen content and look for general clues before deciding whether to digest or reject: – Books: look at title, keywords, contents, index – Journal: look at abstract, keywords, descriptors – Websites: look at title bar, document title, links to and from the site, author
  30. 30. Relevance (contd.) Focus on: • Level: detailed/general/simple? • Emphasis: not always obvious from keywords/abstract • Geographical: what countries/regions are included? • Context of your own research: a unique insight into an aspect of your own research? Are ideas confirmed/refuted by the research? N.B. save time, reduce information overload
  31. 31. 2. Authority • Author: someone whose opinion or testimony is accepted; affiliated, qualified expert, academic credentials OR journalist/other author/commentator. Has their research been frequently cited? Is there a ‘H index’? In what publications? Peer- reviewed? Where indexed? Impact factor? (see Citation Analysis) For books: who is the publisher? Established academic publisher within your field, university press? etc. • Organisation: is it commercial, non-profit, government, research/educational? Vested interest? Contactable?
  32. 32. Authority (contd.) Web Sites: (see also the ‘Internet Detective’) • Look at the ‘about’ or ‘who are we’ section of the site • How well established is it and when was the site last updated? • Evidence of sponsorship? • Is it clear who wrote the content and is that person contactable? • URL will often give a clue to type of organisation/country of origin: - Commercial company? (.com or or .biz) - A non-profit organisation? (.org) - A government body? (.gov) - A research/educational organisation? (.org or or .edu) e.g. v
  33. 33. 3. Method of Production & Methodology • Type of Publication (Rem: anyone can publish on the web) - Peer-review procedure and instructions for authors Peer review/Refereed/Juried: process by which an academic journal passes a paper submitted for publication to independent experts for comments on its suitability and worth; refereeing. e.g. About the BMJ , Resources for Authors e.g. Addiction - Editorial board members check journal website e.g. BMJ Editorial Advisory Board See: Google Scholar & Junk Science See also: Ulrich’s Periodicals Directory
  34. 34. Method of Production & Methodology (contd.) • Research Methodology: - Validity: tests what it aims to test? claims made that are trustworthy? Applicable to the case or circumstances? - Reliability: Extent to which a measurement made repeatedly in identical circumstances will yield concordant results, consistent. Methods appropriate in terms of sample size, use of control groups, quantitative or qualitative survey design etc. to ensure reproducibility?
  35. 35. 4. Objectivity: What is the objective? To further understanding? To contribute to the field OR news, entertainment, opinion; editorial agenda/viewpoint Sponsors? bias/vested interests? Hidden bias, whether deliberate or not (GM foods, organic foods, climate change). Backed up with evidence? Clearly referenced in a bibliography? Has the evidence been interpreted in a balanced/unbiased manner?
  36. 36. How is the information presented? - Colour & font, general appearance - Language and writing style serious/sober OR glossy appearance - Use of diagrams and images - Structure and layout; logical? - Quality of reproduction - Advertising: minimal/substantial; target audience 5. Presentation
  37. 37. 6. Currency How up to date is the information/research? • The importance of currency depends on the context within which you plan to refer to that research in your literature review. • Factors to consider include: - Is it clear when the information was produced? - Does the date of the information meet my requirements? - Is it obsolete or superseded?
  38. 38. Scholarly Communication Process • “Scholarly communication is the system through which research and other scholarly writings are created, evaluated for quality, disseminated to the scholarly community, and preserved for future use.” Source: ACRL (Association of College and Research Libraries) • A Process: From funding to the eventual dissemination of research results through formal/traditional and less formal mechanisms
  39. 39. Scholarly Communication • Traditional Approach: from informal to formal with communication of research occurring at every stage: Informal: meetings, discussions, seminars, emails, blogs, social networking sites etc. Report on-going research at conference (conference proceedings) Publication in an academic journal, book AND/OR completion of thesis Indexed in research databases, catalogues, repositories • Recent changes in scholarly communication: new opportunities and challenges e.g. self-archiving, repositories, open access publishing, digital humanities, social web
  40. 40. Peer-review • ‘Peer-reviewed’/‘refereed’ – academic/scholarly ‘The process by which an academic journal passes a paper submitted for publication to independent experts for comments on it’s suitability and worth; refereeing’ • Accepted/rejected: contribution to the field/new ideas, bias/conflicts of interest, suitability for journal • Types: Double-blind, Single-bind, Open • Future: Open online peer-review? e.g. blogs, Twitter
  41. 41. Other Periodicals (i.e. Not peer-reviewed): 1. Popular Magazines: (written by journalists/commentators) - Newspapers - Substantive news e.g. ‘The Economist’, ‘National Geographic’, ‘New Scientist’ - Other Magazines 2. Trade Journals (specific industry, enable practitioners share market and product information within an industry) e.g. ‘Macworld’, ‘Restaurant Business’, ‘Chemical Week’. ‘Beverage World’, ‘Computerworld’ etc.
  42. 42. Citation Analysis (Bibliometrics) • Number of times a paper or researcher is cited by other scholars in the field; assumes influential researchers/authors and important works cited more often. Citations can be used to develop metrics such as h-index, impact factor etc. • Where to find these metrics: 1. Web of Science (Thomson Reuters) 9,000+ peer-reviewed journals 2. Scopus (Elsevier) 16,000+ peer-reviewed journals; more than 4,000 international publishers; 1996 on 3. Scimago Country & Journal Rank Database journals and country-specific scientific indicators developed from data contained in ‘Scopus.’ 4. Google Scholar provides links to ‘cited by’ information. ‘Publish or Perish’ software can generate metrics based on this.
  43. 43. 3. Method of Production & Methodology • Type of Publication (Rem: anyone can publish on the web) - Peer-review procedure and instructions for authors Peer review/Refereed/Juried: process by which an academic journal passes a paper submitted for publication to independent experts for comments on its suitability and worth; refereeing. e.g. About the BMJ , Resources for Authors e.g. Addiction - Editorial board members check journal website e.g. BMJ Editorial Advisory Board See: Google Scholar & Junk Science See also: Ulrich’s Periodicals Directory
  44. 44. Example: Number of times cited Article Title: Identification of pancreatic cancer stem cells Author(s): Li CW, Heidt DG, Dalerba P, et al. Source: CANCER RESEARCH Volume: 67 Issue: 3 Pages: 1030-1037 Published: FEB 1 2007 1. Go to Web of Knowledge 2. Pick ‘Web of Science’ 3. Enter the Title to locate article 4. Note Number of ‘Times Cited’ 5. Go to ‘Scopus’ 6. Enter the Title to locate article 7. Note number of ‘citations’ 5. Go to Google Scholar 6. Type article title in box 7. Note ‘Cited by -’ N.B Number of cites can only provide an indication of ‘quality’.
  45. 45. Exercise: Mullally B.J. et al. (2008) ‘Prevalence of smoking among bar workers prior to the Republic of Ireland smokefree workplace legislation’, Irish Journal of Medical Science, 177(4): 309-16 • How many times has the same article been cited in Web of Science, Scopus and Google Scholar?
  46. 46. Journal Impact • Journal Impact Factor: average number of times articles from the journal published in past two years have been cited in JCR year Check: • ‘Journal Citation Reports’ (Journal Performance Metrics) includes impact factors. Part of Thomson Reuter’s Web of Science • Scimago Country & Journal Rank Database: (free) journals and country scientific indicators developed from information in ‘Scopus’
  47. 47. Journal impact Scopus and Scimago Scimago Country & Journal Rank Database: (free) journals and country scientific indicators developed from information in ‘Scopus’ - better for humanities and social sciences • N.B. SJR: ‘Scimago Journal Rank’ reflects prestige of source where such citations come from. Covers a 3 year citation window. • SNIP: ‘Source normalized impact per paper’: corrects for differences in the frequency of citation across research fields- citations given a weighting based on the total number of citations in that field. N.B. Scopus only
  48. 48. Example: JCR impact factor 1. What is the impact factor of the Journal ‘The Lancet’? 1. Go to Web of Science 2. Click on ‘Journal Citation Reports’ at the top of the page 3. In the ‘Go to Journal Profile’ box type in the journal title & Search 4. Once the journal is displayed, scroll down to see the Key Indicators 2. What are the highest impact journals for Oncology? 1. Go to Web of Science 2. Click on ‘Journal Citation Reports’ at the top of the page 3. Click on ‘Select Categories’ on the left of the screen 4. Choose a category by ticking the box 5. Click on ‘Submit’ 6. Scroll down to find the journal metrics for the chosen subject
  49. 49. Exercise • What is the 2014 impact factor for the journal ‘Cell’? • What Psychiatry journal had the highest impact factor for 2014?
  50. 50. H-Index A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-h) papers have no more than h citations each •highest number of papers a scientist has that have each received at least that number of citations e.g. someone with h-index 50 has written 50 papers that have each had at least 50 citations. Where to find it: – Web of Science – Scopus – Google Scholar – Publish or Perish
  51. 51. Other Metrics • InCites: uses Web of Science data to generate metrics for institutions and units within those institutions - benchmarking • Essential Science Indicators: Institution rankings and top papers derived from Wed of Science Data ‘NATL UNIV IRELAND UNIV COLL CORK’ • Webometrics: quantifying the impact of web sites • Altmetrics: attempt to track the impact of scholarly works in the social web • National systems to classify journals, publishers
  52. 52. How many times cited? Stapel, DA. And Koomen, W (2001). “I, we, and the effects of others on me: How self- construal level moderates social comparison effects.” Journal of Personality and Social Psychology 80(5): 766- 781. Note: Citation data is then used to create h-index for authors, impact factor of the journal, metrics for the university etc. etc.
  53. 53. Bibliometrics- limitations Useful tools, but not a measure of true quality e.g.: • current popularity of topic & availability of article/journal: increases citations / higher IF • IF based on average over all articles: underestimates citations of top cited articles, exaggerates number of citations of the average article • comparison of impact factors between different fields is invalid e.g. not as relevant for literature (books citing other books) • coverage differs in each resource (Web of Science, Scopus, Google Scholar); none is complete and reliable e.g. for IF only ISI database journals used; undercounts number of citations from other journals e.g. ‘less-developed’ countries, other languages
  54. 54. Bibliometrics- limitations (contd.) • Why have the papers been cited? e.g. Review articles, Wakefield • Well over 50% of papers are never cited • What about books, book chapters, conference papers, digital humanities etc.? • Takes time for papers to collect citations, especially in some fields • H-index is a poor metric for young researchers • H-index underestimates the importance of seminal articles • Pressure on academics to publish may lead to a drop in quality? • Google metrics are not yet reliable enough in isolation
  55. 55. Impact Factor- limitations (contd.) • ‘Like nuclear energy, the impact factor is a mixed blessing.’ • ‘The use of journal impacts in evaluating individuals has its inherent dangers. In an ideal world, evaluators would read each article and make personal judgments.’ Garfield, E. (2006) ‘The History and Meaning of the Impact Factor’. JAMA, 295(1): pp 90-93 [Garfield invented scholarly citation in the 1960’s]
  56. 56. H-index - limitations (contd.) ‘Obviously a single number can never give more than a rough approximation to an individual’s multifaceted profile, and many other factors should be considered in combination in evaluating an individual.’ Hirsch, J.E. (2005) ‘An index to quantify an individual's scientific research output’. arXiv:physics/0508025 v5 29 Sep 2006 (Hirsch proposed the H-index in 2005)
  57. 57. Final word: • Can’t always rely on citation data etc. • Each researcher must make their own judgment about the ‘quality’ and suitability of any article or information source. • Depends a lot on the context i.e. how the research is to be used or referred to in your literature review. You may need to draw on material from newspapers, conferences, web material etc. with no peer review available. • Refer to the framework/checklist outlined above and check with your supervisor when in doubt. • Evaluation is an art; there no perfect indicator of quality. You need to look for clues, and ultimately judge on the basis of usefulness for your research question
  58. 58. Slides: information Further reading: