Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Chain of Trust, a web quality assessment tool


Published on

Presentation for the School of Dentistry Bootcamp series on April 23, 2008. Uploaded originally at that time, but Slideshare for some inexplicable reason deleted the file. Hope it sticks this time. The Chain of Trust / Levels of Evidence tool was originally developed for use by college undergraduate students, shown adapted here for use in healthcare. It is appropriate for a wide variety of audiences.

  • Be the first to comment

  • Be the first to like this

Chain of Trust, a web quality assessment tool

  1. 1. The Chain of Trust — Quality and Credibility in Health Information Sources PF Anderson, [email_address] April 23, 2008 © 2008 Regents of the University of Michigan. All rights reserved.
  2. 2. Background <ul><li>In 2006 I was asked by two nursing faculty to develop: </li></ul><ul><ul><li>a quality assessment tool </li></ul></ul><ul><ul><li>for both health information and term paper sources </li></ul></ul><ul><ul><li>that could be used with incoming freshman </li></ul></ul><ul><ul><li>as an assignment. </li></ul></ul><ul><li>Existing tools were either too specific, too difficult to use, or not appropriate for the target audience. </li></ul>
  3. 3. My earlier tools <ul><li>Evaluation Tool for Clinician-Oriented Web Sites </li></ul><ul><ul><li> </li></ul></ul><ul><li>Evaluation Tool for Patient-Oriented Web Sites </li></ul><ul><ul><li> </li></ul></ul><ul><li>NOTE: In prior testing, these tools had proven effective, but difficult to use in teaching, and time consuming for scoring. </li></ul>
  4. 4. Why? <ul><li>Everyone wants the good information </li></ul><ul><li>Nobody agrees what that is </li></ul><ul><li>Like “match the tool to the task”, information one person recommends won’t be right for another person </li></ul><ul><li>Censoring what information is available or “acceptable” creates a lack of trust </li></ul><ul><li>Criticizing information choices impairs communication (and trust) </li></ul>
  5. 5. Why? (cont.) <ul><li>What’s good depends on: </li></ul><ul><ul><li>The question </li></ul></ul><ul><ul><li>Who’s asking </li></ul></ul><ul><ul><li>Literacy </li></ul></ul><ul><ul><li>Prior educational context </li></ul></ul><ul><ul><li>Prior knowledge in this topic </li></ul></ul><ul><ul><li>Prior experiences </li></ul></ul><ul><ul><li>Attitudes & assumptions </li></ul></ul><ul><ul><li>Intangible personal preferences </li></ul></ul><ul><ul><li>Other context </li></ul></ul>
  6. 6. What’s missing <ul><li>Professional codes define professional responsibilities with ever-greater accuracy. Huge efforts also go into ensuring trustworthy performance. . . .The efforts to prevent abuse of trust are gigantic, relentless and expensive; and inevitably their results are always less than perfect. Have these countermeasures begun to restore trust, or just to reduce suspicion? . . . Patients, it is said, no longer trust doctors . . . and in particular no longer trust hospitals or hospital consultants. &quot;Loss of trust&quot; is in short a clich é of our times. </li></ul><ul><ul><li>O'Neill, Onora. A Question of Trust. (BBC Reith Lectures 2002: Lecture 1. &quot;Spreading Suspicion,&quot; April 6, 2002, pp. 2 - 3.) Retrieved February 3, 2004, from; also available at </li></ul></ul><ul><ul><li>Cited in: MLA Guide, Introduction: </li></ul></ul>
  7. 7. What’s needed <ul><li>Trust </li></ul><ul><li>Respectful dialogue </li></ul><ul><li>Tools to support dialogue </li></ul>
  8. 8. Solution? <ul><li>Yes, “trust your gut”, but … </li></ul><ul><li>Train the “gut feeling” </li></ul><ul><ul><li>Discuss & define common elements </li></ul></ul><ul><ul><li>Articulate contextual elements </li></ul></ul><ul><ul><li>Describe environmental constraints </li></ul></ul><ul><ul><li>Articulate criteria </li></ul></ul><ul><li>In other words </li></ul><ul><ul><li>Spell it out </li></ul></ul>
  9. 9. Potential audience <ul><li>Students, undergrad </li></ul><ul><li>Students, grad & professional </li></ul><ul><li>Patients </li></ul><ul><li>Healthcare professionals </li></ul><ul><li>Friends and family </li></ul><ul><li>Personal use </li></ul><ul><li>Anyone </li></ul>
  10. 10. Challenges <ul><li>Easy to use </li></ul><ul><li>Easy to remember </li></ul><ul><li>Broad audience / multipurpose </li></ul><ul><ul><li>Incoming freshmen </li></ul></ul><ul><ul><li>Training of healthcare professionals </li></ul></ul><ul><li>Broad range of information available </li></ul><ul><li>Blurred media boundaries </li></ul><ul><ul><li>traditional print peer-reviewed publications </li></ul></ul><ul><ul><li>popular press & news media </li></ul></ul><ul><ul><li>general web </li></ul></ul><ul><li>Potential range of literacy skills </li></ul><ul><li>Potential range of awareness of health information quality issues & importance </li></ul>
  11. 11. Process <ul><li>Review as many existing similar tools as discoverable, for all audiences, from elementary school children to professionals </li></ul><ul><li>Collect and organize concepts represented in the various tools </li></ul><ul><ul><li>Note frequency of distribution for common concepts </li></ul></ul><ul><ul><li>Note unique and valuable concepts </li></ul></ul><ul><li>Prioritize </li></ul><ul><li>Seek unifying structure </li></ul>
  12. 12. Other information quality assessment tools <ul><li>ABC (The Good, The Bad, & the Ugly): </li></ul><ul><ul><li>Accuracy, authority, bias/objectivity, currency, coverage </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Internet Detective </li></ul><ul><ul><li> </li></ul></ul><ul><li>Five criteria (Cornell) </li></ul><ul><ul><li> </li></ul></ul><ul><li>MedlinePlus: Guide to Healthy Web Surfing: </li></ul><ul><ul><li> </li></ul></ul>
  13. 13. Aha moment - “Great home page” Ahmad Risk <ul><li>Full text: </li></ul><ul><ul><li><> </li></ul></ul><ul><li>Candour </li></ul><ul><li>Honesty </li></ul><ul><li>Quality </li></ul><ul><li>Informed consent </li></ul><ul><li>Privacy </li></ul><ul><li>Professionalism </li></ul><ul><li>Responsible partnering </li></ul><ul><ul><li>That the website chooses its partner with care and that those partners strengthen the chain of trust among its community. </li></ul></ul><ul><li>Accountability </li></ul>
  14. 14. Strategy <ul><li>Keyword based </li></ul><ul><li>Concept clustering </li></ul><ul><li>Likert scale </li></ul><ul><li>Levels of evidence - extended </li></ul>
  15. 15. Assumptions <ul><li>All information sources have some good information </li></ul><ul><li>All information sources have some bad information </li></ul><ul><li>Peer review tends to be better information, but cannot be guaranteed </li></ul><ul><li>Branding or certificate programs likewise tend to be better but cannot be guaranteed </li></ul><ul><li>Who watches the watchers? </li></ul>
  16. 16. Chain <ul><li>Candor </li></ul><ul><li>Honesty </li></ul><ul><li>Accountability </li></ul><ul><li>Information quality </li></ul><ul><li>Neighborly </li></ul>
  17. 17. Trust <ul><li>Timeliness </li></ul><ul><li>Relevant </li></ul><ul><li>Unbiased </li></ul><ul><li>Scope </li></ul><ul><li>Trustworthy </li></ul>
  18. 18. Scoring <ul><li>0 = No way </li></ul><ul><li>5 = I really really like it! </li></ul><ul><li>Lowest possible score = 0 </li></ul><ul><li>Highest possible score = 50 </li></ul><ul><li>Perfectly alright to use the tool without the numbers - it is the ideas that are important. </li></ul>
  19. 19. Connect your gut feeling to the evidence <ul><li>Places information in context </li></ul><ul><li>Makes more overt aspects such as </li></ul><ul><ul><li>Type of question being asked </li></ul></ul><ul><ul><li>Range of information available </li></ul></ul><ul><ul><li>What is the best evidence available </li></ul></ul><ul><ul><li>Environmental factors or constraints </li></ul></ul>
  20. 20. Levels of evidence in EBHC
  21. 21. A broader view
  22. 22. How do they relate?
  23. 23. An alternate view
  24. 24. What is “best available evidence”? <ul><li>Example 1: Common condition, common topic, large research base? </li></ul><ul><ul><li>Gold standard: systematic reviews, meta-analyses, randomized controlled clinical trials (RCTs, CCTs) </li></ul></ul><ul><li>Example 2: Rare condition, little to no research? </li></ul><ul><ul><li>Expert opinion, If you can find it </li></ul></ul><ul><ul><li>Patient perspective, support groups, personal web pages </li></ul></ul><ul><li>… and all the range between the two … </li></ul>
  25. 25. Get your own copy <ul><li>From this page: </li></ul><ul><ul><li> </li></ul></ul><ul><li>Horizontal version: </li></ul><ul><ul><li> </li></ul></ul><ul><li>Vertical version: </li></ul><ul><ul><li> </li></ul></ul>
  26. 26. More information <ul><li>Links: </li></ul><ul><ul><li> </li></ul></ul>