Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Wendy Turell, DrPH, CHCP
Senior Director, Outcomes and Analytics
PlatformQ Health Education
• Measurement about how people think or act gathered via Q&A process
• Can be used in experimental, quasi-experimental, an...
 Measurement Validity
 How accurately the instrument measures the construct you are trying to measure
 Examples:
 A Ru...
 Many errors “bias” research results
 Examples: sampling error, nonresponse error, self-presentation/social desirability...
 Pilot your questions
 Psychometric review
 validate your questions against an external construct (e.g., correlate with...
TIP 1:
 Question:
 Measurement studies have shown that 5 option MCQ formats are more reliable and
valid than dichotomous...
TIP 2:
 Examples:
 All of the Above, None of the Above, A and E, All except D
 More likely to lead to measurement error...
TIP 3:
 Novel targets in clear cell RCC include all of the following EXCEPT:
a. PD-1
b. PD-L1
c. MET
d. WNT pathway
 Whi...
TIP 4:
 Questions that contain multiple concepts
Do you recommend changes in diet and exercise for all of your overweigh...
TIP 5:
 I can rarely differentiate a diagnosis of patients with schizophrenia from those
with schizoaffective disorder.
...
 Which question format do you prefer?
 Question 1: Basal insulin should provide about ______ of daily insulin requiremen...
 Which question format do you prefer?
 Question 1: Some ER/LA opiates have eliminated risk of abuse.
a. True
b. False
 ...
 Based on what you learned in this session, rewrite this question:
 Which is NOT your favorite color crayon?
 Yellow
 ...
 Based on what you learned in this session, rewrite this question:
 The following outcomes level is the most difficult f...
Thank you & good luck!
Contact: wturell@platformq.com
Wednesday April 8th, 2015
1:30 PM ET
5 Tips to Make You a Survey Measurement Rock Star
Did you know that there are measure...
5 Tips to Make you a Survey Measurement Rock Star
Upcoming SlideShare
Loading in …5
×

5 Tips to Make you a Survey Measurement Rock Star

867 views

Published on

Conference Abstract: Did you know that there are measurement experts who spend their lives researching the validity of different question styles and formats? How ’bout we spend 30 minutes reviewing some of the most relevant findings in survey methods research!? During this presentation, you will learn about fascinating topics such as the validity of true/false questions, when to use “none of the above” style questions, and which question styles will help you assess the information you are actually trying to asses (read= minimize bias and error)! Participants will be asked to vote on sample question strength, and recraft “before’s” into highly valid “after’s.”

Published in: Data & Analytics
  • Hello! I can recommend a site that has helped me. It's called ⇒ www.WritePaper.info ⇐ They helped me for writing my quality research paper.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Überprüfen Sie die Quelle ⇒ www.WritersHilfe.com ⇐ . Diese Seite hat mir geholfen, eine Diplomarbeit zu schreiben.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • My experience with essay services has generally been very positive. I requested a writer from HelpWriting.net to write my English essay on Jane Eyre because I was really short on time (that week my family had to move houses unexpectedly and I had no time whatsoever to sit down at a computer and do work.) Needless to say the services were good and my essay was done really quickly. The writer that I picked followed directions well. Of course, if you can do it yourself, then do so but if it’s an end-all-be-all situation I definitely recommend HelpWriting.net
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Easy and hassle free way to make money online! I have just registered with this site and straight away I was making money! It doesn't get any better than this. Thank you for taking out all the hassle and making money answering surveys as easy as possible even for non-techie guys like me! ♥♥♥ http://t.cn/AieXAuZz
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Greek Ritual REVERSES Diabetes (do this before bed) Before you go to bed tonight, do this ONE "stupidly simple" Greek ritual to reverse your diabetes... ♥♥♥ http://t.cn/AiBhrtGG
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

5 Tips to Make you a Survey Measurement Rock Star

  1. 1. Wendy Turell, DrPH, CHCP Senior Director, Outcomes and Analytics PlatformQ Health Education
  2. 2. • Measurement about how people think or act gathered via Q&A process • Can be used in experimental, quasi-experimental, and non experimental research • Initially developed in the 1920s, now have theories and principles that guide survey science • Survey Methods = commonly use in CEHP • Surveys: • Can be administered via self administered questionnaires or interviews • Paper/pencil, verbally, online • Are most often quantitative (but can be qualitative) • Are a standardized measurement instrument • Replicable • Can measure: • Attitudes and opinions • Behaviors • Motivations • Emotions • Satisfaction
  3. 3.  Measurement Validity  How accurately the instrument measures the construct you are trying to measure  Examples:  A Ruler = High measurement validity  Measurement reliability  How consistently your instrument measures what you are trying to measure.  Does your testing instrument generate the same result in similar circumstances?
  4. 4.  Many errors “bias” research results  Examples: sampling error, nonresponse error, self-presentation/social desirability bias, acquiescence bias, etc.  Today: Focus on measurement error  The difference between the estimate produced and the true value because respondents gave inaccurate answers to survey questions.  Occurs when respondents are unable / unwilling to provide accurate answers  May be due several factors……..including poor question design  Does the question adequately measure the idea or concept of interest?  If not = specification error (low construct validity)
  5. 5.  Pilot your questions  Psychometric review  validate your questions against an external construct (e.g., correlate with a measure that is known to measure the construct of interest very well)  examine internal consistency of constructs  To begin with………..write solid questions by using question structures that have been shown to elicit more valid & reliable responses!
  6. 6. TIP 1:  Question:  Measurement studies have shown that 5 option MCQ formats are more reliable and valid than dichotomous answer “true/false” style questions: a. True b. False  True/false; Yes/no; Agree/disagree  Probability of guessing correctly = .5 (dichotomous) versus .2 (5 choice MCQ)  “Acquiescence” = tendency to agree as a response style  When guessing, examinees tend to lean towards “true” > “false” Grosse, Martin E.; Wright, Benjamin D. (1985) Validity and Reliability of True-False Tests. Educational and Psychological Measurement, 45(1): 1-13 Norcini JJ, Swanson DB, Grosso LJ, Webster GD. (1985). Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence. Med Educ. 19(3):238-47.
  7. 7. TIP 2:  Examples:  All of the Above, None of the Above, A and E, All except D  More likely to lead to measurement error  More challenging to think of good distractor answers  All of the Above:  When studied, responses skew more towards “all of the above” response (typically artificially elevating scores)  Loss of discriminatory power in reviewing other response options Cuing Effect of "All of the above" on the Reliability and Validity of Multiple-Choice Test Items P. H. Harasym,E. J. Leong, C. Violato, R. Brant,and F. L. Lorscheider. Evaluation & the Health Professions, March 1998; vol. 21, 1: pp. 120-133.
  8. 8. TIP 3:  Novel targets in clear cell RCC include all of the following EXCEPT: a. PD-1 b. PD-L1 c. MET d. WNT pathway  Which of the following prognostic features is associated with WORST overall survival? a. Low Ki-67 proliferative index b. Morphology of small cells resembling follicular leukemia c. Presence of 3 or more complex karyotype abnormalities d. Translocation involving chromosomal 11  “not”, “except”, “least likely” etc.  Flip cognitive reasoning  > Taxing = > measurement error
  9. 9. TIP 4:  Questions that contain multiple concepts Do you recommend changes in diet and exercise for all of your overweight patients? In the past month, how frequently have you self-monitored your blood sugar and blood pressure? How satisfied are you with the accuracy and delivery of the content? Do you desire more live and online CME activities in oncology? Rewrite: Do you recommend changes in diet for all of your overweight patients? Do you recommend changes in exercise for all of your overweight patients?
  10. 10. TIP 5:  I can rarely differentiate a diagnosis of patients with schizophrenia from those with schizoaffective disorder.  how to define “rarely”  My health has improved since participating in the educational session.  how to define “health”  I have visited my doctor in the past 6 months.  how to define “doctor” – do other healthcare providers count?  how to define “visit” – do phone/email consults count? What if you just saw the nurse?
  11. 11.  Which question format do you prefer?  Question 1: Basal insulin should provide about ______ of daily insulin requirement. a. 10% b. 25% c. 50% d. 75%  Question 2: In the evaluation and diagnosis of a person with mild cognitive impairment, which of the following biomarkers are considered most likely to indicate that MCI is due to AD: a. A positive CSF Aβ biomarker only b. A positive Aβ biomarker and a positive tau, FDG, or sMRI biomarker c. Positive tau and FDG biomarkers d. A positive FDG biomarker only
  12. 12.  Which question format do you prefer?  Question 1: Some ER/LA opiates have eliminated risk of abuse. a. True b. False  Question 2: Patients with an unprovoked VTE are at risk for a recurrence for what amount of time? a. 3 months b. 6 months c. 12 months d. Indefinitely
  13. 13.  Based on what you learned in this session, rewrite this question:  Which is NOT your favorite color crayon?  Yellow  Blue  Green  Brick Red
  14. 14.  Based on what you learned in this session, rewrite this question:  The following outcomes level is the most difficult for me to evaluate and report:  Level 3: Knowledge  Level 4: Competence  Level 5: Performance  Level 6: Patient Health  Level 7: Community Health
  15. 15. Thank you & good luck! Contact: wturell@platformq.com
  16. 16. Wednesday April 8th, 2015 1:30 PM ET 5 Tips to Make You a Survey Measurement Rock Star Did you know that there are measurement experts who spend their lives researching the validity of different question styles and formats? How ’bout we spend 30 minutes reviewing some of the most relevant findings in survey methods research!? During this presentation, you will learn about fascinating topics such as the validity of true/false questions, when to use “none of the above” style questions, and which question styles will help you assess the information you are actually trying to asses (read= minimize bias and error)! Participants will be asked to vote on sample question strength, and recraft “before’s” into highly valid “after’s.”

×