Yoni Donner - Valid Assessment


Published on

At a "Quantified Self and Science" meetup, Yoni Donner talks about the issue of valid assessment.

1 Comment
  • Live notes from Eri Gentry watching Yoni's presentation:
    Good heuristic for individuals: ask scientific friends, who won’t be afraid to judge your results harshly, for help
    How do we make sure our results are reliable and relevant? Cognitive assessments
    Quantified Mind (QM): 20 cognitive tests to cover multiple skills, grouped into batteries
    Validity of quantified mind: if we measure the same thing twice, we want to get the same result twice -> reliability. Weight = high reliability (.999). Psychological tests are nowhere near that reliability. .7 is considered good. For QM, most test were over .8, some over .9
    Face validity: Gee, that’s obvious! A reaction time test that measures reaction time. Mental rotation. Obvious that it’s a spatial skill.
    Not everything has face validity or reliability, so we can rely on existing tests and data -> Borrowed Validity
    Criterion validity: already established as valuable. This takes a lot of work, collaborations, participation, etc
    Experimental results: results. Not that we understand what they mean but they could….
    Future experiments: functional validation, etc
    Gary: how can I jump to most validity right away? Eg what about those 1-5 scales? Gary made his fear of error less by using a 1-3 scale. More accuracy with labeling of internal state. Drive validity to its simplest state
    Comment: How do you deal with outliers in your own data? - Depends on what you’re measuring. (if you are dealing with multiple people, can use z scaling to make data relative)
    Comment: Terms reliability and face validity are not correct terms. Suggestion: specificity/sensitivity, plotting both on ROC curve. Sensitivity = probability of observing a true positive, Specificity = … undefined. Want false negative, false positive. Reliability wrong word
    Yoni meant what he said…. Stopping now before it gets too far into semantics to help us
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Yoni Donner - Valid Assessment

  1. 1. Valid assessment• We want our measurements to be reliable and relevant. How do we verify that?• Cognitive assessments• Quantified Mind: www.quantified-mind.com• Over 20 cognitive tests aiming to cover multiple distinct cognitive skills, grouped into short batteries• Tests are short and fun, but...• How do we know that they measure what we care about?
  2. 2. Validity of Quantified Mind (www.quantified-mind.com)• Reliabilityo For Quantified Mind, most tests have reliability > 0.8, some even over 0.9§ Affected by duration - not bad for 1-minute tests• Face validityo E.g. reaction time, mental rotationo Works best in combination with high reliability• Borrowed validity• Criterion validity• Experimental results• Future experiments• And much more: yonid@stanford.edu