The document discusses various aspects of instrument validity and reliability that must be established when developing a new assessment tool. It defines reliability as the stability and consistency of a measure, and validity as how well a measure captures the intended construct. It describes different types of validity such as face validity, convergent/concurrent validity, discriminant validity, and predictive validity that should be evaluated using data like expert ratings, comparisons to other measures, and longitudinal outcomes. Establishing reliability is a prerequisite before validity can be assessed.

1. # 7 Location In The Research ProcessFormulatingResearchQuestionsAppropriateDesignCraftKnowledgeAppropriateSamplingDecisionsPreviousResearchEnsureResearchEthicsDataCollection&ManagementDataAnalysisOngoingResearchInterpretationDissemination

2. Measurement IIWhat did we catch?Was it the same as the others?How do we know we caught it?

3. Properties of InstrumentsReliabilityThe stability and consistency of a measureValidityHow we know that our measure captures what we're interested in

4. AN INSTRUMENT MUST HAVE RELIABILITY BEFORE IT CAN HAVE VALIDITY!

5. ReliabilityThe stability and consistency of an instrumentStabilitySimilar results over timeConsistencyTest items will be consistent with each otherItems will be consistent with the overall score on the test

6. StabilityMeasured using test-retest reliability checksA participant completes the test at one point in timeAfter a period of time elapses they complete the test againUsually at least 6 weeksAssessed using correlation coefficientsPearson rSpearman 

7. ConsistencyInternal consistencyThe items all measure the same core constructMeasured using the Cronbach's Alpha Coefficient A form of multiple correlationMeasures how much the individual items in a test or scale have in common statistically

8. First QuestionThe Juvenile Court Administrator wants to evaluate a new assessment tool to find out if you can tell which kids, once released from Detention, will recidivate or not.What is the first thing that must be done to solve your groups research problem?How do you propose to do it?

9. ValidityHow we know that our instrument captures what we're interested inConcept translationHow well our instrument translates a concept into a variableValidity is reported as correlations of varying complexity

10. ValidityFace ValidityConvergent / Concurrent ValidityDiscriminant ValidityPredictive Validity

11. Face ValidityHow well an instrument apparently accomplishes concept translationHow completely an instrument samples all of the components of a conceptPanels of expertsIn the domain being measuredWho have experience with the target

12. Convergent / Concurrent ValidityHow well an instrument relates to other measures of the same or similar conceptsCompare an instrument with:Other existing instrumentsExpert opinionsBehavioral observationsAssessed when the measure is given to subjects

13. Discriminant ValidityHow well an instrument discriminates between the presence or absence of concept componentsAn instrument can discriminate between known groups, assessed when the measure is given to subjects

14. Predictive ValidityHow well an instrument predicts future events or behaviorUseful concepts will predict phenomenaPredictive validity assessment - examines how we know that our measure captures what we are interested inCompare the instrument with other predictive factorsTest prediction of the measure aloneAssessed longitudinally, after the instrument is given to subjects

15. Second QuestionThe DCFS Area Regional Director wants to be able to tell how many future CPS referrals families are likely to have after their first CPS referral. He devises a questionnaire for the child’s mother to complete.What type of validity is involved in your research problem?What data might you use to address the validity question?