SlideShare a Scribd company logo
1 of 25
Download to read offline
The Art and Science of Test Development—Part D

        Develop norm (standardization) sample plan


                          Kevin S. McGrew, PhD.

                           Educational Psychologist

                          Research Director
                      Woodcock-Muñoz Foundation




The basic structure and content of this presentation is grounded extensively on the test
            development procedures developed by Dr. Richard Woodcock
Develop norm (standardization) sampling plan




                                   
                                  
                                   
                                  
                                  
                                  
                                  
                                
                                 
                               
                                
                                  
                                  

                                    The goal:
                       A nationally representative sample
                       from which to develop test norms
Implementation of sampling plan in practical test
          development framework




                 
                
                 
               
                
               
                
              
               
             
              
                
                

Gather the norm (standardization) data and use
    for Rasch scaling, norm development,
    psychometric, and statistical analyses
Implementation of sampling plan in
Conceptual Psychometric Validity Framework




                                                       
                                                        
                                                        
    Gather the norm (standardization) data and use     
                                                        
                                                       
                                                       
                                                        
        for Rasch scaling, norm development,          
                                                       
                                                        
        psychometric, and statistical analyses        
                                                      
                                                       
                                                       
                                                     
                                                      
Develop norm (standardization) sample specifications for
   country/nation where test is intended to be used




                         
                        
                         
                        
                        
                        
                        
                      
                       
                     
                      
                        
                        

                          The goal:
             A nationally representative sample
             from which to develop test norms
Sampling plan should be based on best available source of
       national statistics (select examples below)
Three-stage sampling plan strategy used for
           WJ batteries in United States


                                                           
                                                          
                                                           
                                                         
                                                          
                                                         
                                                          
Stage 1: Sampling of communities                        
                                                         
                                                       
                                                        
                                                          
                                                          
                                                             The goal:
Stage 2: Sampling of schools                    A nationally representative sample
                                                from which to develop test norms



Stage 3: Sampling of subjects

   • School-age subjects – random sampling in grades

   • Preschool, university and adult subjects - quotas selected as
   per selected US Census variables

                                      
                                       
                                      
                                      
      Goal is a nationally            
                                      
 representative sample upon         
                                     
which to base the norms for the    
                                    
  measures (tests, clusters)          
                                      
No sampling plan is perfect: “Tweak” final norm data via subject weighting




    A few tips/cautions

•Oversample small groups
 and then down-weight

•No amount of creative
 weighting can “fix” a poorly
 executed sampling plan

•Continuous sample analysis:
  Make sure to frequently
 monitor data as it is collected
 to see how close the “fit”
 between the sampling plan
 and actual subject
 characteristics. Important so
 you can make adjustments
 during the data collection
 (before it is too late)
Applied Psych Test Design: Part D--Develop norm (standardization) plan
Applied Psych Test Design: Part D--Develop norm (standardization) plan
Applied Psych Test Design: Part D--Develop norm (standardization) plan
WJ III Three-stage sampling plan strategy: Stage 1

Sampling of communities – communities sampled according
to 10 different community characteristics
Important question to think about


                                                                              $$




                                                                          $$$$$$$$




Are students who are categorized as “low SES” (at parent/family level) from $$ Community
likely to be having the same community/life/family/educational experiences as “low SES”
students (at parent/family level) from Community $$$$$$$$ ?
Illustration of potential threat to representative sample when community
                                                SES is ignored and only family/parent SES is used in sampling plan

                                                            Level of abilities of selected students
                                   Cmnty C
                            High
  Community SES classification

                                   Cmnty B
            Middle


                                   Cmnty A
Low




                                                             Family/Parent SES classification

                                               Low                           Middle                            High
Community SES is one of the most important, yet
 most frequently overlooked variables in a test
         standardization sampling plan
Sample research/norm
demographic and data
 file variable coding
     sheet: WJ III

      Tips/Cautions

• Spend considerable time
designing this data
collection form (be OCD)

• Think of all future
analyses when deciding
what to include/code

• If in doubt about a
variable…include it (you
typically can’t go back
later to get information)

• Create coding system
with quality data entry
procedures in mind

• You can’t analyze what
you haven’t gathered and
coded from the beginning
Development and implementation of sampling plan is probably the most critical
phase of test development
     • GIGO – garbage ingarbage out

Retain as much control over all aspects of data collection and data entry as
possible
Quality of data is only as good as your pool of examiners
     • Recruiting, training, supervising, and retaining good examiners requires major
     attention and is very important

     • Examiners need supervision

     • Terminate poor examiners as soon as you can

     • Central office should review every single piece of information on submitted test
     records, especially at the beginning of an examiners testing (you don’t want them
     practicing errors)

     • Run special Rasch “person fit” reports to flag test records that look suspicious (and
     see if they come from certain examiners)

     • Best examiners are not necessarily psychologists or doctoral students.
                                                                                 (cont. next slide)
Spend considerable time developing the test record and variable coding
sheets
Run “continuous sample analysis reports” to monitor sampling plan
adherence or drift – so you can make changes quickly before it is too late
There is no single-purpose system, or collection of published software
programs, that can handle the detailed and fluid work of data entry, editing
and monitoring. Be prepared to develop and pay for custom software.
Dedicated professional data-entry software is a must. SPSS, Excel, etc.
won’t cut it.
     • Double data entry verification a must
     • Only have a few well trained and diligent individuals enter the data

Don’t succumb to samples of convenience
Oversample small groups – and then down weight
Seriously consider three-stage sampling plan that controls for community
SES
Planned “incomplete” (missing) data collection

Matrix sampling (Partial)




                                      Multiple and EM-based data imputation
Recent example: Australian WJ IIII standardization partial matrix sampling plan
                                     Test                    1     A     B     C     D     E         X             Total n

                    1. Verbal Comprehension                  C     C     C     C     C     C                       1346

                    2. Visual-Auditory Learning              C     C     C     C     C     C                       1336

                    3. Spatial Relations                     C     C     C     C     C     C                       1375

                    4. Sound Blending                        C     C     C     C     C     C                       1382
 Yellow column
 (1) represents     5. Concept Formation                     C     C     C     C     C     C                       1300
  initial “core”    6. Visual Matching                       C     C     C     C     C     C                       1364
 battery prior to
                    7. Numbers Reversed                      C     C     C     C     C     C                       1325
implementation
   of “core +       8. Incomplete Words                      C                             M                        627

 matrix” battery    9. Auditory Working Memory               C                       M                              548




                                                                                                 Mixed batteries
       plan         11. General Information                              M                 M                        389

                    12. Retrieval Fluency                                M     M                                    388

                    13. Picture Recognition                              M     M                                    390

                    14. Auditory Attention                                           M     M                        388

                    15. Analysis-Synthesis                                     M     M                              362

                    16. Decision Speed                                   M           M                              376

                    17. Memory for Words                                       M     M                              364

                    1. Letter-Word Identification            C     C     C     C     C     C                       1323

                    2. Reading Fluency                       C     M                                                588

                    5. Calculation                           C     C     C     C     C     C                       1210

                    6. Math Fluency                          C                 M                                    588

                    7. Spelling                              C     M                                                557

                    8. Writing Fluency                             M     M                                          352

                    9. Passage Comprehension                 C     C     C     C     C     C                       1203

                    10. Applied Problems                     C     C     C     C     C     C                        964

                    11. Writing Samples                            M                       M                        342
                                                    Totals   450   102   117   127   108   120   372               1396
End of Part D
  Additional steps in test development process will be
presented in subsequent modules as they are developed
Applied Psych Test Design: Part D--Develop norm (standardization) plan
Applied Psych Test Design: Part D--Develop norm (standardization) plan
Applied Psych Test Design: Part D--Develop norm (standardization) plan

More Related Content

What's hot

Blueprinting Liz Norman ANZCVS 2021
Blueprinting Liz Norman ANZCVS 2021Blueprinting Liz Norman ANZCVS 2021
Blueprinting Liz Norman ANZCVS 2021Liz Norman
 
Qualitative and quantitative analysis
Qualitative and quantitative analysisQualitative and quantitative analysis
Qualitative and quantitative analysisNellie Deutsch (Ed.D)
 
Drafting written questions and marking schemes Liz Norman ANZCVS 2020
Drafting written questions and marking schemes Liz Norman ANZCVS 2020Drafting written questions and marking schemes Liz Norman ANZCVS 2020
Drafting written questions and marking schemes Liz Norman ANZCVS 2020Liz Norman
 
Question clarity and why its important Liz Norman ANZCVS 2020
Question clarity and why its important Liz Norman ANZCVS 2020Question clarity and why its important Liz Norman ANZCVS 2020
Question clarity and why its important Liz Norman ANZCVS 2020Liz Norman
 
Blueprinting and drafting questions Liz Norman ANZCVS 2019
Blueprinting and drafting questions Liz Norman ANZCVS 2019Blueprinting and drafting questions Liz Norman ANZCVS 2019
Blueprinting and drafting questions Liz Norman ANZCVS 2019Liz Norman
 
Grading criteria and marking schemes Liz Norman ANZCVS 2019
Grading criteria and marking schemes Liz Norman ANZCVS 2019Grading criteria and marking schemes Liz Norman ANZCVS 2019
Grading criteria and marking schemes Liz Norman ANZCVS 2019Liz Norman
 
Oral exams Liz Norman ANZCVS 2020
Oral exams Liz Norman ANZCVS 2020Oral exams Liz Norman ANZCVS 2020
Oral exams Liz Norman ANZCVS 2020Liz Norman
 
Notes from frankel and wallen
Notes from frankel  and wallenNotes from frankel  and wallen
Notes from frankel and wallenRose Jedin
 
Quantitative Research Process
Quantitative Research ProcessQuantitative Research Process
Quantitative Research ProcessAIMS Education
 
Assessment 15 Annotated
Assessment 15 AnnotatedAssessment 15 Annotated
Assessment 15 AnnotatedJames Atherton
 
Qualitative Research Chapter 3 g11 Research Method and Procedures
Qualitative Research Chapter 3 g11 Research Method and ProceduresQualitative Research Chapter 3 g11 Research Method and Procedures
Qualitative Research Chapter 3 g11 Research Method and ProceduresGhail RC
 
Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17tjcarter
 
Ed200 research chapter 3 methodology(jan282012)
Ed200 research  chapter 3 methodology(jan282012)Ed200 research  chapter 3 methodology(jan282012)
Ed200 research chapter 3 methodology(jan282012)Maria Theresa
 
Business Research Method - Unit III, AKTU, Lucknow Syllabus
Business Research Method - Unit III, AKTU, Lucknow SyllabusBusiness Research Method - Unit III, AKTU, Lucknow Syllabus
Business Research Method - Unit III, AKTU, Lucknow SyllabusKartikeya Singh
 
Research methodology week03
Research methodology week03Research methodology week03
Research methodology week03swati kundabor
 

What's hot (20)

Blueprinting Liz Norman ANZCVS 2021
Blueprinting Liz Norman ANZCVS 2021Blueprinting Liz Norman ANZCVS 2021
Blueprinting Liz Norman ANZCVS 2021
 
Qualitative and quantitative analysis
Qualitative and quantitative analysisQualitative and quantitative analysis
Qualitative and quantitative analysis
 
Drafting written questions and marking schemes Liz Norman ANZCVS 2020
Drafting written questions and marking schemes Liz Norman ANZCVS 2020Drafting written questions and marking schemes Liz Norman ANZCVS 2020
Drafting written questions and marking schemes Liz Norman ANZCVS 2020
 
Chap2 latest
Chap2 latestChap2 latest
Chap2 latest
 
Question clarity and why its important Liz Norman ANZCVS 2020
Question clarity and why its important Liz Norman ANZCVS 2020Question clarity and why its important Liz Norman ANZCVS 2020
Question clarity and why its important Liz Norman ANZCVS 2020
 
Blueprinting and drafting questions Liz Norman ANZCVS 2019
Blueprinting and drafting questions Liz Norman ANZCVS 2019Blueprinting and drafting questions Liz Norman ANZCVS 2019
Blueprinting and drafting questions Liz Norman ANZCVS 2019
 
Handout #1 - QIAMay4
Handout #1 - QIAMay4 Handout #1 - QIAMay4
Handout #1 - QIAMay4
 
Grading criteria and marking schemes Liz Norman ANZCVS 2019
Grading criteria and marking schemes Liz Norman ANZCVS 2019Grading criteria and marking schemes Liz Norman ANZCVS 2019
Grading criteria and marking schemes Liz Norman ANZCVS 2019
 
Research methodology
Research methodologyResearch methodology
Research methodology
 
Oral exams Liz Norman ANZCVS 2020
Oral exams Liz Norman ANZCVS 2020Oral exams Liz Norman ANZCVS 2020
Oral exams Liz Norman ANZCVS 2020
 
Notes from frankel and wallen
Notes from frankel  and wallenNotes from frankel  and wallen
Notes from frankel and wallen
 
Quantitative Research Process
Quantitative Research ProcessQuantitative Research Process
Quantitative Research Process
 
Assessment 15 Annotated
Assessment 15 AnnotatedAssessment 15 Annotated
Assessment 15 Annotated
 
Qualitative Research Chapter 3 g11 Research Method and Procedures
Qualitative Research Chapter 3 g11 Research Method and ProceduresQualitative Research Chapter 3 g11 Research Method and Procedures
Qualitative Research Chapter 3 g11 Research Method and Procedures
 
2 stat critical_1
2 stat critical_12 stat critical_1
2 stat critical_1
 
Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17
 
Ed200 research chapter 3 methodology(jan282012)
Ed200 research  chapter 3 methodology(jan282012)Ed200 research  chapter 3 methodology(jan282012)
Ed200 research chapter 3 methodology(jan282012)
 
Business Research Method - Unit III, AKTU, Lucknow Syllabus
Business Research Method - Unit III, AKTU, Lucknow SyllabusBusiness Research Method - Unit III, AKTU, Lucknow Syllabus
Business Research Method - Unit III, AKTU, Lucknow Syllabus
 
Finalreview
FinalreviewFinalreview
Finalreview
 
Research methodology week03
Research methodology week03Research methodology week03
Research methodology week03
 

Similar to Applied Psych Test Design: Part D--Develop norm (standardization) plan

The Research Process
The Research ProcessThe Research Process
The Research ProcessZain Mushtaq
 
Kapruch steve
Kapruch steveKapruch steve
Kapruch steveNASAPMC
 
LIVES research framework
LIVES research frameworkLIVES research framework
LIVES research frameworkILRI
 
Conceipt of experimental design sixsigma1
Conceipt of experimental design sixsigma1Conceipt of experimental design sixsigma1
Conceipt of experimental design sixsigma1Mujtaba Masood
 
Data mining introduction
Data mining introductionData mining introduction
Data mining introductionBasma Gamal
 
7 QC Tools training presentation
7 QC Tools training presentation7 QC Tools training presentation
7 QC Tools training presentationPRASHANT KSHIRSAGAR
 
MODULE 1_Introduction to Data analytics and life cycle..pptx
MODULE 1_Introduction to Data analytics and life cycle..pptxMODULE 1_Introduction to Data analytics and life cycle..pptx
MODULE 1_Introduction to Data analytics and life cycle..pptxnikshaikh786
 
Rd online to-t_module-2.1-2.2_research-methods
Rd online to-t_module-2.1-2.2_research-methodsRd online to-t_module-2.1-2.2_research-methods
Rd online to-t_module-2.1-2.2_research-methodsSaadiaMahmood3
 
RD_Online-ToT_Module-2.1-2.2_Research-methods.pptx
RD_Online-ToT_Module-2.1-2.2_Research-methods.pptxRD_Online-ToT_Module-2.1-2.2_Research-methods.pptx
RD_Online-ToT_Module-2.1-2.2_Research-methods.pptxCristelAnnVerayoDesc
 
What is Research problem.pptx
What is Research problem.pptxWhat is Research problem.pptx
What is Research problem.pptxPratikshaSurve4
 
ACRL 2011 Data-Driven Library Web Design
ACRL 2011 Data-Driven Library Web DesignACRL 2011 Data-Driven Library Web Design
ACRL 2011 Data-Driven Library Web DesignAmanda Dinscore
 
NG BB 49 Risk Assessment
NG BB 49 Risk AssessmentNG BB 49 Risk Assessment
NG BB 49 Risk AssessmentLeanleaders.org
 
Unaids size estimation_training_manual_ver2
Unaids size estimation_training_manual_ver2Unaids size estimation_training_manual_ver2
Unaids size estimation_training_manual_ver2Bhanu Sharadha
 
Research protocol & Systematic Review.docx
Research protocol & Systematic Review.docxResearch protocol & Systematic Review.docx
Research protocol & Systematic Review.docxAmaraZahid
 

Similar to Applied Psych Test Design: Part D--Develop norm (standardization) plan (20)

The Research Process
The Research ProcessThe Research Process
The Research Process
 
Kapruch steve
Kapruch steveKapruch steve
Kapruch steve
 
data analysis.ppt
data analysis.pptdata analysis.ppt
data analysis.ppt
 
data analysis.pptx
data analysis.pptxdata analysis.pptx
data analysis.pptx
 
LIVES research framework
LIVES research frameworkLIVES research framework
LIVES research framework
 
Conceipt of experimental design sixsigma1
Conceipt of experimental design sixsigma1Conceipt of experimental design sixsigma1
Conceipt of experimental design sixsigma1
 
ES_140_METHODS_OF_RESEARCH.pdf
ES_140_METHODS_OF_RESEARCH.pdfES_140_METHODS_OF_RESEARCH.pdf
ES_140_METHODS_OF_RESEARCH.pdf
 
Data mining introduction
Data mining introductionData mining introduction
Data mining introduction
 
Overseas mkt research 2
Overseas mkt research 2Overseas mkt research 2
Overseas mkt research 2
 
7 QC Tools training presentation
7 QC Tools training presentation7 QC Tools training presentation
7 QC Tools training presentation
 
MODULE 1_Introduction to Data analytics and life cycle..pptx
MODULE 1_Introduction to Data analytics and life cycle..pptxMODULE 1_Introduction to Data analytics and life cycle..pptx
MODULE 1_Introduction to Data analytics and life cycle..pptx
 
Rd online to-t_module-2.1-2.2_research-methods
Rd online to-t_module-2.1-2.2_research-methodsRd online to-t_module-2.1-2.2_research-methods
Rd online to-t_module-2.1-2.2_research-methods
 
RD_Online-ToT_Module-2.1-2.2_Research-methods.pptx
RD_Online-ToT_Module-2.1-2.2_Research-methods.pptxRD_Online-ToT_Module-2.1-2.2_Research-methods.pptx
RD_Online-ToT_Module-2.1-2.2_Research-methods.pptx
 
What is Research problem.pptx
What is Research problem.pptxWhat is Research problem.pptx
What is Research problem.pptx
 
ACRL 2011 Data-Driven Library Web Design
ACRL 2011 Data-Driven Library Web DesignACRL 2011 Data-Driven Library Web Design
ACRL 2011 Data-Driven Library Web Design
 
NG BB 49 Risk Assessment
NG BB 49 Risk AssessmentNG BB 49 Risk Assessment
NG BB 49 Risk Assessment
 
De carlo rizk 2010 icelw
De carlo rizk 2010 icelwDe carlo rizk 2010 icelw
De carlo rizk 2010 icelw
 
Stratification anshul
Stratification anshulStratification anshul
Stratification anshul
 
Unaids size estimation_training_manual_ver2
Unaids size estimation_training_manual_ver2Unaids size estimation_training_manual_ver2
Unaids size estimation_training_manual_ver2
 
Research protocol & Systematic Review.docx
Research protocol & Systematic Review.docxResearch protocol & Systematic Review.docx
Research protocol & Systematic Review.docx
 

More from Kevin McGrew

The Model of Achievement Competence Motivation (MACM) Part E: Crossing the R...
The Model of Achievement Competence Motivation (MACM) Part E:  Crossing the R...The Model of Achievement Competence Motivation (MACM) Part E:  Crossing the R...
The Model of Achievement Competence Motivation (MACM) Part E: Crossing the R...Kevin McGrew
 
The Model of Achievement Competence Motivation (MACM): Part D: The volition ...
The Model of Achievement Competence Motivation (MACM): Part D:  The volition ...The Model of Achievement Competence Motivation (MACM): Part D:  The volition ...
The Model of Achievement Competence Motivation (MACM): Part D: The volition ...Kevin McGrew
 
The Model of Achievement Competence Motivation (MACM) Part C: The motivation...
The Model of Achievement Competence Motivation (MACM) Part C:  The motivation...The Model of Achievement Competence Motivation (MACM) Part C:  The motivation...
The Model of Achievement Competence Motivation (MACM) Part C: The motivation...Kevin McGrew
 
The Model of Achievement Competence Motivation (MACM): Part B - An overview ...
The Model of Achievement Competence Motivation (MACM):  Part B - An overview ...The Model of Achievement Competence Motivation (MACM):  Part B - An overview ...
The Model of Achievement Competence Motivation (MACM): Part B - An overview ...Kevin McGrew
 
The Model of Achievement Competence Motivation (MACM): Part A Introduction o...
The Model of Achievement Competence Motivation (MACM):  Part A Introduction o...The Model of Achievement Competence Motivation (MACM):  Part A Introduction o...
The Model of Achievement Competence Motivation (MACM): Part A Introduction o...Kevin McGrew
 
The WJ IV Cognitive GIA in iintellectual disability (ID) assessment
The WJ IV Cognitive GIA in iintellectual disability (ID) assessmentThe WJ IV Cognitive GIA in iintellectual disability (ID) assessment
The WJ IV Cognitive GIA in iintellectual disability (ID) assessmentKevin McGrew
 
The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence: Schne...
The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence:  Schne...The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence:  Schne...
The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence: Schne...Kevin McGrew
 
Beyond cognitive abilities: An integrative model of learning-related persona...
Beyond cognitive abilities:  An integrative model of learning-related persona...Beyond cognitive abilities:  An integrative model of learning-related persona...
Beyond cognitive abilities: An integrative model of learning-related persona...Kevin McGrew
 
What about executive functions and CHC theory: New research for discussion
What about executive functions and CHC theory:  New research for discussionWhat about executive functions and CHC theory:  New research for discussion
What about executive functions and CHC theory: New research for discussionKevin McGrew
 
"Intelligent" intelligence testing with the WJ IV COG: Why do some individua...
"Intelligent" intelligence testing with the WJ IV COG:  Why do some individua..."Intelligent" intelligence testing with the WJ IV COG:  Why do some individua...
"Intelligent" intelligence testing with the WJ IV COG: Why do some individua...Kevin McGrew
 
CHC model of inteligence revised (v2.4). Has Glr been incorrectly conceptual...
CHC model of inteligence revised (v2.4).  Has Glr been incorrectly conceptual...CHC model of inteligence revised (v2.4).  Has Glr been incorrectly conceptual...
CHC model of inteligence revised (v2.4). Has Glr been incorrectly conceptual...Kevin McGrew
 
What is "intelligent" intelligence testing
What is "intelligent" intelligence testingWhat is "intelligent" intelligence testing
What is "intelligent" intelligence testingKevin McGrew
 
"intelligent" intelligence testing: Why do some individuals obtain markedly ...
"intelligent" intelligence testing:  Why do some individuals obtain markedly ..."intelligent" intelligence testing:  Why do some individuals obtain markedly ...
"intelligent" intelligence testing: Why do some individuals obtain markedly ...Kevin McGrew
 
"intelligent" intelligence testing: Evaluating wihtin CHC domain test score ...
"intelligent" intelligence testing:  Evaluating wihtin CHC domain test score ..."intelligent" intelligence testing:  Evaluating wihtin CHC domain test score ...
"intelligent" intelligence testing: Evaluating wihtin CHC domain test score ...Kevin McGrew
 
How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...
How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...
How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...Kevin McGrew
 
The WJ IV and Beyond CHC Theory: Kevin McGrew's NASP mini-skills workshop
The WJ IV and Beyond CHC Theory:  Kevin McGrew's NASP mini-skills workshopThe WJ IV and Beyond CHC Theory:  Kevin McGrew's NASP mini-skills workshop
The WJ IV and Beyond CHC Theory: Kevin McGrew's NASP mini-skills workshopKevin McGrew
 
The WJ IV Measurement of Auditory Processing (Ga)
The WJ IV Measurement of Auditory Processing (Ga)The WJ IV Measurement of Auditory Processing (Ga)
The WJ IV Measurement of Auditory Processing (Ga)Kevin McGrew
 
Overview of the WJ IV Cognitive Battery: GIA and CHC Clusters
Overview of the WJ IV Cognitive Battery: GIA and CHC ClustersOverview of the WJ IV Cognitive Battery: GIA and CHC Clusters
Overview of the WJ IV Cognitive Battery: GIA and CHC ClustersKevin McGrew
 
CHC Theory Codebook 2: Cognitive definitions
CHC Theory Codebook 2:  Cognitive definitionsCHC Theory Codebook 2:  Cognitive definitions
CHC Theory Codebook 2: Cognitive definitionsKevin McGrew
 
CHC Theory Codebook 1: Cognitive definitions
CHC Theory Codebook 1:  Cognitive definitionsCHC Theory Codebook 1:  Cognitive definitions
CHC Theory Codebook 1: Cognitive definitionsKevin McGrew
 

More from Kevin McGrew (20)

The Model of Achievement Competence Motivation (MACM) Part E: Crossing the R...
The Model of Achievement Competence Motivation (MACM) Part E:  Crossing the R...The Model of Achievement Competence Motivation (MACM) Part E:  Crossing the R...
The Model of Achievement Competence Motivation (MACM) Part E: Crossing the R...
 
The Model of Achievement Competence Motivation (MACM): Part D: The volition ...
The Model of Achievement Competence Motivation (MACM): Part D:  The volition ...The Model of Achievement Competence Motivation (MACM): Part D:  The volition ...
The Model of Achievement Competence Motivation (MACM): Part D: The volition ...
 
The Model of Achievement Competence Motivation (MACM) Part C: The motivation...
The Model of Achievement Competence Motivation (MACM) Part C:  The motivation...The Model of Achievement Competence Motivation (MACM) Part C:  The motivation...
The Model of Achievement Competence Motivation (MACM) Part C: The motivation...
 
The Model of Achievement Competence Motivation (MACM): Part B - An overview ...
The Model of Achievement Competence Motivation (MACM):  Part B - An overview ...The Model of Achievement Competence Motivation (MACM):  Part B - An overview ...
The Model of Achievement Competence Motivation (MACM): Part B - An overview ...
 
The Model of Achievement Competence Motivation (MACM): Part A Introduction o...
The Model of Achievement Competence Motivation (MACM):  Part A Introduction o...The Model of Achievement Competence Motivation (MACM):  Part A Introduction o...
The Model of Achievement Competence Motivation (MACM): Part A Introduction o...
 
The WJ IV Cognitive GIA in iintellectual disability (ID) assessment
The WJ IV Cognitive GIA in iintellectual disability (ID) assessmentThe WJ IV Cognitive GIA in iintellectual disability (ID) assessment
The WJ IV Cognitive GIA in iintellectual disability (ID) assessment
 
The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence: Schne...
The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence:  Schne...The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence:  Schne...
The Evolution of the Cattell-Horn-Carrol (CHC) Theory of Intelligence: Schne...
 
Beyond cognitive abilities: An integrative model of learning-related persona...
Beyond cognitive abilities:  An integrative model of learning-related persona...Beyond cognitive abilities:  An integrative model of learning-related persona...
Beyond cognitive abilities: An integrative model of learning-related persona...
 
What about executive functions and CHC theory: New research for discussion
What about executive functions and CHC theory:  New research for discussionWhat about executive functions and CHC theory:  New research for discussion
What about executive functions and CHC theory: New research for discussion
 
"Intelligent" intelligence testing with the WJ IV COG: Why do some individua...
"Intelligent" intelligence testing with the WJ IV COG:  Why do some individua..."Intelligent" intelligence testing with the WJ IV COG:  Why do some individua...
"Intelligent" intelligence testing with the WJ IV COG: Why do some individua...
 
CHC model of inteligence revised (v2.4). Has Glr been incorrectly conceptual...
CHC model of inteligence revised (v2.4).  Has Glr been incorrectly conceptual...CHC model of inteligence revised (v2.4).  Has Glr been incorrectly conceptual...
CHC model of inteligence revised (v2.4). Has Glr been incorrectly conceptual...
 
What is "intelligent" intelligence testing
What is "intelligent" intelligence testingWhat is "intelligent" intelligence testing
What is "intelligent" intelligence testing
 
"intelligent" intelligence testing: Why do some individuals obtain markedly ...
"intelligent" intelligence testing:  Why do some individuals obtain markedly ..."intelligent" intelligence testing:  Why do some individuals obtain markedly ...
"intelligent" intelligence testing: Why do some individuals obtain markedly ...
 
"intelligent" intelligence testing: Evaluating wihtin CHC domain test score ...
"intelligent" intelligence testing:  Evaluating wihtin CHC domain test score ..."intelligent" intelligence testing:  Evaluating wihtin CHC domain test score ...
"intelligent" intelligence testing: Evaluating wihtin CHC domain test score ...
 
How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...
How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...
How to evaulate the unusualness (base rate) of WJ IV cluster or test score di...
 
The WJ IV and Beyond CHC Theory: Kevin McGrew's NASP mini-skills workshop
The WJ IV and Beyond CHC Theory:  Kevin McGrew's NASP mini-skills workshopThe WJ IV and Beyond CHC Theory:  Kevin McGrew's NASP mini-skills workshop
The WJ IV and Beyond CHC Theory: Kevin McGrew's NASP mini-skills workshop
 
The WJ IV Measurement of Auditory Processing (Ga)
The WJ IV Measurement of Auditory Processing (Ga)The WJ IV Measurement of Auditory Processing (Ga)
The WJ IV Measurement of Auditory Processing (Ga)
 
Overview of the WJ IV Cognitive Battery: GIA and CHC Clusters
Overview of the WJ IV Cognitive Battery: GIA and CHC ClustersOverview of the WJ IV Cognitive Battery: GIA and CHC Clusters
Overview of the WJ IV Cognitive Battery: GIA and CHC Clusters
 
CHC Theory Codebook 2: Cognitive definitions
CHC Theory Codebook 2:  Cognitive definitionsCHC Theory Codebook 2:  Cognitive definitions
CHC Theory Codebook 2: Cognitive definitions
 
CHC Theory Codebook 1: Cognitive definitions
CHC Theory Codebook 1:  Cognitive definitionsCHC Theory Codebook 1:  Cognitive definitions
CHC Theory Codebook 1: Cognitive definitions
 

Recently uploaded

How The Hustle Milestone Referral Program Got 300K Subscribers
How The Hustle Milestone Referral Program Got 300K SubscribersHow The Hustle Milestone Referral Program Got 300K Subscribers
How The Hustle Milestone Referral Program Got 300K SubscribersFlyyx Tech
 
ICv2 Hobby Games White Paper 2024 - State of the Industry
ICv2 Hobby Games White Paper 2024 - State of the IndustryICv2 Hobby Games White Paper 2024 - State of the Industry
ICv2 Hobby Games White Paper 2024 - State of the IndustryDennisViau
 
Shravan Kumaran and sanjay kumaran.pdf..
Shravan Kumaran and sanjay kumaran.pdf..Shravan Kumaran and sanjay kumaran.pdf..
Shravan Kumaran and sanjay kumaran.pdf..ranjithapriya2
 
CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)
CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)
CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)CXO 2.0 Conference
 
Presented by Sabri international .......
Presented by Sabri international .......Presented by Sabri international .......
Presented by Sabri international .......SABRI INTERNATIONAL
 
pitchdeck ORPC 2019 data info turine.pdf
pitchdeck ORPC 2019 data info turine.pdfpitchdeck ORPC 2019 data info turine.pdf
pitchdeck ORPC 2019 data info turine.pdflebob12
 
A Case Study On SQUARE GROUP Bangladesh.pdf
A Case Study On SQUARE GROUP Bangladesh.pdfA Case Study On SQUARE GROUP Bangladesh.pdf
A Case Study On SQUARE GROUP Bangladesh.pdfmeftaul987
 
0311 National Accounts Online Giving Trends.pdf
0311 National Accounts Online Giving Trends.pdf0311 National Accounts Online Giving Trends.pdf
0311 National Accounts Online Giving Trends.pdfBloomerang
 
unfinished legacy it is a clothing brand
unfinished legacy it is a clothing brandunfinished legacy it is a clothing brand
unfinished legacy it is a clothing brandakashm530190
 
NVIDIA's overall business overview Presentation.pptx
NVIDIA's overall business overview Presentation.pptxNVIDIA's overall business overview Presentation.pptx
NVIDIA's overall business overview Presentation.pptxKrutik Rakade
 
Mist Cooling & Fogging System Company in Egypt
Mist Cooling & Fogging System Company in EgyptMist Cooling & Fogging System Company in Egypt
Mist Cooling & Fogging System Company in Egyptopstechsanjanasingh
 
Meet Raj Shamani: A Trailblazing Entrepreneur
Meet Raj Shamani: A Trailblazing EntrepreneurMeet Raj Shamani: A Trailblazing Entrepreneur
Meet Raj Shamani: A Trailblazing Entrepreneurramya202104
 
Optimize Your CRM Customization and Beyond
Optimize Your CRM Customization and BeyondOptimize Your CRM Customization and Beyond
Optimize Your CRM Customization and BeyondBoundify
 
Young Woman Entrepreneur - Kaviya Cherian
Young Woman Entrepreneur - Kaviya CherianYoung Woman Entrepreneur - Kaviya Cherian
Young Woman Entrepreneur - Kaviya CherianCDEEPANVITA
 
Pitch Deck Teardown: SuperScale's $5.4M Series A deck
Pitch Deck Teardown: SuperScale's $5.4M Series A deckPitch Deck Teardown: SuperScale's $5.4M Series A deck
Pitch Deck Teardown: SuperScale's $5.4M Series A deckHajeJanKamps
 
Strategic Resources Corporate Presentation - March 2024 Update
Strategic Resources Corporate Presentation - March 2024 UpdateStrategic Resources Corporate Presentation - March 2024 Update
Strategic Resources Corporate Presentation - March 2024 UpdateAdnet Communications
 
Unleashing the Power of Fandom: A Short Guide to Fan Business
Unleashing the Power of Fandom: A Short Guide to Fan BusinessUnleashing the Power of Fandom: A Short Guide to Fan Business
Unleashing the Power of Fandom: A Short Guide to Fan Businesstompeter3736
 
We are inviting you on board, to move forward together in the Right Direction
We are inviting you on board, to move forward together in the Right DirectionWe are inviting you on board, to move forward together in the Right Direction
We are inviting you on board, to move forward together in the Right DirectionRight Direction Aero
 

Recently uploaded (20)

How The Hustle Milestone Referral Program Got 300K Subscribers
How The Hustle Milestone Referral Program Got 300K SubscribersHow The Hustle Milestone Referral Program Got 300K Subscribers
How The Hustle Milestone Referral Program Got 300K Subscribers
 
ICv2 Hobby Games White Paper 2024 - State of the Industry
ICv2 Hobby Games White Paper 2024 - State of the IndustryICv2 Hobby Games White Paper 2024 - State of the Industry
ICv2 Hobby Games White Paper 2024 - State of the Industry
 
Shravan Kumaran and sanjay kumaran.pdf..
Shravan Kumaran and sanjay kumaran.pdf..Shravan Kumaran and sanjay kumaran.pdf..
Shravan Kumaran and sanjay kumaran.pdf..
 
CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)
CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)
CXO 2.0 Conference (Event Information Deck | Dec'24-Mar'25)
 
Presented by Sabri international .......
Presented by Sabri international .......Presented by Sabri international .......
Presented by Sabri international .......
 
WAM Corporate Presentation Mar 12 2024.pdf
WAM Corporate Presentation Mar 12 2024.pdfWAM Corporate Presentation Mar 12 2024.pdf
WAM Corporate Presentation Mar 12 2024.pdf
 
pitchdeck ORPC 2019 data info turine.pdf
pitchdeck ORPC 2019 data info turine.pdfpitchdeck ORPC 2019 data info turine.pdf
pitchdeck ORPC 2019 data info turine.pdf
 
A Case Study On SQUARE GROUP Bangladesh.pdf
A Case Study On SQUARE GROUP Bangladesh.pdfA Case Study On SQUARE GROUP Bangladesh.pdf
A Case Study On SQUARE GROUP Bangladesh.pdf
 
0311 National Accounts Online Giving Trends.pdf
0311 National Accounts Online Giving Trends.pdf0311 National Accounts Online Giving Trends.pdf
0311 National Accounts Online Giving Trends.pdf
 
unfinished legacy it is a clothing brand
unfinished legacy it is a clothing brandunfinished legacy it is a clothing brand
unfinished legacy it is a clothing brand
 
NVIDIA's overall business overview Presentation.pptx
NVIDIA's overall business overview Presentation.pptxNVIDIA's overall business overview Presentation.pptx
NVIDIA's overall business overview Presentation.pptx
 
Mist Cooling & Fogging System Company in Egypt
Mist Cooling & Fogging System Company in EgyptMist Cooling & Fogging System Company in Egypt
Mist Cooling & Fogging System Company in Egypt
 
Meet Raj Shamani: A Trailblazing Entrepreneur
Meet Raj Shamani: A Trailblazing EntrepreneurMeet Raj Shamani: A Trailblazing Entrepreneur
Meet Raj Shamani: A Trailblazing Entrepreneur
 
Optimize Your CRM Customization and Beyond
Optimize Your CRM Customization and BeyondOptimize Your CRM Customization and Beyond
Optimize Your CRM Customization and Beyond
 
Young Woman Entrepreneur - Kaviya Cherian
Young Woman Entrepreneur - Kaviya CherianYoung Woman Entrepreneur - Kaviya Cherian
Young Woman Entrepreneur - Kaviya Cherian
 
WAM Corporate Presentation Mar 12 2024_Video.pdf
WAM Corporate Presentation Mar 12 2024_Video.pdfWAM Corporate Presentation Mar 12 2024_Video.pdf
WAM Corporate Presentation Mar 12 2024_Video.pdf
 
Pitch Deck Teardown: SuperScale's $5.4M Series A deck
Pitch Deck Teardown: SuperScale's $5.4M Series A deckPitch Deck Teardown: SuperScale's $5.4M Series A deck
Pitch Deck Teardown: SuperScale's $5.4M Series A deck
 
Strategic Resources Corporate Presentation - March 2024 Update
Strategic Resources Corporate Presentation - March 2024 UpdateStrategic Resources Corporate Presentation - March 2024 Update
Strategic Resources Corporate Presentation - March 2024 Update
 
Unleashing the Power of Fandom: A Short Guide to Fan Business
Unleashing the Power of Fandom: A Short Guide to Fan BusinessUnleashing the Power of Fandom: A Short Guide to Fan Business
Unleashing the Power of Fandom: A Short Guide to Fan Business
 
We are inviting you on board, to move forward together in the Right Direction
We are inviting you on board, to move forward together in the Right DirectionWe are inviting you on board, to move forward together in the Right Direction
We are inviting you on board, to move forward together in the Right Direction
 

Applied Psych Test Design: Part D--Develop norm (standardization) plan

  • 1. The Art and Science of Test Development—Part D Develop norm (standardization) sample plan Kevin S. McGrew, PhD. Educational Psychologist Research Director Woodcock-Muñoz Foundation The basic structure and content of this presentation is grounded extensively on the test development procedures developed by Dr. Richard Woodcock
  • 2. Develop norm (standardization) sampling plan                      The goal: A nationally representative sample from which to develop test norms
  • 3. Implementation of sampling plan in practical test development framework                     Gather the norm (standardization) data and use for Rasch scaling, norm development, psychometric, and statistical analyses
  • 4. Implementation of sampling plan in Conceptual Psychometric Validity Framework    Gather the norm (standardization) data and use      for Rasch scaling, norm development,    psychometric, and statistical analyses        
  • 5. Develop norm (standardization) sample specifications for country/nation where test is intended to be used                      The goal: A nationally representative sample from which to develop test norms
  • 6. Sampling plan should be based on best available source of national statistics (select examples below)
  • 7. Three-stage sampling plan strategy used for WJ batteries in United States        Stage 1: Sampling of communities              The goal: Stage 2: Sampling of schools A nationally representative sample from which to develop test norms Stage 3: Sampling of subjects • School-age subjects – random sampling in grades • Preschool, university and adult subjects - quotas selected as per selected US Census variables
  • 8.     Goal is a nationally    representative sample upon   which to base the norms for the   measures (tests, clusters)         
  • 9. No sampling plan is perfect: “Tweak” final norm data via subject weighting A few tips/cautions •Oversample small groups and then down-weight •No amount of creative weighting can “fix” a poorly executed sampling plan •Continuous sample analysis: Make sure to frequently monitor data as it is collected to see how close the “fit” between the sampling plan and actual subject characteristics. Important so you can make adjustments during the data collection (before it is too late)
  • 13. WJ III Three-stage sampling plan strategy: Stage 1 Sampling of communities – communities sampled according to 10 different community characteristics
  • 14. Important question to think about $$ $$$$$$$$ Are students who are categorized as “low SES” (at parent/family level) from $$ Community likely to be having the same community/life/family/educational experiences as “low SES” students (at parent/family level) from Community $$$$$$$$ ?
  • 15. Illustration of potential threat to representative sample when community SES is ignored and only family/parent SES is used in sampling plan Level of abilities of selected students Cmnty C High Community SES classification Cmnty B Middle Cmnty A Low Family/Parent SES classification Low Middle High
  • 16. Community SES is one of the most important, yet most frequently overlooked variables in a test standardization sampling plan
  • 17. Sample research/norm demographic and data file variable coding sheet: WJ III Tips/Cautions • Spend considerable time designing this data collection form (be OCD) • Think of all future analyses when deciding what to include/code • If in doubt about a variable…include it (you typically can’t go back later to get information) • Create coding system with quality data entry procedures in mind • You can’t analyze what you haven’t gathered and coded from the beginning
  • 18. Development and implementation of sampling plan is probably the most critical phase of test development • GIGO – garbage ingarbage out Retain as much control over all aspects of data collection and data entry as possible Quality of data is only as good as your pool of examiners • Recruiting, training, supervising, and retaining good examiners requires major attention and is very important • Examiners need supervision • Terminate poor examiners as soon as you can • Central office should review every single piece of information on submitted test records, especially at the beginning of an examiners testing (you don’t want them practicing errors) • Run special Rasch “person fit” reports to flag test records that look suspicious (and see if they come from certain examiners) • Best examiners are not necessarily psychologists or doctoral students. (cont. next slide)
  • 19. Spend considerable time developing the test record and variable coding sheets Run “continuous sample analysis reports” to monitor sampling plan adherence or drift – so you can make changes quickly before it is too late There is no single-purpose system, or collection of published software programs, that can handle the detailed and fluid work of data entry, editing and monitoring. Be prepared to develop and pay for custom software. Dedicated professional data-entry software is a must. SPSS, Excel, etc. won’t cut it. • Double data entry verification a must • Only have a few well trained and diligent individuals enter the data Don’t succumb to samples of convenience Oversample small groups – and then down weight Seriously consider three-stage sampling plan that controls for community SES
  • 20. Planned “incomplete” (missing) data collection Matrix sampling (Partial) Multiple and EM-based data imputation
  • 21. Recent example: Australian WJ IIII standardization partial matrix sampling plan Test 1 A B C D E X Total n 1. Verbal Comprehension C C C C C C 1346 2. Visual-Auditory Learning C C C C C C 1336 3. Spatial Relations C C C C C C 1375 4. Sound Blending C C C C C C 1382 Yellow column (1) represents 5. Concept Formation C C C C C C 1300 initial “core” 6. Visual Matching C C C C C C 1364 battery prior to 7. Numbers Reversed C C C C C C 1325 implementation of “core + 8. Incomplete Words C M 627 matrix” battery 9. Auditory Working Memory C M 548 Mixed batteries plan 11. General Information M M 389 12. Retrieval Fluency M M 388 13. Picture Recognition M M 390 14. Auditory Attention M M 388 15. Analysis-Synthesis M M 362 16. Decision Speed M M 376 17. Memory for Words M M 364 1. Letter-Word Identification C C C C C C 1323 2. Reading Fluency C M 588 5. Calculation C C C C C C 1210 6. Math Fluency C M 588 7. Spelling C M 557 8. Writing Fluency M M 352 9. Passage Comprehension C C C C C C 1203 10. Applied Problems C C C C C C 964 11. Writing Samples M M 342 Totals 450 102 117 127 108 120 372 1396
  • 22. End of Part D Additional steps in test development process will be presented in subsequent modules as they are developed