![The plot of Level-1 residuals at TIME 0 against the predictor FINAL GRADE reveals that among students with a higher grade, residuals are scattered around mean zero. However, for students with lower grades, there is a propensity for greater overestimation - this is a ceiling effect. Terrence Jorgensen Amanda Rollason Max Mitchell Rachel Bishop Mallika Bandyopadhyay Measuring the Accuracy of Self-Assessment SUMMARY of RESULTS Data Sources : Two psychology professors received IRB approval to track the performance of 258 Kennesaw State University undergraduate students during Spring 2008. With their students’ permission, eight participating professors teaching three various levels of psychology courses provided anonymous exam scores, final grades, cumulative GPAs, and demographics. Before and after each of three or four noncumulative exams, students provided an estimate of what their exam score would be. The response of interest (DIFF) was calculated by subtracting the actual exam score from the predicted and postdicted scores, differentiated by the variable AFTER. Analysis : Using SAS PROC MIXED, we developed a mixed effects model to describe the change in the difference between actual and estimated scores, and the effects of taking the exam, the students’ class standing and overall course performance on the average change trajectories. Various error covariance structures were examined to appropriately model the within-subject variability. SAMPLE SAS 9.2 CODE INTRODUCTION METHOD Ideally, accurate self-assessments alert students to the gap between their own knowledge versus the expected standard of performance, at which point they choose to address these deficits through altering learning strategies, changing the focus of study, or deliberately seeking help and collaboration. Before interventions on self-assessment skills training are designed, the existing level of this skill among students must be examined. We are curious how the “illusion of knowing” changes throughout a course and what factors might influence those changes, so four research questions were formulated: FIXED EFFECTS, VARIANCE COMPONENTS, and GOODNESS of FIT ~ significant at p < .10 level; * significant at p < .05 level; ** significant at p < .01 level; *** significant at p < .001 level a There is no variance component associated with the level-1 predictor Pre–Post because its effect is constant across individuals. b rGrade is the recentered Final Course Grade at the mean value ( μ = 77.92) . Model G is: Accuracy in estimation of one’s own understanding of course material was best explained by a small quadratic effect of time- seniors seemed to catch on more quickly than underclassmen in terms of how accurate their perceptions are. Seniors’ trajectories indicate that immediately after the initial exam, they began estimating their understanding more conservatively, whereas underclassmen seemed to remain constant during the first half of the semester. However, all students experienced a trend toward overestimation near finals. This is to be expected when there are multiple simultaneous exams during the students’ schedules. Thus, although the effect of the exam was consistent over time—students’ estimates were more accurate after the exam than before—the interaction term reveals that A-students, who typically underestimate, grow less conservative after an exam as compared to C-students, who typically overestimate. SAS 9.2 SGPLOT: EXPLORATION, DIAGNOSTICS, and VISUALIZATION PARAMETER ESTIMATES Comparison of Covariance Matrix Structures The fit statistics for our final model (Model G) had the lowest values from an unstructured covariance matrix. C Students A Students Seniors Underclassmen Do students guess their exam scores more accurately before or after the exam? Does accuracy increase over the course of the semester? Is the level of the course or the class-standing a factor in explaining the level of accuracy? Is it the overall performance in college or in the specific course that is better for describing the level of accuracy? Getting the Data ... TITLE 'Wide Format' ; PROC IMPORT DATAFILE = '*.xlsx' OUT =meta REPLACE DBMS = EXCEL; SHEET = "META" ; ... RUN ; ... TITLE 'Long Format' ; DATA long; SET wide; ARRAY times [ 1 : 4 ] time1-time4; ARRAY exams [ 1 : 4 ] exam1-exam4; ... DO wave = 1 TO 4 ; time = times[wave]; exam = exams[wave]; ... OUTPUT ; END ; KEEP id year rgrade wave time ... ; RUN ; ... Creating a Random Subsample ... DATA wide ; SET wide ; idrand + 1 ; rand = RANUNI( 8675309 ); PROC SORT DATA = wide ; BY rand ; DATA sample ; SET wide ; IF _N_ <= 16 ; PROC PRINT DATA = sample ; VAR id idrand ; RUN ; ... Exploring the Data ... ODS HTML ; PROC SGPANEL DATA =sample NOAUTOLEGEND ; PANELBY idrand / COLUMNS = 4 ROWS = 4 ; TITLE 'Individual Linear and Quadratic Trajectories' ; TITLE2 'Random Subsample of n = 16' ; REFLINE 0 / AXIS = y LINEATTRS =( PATTERN = 34 THICKNESS = .5 ); REG Y =diff X =time / LINEATTRS =( PATTERN = 1 COLOR =red THICKNESS = 1 ); REG Y =diff X =time / DEGREE = 2 LI NEATTRS =( PATTERN = 1 COLOR =blue THICKNESS = 1 ); RUN ; ODS HTML CLOSE ; ... Selecting a Fixed Effects Model ... ODS RTF ; ODS OUTPUT FitStatistics=RIS; ODS GRAPHICS ON ; PROC MIXED DATA =long METHOD =ML COVTEST ; TITLE 'Model F' ; CLASS id; MODEL diff = time time*time after time*senior rgrade rgrade*after / OUTP =p1 OUTPM =pm S CHISQ ; RANDOM intercept time time*time / TYPE =UN SUB =id S ; ODS OUTPUT SolutionR=p2; RUN ; ODS GRAPHICS OFF ; PROC REG DATA =pm ; MODEL diff=pred; RUN ; QUIT ; ODS RTF CLOSE ; ... Modeling the Covariance Structure ... %MACRO fits(Type=, data=); ODS OUTPUT FitStatistics=&data; ... DATA &data; SET &data; RENAME value=&date; IF _N_ = 1 THEN DELETE; RUN; %MEND fits; % FITS (type=UN, data=un) % FITS (type=CS, data=cs) % FITS (type=CSH, data=csh) % FITS (type=AR( 1 ), data=ar) % FITS (type=ARH( 1 ), data=arh) % FITS (type=TOEP, data=toep) ... Analyzing Residuals ... DATA _NULL_ ; SET corr; IF _N_ = 1 THEN DO ; CALL SYMPUT ( 'int_sen' ,PUT(senior, 4.2 )); CALL SYMPUT ( 'int_rgr' ,PUT(rgrade, 4.2 )); END ; IF _N_ = 2 THEN DO ; CALL SYMPUT ( 'time_sen' , PUT(senior, 4.2 )); CALL SYMPUT ( 'time_rgr' ,PUT(rgrade, 4.2 )); END ; RUN ; ... PROC SGPLOT DATA = merged ; YEAXIS LABEL = 'Intercept for Linear Time' ; XEAXIS LABEL = 'Final Grade' ; SCATTER X =rgrade Y =intercept / MARKERATTRS =( SYMBOL =circlefilled); REFLINE = 0 / AXIS = Y ; INSET "R = &int_rgr" / NOBORDER POSITION =TOPRIGHT; RUN ; ODS RTF CLOSE ; ... RIS UN CS CSH AR ARH TOEP AIC (small is better) 13097.6 12729.3 13274.2 13265.3 13083.7 13041.9 13061.7 BIC (small is better) 13147.1 12881.6 13281.3 13297.2 13090.8 13073.8 13090.1 Parameters Model A (UMM) Model B1 (UGM) Model B2 (Time 2 ) Model B3 (After) Model C (Year) Model D (Year) Model E (rGrade) b Model F (rGrade) b Model G (Final) Fixed Effects Initial Status Intercept for π 0i γ 00 4.35*** (0.66) 3.91*** (0.70) 5.22*** (0.75) 5.63*** (0.76) 10.59*** (1.48) 10.44*** (1.35) 4.94*** (1.17) 5.74*** (0.66) 6.18*** (0.61) rGrade γ 01 -0.59*** (0.05) -0.60*** (0.04) -0.61*** (0.04) Sophomore γ 02 -5.67** (2.10) -5.18** (1.81) -0.29 (1.44) Junior γ 03 -6.26** (2.14) -6.34*** (1.86) 1.56 (1.55) Senior γ 04 -7.50*** (1.91) -7.43*** (1.81) 1.66 (1.67) Linear Rate of Change (Time) Intercept for π 1i γ 10 0.09 (0.07) -1.04*** (0.23) -1.01*** (0.23) -0.74** (0.26) -0.79** (0.23) -0.80*** (0.23) -0.81** (0.23) -0.79*** (0.23) rGrade γ 11 -0.01 (0.01) Sophomore γ 12 -0.02 (0.21) Junior γ 13 -0.13 (0.21) Senior γ 14 -0.71*** (0.19) -0.66*** (0.15) -0.59*** (0.16) -0.55*** (0.12) -0.51*** (0.12) Quadratic Rate of Change (Time 2 ) Intercept for π 2i γ 20 0.10*** (0.02) 0.10*** (0.02) 0.09*** (0.02) 0.10*** (0.02) 0.09*** (0.02) 0.09*** (0.02) 0.09*** (0.02) Effect of Exam (Pre v. Post) a Intercept for π 3i γ 30 -0.90* (0.37) -2.12** (0.78) -1.31** (0.46) -1.00* (0.46) -0.89* (0.37) -1.20*** (0.26) rGrade γ 31 0.13*** (0.03) 0.14*** (0.03) 0.14*** (0.02) Sophomore γ 32 1.37 (1.11) Junior γ 33 1.14 (1.12) Senior γ 34 2.28* (1.01) 1.47~ (0.79) 0.40 (0.82) Variance Components Level 1 Within- Person σ 2 ε 91.67*** (3.33) 75.40*** (3.00) 57.09*** (2.56) 56.83*** (2.55) 57.72*** (2.55) 56.83*** (2.55) 55.89*** (2.51) 55.91*** (2.51) Level 2 In initial status σ 2 0 98.89*** (9.92) 97.46*** (11.16) 113.88*** (12.61) 113.9*** (12.60) 109.1*** (12.20) 109.4*** (12.24) 73.21*** (8.99) 74.46*** (9.10) In linear rate of change σ 2 1 0.75*** (0.12) 7.91*** (1.19) 7.95*** (1.19) 7.90*** (1.19) 7.90*** (1.19) 7.98*** (1.19) 7.99*** (1.19) In quadratic rate of change σ 2 2 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) Covariance σ 01 -1.57~ (0.87) -10.70*** (2.96) -10.70*** (2.96) -11.62*** (2.96) -11.68*** (2.96) -11.14*** (2.60) -11.28*** (2.61) σ 02 0.70** (0.23) 0.70** (0.23) 0.72** (0.23) 0.73** (0.23) 0.62** (0.20) 0.63** (0.20) σ 12 -0.58*** (0.09) -0.58*** (0.09) -0.58*** (0.09) -0.58*** (0.09) -0.59*** (0.09) -0.59*** (0.09) Pseudo R 2 Statistics and Goodness of Fit R 2 yŷ 0.0000 0.0047 0.0067 0.0878 0.0866 0.3017 0.2971 0.2974 R 2 ε 0.1775 0.3772 0.3801 0.3704 0.3801 0.3903 0.3901 R 2 0 0.0422 0.0399 0.3574 0.3464 R 2 1 0.0063 0.0063 — — Deviance 13588.1 13492.9 13355.1 13349.2 13215.2 13217.6 13064.3 13069.6 12643.3 AIC 13594.1 13504.9 13375.1 13371.2 13255.2 13249.6 13102.3 13097.6 12729.3 BIC 13604.8 13526.2 13410.6 13410.3 13326.1 13306.3 13169.5 13147.1 12881.6 The prototypical change trajectories clearly illustrate the many fixed effects of the final model. Main effect of AFTER Four sets of parallel curves indicate Pre–Post Exam effect for four distinct groups. Main effect of GRADE A-students typically estimate more conservatively than C-students, but both groups’ estimates are more accurate after an exam than before. AFTER × GRADE Interaction C-students are more conservative after an exam; A-students are less conservative. TIME × YEAR Interaction Seniors and underclassmen begin similarly, but seniors grow more conservative over time until finals week, whereas underclassmen stay somewhat steady until finals week.](https://image.slidesharecdn.com/selfassessmenttdjorgensen-13284002459356-phpapp01-120204181438-phpapp01/85/Self-Assessment-1-320.jpg)
![The plot of Level-1 residuals at TIME 0 against the predictor FINAL GRADE reveals that among students with a higher grade, residuals are scattered around mean zero. However, for students with lower grades, there is a propensity for greater overestimation - this is a ceiling effect. Terrence Jorgensen Amanda Rollason Max Mitchell Rachel Bishop Mallika Bandyopadhyay Measuring the Accuracy of Self-Assessment SUMMARY of RESULTS Data Sources : Two psychology professors received IRB approval to track the performance of 258 Kennesaw State University undergraduate students during Spring 2008. With their students’ permission, eight participating professors teaching three various levels of psychology courses provided anonymous exam scores, final grades, cumulative GPAs, and demographics. Before and after each of three or four noncumulative exams, students provided an estimate of what their exam score would be. The response of interest (DIFF) was calculated by subtracting the actual exam score from the predicted and postdicted scores, differentiated by the variable AFTER. Analysis : Using SAS PROC MIXED, we developed a mixed effects model to describe the change in the difference between actual and estimated scores, and the effects of taking the exam, the students’ class standing and overall course performance on the average change trajectories. Various error covariance structures were examined to appropriately model the within-subject variability. SAMPLE SAS 9.2 CODE INTRODUCTION METHOD Ideally, accurate self-assessments alert students to the gap between their own knowledge versus the expected standard of performance, at which point they choose to address these deficits through altering learning strategies, changing the focus of study, or deliberately seeking help and collaboration. Before interventions on self-assessment skills training are designed, the existing level of this skill among students must be examined. We are curious how the “illusion of knowing” changes throughout a course and what factors might influence those changes, so four research questions were formulated: FIXED EFFECTS, VARIANCE COMPONENTS, and GOODNESS of FIT ~ significant at p < .10 level; * significant at p < .05 level; ** significant at p < .01 level; *** significant at p < .001 level a There is no variance component associated with the level-1 predictor Pre–Post because its effect is constant across individuals. b rGrade is the recentered Final Course Grade at the mean value ( μ = 77.92) . Model G is: Accuracy in estimation of one’s own understanding of course material was best explained by a small quadratic effect of time- seniors seemed to catch on more quickly than underclassmen in terms of how accurate their perceptions are. Seniors’ trajectories indicate that immediately after the initial exam, they began estimating their understanding more conservatively, whereas underclassmen seemed to remain constant during the first half of the semester. However, all students experienced a trend toward overestimation near finals. This is to be expected when there are multiple simultaneous exams during the students’ schedules. Thus, although the effect of the exam was consistent over time—students’ estimates were more accurate after the exam than before—the interaction term reveals that A-students, who typically underestimate, grow less conservative after an exam as compared to C-students, who typically overestimate. SAS 9.2 SGPLOT: EXPLORATION, DIAGNOSTICS, and VISUALIZATION PARAMETER ESTIMATES Comparison of Covariance Matrix Structures The fit statistics for our final model (Model G) had the lowest values from an unstructured covariance matrix. C Students A Students Seniors Underclassmen Do students guess their exam scores more accurately before or after the exam? Does accuracy increase over the course of the semester? Is the level of the course or the class-standing a factor in explaining the level of accuracy? Is it the overall performance in college or in the specific course that is better for describing the level of accuracy? Getting the Data ... TITLE 'Wide Format' ; PROC IMPORT DATAFILE = '*.xlsx' OUT =meta REPLACE DBMS = EXCEL; SHEET = "META" ; ... RUN ; ... TITLE 'Long Format' ; DATA long; SET wide; ARRAY times [ 1 : 4 ] time1-time4; ARRAY exams [ 1 : 4 ] exam1-exam4; ... DO wave = 1 TO 4 ; time = times[wave]; exam = exams[wave]; ... OUTPUT ; END ; KEEP id year rgrade wave time ... ; RUN ; ... Creating a Random Subsample ... DATA wide ; SET wide ; idrand + 1 ; rand = RANUNI( 8675309 ); PROC SORT DATA = wide ; BY rand ; DATA sample ; SET wide ; IF _N_ <= 16 ; PROC PRINT DATA = sample ; VAR id idrand ; RUN ; ... Exploring the Data ... ODS HTML ; PROC SGPANEL DATA =sample NOAUTOLEGEND ; PANELBY idrand / COLUMNS = 4 ROWS = 4 ; TITLE 'Individual Linear and Quadratic Trajectories' ; TITLE2 'Random Subsample of n = 16' ; REFLINE 0 / AXIS = y LINEATTRS =( PATTERN = 34 THICKNESS = .5 ); REG Y =diff X =time / LINEATTRS =( PATTERN = 1 COLOR =red THICKNESS = 1 ); REG Y =diff X =time / DEGREE = 2 LI NEATTRS =( PATTERN = 1 COLOR =blue THICKNESS = 1 ); RUN ; ODS HTML CLOSE ; ... Selecting a Fixed Effects Model ... ODS RTF ; ODS OUTPUT FitStatistics=RIS; ODS GRAPHICS ON ; PROC MIXED DATA =long METHOD =ML COVTEST ; TITLE 'Model F' ; CLASS id; MODEL diff = time time*time after time*senior rgrade rgrade*after / OUTP =p1 OUTPM =pm S CHISQ ; RANDOM intercept time time*time / TYPE =UN SUB =id S ; ODS OUTPUT SolutionR=p2; RUN ; ODS GRAPHICS OFF ; PROC REG DATA =pm ; MODEL diff=pred; RUN ; QUIT ; ODS RTF CLOSE ; ... Modeling the Covariance Structure ... %MACRO fits(Type=, data=); ODS OUTPUT FitStatistics=&data; ... DATA &data; SET &data; RENAME value=&date; IF _N_ = 1 THEN DELETE; RUN; %MEND fits; % FITS (type=UN, data=un) % FITS (type=CS, data=cs) % FITS (type=CSH, data=csh) % FITS (type=AR( 1 ), data=ar) % FITS (type=ARH( 1 ), data=arh) % FITS (type=TOEP, data=toep) ... Analyzing Residuals ... DATA _NULL_ ; SET corr; IF _N_ = 1 THEN DO ; CALL SYMPUT ( 'int_sen' ,PUT(senior, 4.2 )); CALL SYMPUT ( 'int_rgr' ,PUT(rgrade, 4.2 )); END ; IF _N_ = 2 THEN DO ; CALL SYMPUT ( 'time_sen' , PUT(senior, 4.2 )); CALL SYMPUT ( 'time_rgr' ,PUT(rgrade, 4.2 )); END ; RUN ; ... PROC SGPLOT DATA = merged ; YEAXIS LABEL = 'Intercept for Linear Time' ; XEAXIS LABEL = 'Final Grade' ; SCATTER X =rgrade Y =intercept / MARKERATTRS =( SYMBOL =circlefilled); REFLINE = 0 / AXIS = Y ; INSET "R = &int_rgr" / NOBORDER POSITION =TOPRIGHT; RUN ; ODS RTF CLOSE ; ... RIS UN CS CSH AR ARH TOEP AIC (small is better) 13097.6 12729.3 13274.2 13265.3 13083.7 13041.9 13061.7 BIC (small is better) 13147.1 12881.6 13281.3 13297.2 13090.8 13073.8 13090.1 Parameters Model A (UMM) Model B1 (UGM) Model B2 (Time 2 ) Model B3 (After) Model C (Year) Model D (Year) Model E (rGrade) b Model F (rGrade) b Model G (Final) Fixed Effects Initial Status Intercept for π 0i γ 00 4.35*** (0.66) 3.91*** (0.70) 5.22*** (0.75) 5.63*** (0.76) 10.59*** (1.48) 10.44*** (1.35) 4.94*** (1.17) 5.74*** (0.66) 6.18*** (0.61) rGrade γ 01 -0.59*** (0.05) -0.60*** (0.04) -0.61*** (0.04) Sophomore γ 02 -5.67** (2.10) -5.18** (1.81) -0.29 (1.44) Junior γ 03 -6.26** (2.14) -6.34*** (1.86) 1.56 (1.55) Senior γ 04 -7.50*** (1.91) -7.43*** (1.81) 1.66 (1.67) Linear Rate of Change (Time) Intercept for π 1i γ 10 0.09 (0.07) -1.04*** (0.23) -1.01*** (0.23) -0.74** (0.26) -0.79** (0.23) -0.80*** (0.23) -0.81** (0.23) -0.79*** (0.23) rGrade γ 11 -0.01 (0.01) Sophomore γ 12 -0.02 (0.21) Junior γ 13 -0.13 (0.21) Senior γ 14 -0.71*** (0.19) -0.66*** (0.15) -0.59*** (0.16) -0.55*** (0.12) -0.51*** (0.12) Quadratic Rate of Change (Time 2 ) Intercept for π 2i γ 20 0.10*** (0.02) 0.10*** (0.02) 0.09*** (0.02) 0.10*** (0.02) 0.09*** (0.02) 0.09*** (0.02) 0.09*** (0.02) Effect of Exam (Pre v. Post) a Intercept for π 3i γ 30 -0.90* (0.37) -2.12** (0.78) -1.31** (0.46) -1.00* (0.46) -0.89* (0.37) -1.20*** (0.26) rGrade γ 31 0.13*** (0.03) 0.14*** (0.03) 0.14*** (0.02) Sophomore γ 32 1.37 (1.11) Junior γ 33 1.14 (1.12) Senior γ 34 2.28* (1.01) 1.47~ (0.79) 0.40 (0.82) Variance Components Level 1 Within- Person σ 2 ε 91.67*** (3.33) 75.40*** (3.00) 57.09*** (2.56) 56.83*** (2.55) 57.72*** (2.55) 56.83*** (2.55) 55.89*** (2.51) 55.91*** (2.51) Level 2 In initial status σ 2 0 98.89*** (9.92) 97.46*** (11.16) 113.88*** (12.61) 113.9*** (12.60) 109.1*** (12.20) 109.4*** (12.24) 73.21*** (8.99) 74.46*** (9.10) In linear rate of change σ 2 1 0.75*** (0.12) 7.91*** (1.19) 7.95*** (1.19) 7.90*** (1.19) 7.90*** (1.19) 7.98*** (1.19) 7.99*** (1.19) In quadratic rate of change σ 2 2 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) Covariance σ 01 -1.57~ (0.87) -10.70*** (2.96) -10.70*** (2.96) -11.62*** (2.96) -11.68*** (2.96) -11.14*** (2.60) -11.28*** (2.61) σ 02 0.70** (0.23) 0.70** (0.23) 0.72** (0.23) 0.73** (0.23) 0.62** (0.20) 0.63** (0.20) σ 12 -0.58*** (0.09) -0.58*** (0.09) -0.58*** (0.09) -0.58*** (0.09) -0.59*** (0.09) -0.59*** (0.09) Pseudo R 2 Statistics and Goodness of Fit R 2 yŷ 0.0000 0.0047 0.0067 0.0878 0.0866 0.3017 0.2971 0.2974 R 2 ε 0.1775 0.3772 0.3801 0.3704 0.3801 0.3903 0.3901 R 2 0 0.0422 0.0399 0.3574 0.3464 R 2 1 0.0063 0.0063 — — Deviance 13588.1 13492.9 13355.1 13349.2 13215.2 13217.6 13064.3 13069.6 12643.3 AIC 13594.1 13504.9 13375.1 13371.2 13255.2 13249.6 13102.3 13097.6 12729.3 BIC 13604.8 13526.2 13410.6 13410.3 13326.1 13306.3 13169.5 13147.1 12881.6 The prototypical change trajectories clearly illustrate the many fixed effects of the final model. Main effect of AFTER Four sets of parallel curves indicate Pre–Post Exam effect for four distinct groups. Main effect of GRADE A-students typically estimate more conservatively than C-students, but both groups’ estimates are more accurate after an exam than before. AFTER × GRADE Interaction C-students are more conservative after an exam; A-students are less conservative. TIME × YEAR Interaction Seniors and underclassmen begin similarly, but seniors grow more conservative over time until finals week, whereas underclassmen stay somewhat steady until finals week.](https://image.slidesharecdn.com/selfassessmenttdjorgensen-13284002459356-phpapp01-120204181438-phpapp01/75/Self-Assessment-1-2048.jpg)
![[M3A4] Data Analysis and Interpretation Specialization](https://cdn.slidesharecdn.com/ss_thumbnails/m3a4assigment-170613165711-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Q1~12]Aclothingstoreisconsideringtwomethodstoreducetheselosses1).docx](https://cdn.slidesharecdn.com/ss_thumbnails/q112aclothingstoreisconsideringtwomethodstoreducetheselosses1-221114184210-53a9c855-thumbnail.jpg?width=640&height=640&fit=bounds)
Self Assessment
- 1. The plot ofLevel-1 residuals at TIME 0 against the predictor FINAL GRADE reveals that among students with a higher grade, residuals are scattered around mean zero. However, for students with lower grades, there is a propensity for greater overestimation - this is a ceiling effect. Terrence Jorgensen Amanda Rollason Max Mitchell Rachel Bishop Mallika Bandyopadhyay Measuring the Accuracy of Self-Assessment SUMMARY of RESULTS Data Sources : Two psychology professors received IRB approval to track the performance of 258 Kennesaw State University undergraduate students during Spring 2008. With their students’ permission, eight participating professors teaching three various levels of psychology courses provided anonymous exam scores, final grades, cumulative GPAs, and demographics. Before and after each of three or four noncumulative exams, students provided an estimate of what their exam score would be. The response of interest (DIFF) was calculated by subtracting the actual exam score from the predicted and postdicted scores, differentiated by the variable AFTER. Analysis : Using SAS PROC MIXED, we developed a mixed effects model to describe the change in the difference between actual and estimated scores, and the effects of taking the exam, the students’ class standing and overall course performance on the average change trajectories. Various error covariance structures were examined to appropriately model the within-subject variability. SAMPLE SAS 9.2 CODE INTRODUCTION METHOD Ideally, accurate self-assessments alert students to the gap between their own knowledge versus the expected standard of performance, at which point they choose to address these deficits through altering learning strategies, changing the focus of study, or deliberately seeking help and collaboration. Before interventions on self-assessment skills training are designed, the existing level of this skill among students must be examined. We are curious how the “illusion of knowing” changes throughout a course and what factors might influence those changes, so four research questions were formulated: FIXED EFFECTS, VARIANCE COMPONENTS, and GOODNESS of FIT ~ significant at p < .10 level; * significant at p < .05 level; ** significant at p < .01 level; *** significant at p < .001 level a There is no variance component associated with the level-1 predictor Pre–Post because its effect is constant across individuals. b rGrade is the recentered Final Course Grade at the mean value ( μ = 77.92) . Model G is: Accuracy in estimation of one’s own understanding of course material was best explained by a small quadratic effect of time- seniors seemed to catch on more quickly than underclassmen in terms of how accurate their perceptions are. Seniors’ trajectories indicate that immediately after the initial exam, they began estimating their understanding more conservatively, whereas underclassmen seemed to remain constant during the first half of the semester. However, all students experienced a trend toward overestimation near finals. This is to be expected when there are multiple simultaneous exams during the students’ schedules. Thus, although the effect of the exam was consistent over time—students’ estimates were more accurate after the exam than before—the interaction term reveals that A-students, who typically underestimate, grow less conservative after an exam as compared to C-students, who typically overestimate. SAS 9.2 SGPLOT: EXPLORATION, DIAGNOSTICS, and VISUALIZATION PARAMETER ESTIMATES Comparison of Covariance Matrix Structures The fit statistics for our final model (Model G) had the lowest values from an unstructured covariance matrix. C Students A Students Seniors Underclassmen Do students guess their exam scores more accurately before or after the exam? Does accuracy increase over the course of the semester? Is the level of the course or the class-standing a factor in explaining the level of accuracy? Is it the overall performance in college or in the specific course that is better for describing the level of accuracy? Getting the Data ... TITLE 'Wide Format' ; PROC IMPORT DATAFILE = '*.xlsx' OUT =meta REPLACE DBMS = EXCEL; SHEET = "META" ; ... RUN ; ... TITLE 'Long Format' ; DATA long; SET wide; ARRAY times [ 1 : 4 ] time1-time4; ARRAY exams [ 1 : 4 ] exam1-exam4; ... DO wave = 1 TO 4 ; time = times[wave]; exam = exams[wave]; ... OUTPUT ; END ; KEEP id year rgrade wave time ... ; RUN ; ... Creating a Random Subsample ... DATA wide ; SET wide ; idrand + 1 ; rand = RANUNI( 8675309 ); PROC SORT DATA = wide ; BY rand ; DATA sample ; SET wide ; IF _N_ <= 16 ; PROC PRINT DATA = sample ; VAR id idrand ; RUN ; ... Exploring the Data ... ODS HTML ; PROC SGPANEL DATA =sample NOAUTOLEGEND ; PANELBY idrand / COLUMNS = 4 ROWS = 4 ; TITLE 'Individual Linear and Quadratic Trajectories' ; TITLE2 'Random Subsample of n = 16' ; REFLINE 0 / AXIS = y LINEATTRS =( PATTERN = 34 THICKNESS = .5 ); REG Y =diff X =time / LINEATTRS =( PATTERN = 1 COLOR =red THICKNESS = 1 ); REG Y =diff X =time / DEGREE = 2 LI NEATTRS =( PATTERN = 1 COLOR =blue THICKNESS = 1 ); RUN ; ODS HTML CLOSE ; ... Selecting a Fixed Effects Model ... ODS RTF ; ODS OUTPUT FitStatistics=RIS; ODS GRAPHICS ON ; PROC MIXED DATA =long METHOD =ML COVTEST ; TITLE 'Model F' ; CLASS id; MODEL diff = time time*time after time*senior rgrade rgrade*after / OUTP =p1 OUTPM =pm S CHISQ ; RANDOM intercept time time*time / TYPE =UN SUB =id S ; ODS OUTPUT SolutionR=p2; RUN ; ODS GRAPHICS OFF ; PROC REG DATA =pm ; MODEL diff=pred; RUN ; QUIT ; ODS RTF CLOSE ; ... Modeling the Covariance Structure ... %MACRO fits(Type=, data=); ODS OUTPUT FitStatistics=&data; ... DATA &data; SET &data; RENAME value=&date; IF _N_ = 1 THEN DELETE; RUN; %MEND fits; % FITS (type=UN, data=un) % FITS (type=CS, data=cs) % FITS (type=CSH, data=csh) % FITS (type=AR( 1 ), data=ar) % FITS (type=ARH( 1 ), data=arh) % FITS (type=TOEP, data=toep) ... Analyzing Residuals ... DATA _NULL_ ; SET corr; IF _N_ = 1 THEN DO ; CALL SYMPUT ( 'int_sen' ,PUT(senior, 4.2 )); CALL SYMPUT ( 'int_rgr' ,PUT(rgrade, 4.2 )); END ; IF _N_ = 2 THEN DO ; CALL SYMPUT ( 'time_sen' , PUT(senior, 4.2 )); CALL SYMPUT ( 'time_rgr' ,PUT(rgrade, 4.2 )); END ; RUN ; ... PROC SGPLOT DATA = merged ; YEAXIS LABEL = 'Intercept for Linear Time' ; XEAXIS LABEL = 'Final Grade' ; SCATTER X =rgrade Y =intercept / MARKERATTRS =( SYMBOL =circlefilled); REFLINE = 0 / AXIS = Y ; INSET "R = &int_rgr" / NOBORDER POSITION =TOPRIGHT; RUN ; ODS RTF CLOSE ; ... RIS UN CS CSH AR ARH TOEP AIC (small is better) 13097.6 12729.3 13274.2 13265.3 13083.7 13041.9 13061.7 BIC (small is better) 13147.1 12881.6 13281.3 13297.2 13090.8 13073.8 13090.1 Parameters Model A (UMM) Model B1 (UGM) Model B2 (Time 2 ) Model B3 (After) Model C (Year) Model D (Year) Model E (rGrade) b Model F (rGrade) b Model G (Final) Fixed Effects Initial Status Intercept for π 0i γ 00 4.35*** (0.66) 3.91*** (0.70) 5.22*** (0.75) 5.63*** (0.76) 10.59*** (1.48) 10.44*** (1.35) 4.94*** (1.17) 5.74*** (0.66) 6.18*** (0.61) rGrade γ 01 -0.59*** (0.05) -0.60*** (0.04) -0.61*** (0.04) Sophomore γ 02 -5.67** (2.10) -5.18** (1.81) -0.29 (1.44) Junior γ 03 -6.26** (2.14) -6.34*** (1.86) 1.56 (1.55) Senior γ 04 -7.50*** (1.91) -7.43*** (1.81) 1.66 (1.67) Linear Rate of Change (Time) Intercept for π 1i γ 10 0.09 (0.07) -1.04*** (0.23) -1.01*** (0.23) -0.74** (0.26) -0.79** (0.23) -0.80*** (0.23) -0.81** (0.23) -0.79*** (0.23) rGrade γ 11 -0.01 (0.01) Sophomore γ 12 -0.02 (0.21) Junior γ 13 -0.13 (0.21) Senior γ 14 -0.71*** (0.19) -0.66*** (0.15) -0.59*** (0.16) -0.55*** (0.12) -0.51*** (0.12) Quadratic Rate of Change (Time 2 ) Intercept for π 2i γ 20 0.10*** (0.02) 0.10*** (0.02) 0.09*** (0.02) 0.10*** (0.02) 0.09*** (0.02) 0.09*** (0.02) 0.09*** (0.02) Effect of Exam (Pre v. Post) a Intercept for π 3i γ 30 -0.90* (0.37) -2.12** (0.78) -1.31** (0.46) -1.00* (0.46) -0.89* (0.37) -1.20*** (0.26) rGrade γ 31 0.13*** (0.03) 0.14*** (0.03) 0.14*** (0.02) Sophomore γ 32 1.37 (1.11) Junior γ 33 1.14 (1.12) Senior γ 34 2.28* (1.01) 1.47~ (0.79) 0.40 (0.82) Variance Components Level 1 Within- Person σ 2 ε 91.67*** (3.33) 75.40*** (3.00) 57.09*** (2.56) 56.83*** (2.55) 57.72*** (2.55) 56.83*** (2.55) 55.89*** (2.51) 55.91*** (2.51) Level 2 In initial status σ 2 0 98.89*** (9.92) 97.46*** (11.16) 113.88*** (12.61) 113.9*** (12.60) 109.1*** (12.20) 109.4*** (12.24) 73.21*** (8.99) 74.46*** (9.10) In linear rate of change σ 2 1 0.75*** (0.12) 7.91*** (1.19) 7.95*** (1.19) 7.90*** (1.19) 7.90*** (1.19) 7.98*** (1.19) 7.99*** (1.19) In quadratic rate of change σ 2 2 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) 0.048*** (0.008) Covariance σ 01 -1.57~ (0.87) -10.70*** (2.96) -10.70*** (2.96) -11.62*** (2.96) -11.68*** (2.96) -11.14*** (2.60) -11.28*** (2.61) σ 02 0.70** (0.23) 0.70** (0.23) 0.72** (0.23) 0.73** (0.23) 0.62** (0.20) 0.63** (0.20) σ 12 -0.58*** (0.09) -0.58*** (0.09) -0.58*** (0.09) -0.58*** (0.09) -0.59*** (0.09) -0.59*** (0.09) Pseudo R 2 Statistics and Goodness of Fit R 2 yŷ 0.0000 0.0047 0.0067 0.0878 0.0866 0.3017 0.2971 0.2974 R 2 ε 0.1775 0.3772 0.3801 0.3704 0.3801 0.3903 0.3901 R 2 0 0.0422 0.0399 0.3574 0.3464 R 2 1 0.0063 0.0063 — — Deviance 13588.1 13492.9 13355.1 13349.2 13215.2 13217.6 13064.3 13069.6 12643.3 AIC 13594.1 13504.9 13375.1 13371.2 13255.2 13249.6 13102.3 13097.6 12729.3 BIC 13604.8 13526.2 13410.6 13410.3 13326.1 13306.3 13169.5 13147.1 12881.6 The prototypical change trajectories clearly illustrate the many fixed effects of the final model. Main effect of AFTER Four sets of parallel curves indicate Pre–Post Exam effect for four distinct groups. Main effect of GRADE A-students typically estimate more conservatively than C-students, but both groups’ estimates are more accurate after an exam than before. AFTER × GRADE Interaction C-students are more conservative after an exam; A-students are less conservative. TIME × YEAR Interaction Seniors and underclassmen begin similarly, but seniors grow more conservative over time until finals week, whereas underclassmen stay somewhat steady until finals week.