This article was downloaded by: [22.214.171.124]On: 07 February 2013, At: 00:41Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK Journal of Clinical Child Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hcap19 Factors influencing elementary school teachers ratings of ADHD and ODD behaviors Jack Stevens & Alexandra L. Quittner Version of record first published: 07 Jun 2010.To cite this article: Jack Stevens & Alexandra L. Quittner (1998): Factors influencing elementary school teachers ratings ofADHD and ODD behaviors, Journal of Clinical Child Psychology, 27:4, 406-414To link to this article: http://dx.doi.org/10.1207/s15374424jccp2704_4PLEASE SCROLL DOWN FOR ARTICLEFull terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditionsThis article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form toanyone is expressly forbidden.The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses shouldbe independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims,proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly inconnection with or arising out of the use of this material.
Journal of Clinical Child Psychology Copyright O 1998 by 1998, Vol. 27, No. 4,406-414 Lawrence Erlbaum Associates, Inc. Factors Influencing Elementary School Teachers Ratings of ADHD and ODD Behaviors Jack Stevens and Alexandra L. Quittner Department of Psychology, Indiana University-Bloomington Howard Abikoff New York University Medical Center Examinedfactors that influence teachers ratings of children with either attention def- icit hyperactivity disorder (ADHD) or oppositional demnt disorder (ODD). 105 teachers watched 2 videotapes-1 depicting a normal child and the other a child with either ADHD or ODD--and rated each child using 2 differentquestionnaires.Results indicated that teachers accurately rated the child on the ADHD versus ODD tape as having significantly more inattention and hyperactivity but significantly lessDownloaded by [126.96.36.199] at 00:41 07 February 2013 oppositionality. However, effectsizes indicated the presence of a unidirectional, neg- ative halo efSect of oppositional behavwrs on ratings of hyperactivity and inattention. Teachers appeared less biased in theirjudgments when using a welboperationalized rating scale. Finally, knowledge, education, and experience with children withADHD generally had no effect on the accuracy of teachers ratings. Elementary school teachers play a major role in childhood behavior. Often they lack the time or ability the assessment of childrens academic and behavioral to notice specific childrens behaviors. Alternatively, problems. Due to their extensive contact with chil- teachers may have frequent opportunities to observe dren in a variety of structured and unstructured set- childrens conduct but may be biased by certain char- tings and their knowledge of age-appropriate skills acteristics of the students. For example, Stevens and behaviors, teachers provide important informa- (1980) found that ethnicity and socioeconomic status tion for both clinical and research purposes (Atkins, produced negative halo effects on teachers ratings. Pelham, & Licht, 1985; Shelton & Barkley, 1995). Specifically, the videotaped behaviors of African Most often, educators are asked to evaluate children American and poor children were evaluated as more with psychological problems by completing standard- deviant than Caucasian and middle-class children, de- ized rating scales (Sandoval, 1981). These ratings are spite identical rates of disruptive behaviors. then used to make decisions concerning diagnosis, treat- Negative halo effects may be associated not only ment, and educational placement (see Brown, 1985). with certain demographic characteristics but also with Teacher ratings are also frequently used in a research the presence of certain childhood behaviors. These bi- context as inclusionary criteria, as a means of monitor- ases have been found to produce errors in teachers ing treatment progress, and as indicators of long-term judgments, particularly in the area of attention deficit outcomes (DuPaul, Guevremont, & Barkley, 1991). hyperactivity disorder (ADHD). Mann et al. (1992) Despite evidence indicating that teachers ratings noted that the diagnosis is "based more on an assess- can discriminate between children and adolescents ment of developmentally inappropriate intensity, fre- with higher versus lower levels of psychological diffi- quency, andor duration of the behavior rather than its culties (e.g., Dalley, Bolocofsky, & Karlin, 1994), mere presence. Such judgments increase the possibil- teachers are not always accurate and objective raters of ity of observer bias" (p. 1539). Researchers have found that although teachers rat- ings can reliably differentiate children with and with- We thank Dr. Janice Bizzari for her support and assistance. We out attention deficits (Atkins, Pelham, & Licht, 1985), also thank the principals and teachers for their time and conscientious effort. including even those who exhibit hyperactivity Alexandra L. Quittner is now at the Department of Clinical and (Brown, 1985), teachers often do not distinguish chil- Health Psychology, University of Florida. dren with ADHD from those with symptomatology of Requests for reprints should be sent to Jack Stevens, Department the other disruptive behavior disorders--conduct dis- of Psychology, Indiana University, Bloomington, IN 47405-1301, order (CD) and ODD. For example, Schachar, E-mail: firstname.lastname@example.org,or to Alexandra L. Quittner, Depart- ment of Clinical and Health Psychology, University of Florida, P.O. Sandberg, and Rutter (1986) investigated the relation Box 100165, Gainesville, FL 32160, E-mail: email@example.com between two respondents-teachers and blind observ-
TEACHERS RATINGS OF ADHD AND ODD BEWAVIORS ers-on measures of ADHD and ODD. They found a We also examined whether teachers ratings would negative halo effect of oppositional behaviors on hy- be less biased when more behaviorally anchored items peractivity ratings but no halo effect of hyperactivity were administered (Abikoff et al., 1993). Qluestion- on oppositional behaviors. Thus, regardless of their ac- naires often contain vague items that require a global tivity level, children exhibiting defiance or aggression rating on some characteristic (e.g., "difficulty in learn- were judged as having ADHD. ing"; Vaughn, Riccio, Hynd, & Hall, 1997). These Abikoff, Courtney, Pelham, and Koplewicz (1993) items are often ambiguous with respect to the target be- found a similar unidirectional bias in which havior and may be interpreted in different ways, inter- oppositional behaviors exerted a negative halo effect fering with teachers ability to make more fine-grained on ratings of ADHD symptomatology. Specifically, distinctions in behavioral ratings. In contrast, descrip- oppositional behaviors in a child on a classroom video- tive items that include greater operationaliiation of the tape spuriously elevated teachers reports of the sever- problem behaviors (e.g., "does not sit still", "blurts out ity of his ADHD behaviors. However, hyperactive and answers") should lead to greater accuracy of teacher inattentive behaviors did not lead teachers to rate an- ratings. Thus, rating scales that employ concrete as op- other child as being defiant or aggressive. This study posed to global items may be less subject to inegative attempted to replicate these findings, testing the hy- halo effects (Mimtz & Collins, 1985). pothesis that ODD behaviors lead teachers to rate chil- Abikoff et al. (1993) found evidence of the unidi- dren as hyperactive or inattentive but that ADHD rectional bias even when teachers rated children using behaviors do not cause teachers to rate these children the Diagnostic and Statistical Manual of Mental Dis-Downloaded by [188.8.131.52] at 00:41 07 February 2013 as oppositional. orders (3rd editi~on, DSM-IZZ-R) ADHD diagnos- rev.; A central issue remains unclear from the Schachar tic category symptoms, which were specific and well et al. (1986) and Abikoff et al. (1993) studies. Spe- operationaked. In that study, however, a categorical cifically, the factors that might lead teachers to appear (all or non~e) approach to assessment was utilized for less biased in their ratings have not been identified. the DSM-III-R symptoms, which did not permit a Certain aspects of the assessment measures or particu- thorough examination of differencesin the magnitudes lar characteristics of the teachers may affect their abil- of teacher ratings. A continuous distribution of ADHD ity to differentiate between the two disorders. This ratings would permit greater detection of differencesin study sough~t address this question by assessing three to teachers ratings. In this study we compared the impact possible determinants of the bias in teachers ratings of of using twlo differentcontinuousbehavioral measures, disruptive blehavior disorders: measurement character- one with and one without global items. We bypothe- istics of the rating scales, teachers knowledge of and sized that teachers would be less biased in their ratings education in ADBD, and teachers professional expe- when using a rating scale that required behaviorally rience with these children. anchored rather than global judgments. First, we hypothesized that two properties of rating A second factor that may influence teachers ability scales may affect teachers ability to distinguish to distinguish between ADHD and ODD is their ADHD from ODD: level of confounding of symptoms knowledge of and educational background in ADHD. across the two disorders and the extent to which items Little is known about how teachers knowledge of the are behaviorally anchored. Several commonly used disorder is associated with their ratings or their use of teacher rating scale factors (e.g., Hyperactivity Factor particular intervention strategies (Greene, 1995). To of the Conners Teachers Rating Scale; Conners, 1990) date, no study has systematically evaluated the relation have individual items that reflect both ADllD and between knowledge of ADHD and teachers ability to ODD (e.g., "disturbs other children") behaviors. discriminate this childhood disorder from others. We Therefore, it is not surprising that teachers ratings of- hypothesized that greater knowledge of the disorder ten do not differentiate between these disorders when would heighten teachers awareness of ADHD individual items include symptoms applying to both. symptomatology, resulting in better distinctil~nsbe- Some researchers have found that pure items can help tween ADHD and ODD. teachers distinguish between different types of prob- Finally, we examined whether professional experi- lem behaviors (Ellen, 1989; Milich Idt Fitzgerald, ence with children with ADHD, their parents, a,ndvari- 1985). However, other studies (e.g., Abikoff et al., ous health care professionals would lead to more 1993; Schachar et al., 1986) have found no evidence accurate ratiugs. Interactions with students with that omitting confounded items reduced the unidirec- ADHLI, as well as in-service presentations and conver- tional bias. These results should be revisited to reex- sations with medical and mental health professionals amine whether items that reflect either ADHD or ODD about children with ADHD may be related to more prove useful in making this distinction. In the current positive perceptions of these children, as was found in study, we expected to find less bias in teacher ratings a study examining teachers attitudes about when utilizing specific items that reflected only hyper- mainstreaming students with learning difficulties activity and inattention as opposed to aggression. (Bender, Vail, & Scott, 1995). We hypothesized that
STEVENS, QUITTNER. & ABIKOPF teachers with greater professional experience with tapes were designed so that the target child in the children with ADHD and their caretakers would be ADHD tape engaged in behaviors characteristic of less susceptible to negative halo effects when rating ADHD and did not behave in an oppositional fashion. children displaying oppositional behaviors. In contrast, the target child in the ODD tape displayed In sum, the central objective of this study was to ex- oppositional behaviors characteristic of the disorder amine whether a unidirectional, negative halo effect but exhibited a rate of ADHD behaviors in the normal was present in teachers ratings of ADHD and ODD range. behaviors. Furthermore, we investigated the role three Several steps were taken to ensure the validity of factors-characteristics of the rating scales, knowl- the videotapes. Two independent sets of observers, edge of ADHD, and experience with ADHD-might each using either the revised Stony Brook classroom play in increasing the accuracy of teachers ratings. observation code (COC; Abikoff, Gittelman-Klein, & Klein, 1977) or the Classroom Observations of Con- duct and Attention Deficit Disorders (COCADD; Method Atkins et al., 1985; Atkins, Pelham, & Licht, 1989) and blind to the condition of the target child, were asked to Participants rate the frequency of ADHD and ODD behaviors. Their ratings were in the expected direction. For exam- One hundred and eight elementary school teachers ple, the rate of hyperactive behaviors was judged to be were recruited from eight public schools through let- twice as high for the target child in the ADHD as op-Downloaded by [184.108.40.206] at 00:41 07 February 2013 ters and phone calls to principals. In addition, three ele- posed to the normal or ODD tape. The frequency and mentary school teachers returning for a graduate types of interactions amang classmates and teacher degree in education also participated. Individual teach- were also controll~d each tape to remove any biases. on ers were not compensated for their participation; how- Finally, to minimize biases related to nonspecific char- ever, schools in which teachers participated were acteristics (e.g., attractiveness of the child), the target offered either a small monetary donation or an child in each psychopathology tape was played by the in-service presentation on ADHD after completion of same actor; a differentchild played the "normal" child. the study. Six of the 111 teachers failed to complete 75% of each of the standardized ratings scales de- scribed in the following, and therefore their data were Procedure excluded from the analyses. The final sample for the current study consisted of Individuals with elementary school teaching expe- 105 elementary school teachers from a medium-sized rience were invited to participate in a study of teachers Midwestern city. The majority of teachers were judgments of childhood behaviors. The study was con- women (90%) and Caucasian (5% were African Amer- ducted in a large room (e.g., library, conference room) ican). Fifty-seven of the participants taught exclu- at each school. After signing informed consent state- sively regular education students, 8 teachers taught ments, groups of teachers, ranging in size from 1 to 19, only special education students, and the remaining 40 were assigned to one of four conditions. taught both regular and special education students. On A between-subject design was employed in which average, teachers had considerable teaching experi- teachers viewed, in counterbalanced order, two 10-rnin ence ( M E 14.75 years, SD = 9.14). Seventy-seven had videotapes designed by Abikoff and colleagues a masters degree, and two held a doctorate. (1993). All teachers viewed a videotape of a normal fourth-grade boy in a classroom setting and either a videotape of a boy with pure ADHD or a videotape of a Stimuli boy with pure ODD. Forty-seven teachers viewed the normal and ADHD tapes (i,e., 17 saw the normal tape The three videotaped stimuli used in this study are first and 30 saw the ADHD tape first). The remaining described briefly in the following paragraphs. A more 58 teachers viewed the normal and ODD tapes (i.e., 28 detailed description of the development and validation saw the normal tape first and 30 saw the ODD tape of the videotapes can be found in Abikoff et al. (1993). first). Thus, the four conditions were: (a) nor- Each 10-min videotape highlighted the activities of maVADHD, (b) ADHDInormal, (c) normaVODD, and one specific 10-year-old boy, along with a teacher and (d) ODDInormal. several peers as they engaged in a structured academic Prior to presentation of the two tapes, teachers were activity (e.g., completing worksheets). The target child given three packets of questionnaires, and the investi- in each tape exhibited different types and frequencies gators read a page of instructions telling teachers that of disruptive childhood behaviors in order to depict a they will be presented with two videotapes showing child who was either "normal," had "pure" ADHD, or children in a fourth-grade classroom containing chil- had "pure" ODD. Specifically, the two "pathology" dren with mixed learning levels and that they can think
TEACHERS RATINGS OF ADHD AND ODD BEHAVIORS of each tape as a 10-min "slice of life" in the school standing of ADHD symptoms. Good internal consis- day. They were instructed to watch a particular young- tency coefficients (i.e., .84 to .95) were obtained with ster on the tape and were not informed of the diagnostic this sample. status (i.e., "normal," "ADHD," and "ODD) of the target children. Full instructions can be found in Abikoff et al. (1993). Knowledge of ADHD. Teachers knowledge of After viewing the first tape, teachers were told to ADHD was assessed using the 17-item Knowledge extrapolate from what they observed on the tape to scale from the Attention Deficit Hyperactivity Disor- questions involving behaviors that were not present on der Knowledge and Opinion Survey-Revised; Bennett, the tape, as they would in a real situation. After view- Power, Rostain, and Carr, 1996).This true-false test in- ing each tape, teachers completed two different rating cluded questions concerning the etiology, assessment, scales-the Conners Teacher Rating Scale (CTRS-28; treatment, and prognosis of the disorder. Correct re- Goyette, Comers, & Ulrich, 1978) and the SNAP-N sponses were coded "1," and incorrect or bllank re- Rating Scde (Swanson, personal communication, De- sponses were coded "0." Item-total correlations in the cember 13, 1996). Finally, teachers completed two present sample of elementary education teachers were questionnares in a third envelope that assessed their poor, ranging from -.22 to 30. Internal consistency knowledge of and experience with ADHD. Teachers was also low (Cronbach a = 22). were not told that the individuals appearing in the tapes were actors until completion of the study. The entireDownloaded by [220.127.116.11] at 00:41 07 February 2013 procedure lasted approximately 45 rnin. Experience with ADHD. A 23-item scale, the ADHD Experience Scale, was developed for this study. This questionnaire expanded on a similar measure de- Measures veloped by Anastopoulos (personal commu~nication, January 23,1997) andcontained three major sections: Conners Teacher Ratings Scale. The CTRS- 28 is a shortened version of the 39-item Conners 1. Educational opportunities to learn about Teacher Questionnaire (Conners, 1969) and is one of ADHD (e.g., in-service presentations). the most frequently used teacher rating scales. 2. Professional experiences with ADHD (e.g., Teachers rate the target childs behavior on a 4-point number of previous students with ADHD, con- scale ranging from not at all to very much. The ferences with parents, number of contacts with CTRS-28 yields three factors (i.e., Conduct Problem, health care professionals about these students. Hyperactivity, and Inattentive-Passive) and a 3. Attitudes about ADHD (e.g., how distressing Hyperkinesis Index, which contains 10 of the most fre- these behaviors are). quently endorsed items by teachers (Goyette et al., 1978). Good internal consistency coefficients Two subscales were created from this measure: a (.74-.94) were obtained for the four subscales of the six-item Educational Background scale (Cronbach a = CIXS-28 for teachers rating either pathology tape in .68) and an eight-item Professional Contact scale this study. (Cronbach a = .75). SNAP-IV Rating Scale. A 39-item version of Results the SNAP-IV Rating Scale was also completed by teachers to rate the presence of ADHD and ODD be- Preliminary Analyses haviors in the target child. Teachers rated the childs behavior on a 4-point scale that was similar to the one There were no systematic differences in teachers used on the CTRS-28. In this version of the SNAP-IV, characteristics across the four conditions. Chi-square there were the nine DSM (4th ed.; DSM-ZV) symptoms tests indicated that the number (ofteachers with special of ADHD-Inattentive Type, the nine DSM-ZV symp- education experience or graduate degrees was similar toms of ADHD-Hyperactive/Impulsive Type, and the across conditions,x2(3, = 105)= 1.72and3.54,p> .30. N eight DSM-Wsymptoms of ODD. The SNAP-IV was Three separate muItivariate analyses of variance chosen because its items utilized more specific opera- (MANOVAs) revealed no significant differences tional definitions and required more descriptive (e.g., across conditions in educational background, profes- "fidgets with hands or feet or squirms in seat") as op- sional contact, and knowledge of ADHD, years of expe- posed to global (e.g., "childish and immatureD) judg- rience, or ratings of distress in h~andlingADHDl behav- ments than did the CTRS-28. In addition, the iors (Hotellings Trangedfrom0.090to 1 . 4 7 5 ,> .05). ~ SNAP-IV specified behaviors according to DSM-N As expected, paired t tests indicated that teachers dif- criteria, which reflects the fields most recent under- ferentiated between the normal and the two pathology
STEVENS, QUITTNER, & ABIKOFF Table 1. Teachers Mean CTRS-28 and SNAP-IV Ratings o Each Tape f Type of Child Depicted on Tape - -- Normal ADHD ODD Scale M SD M SD M SD Effect Size Indexs CTRS-28 HA 0.19 0.08 1.91 0.57 1.66 0.55 0.27 IN-PA 0.36 0.42 1.56 0.61 1.36 0.58 0.17 HI 0.27 0.28 1.73 0.57 1.78 0.54 0.03 CP 0.03 0.08 0.74 0.53 2.05 0.57 1.23 SNAP-IV HYAMP 0.17 0.17 1.72 0.66 1.10 0.66 0.50 INATT 0.38 0.46 1.90 0.67 1.38 0.72 0.38 ODD 0.03 0.21 0.77 0.67 2.13 0.62 1.08 - - Note: N= 105for normal tape, N = 47 for ADHD tape, and N= 58 for ODD tape. CTRS-28 = Conners Teachers Rating Scale (28-itemversion);HA =Hyperactivity factor; IN-PA = Inattentive-Passive factor; HI = Hyperkinesis Index; CP = Conduct Problem factor; HYIIMP =DSM-IV ADHD Hyperactivity/Impdsivity scale; INATT = DSM-IV ADHD Inattention scale; ODD = DSM-IVOppositional Defiant Disorder scale. %ffect size = differencesin ratings between the ADHD and ODD tape controlling for the order of the normal tape, special education experience, and ratings of the normal tape. Effect sizes < S O are modest, .SO-.70 are moderate, and > .70 are large.Downloaded by [18.104.22.168] at 00:41 07 February 2013 tapes. As can be seen in Table 1,teachers rated the target trary to our prediction, the ADHD tape was rated child in the ADHD tape versus the normal tape as having significantly higher than the ODD tape on the greater symptomatology on the CTRS-28 Hyperactiv- CTRS-28 Hyperactivity Factor, F(1, 96) = 7.17, p < ity, Inattentive-Passive, andHyperkinesis scales andon .01. Note, however, that the mean CTRS Hyperactivity the SNAP-IV Hyperactivity-Impulsivity, and Inatten- Factor scores were still above the clinical cutoff of 1.5 tion scales; t(46) ranged from 12.68 to 21.47, p < .001. on both tapes. Finally, the child in the ADHD versus Similarly,teachersratedthe target child intheODD tape ODD tape was judged to have greater symptomatology versus the normal tape as having more severe on the SNAP-IV Inattention and Hyperactiv- oppositional behaviors on the Conners Conduct Prob- ity-Impulsivity scales, F(1, 96) = 13.76 and 23.92 re- lem scale and on the SNAP-IV ODD scale, t(57) =27.83 spectively,p < .001. and 24.76, p < .001. Ratings of oppositional behaviors. As predicted, the average CTRS-28 Conduct Problem and SNAP-IV ODD scores were significantly lower for teachers Ratings of Behavior on the watching the ADHD versus ODD tape, F(1, 96) = ADHD Versus ODD Tapes 145.66 and 103.24 respectively, p < .001. To test the presence of a negative halo effect (Hy- Effect sizes for judgments of ADHD versus ODD pothesis I), a 2 (Type of Tape: ADHD vs. ODD) x 2 behaviors. The effect size of type of tape (ADHD (Order of the Normal Tape: First vs. Second) x 2 (Type vs. ODD) was calculated1after controlling for order, of Teacher: Special Education/Inclusion vs. Regular special education experience, and behavior ratings on Education) analysis of covariance (ANCOVA) was the normal tape (see Table 1). As mentioned previ- conducted on each of the four CTRS-28 scales and ously, teachers ratings of the child on the ADHD ver- three SNAP-IV scales. To account for differences in sus ODD tape were significantlydifferent on two of the individual response styles across teachers, their ratings four CTRS-28 scales (Hyperactivity Factor, Conduct on the normal tape were used as covariates. Across all Problem Factor) and on all three of the SNAP-IV scales, we found no significant effects of order of tape scales. Thus, although several statistically reliable dif- or teachers special education experience as well as no ferences were found, the magnitude of the main effects significant two-way or three-way interactions. F(1,96) for type of tape was considerably less for measures of ranged from .OO to 3.35, p > .05. inattention and hyperactivity than for measures of oppositional behavior. In sum, there was substantially less discrepancy between the two pathology tapes in Ratings of hyperactive and inattentive behaviors. ratings of ADHD versus ODD behaviors, suggesting As expected, no significant differences were found be- tween ratings of the ADHD versus ODD tape on the In- ~ffect indexes were calculated using the proportion of vari- size attentive-Passive scale, F(l, 96) = 2.84, p > .09, or on ance in ratings accounted for by the type of tape. See Cohen (1992) the Hyperkinesis Index, F(l, 96) = .052, p > 3 0 . Con- for further details on effect sizes.
TEACHERS RATINGS OF ADHD AND ODD BEHAVIORS that oppositional behaviors did exert a negative halo ef- 1985). To assess the impact of these problemaltic items fect on ratings of hyperactivity and inattention. on teachers ratings, these four items were omitted from the CTRS-28 Inattentivepassive scale, and the remaining four items were subjected to an analysis of Impact of Operationalization of the covariance. As expected, teachers rating the ADHD Behavioral Measures on versus ODD tape endorsed significantly moire severe Inattention and Hyperactivity Ratings inattention o this modified scale, F(1,96) = 8.89, p < m .01. In sum, negative halo effects appear to be at least Teachers were expected to evidence less bias in partly due to measurement problems in these com- their ratings of inattention and hyperactivity as monly used ratings subscales. operationalization of the rating scale increased. This hypothesis was supported. As mentioned previously, differences in the predicted direction were obtained Factors Related to Teachers ADHD on both of the well-operationalized ADHD scales of Ratings of the Pathology Tapes the SNAP-IV but on only one of the three ADHD scales of the less behaviorally anchored CTRS-28. Teachers with greater knowledge about and more Thus, item content and wording of the questions ap- professional experience with ADHD were expected to peared to contribute significantly to the bias. rate the two pathology tapes more accurately. To ex- Two pauticular problems with the CTRS-28 were amine the relation between these teacher chmacteris-Downloaded by [22.214.171.124] at 00:41 07 February 2013 noted: confounding of content across the scales and the tics and their ratings of hyperactive and inattentive global wording of some items, First, in terms of over- behaviors, hierarchical regression analyses were per- lapping content, both the Hyperactivity factor and the formed separately for the PDHD and ODlD tapes. Hyperkinesis Index of the CTRS-28 contained items Three-step hierarchical regressions were eomputed that assessied both ADHD and ODD behaviors simulta- with the CTRS-28 InattentivePassive and Hyperac- neously (e.g., "disturbs other children"). A measure of tivity scales and the SNAP-IV Inattention and Hyper- pure inattention and hyperactivity, not confounded by activity-Impulsivity scales as dependent variables. items reflecting aggression and noncompliance, might Ratings of the normal tape andl order of the tape presen- reduce the occurrence of biased judgments of children tation were entered on the first and second steps, re- with ODD. The five InattentionlOveractivity (10) spectively. Next, five variables reflecting teachers items from the IOWA Conners Teachers Rating Scale backgrounds and experience were entered on the third (Loney & Milich, 1982) that were empirically related step. These teacher factors were: (a) ADHD Educa- to hyperactivity but not aggression were subjected to tional Background, (b) ADHD Professional Contact, an analysis of c~variance.~ partialling out ratings After (c) ADHD Knowledge soore, (d) Special Elducation of the normal tape, significantly higher ratings on these Experience, and (e) Distress Ratings (on ;I 5-point five items for the target child on the ADHD versus scale) caused by ADHD behaviors.3 ODD tape were found, M = 10.8 and 8.3 respectively, The regression analyses indicated that neither F(1,96) = 16.76, p < ,001. This indicated that the bias teachers individual rating style (Step 1) nor the order was weaker when using a scale with items reflecting of tape presentation (Step 2) was significantlly related only hyperactivity and inattention. to any of the scale scores for teachers rating either pa- Second, problems with the wording of the inatten- thology tape. The total R2 ranged from .02 to .07. The tion items were noted in an item analysis of the only exception was that highes hyperactivity ratings of CTRS-28 Inattentive-Passive scale. The Inatten- the normal tape were significantly related ito higher tive-Passive scale contains two items reflecting social CTRS-28 hyperactivity ratings of the ODD tape O)< skills as opposed to inattention (i.e., "appears to be eas- .05; total R2 = .19). On the third step, ADHD Profes- ily led by other children" and "appears to lack leader- sional Contact, ADHD Knwwledge Test score, and ship"). Two items also required global rather than Level of Distress caused by ADHD behaviors were not descriptive judgments (i.e., "childish and immature" significantly associated with ratings of the M)HDtape and "difficulty in learning"), which may be more sus- ( p > .05). However, two variables were significantly ceptible to negative halo effects (Mintz & Collins, related to teachers ratings of the ADHD tape, but in the direction opposite to our prediction: 2 ~ our study, "restless in the squirmy sense," "makes inappro- n priate noises when slhe shouldnt,""distractibilityor attention span a problem," "excitable, impulsive," and "fails to finish things that slhe 3~ulticollinearity among the predictor variables did nc>tappear to starts" were utilized. The items frrom the original (1982) scale were as be a problem. For ratings of the ADHD tape, wrrelati~ons among follows: "fidgeting," "hums and makes other odd noises," "inatten- these variables = -0.26 to 0.57. For ratings of the ODD tape, wrrela- tive, easily distracted," "excitable, impulsive," and "fails to finish tions among these variables=-0.27to 0.44. Multicollinearityis more things he stants (short attention span)." likely when correlationsare above .70 (Hanushek & Jackson, 1977).
STEVENS,QUITTNER, & ABIKOFF 1. Teachers with more special education experi- mainstreaminghas occurred,resulting in more children ence endorsed less symptomatology on the with various problems in regular education classrooms CTRS-28 InattentivePassive scale. (Stainback & Stainback, 1995;Westwood, 1997). This 2. Teachers with more ADHD education had sig- shift has given a broad spectrum of teachers greater ex- nificantly lower CTRS-28 Hyperactivity rat- posure to children with ADHD, which may have de- ings ( p < .05; total R~ = .26 and .18, respec- creased the strength of the unidirectional bias in this tively; A R~ coefficients = .21 and .l3). study. We also found strong support for the second hypoth- On the third step of the analyses of the ratings of the esis. As expected, teachers evidenced less bias when ODD tape, ADHD Professional Contact, Special Edu- completing a well-operationalized rating scale, such as cation Experience, and Level of Distress caused by the SNAP-IV, and exhibited greater bias on the less ADHD behaviors were not significant predictors. well-operationalized CTRS-28. More fine-grained However, as predicted, higher scores on the ADHD analyses indicated that individual items reflecting both Knowledge test were associated with significantly ADHD and ODD behaviors from the CTRS-28 lower CTRS-28 InattentivePassive ratings of the Hyperkinesis scale (e.g., "disturbs other children") ap- ODD tape. Moreover, ADHD Educational Back- peared to contribute to these inaccuracies. However, ground was a significant predictor for all four scales (p when items reflecting only ADHD or ODD < 05; total Rz ranged from .18 to .31; A R2 coefficients symptomatology were utilized, teachers accurately en- ranged from "10 to .28). Contrary to Hypothesis 3, dorsed greater inattention and hyperactivity for theDownloaded by [126.96.36.199] at 00:41 07 February 2013 teachers with more ADHD education endorsed more child in the ADHD versus ODD tape. Note that Abikoff inattention and hyperactivity on the ODD tape. et al. (1993) still foundevidence of a unidirectionalbias when utilizing ascalecontaining only hyperactivity and inattention items, Because that study found generally Discussion stronger evidence of a bias than the present one, purer item content may not have played asr significant arolein We obtained support for our first hypothesis that the attenuatingthis bias. Given the less extreme bias found presence of oppositional behaviors would exert a uni- in the current study,purer items have appeamd to redu~e directional, negative halo effect on teachers ratings of this problem. hyperactivity and inattention. In this study, teachers In addition, teachers were not able to differentiate generally rated the child on the ADHD versus ODD the two symptomatic children on the CTRS-28 Inat- tape as having significaqtly greater hyperactivity and tentive-Passive scale, suggesting that this measure of inattention and significantly less oppositionality. inattention lacks discriminant validity. Other research- However, an examination of the effect sizes for type of ershave also found that the Conners Teacher Rating tape indicated inaccuracies in teachers judgments. Scales lacked specificity and were not able to identify Small differences in ratings of hyperactive and inatten- those children who had attention deficits per se from tive behaviors were found between pathology tapes, in those who exhibit hyperactivity or aggression contrast to the large differences obtained in (Ullmann, Sleator, & Sprague, 1985). However, when oppositional ratings between tapes. Thus, teachers of- we excluded items that reflected social skills or were ten erroneously rated the child exhibiting only global in nature, teachers correctly rated the target oppositional behaviors as having substantial symp- child with ADHD versus ODD as more inattentive. toms of ADHD. This is consistent with our premise that item$ reflect- These findings differ somewhat from those reported ing specific behaviors, rather than global characteris- by Abikoff etal. (1993). In this study, differences in the tics, improve the accuracy of teachers ratings. ratings of hyperactive and inattentive behaviors be- Finally, we found no support for our third hypoth- tween tapes were small but statistically significant. esis. Having greater knowledge and experience with However, in the Abikoff et al. (1993) study, ADHD was not generally associated with more accu- nonsignificant differences in ratings of the two tapes rate ratings. The lack of a relation betwean knowl- were generally found, suggesting a stronger negative edge of ADHD and teacher ratings should be inter- halo effect. One possible explanation for these discrep- preted with caution, because the knowledge measure ant results is that in recent years teachers have become we used had significant limitations. Fir& item-total more aware that there are different types of childhood correlations were low, suggesting that individual behavior disorders. The data from the Abikoff et al. items were not tapping the same central construct. (1993) study were callected almost 8 years ago, and in Second, the majority of guestbns focused an factual the meantime teachers may have had greater access to information about the prevalence, etidagy, treatment, information about child behavior problems, which may and prognosis of the disordw, as opposed to knowl- have attenuated the bias. Similsuly, in the past decade a edge about relevant symptoms, T@aohers7under- strong shift in educational philosophy with regard to standing of the specific behaviors that do anid do not
TEACHEXIS RATINGS OF ADHD AND ODD BEHAVIORS constitute ADHD, rather than general facts, may be a would be advised when selecting rating scales to uti- better indicator of their ability to recognize and rate lize well-operationalized questionnaires, which appear ADHD symptoms accurately. However, the design of to facilitate differentiation of childhood behavior dis- this stu4y precluded administration of this type of orders. Mote that Conners (1997) has recently revised measure because the videotaped stimuli would have both the parent and teacher questionnaires to reflect the influenced those responses. better-operationalized DSM-N criteria. R~esearchers We also foundlittle evidencethat greater exposureto should further investigate whether improving the educational materials about ADHD was related to more operationalization of questionnaires increases their accurate ratings. In fact, our findings were in the oppo- discriminant validity, not only in the domain of child- site direction, indicating that teachers exposed to more hood disruptive behaviors but also across all areas of information about ADHD tended to rate the child with child and adult psychopathology. ODD as bavingmoreinattentionandhyperactivity. This Second, factual knowledge and professioinal experi- finding held true regardless of the operationalization of ence with ADMD do not appear to be sufficient for im- theratings scales. Perhapsteacherswithmoreeducation proving teachers recognition of ADHD and ODD relevant to ADHD recognize that the behavior of chil- symptomatology. This is an important issue because dren with ODD is problematic but overgeneralize all be- teachers are often the earliest and most frequent source havior disorders to the one they "know bestY-ADHD. for identifying children with behavior problems. Two Many h v e implied that the extensive coverage of important steps should be taken. First, a ADHD by the media and other sources has led to an psychometrically sound measure of teachers knowl-Downloaded by [188.8.131.52] at 00:41 07 February 2013 overdiagnosis of the disorder (Hancock, 1996). Thus, edge of A O H D includes questions relevant to the that childrenwith diverseproblems (e.g., learningdisorders, discrimination of different childhood disord~rs should aggression) may oftenbe incorrectly labeled ,as ADHD. be developed. Second, through college and continuing Sirnildy, more contact with experts on ADHD and education courses teachers should be trained specifi- children with this disorder was not related to the accu- cally to recognize the symptoms of ADHD, ODD, and racy of teachers ratings of the ODD tape. It is possible other psychol~ogicaldisorders (Abikoff et al., 1993). that exposureto professionals and parents does not lead Other factors besides knowledge and experience to bemr differentiation of specific childhood disorders. should also be considered, such as teachers personali- For example, contact with children with ADHD and ties and attitudes toward these students. For example, their caretakers may sengitizeteachers to the key symp- teachers preference for structured versus unstructured toms of the disorder,but it may also result in greater ac- classrooms may influencetheir ratings of chiildren with ceptance of childrens behavior problems, and hence, hyperactivity (Eddowes, Aldridge, & (Julpepper, less extreme ratings. In addition, it is widely acknowl- 1994). Given the national move toward the edged that health care professionals often hold different mainstreaming of children with a variety crf disabili- views with regard to diagnosis and appropriate treat- ties, it is vitally important that teachers have the ment. Therefore, contact with these professionals may knowledge and experience relevant to facing these not result in better-informed teachers. new challenges. Two limitations of this study are worth noting. First, it is not clear how generalizable findings based on rat- ings of videotaped segments are to the more typical References classroom setting.Teachers can rarely focus on one stu- dent for extended period of time, as they did when Abikoff, H., Courtney, M., Pelham, W. E., & Kopl~ewicz,H. S. viewing these videotapes. Moreover, teachers typically (1993). Teachers rating of disruptive behaviors: The influence evaluate their students in a variety of situations, from of halo effects. Journal of Abnormal Child Psychology, 21, structured academic activities to less organized free 519-533. Abikoff, H., Gittelman-Klein, R., L Klein, D. F.(1977). Validation play. Examining teachers ratings with children in ac- of a classroom observation code for hyperactive children. Jour- tual settings, as their behavior varies across situations, nal of Consulting and Clinical Psychology, 4,772-783. would increasethe generalizability of our findings.No- Atkins,M. S., Pelham,W. E., &Licht,M. H. (1985). Acomparisonof tably, however, our results convergewith those investi- objective classroom measures and teacher ratings;of attention gators who found a unidirectional bias in a classroom deficit disorder. Journal of Abnormal Child Psychology, 13, 155-167. setting (Schachar et al., 1986). Second,in this study we Atkins, M. S., Pelham, W. E., & Licht, M. H. (1989). The differential did not investigate teachers knowledge of ODD, an- validity of teacher ratings of inattentiontovemtivity and ag- other factor that may have influenced their ratings. gression. JoumalofAbnormal ChildPsychology, 17, 423--435. Greater knowledge of ODD could have increased the Bender, W. N., Vail, C. O., &Scott, I<. (1995). Teachersattitudesto- accuracy of their ratings of children displaying only ward increased mainstreaming:Implementingeffective instruc- tion for students with learning disabilities. Journar of Learning oppositioinal behaviors. Disabilities, 28, 87-94. Two important implications may be drawn from Bennett, D. S., Power, T. J., Rostain, A. L., &Cam,D. E. (1996). Par- this investigation. First, clinicians and researchers ent aciceptability and feasibility of ADHD interventions: As-
STEVENS, QUITTNER, & ABIKOFF sessment, correlates, and predictive validity. Journal of Pediat- Milich, R., & Fitzgerald, G. (1985).Validation of inattentionloverac- ric Psychology, 21,643-657. tivity and aggression ratings with classroom observations. Jour- Brown, R. T. (1985). The validity of teacherratings in differentiating nal of Clinical and Consulting Psychology, 53, 139-140. between two subgroups of attention deficit disordered children Mintz, L. I, & Collins, B. (1985).Qualitative influences on the per- . with or without hyperactivity. Educational and Psychological ception of movement: An experimental study. Journal of Ab- Measurement, 45,661-669. normal Child Psychology, 13, 143-153. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, Sandoval, J. (1981). Format effects in two teacher rating scales of hy- 155-159. peractivity. Journal ofAbnorma1Child Psychology, 9,203-21 8. Conners, C. K. (1969). A teacher rating scale for use in drug studies Schachar, R., Sandberg, S., &Rutter, M. (1986).Agreement between with children. American Journal of Psychiatry, 126, 884-888. teachers ratings and observations of hyperactivity, inattentive- Conners, C. K. (1990). Conners Rating Scalesmanual. Toronto, On- ness, and defiance. Journal ofAbnorma1 Child Psychology, 14, tario, Canada: Multi-Health Systems, Inc. 331-345. Comers, C. K. (1997). Conners Ratings Scales-Revised North Shelton, T. L., & Barkley, R. A. (1995). The assessment and treat- Tonawanda, NY:Multi-health Systems, h c . ment of attention-deficit/hyperactivitydisorder in children. In Dalley, M. B., Bolocofsky, D. N., & Karlin, N. J. (1994). M. C. Roberts (Ed.), Handbook of pediatric psychology (pp. Teacher-ratings and self-ratings of social competencyin adoles- 633-654). New York: Guilford. cents with low- and high-depressive symptoms. Journal of Ab- Stainback, W., & Stainback, S. (1995). Contemplating inclusiveedn- normal Child Psychology, 22,477-485. cation from a historical perspective. In R. Villa & J. Thousand DuPual, G. J.,Guevremont, D. C., & Barkley, R. A. (1991). Attention (Eds.), Creating an inclusive school (chap. 2). Alexandria, VA: deficit-hyperactivity disorder in adolescence: Critical assess- Association for Supervision and Curriculum Development. ment parameters. Clinical Psychology Review, 11, 231-245. Stevens, G. (1980). Bias in attributivns of positive and negative be- Eddowes, E. A., Aldridge, J., & Culpepper, S. (1994). Primary teach- havior in children by school psychologists, parents, and teach- ers classroom practices and their perceptions of childrens at- ers. Perceptual and Motor Skills, 50, 1283-1290.Downloaded by [184.108.40.206] at 00:41 07 February 2013 tention problems. Perceptual and Motor Skills, 79, 787-790. Ullmann, R. K., Sleator, E. K., & Sprague, R. L. (1985). A change in Ellen, Arthur S. (1989). Discriminant validity of teacher ratings for mind: The Conners Abbreviated Rating Scales reconsidered. n d , learningdisabled, and emotionally-handicapped boys. Journal of Abnormal Child Psychology, 13,553-565. Journal qfScho01 Psychology, 27,15-25. Vaughn, M. L., Riccio, C. A., Hynd, G. W., &Hall, J. (1997).Diag- Goyette, C. H., Conners, C. I. & Ulrich, R. F. (1978). Normative S, nosing ADHD (predominately Inattentiveand Combined Type data on the Revised Conners Parent and Teacher Rating Scales. subtypes): Discriminant validity of the Behavior Assessment Journal ofAbnorma1 Child Psychology, 6,221-236. System for Children and the Achenbach Parent and Teacher Greene, R. W. (1995). Students with ADHD in school classrooms: Rating Scales. Journal of Clinical Child Psychology, 26, Teacher factors related to compatibility, assessment, and inter- 349-357. vention. School Psychology Review, 24, 81-93. Westwood, P. (1997). Commonsense methodsfor children with spe- Hancock, L. (1996, March 18). Mothers Little helper. Newsweek, cial needs. New York: Routledge. I2?,51-56. Wilson, C. C. (1980). Behavioral assessment: Questionnaires. Hanushek, E. A., &Jackson, J. E. (1977).Statistical methodsfor so- School Psychology Review, 9, 58-66. cial scientists. New York: Academic. Loney, J., & Milich, R. (1982). Hyperactivity, inattention, and ag- gression in clinical practice. In M. Wolraich & D. K. Routh (Eds.), Advances in developmental and behavioral pediatrics (Vol. 12, pp. 221-251). New York: Plenum. Mann, E. M., Ikeda,Y., Mueller, C.W., Takahashi, A., Tao, K. T., Hnmris, E., Li, B. L., &Chin, D. (1992). Cross-cultural differ- ences in rating hyperactive-disruptive behaviors in children. Manuscript received November 11,1997 American Journal of Psychiatry, 149, 153S1542. Final revision received April 13, 1998