Handbook of psychology_vol_2_-_research_methods_in_psychology

HANDBOOK of PSYCHOLOGY VOLUME 2RESEARCH METHODS IN PSYCHOLOGY John A. Schinka Wayne F. Velicer Volume Editors Irving B. Weiner Editor-in-Chief John Wiley & Sons, Inc.

HANDBOOK ofPSYCHOLOGY

HANDBOOK of PSYCHOLOGY VOLUME 2RESEARCH METHODS IN PSYCHOLOGY John A. Schinka Wayne F. Velicer Volume Editors Irving B. Weiner Editor-in-Chief John Wiley & Sons, Inc.

➇This book is printed on acid-free paper.Copyright © 2003 by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights reserved.Published simultaneously in Canada.No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic,mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 UnitedStates Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of theappropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400,fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to thePermissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008,e-mail: permcoordinator@wiley.com.Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, theymake no representations or warranties with respect to the accuracy or completeness of the contents of this book and specificallydisclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended bysales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation.You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit orany other commercial damages, including but not limited to special, incidental, consequential, or other damages.This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is soldwith the understanding that the publisher is not engaged in rendering professional services. If legal, accounting, medical,psychological or any other expert assistance is required, the services of a competent professional person should be sought.Designations used by companies to distinguish their products are often claimed as trademarks. In all instances where John Wiley &Sons, Inc. is aware of a claim, the product names appear in initial capital or all capital letters. Readers, however, should contact theappropriate companies for more complete information regarding trademarks and registration.For general information on our other products and services please contact our Customer Care Department within the U.S. at(800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available inelectronic books.Library of Congress Cataloging-in-Publication Data:Handbook of psychology / Irving B. Weiner, editor-in-chief. p. cm. Includes bibliographical references and indexes. Contents: v. 1. History of psychology / edited by Donald K. Freedheim — v. 2. Research methods in psychology / edited by John A. Schinka, Wayne F. Velicer — v. 3. Biological psychology / edited by Michela Gallagher, Randy J. Nelson — v. 4. Experimental psychology / edited by Alice F. Healy, Robert W. Proctor — v. 5. Personality and social psychology / edited by Theodore Millon, Melvin J. Lerner — v. 6. Developmental psychology / edited by Richard M. Lerner, M. Ann Easterbrooks, Jayanthi Mistry — v. 7. Educational psychology / edited by William M. Reynolds, Gloria E. Miller — v. 8. Clinical psychology / edited by George Stricker, Thomas A. Widiger — v. 9. Health psychology / edited by Arthur M. Nezu, Christine Maguth Nezu, Pamela A. Geller — v. 10. Assessment psychology / edited by John R. Graham, Jack A. Naglieri — v. 11. Forensic psychology / edited by Alan M. Goldstein — v. 12. Industrial and organizational psychology / edited by Walter C. Borman, Daniel R. Ilgen, Richard J. Klimoski. ISBN 0-471-17669-9 (set) — ISBN 0-471-38320-1 (cloth : alk. paper : v. 1) — ISBN 0-471-38513-1 (cloth : alk. paper : v. 2) — ISBN 0-471-38403-8 (cloth : alk. paper : v. 3) — ISBN 0-471-39262-6 (cloth : alk. paper : v. 4) — ISBN 0-471-38404-6 (cloth : alk. paper : v. 5) — ISBN 0-471-38405-4 (cloth : alk. paper : v. 6) — ISBN 0-471-38406-2 (cloth : alk. paper : v. 7) — ISBN 0-471-39263-4 (cloth : alk. paper : v. 8) — ISBN 0-471-38514-X (cloth : alk. paper : v. 9) — ISBN 0-471-38407-0 (cloth : alk. paper : v. 10) — ISBN 0-471-38321-X (cloth : alk. paper : v. 11) — ISBN 0-471-38408-9 (cloth : alk. paper : v. 12) 1. Psychology. I. Weiner, Irving B. BF121.H1955 2003 150—dc21 2002066380Printed in the United States of America.10 9 8 7 6 5 4 3 2 1

Editorial BoardVolume 1 Volume 5 Volume 9History of Psychology Personality and Social Psychology Health PsychologyDonald K. Freedheim, PhD Theodore Millon, PhD Arthur M. Nezu, PhDCase Western Reserve University Institute for Advanced Studies in Christine Maguth Nezu, PhDCleveland, Ohio Personology and Psychopathology Pamela A. Geller, PhD Coral Gables, Florida Drexel University Melvin J. Lerner, PhD Philadelphia, PennsylvaniaVolume 2 Florida Atlantic UniversityResearch Methods in Psychology Boca Raton, Florida Volume 10 Assessment PsychologyJohn A. Schinka, PhDUniversity of South Florida Volume 6 John R. Graham, PhDTampa, Florida Developmental Psychology Kent State University Richard M. Lerner, PhD Kent, OhioWayne F. Velicer, PhDUniversity of Rhode Island M. Ann Easterbrooks, PhD Jack A. Naglieri, PhDKingston, Rhode Island Jayanthi Mistry, PhD George Mason University Tufts University Fairfax, Virginia Medford, Massachusetts Volume 11Volume 3 Forensic PsychologyBiological Psychology Volume 7 Educational Psychology Alan M. Goldstein, PhDMichela Gallagher, PhD John Jay College of CriminalJohns Hopkins University William M. Reynolds, PhD Justice–CUNYBaltimore, Maryland Humboldt State University New York, New YorkRandy J. Nelson, PhD Arcata, CaliforniaOhio State University Gloria E. Miller, PhD Volume 12Columbus, Ohio University of Denver Industrial and Organizational Denver, Colorado Psychology Walter C. Borman, PhDVolume 4 Volume 8 University of South FloridaExperimental Psychology Clinical Psychology Tampa, FloridaAlice F. Healy, PhD George Stricker, PhD Daniel R. Ilgen, PhDUniversity of Colorado Adelphi University Michigan State UniversityBoulder, Colorado Garden City, New York East Lansing, MichiganRobert W. Proctor, PhD Thomas A. Widiger, PhD Richard J. Klimoski, PhDPurdue University University of Kentucky George Mason UniversityWest Lafayette, Indiana Lexington, Kentucky Fairfax, Virginia v

My efforts in this work are proudly dedicated to Katherine, Christy, and John C. Schinka. J. A. S. This work is dedicated to Sue, the perfect companion for life’s many journeys and the center of my personal universe. W. F. V.

Handbook of Psychology PrefacePsychology at the beginning of the twenty-first century has A second unifying thread in psychology is a commitmentbecome a highly diverse field of scientific study and applied to the development and utilization of research methodstechnology. Psychologists commonly regard their discipline suitable for collecting and analyzing behavioral data. Withas the science of behavior, and the American Psychological attention both to specific procedures and their applicationAssociation has formally designated 2000 to 2010 as the in particular settings, Volume 2 addresses research methods“Decade of Behavior.” The pursuits of behavioral scientists in psychology.range from the natural sciences to the social sciences and em- Volumes 3 through 7 of the Handbook present the sub-brace a wide variety of objects of investigation. Some psy- stantive content of psychological knowledge in five broadchologists have more in common with biologists than with areas of study: biological psychology (Volume 3), experi-most other psychologists, and some have more in common mental psychology (Volume 4), personality and social psy-with sociologists than with most of their psychological col- chology (Volume 5), developmental psychology (Volume 6),leagues. Some psychologists are interested primarily in the be- and educational psychology (Volume 7). Volumes 8 throughhavior of animals, some in the behavior of people, and others 12 address the application of psychological knowledge inin the behavior of organizations. These and other dimensions five broad areas of professional practice: clinical psychologyof difference among psychological scientists are matched by (Volume 8), health psychology (Volume 9), assessment psy-equal if not greater heterogeneity among psychological practi- chology (Volume 10), forensic psychology (Volume 11), andtioners, who currently apply a vast array of methods in many industrial and organizational psychology (Volume 12). Eachdifferent settings to achieve highly varied purposes. of these volumes reviews what is currently known in these Psychology has been rich in comprehensive encyclope- areas of study and application and identifies pertinent sourcesdias and in handbooks devoted to specific topics in the field. of information in the literature. Each discusses unresolved is-However, there has not previously been any single handbook sues and unanswered questions and proposes future direc-designed to cover the broad scope of psychological science tions in conceptualization, research, and practice. Each of theand practice. The present 12-volume Handbook of Psychol- volumes also reflects the investment of scientific psycholo-ogy was conceived to occupy this place in the literature. gists in practical applications of their findings and the atten-Leading national and international scholars and practitioners tion of applied psychologists to the scientific basis of theirhave collaborated to produce 297 authoritative and detailed methods.chapters covering all fundamental facets of the discipline, The Handbook of Psychology was prepared for the pur-and the Handbook has been organized to capture the breadth pose of educating and informing readers about the presentand diversity of psychology and to encompass interests and state of psychological knowledge and about anticipated ad-concerns shared by psychologists in all branches of the field. vances in behavioral science research and practice. With this Two unifying threads run through the science of behavior. purpose in mind, the individual Handbook volumes addressThe first is a common history rooted in conceptual and em- the needs and interests of three groups. First, for graduate stu-pirical approaches to understanding the nature of behavior. dents in behavioral science, the volumes provide advancedThe specific histories of all specialty areas in psychology instruction in the basic concepts and methods that define thetrace their origins to the formulations of the classical philoso- fields they cover, together with a review of current knowl-phers and the methodology of the early experimentalists, and edge, core literature, and likely future developments. Second,appreciation for the historical evolution of psychology in all in addition to serving as graduate textbooks, the volumesof its variations transcends individual identities as being one offer professional psychologists an opportunity to read andkind of psychologist or another. Accordingly, Volume 1 in contemplate the views of distinguished colleagues concern-the Handbook is devoted to the history of psychology as ing the central thrusts of research and leading edges of prac-it emerged in many areas of scientific study and applied tice in their respective fields. Third, for psychologists seekingtechnology. to become conversant with fields outside their own specialty ix

x Handbook of Psychology Prefaceand for persons outside of psychology seeking informa- valuable contributions to the literature. I would like finally totion about psychological matters, the Handbook volumes express my appreciation to the editorial staff of John Wileyserve as a reference source for expanding their knowledge and Sons for the opportunity to share in the development ofand directing them to additional sources in the literature. this project and its pursuit to fruition, most particularly to The preparation of this Handbook was made possible by Jennifer Simon, Senior Editor, and her two assistants, Marythe diligence and scholarly sophistication of the 25 volume Porterfield and Isabel Pratt. Without Jennifer’s vision of theeditors and co-editors who constituted the Editorial Board. Handbook and her keen judgment and unflagging support inAs Editor-in-Chief, I want to thank each of them for the plea- producing it, the occasion to write this preface would notsure of their collaboration in this project. I compliment them have arrived.for having recruited an outstanding cast of contributors totheir volumes and then working closely with these authors to IRVING B. WEINERachieve chapters that will stand each in their own right as Tampa, Florida

Volume PrefaceA scientific discipline is defined in many ways by the re- computers to perform complex methods of data analysis, in-search methods it employs. These methods can be said to rep- creased computer capacity allowing for more intense analysisresent the common language of the discipline’s researchers. of larger datasets, computer simulations that permit the eval-Consistent with the evolution of a lexicon, new research uation of procedures across a wide variety of situations, newmethods frequently arise from the development of new approaches to data analysis and statistical control, and ad-content areas. By every available measure—number of re- vances in companion sciences that opened pathways to thesearchers, number of publications, number of journals, num- exploration of behavior and created new areas of researchber of new subdisciplines—psychology has undergone a specialization and collaboration.tremendous growth over the last half-century. This growth is Consider the advances since the publication of thereflected in a parallel increase in the number of new research first edition of Kirk’s (1968) text on experimental design.methods available. At that time most studies were relatively small N experiments As we were planning and editing this volume, we dis- that were conducted in psychology laboratories. Research ac-cussed on many occasions the extent to which psychology tivity has subsequently exploded in applied and clinicaland the available research methods have become increasing areas, with a proliferation of new journals largely dedicatedcomplex over the course of our careers. When our generation to quasi-experimental studies and studies in the natural envi-of researchers began their careers in the late 1960s and early ronment (e.g., in neuropsychology and health psychology).1970s, experimental design was largely limited to simple Techniques such as polymerase chain reaction allow psychol-between-group designs, and data analysis was dominated by ogists to test specific genes as risk candidates for behaviorala single method, the analysis of variance. A few other ap- disorders. These studies rely on statistical procedures that areproaches were employed, but by a limited number of re- still largely ignored by many researchers (e.g., logistic re-searchers. Multivariate statistics had been developed, but gression, structural equation modeling). Brain imagingmultiple regression analysis was the only method that was procedures such as magnetic resonance imaging, magnetoen-applied with any frequency. Factor analysis was used almost cephalography, and positron-emission tomography provideexclusively as a method in scale development. Classical test cognitive psychologists and neuropsychologists the opportu-theory was the basis of most psychological and educational nity to study cortical activity on-line. Clinical trials involvingmeasures. Analysis of data from studies that did not meet behavioral interventions applied to large, representative sam-either the design or measurement assumptions required for an ples are commonplace in health psychology. Research em-analysis of variance was covered for most researchers by a ploying each of these procedures requires not only highlysingle book on nonparametric statistics by Siegel (1956). As specific and rigorous research methods, but also speciala review of the contents of this volume illustrates, the choice methods for handling and analyzing extremely large volumesof experimental and analytic methods available to the of data. Even in more traditional areas of research that con-present-day researcher is much broader. It would be fair to tinue to rely on group experimental designs, issues of mea-say that the researcher in the 1960s had to formulate research suring practical significance, determination of sample sizequestions to fit the available methods. Currently, there are re- and power, and procedures for handling nuisance variablessearch methods available to address most research questions. are now important concerns. Not surprisingly, the third edi- In the history of science, an explosion of knowledge is tion of Kirk’s (1995) text has grown in page length by 60%.usually the result of an advance in technology, new theoreti- Our review of these trends leads to several conclusions,cal models, or unexpected empirical findings. Advances in which are reflected in the selection of topics covered by theresearch methods have occurred as the result of all three fac- chapters in this volume. Six features appear to characterizetors, typically in an interactive manner. Some of the specific the evolution in research methodology in psychology.factors include advances in instrumentation and measure- First, there has been a focus on the development of proce-ment technology, the availability of inexpensive desktop dures that employ statistical control rather than experimental xi

xii Volume Prefacecontrol. Because most of the recent growth involves research to focus on the individual and model individual differences.in areas that preclude direct control of independent variables, This becomes increasingly important as we recognize that in-multivariate statistics and the development of methods such terventions do not affect everyone in exactly the same waysas path analysis and structural equation modeling have been and that interventions become more and more tailored to thecritical developments. The use of statistical control has al- individual.lowed psychology to move from the carefully controlled con- The text is organized into four parts. The first part, titledfines of the laboratory to the natural environment. “Foundations of Research,” addresses issues that are funda- Second, there has been an increasing focus on construct- mental to all behavioral science research. The focus is ondriven, or latent-variable, research. A construct is defined by study design, data management, data reduction, and data syn-multiple observed variables. Constructs can be viewed as thesis. The first chapter, “Experimental Design” by Roger E.more reliable and more generalizable than a single observed Kirk, provides an overview of the basic considerations thatvariable. Constructs serve to organize a large set of observed go into the design of a study. Once, a chapter on this topicvariables, resulting in parsimony. Constructs are also theoret- would have had to devote a great deal of attention to compu-ically based. This theory-based approach serves to guide tational procedures. The availability of computers permits astudy design, the choice of variables, the data analysis, and shift in focus to the conceptual rather than the computationalthe data interpretation. issues. The second chapter, “Exploratory Data Analysis” by Third, there has been an increasing emphasis on the de- John T. Behrens and Chong-ho Yu, reminds us of the funda-velopment of new measures and new measurement models. mental importance of looking at data in the most basic waysThis is not a new trend but an acceleration of an old trend. as a first step in any data analysis. In some ways this repre-The behavioral sciences have always placed the most empha- sents a “back to the future” chapter. Advances in computer-sis on the issue of measurement. With the movement of the based graphical methods have brought a great deal of sophis-field out of the laboratory combined with advances in tech- tication to this very basic first step.nology, the repertoire of measures, the quality of the mea- The third chapter, “Power: Basics, Practical Problems,sures, and the sophistication of the measurement models have and Possible Solutions” by Rand R. Wilcox, reflects the crit-all increased dramatically. ical change in focus for psychological research. Originally, Fourth, there is increasing recognition of the importance of the central focus of a test of significance was on controllingthe temporal dimension in understanding a broad range of psy- Type I error rates. The late Jacob Cohen emphasized that re-chological phenomena. We have become a more intervention- searchers should be equally concerned by Type II errors.oriented science, recognizing not only the complexity of This resulted in an emphasis on the careful planning of atreatment effects but also the importance of the change in pat- study and a concern with effect size and selecting the appro-terns of the effects over time. The effects of an intervention priate sample size. Wilcox updates and extends these con-may be very different at different points in time. New statisti- cepts. Chapter 4, “Methods for Handling Missing Data” bycal models for modeling temporal data have resulted. John W. Graham, Patricio E. Cumsille, and Elvira Elek-Fisk, Fifth, new methods of analysis have been developed describes the impressive statistical advances in addressingthat no longer require the assumption of a continuous, equal- the common practical problem of missing observations.interval, normally distributed variable. Previously, re- Previously, researchers had relied on a series of ad hoc pro-searchers had the choice between very simple but limited cedures, often resulting in very inaccurate estimates. The newmethods of data analysis that corresponded to the properties statistical procedures allow the researcher to articulate theof the measure or more complex sophisticated methods of assumptions about the reason the data is missing and makeanalysis that assumed, often inappropriately, that the measure very sophisticated estimates of the missing value based on allmet very rigid assumptions. New methods have been devel- the available information. This topic has taken on even moreoped for categorical, ordinal, or simply nonnormal variables importance with the increasing emphasis on longitudinalthat can perform an equally sophisticated analysis. studies and the inevitable problem of attrition. Sixth, the importance of individual differences is increas- The fifth chapter, “Preparatory Data Analysis” by Linda S.ingly emphasized in intervention studies. Psychology has Fidell and Barbara G. Tabachnick, describes methods of pre-always been interested in individual differences, but meth- processing data before the application of other methods ofods of data analysis have focused almost entirely on the rela- statistical analysis. Extreme values can distort the results oftionships between variables. Individuals were studied as the data analysis if not addressed. Diagnostic methods canmembers of groups, and individual differences served only to preprocess the data so that complex procedures are not un-inflate the error variance. New techniques permit researchers duly affected by a limited number of cases that often are the

Volume Preface xiiiresult of some type of error. The last two chapters in this part, findings that are of interest to psychologists in many fields.“Factor Analysis” by Richard L. Gorsuch and “Clustering The major goal of Chapter 11, “Animal Learning” by Russelland Classification Methods” by Glenn W. Milligan and M. Church, is to transfer what is fairly common knowledge inStephen C. Hirtle, describe two widely employed parsimony experimental animal psychology to investigators with limitedmethods. Factor analysis operates in the variable domain and exposure to this area of research. In Chapter 12, “Neuropsy-attempts to reduce a set of p observed variables to a smaller chology,” Russell M. Bauer, Elizabeth C. Leritz, and Dawnset of m factors. These factors, or latent variables, are more Bowers provide a discussion of neuropsychological inference,easily interpreted and thus facilitate interpretation. Cluster an overview of major approaches to neuropsychological re-analysis operates in the person domain and attempts to reduce search, and a review of newer techniques, including functionala set of N individuals to a set of k clusters. Cluster analysis neuroimaging, electrophysiology, magnetoencephalography,serves to explore the relationships among individuals and or- and reversible lesion methods. In each section, they describeganize the set of individuals into a limited number of sub- the conceptual basis of the technique, outline its strengths andtypes that share essential features. These methods are basic to weaknesses, and cite examples of how it has been used inthe development of construct-driven methods and the focus addressing conceptual problems in neuropsychology.on individual differences. Whatever their specialty area, when psychologists evalu- The second part, “Research Methods in Specific Content ate a program or policy, the question of impact is often at cen-Areas,” addresses research methods and issues as they apply ter stage. The last chapter in this part, “Program Evaluation”to specific content areas. Content areas were chosen in part to by Melvin M. Mark, focuses on key methods for estimatingparallel the other volumes of the Handbook. More important, the effects of policies and programs in the context of evalua-however, we attempted to sample content areas from a broad tion. Additionally, Mark addresses several noncausal formsspectrum of specialization with the hope that these chapters of program evaluation research that are infrequently ad-would provide insights into methodological concerns and dressed in methodological treatises.solutions that would generalize to other areas. Chapter 8, The third part is titled “Measurement Issues.” Advances in“Clinical Forensic Psychology” by Kevin S. Douglas, Randy measurement typically combine innovation in technologyK. Otto, and Randy Borum, addresses research methods and and progress in theory. As our measures become more so-issues that occur in assessment and treatment contexts. For phisticated, the areas of application also increase.each task that is unique to clinical forensic psychology Mood emerged as a seminal concept within psychologyresearch, they provide examples of the clinical challenges during the 1980s, and its prominence has continued unabatedconfronting the psychologist, identify problems faced when ever since. In Chapter 14, “Mood Measurement: Currentresearching the issues or constructs, and describe not only re- Status and Future Directions,” David Watson and Jatin Vaidyasearch strategies that have been employed but also their examine current research regarding the underlying structurestrengths and limitations. In Chapter 9, “Psychotherapy Out- of mood, describe and evaluate many of the most importantcome Research,” Evelyn S. Behar and Thomas D. Borkovec mood measures, and discuss several issues related to theaddress the methodological issues that need to be considered reliability and construct validity of mood measurement. Infor investigators to draw the strongest and most specific Chapter 15, “Measuring Personality and Psychopathology,”cause-and-effect conclusions about the active components of Leslie C. Morey uses objective self-report methods of mea-treatments, human behavior, and the effectiveness of thera- surement to illustrate contemporary procedures for scalepeutic interventions. development and validation, addressing issues critical to all The field of health psychology is largely defined by three measurement methods such as theoretical articulation, situa-topics: the role of behavior (e.g., smoking) in the develop- tional context, and the need for discriminant validity.ment and prevention of disease, the role of stress and emotion The appeal of circular models lies in the combination of aas psychobiological influences on disease, and psychological circle’s aesthetic (organizational) simplicity and its powerfulaspects of acute and chronic illness and medical care. Insight potential to describe data in uniquely compelling substantiveinto the methodological issues and solutions for research in and geometric ways, as has been demonstrated in describ-each of these topical areas is provided by Timothy W. Smith ing interpersonal behavior and occupational interests. Inin Chapter 10, “Health Psychology.” Chapter 16, “The Circumplex Model: Methods and Research At one time, most behavioral experimentation was con- Applications,” Michael B. Gurtman and Aaron L. Pincus dis-ducted by individuals whose training focused heavily on ani- cuss the application of the circumplex model to the descrip-mal research. Now many neuroscientists, trained in various tions of individuals, comparisons of groups, and evaluationsfields, conduct research in animal learning and publish of constructs and their measures.

xiv Volume Preface Chapter 17, “Item Response Theory and Measuring Abili- Velicer and Joseph L. Fava describe a method for studyingties” by Karen M. Schmidt and Susan E. Embretson, de- the change in a single individual over time. Instead of a sin-scribes the types of formal models that have been designed to gle observation on many subjects, this method relies on manyguide measure development. For many years, most tests of observations on a single subject. In many ways, this methodability and achievement have relied on classical test theory as is the prime exemplar of longitudinal research methods.a framework to guide both measure development and mea- Chapter 24, “Structural Equation Modeling” by Jodie B.sure evaluation. Item response theory updates this model in Ullman and Peter M. Bentler, describes a very generalmany important ways, permitting the development of a new method that combines three key themes: constructs or latentgeneration of measures of abilities and achievement that are variables, statistical control, and theory to guide data analy-particularly appropriate for a more interactive model of as- sis. First employed as an analytic method little more thansessment. The last chapter of this part, “Growth Curve Analy- 20 years ago, the method is now widely disseminated in thesis in Contemporary Psychological Research” by John J. behavioral sciences. Chapter 25, “Ordinal Analysis of Behav-McArdle and John R. Nesselroade, describes new quantita- ioral Data” by Jeffrey D. Long, Du Feng, and Norman Cliff,tive methods for the study of change in development psy- discusses the assumptions that underlie many of the widelychology. The methods permit the researcher to model a wide used statistical methods and describes a parallel series ofvariety of different patterns of developmental change over methods of analysis that only assume that the measure pro-time. vides ordinal information. The last chapter, “Latent Class and The final part, “Data Analysis Methods,” addresses statis- Latent Transition Analysis” by Stephanie L. Lanza, Brian P.tical procedures that have been developed recently and are Flaherty, and Linda M. Collins, describes a new method forstill not widely employed by many researchers. They are typ- analyzing change over time. It is particularly appropriateically dependent on the availability of high-speed computers when the change process can be conceptualized as a series ofand permit researchers to investigate novel and complex re- discrete states.search questions. Chapter 19, “Multiple Linear Regression” In completing this project, we realized that we were veryby Leona Aiken, Stephen G. West, and Steven C. Pitts, de- fortunate in several ways. Irving Weiner’s performance asscribes the advances in multiple linear regression that permit editor-in-chief was simply wonderful. He applied just the rightapplications of this very basic method to the analysis of com- mix of obsessive concern and responsive support to keep thingsplex data sets and the incorporation of conceptual models to on schedule. His comments on issues of emphasis, perspective,guide the analysis. The testing of theoretical predictions and and quality were insightful and inevitably on target.the identification of implementation problems are the two We continue to be impressed with the professionalism ofmajor foci of this chapter. Chapter 20, “Logistic Regression” the authors that we were able to recruit into this effort.by Alfred DeMaris, describes a parallel method to multiple Consistent with their reputations, these individuals deliv-regression analysis for categorical variables. The procedure ered chapters of exceptional quality, making our burdenhas been developed primarily outside of psychology and is pale in comparison to other editorial experiences. Because ofnow being used much more frequently to address psycholog- the length of the project, we shared many contributors’ical questions. Chapter 21, “Meta-Analysis” by Frank L. experiences-marriages, births, illnesses, family crises. A def-Schmidt and John E. Hunter, describes procedures that have inite plus for us has been the formation of new friendshipsbeen developed for the quantitative integration of research and professional liaisons.findings across multiple studies. Previously, research findings Our editorial tasks were also aided greatly by the generouswere integrated in narrative form and were subject to the bi- assistance of our reviewers, most of whom will be quicklyases of the reviewer. The method also focuses attention on the recognized by our readers for their own expertise in researchimportance of effect size estimation. methodology. We are pleased to thank James Algina, Phipps Chapter 22, “Survival Analysis” by Judith D. Singer and Arabie, Patti Barrows, Betsy Jane Becker, Lisa M. Brown,John B. Willett, describes a recently developed method for Barbara M. Byrne, William F. Chaplin, Pat Cohen, Patrick J.analyzing longitudinal data. One approach is to code whether Curren, Glenn Curtiss, Richard B. Darlington, Susanan event has occurred at a given occasion. By switching the Duncan, Brian Everitt, Kerry Evers, Ron Gironda, Lisafocus on the time to the occurrence of the event, a much more Harlow, Michael R. Harwell, Don Hedeker, David Charlespowerful and sophisticated analysis can be performed. Again, Howell, Lawrence J. Hubert, Bradley E. Huitema, Beththe development of this procedure has occurred largely out- Jenkins, Herbert W. Marsh, Rosemarie A. Martin, Scott E.side psychology but is being employed much more fre- Maxwell, Kevin R. Murphy, Gregory Norman, Daniel J.quently. In Chapter 23, “Time Series Analysis,” Wayne Ozer, Melanie Page, Mark D. Reckase, Charles S. Reichardt,

Volume Preface xvSteven Reise, Joseph L. Rogers, Joseph Rossi, James or supported by differing approaches. For flaws in the text,Rounds, Shlomo S. Sawilowsky, Ian Spence, James H. however, the usual rule applies: We assume all responsibility.Steiger, Xiaowu Sun, Randall C. Swaim, David Thissen,Bruce Thompson, Terence J. G. Tracey, Rod Vanderploeg, JOHN A. SCHINKAPaul F. Velleman, Howard Wainer, Douglas Williams, and WAYNE F. VELICERseveral anonymous reviewers for their thorough work andgood counsel. We finish this preface with a caveat. Readers will in- REFERENCESevitably discover several contradictions or disagreementsacross the chapter offerings. Inevitably, researchers in differ- Kirk, Roger E. (1968). Experimental design: Procedures for theent areas solve similar methodological problems in different behavioral sciences. Pacific Grove, CA: Brooks/Cole.ways. These differences are reflected in the offerings of this Kirk, Roger E. (1995). Experimental design: Procedures for thetext, and we have not attempted to mediate these differing behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole.viewpoints. Rather, we believe that the serious researcher Siegel, S. (1956). Nonparametric statistics for the behavioralwill welcome the opportunity to review solutions suggested sciences. New York: McGraw-Hill.

ContentsHandbook of Psychology Preface ix Irving B. WeinerVolume Preface xi John A. Schinka and Wayne F. VelicerContributors xxi PA RT O N E FOUNDATIONS OF RESEARCH ISSUES: STUDY DESIGN, DATA MANAGEMENT, DATA REDUCTION, AND DATA SYNTHESIS1 EXPERIMENTAL DESIGN 3 Roger E. Kirk2 EXPLORATORY DATA ANALYSIS 33 John T. Behrens and Chong-ho Yu3 POWER: BASICS, PRACTICAL PROBLEMS, AND POSSIBLE SOLUTIONS 65 Rand R. Wilcox4 METHODS FOR HANDLING MISSING DATA 87 John W. Graham, Patricio E. Cumsille, and Elvira Elek-Fisk5 PREPARATORY DATA ANALYSIS 115 Linda S. Fidell and Barbara G. Tabachnick6 FACTOR ANALYSIS 143 Richard L. Gorsuch7 CLUSTERING AND CLASSIFICATION METHODS 165 Glenn W. Milligan and Stephen C. Hirtle PA RT T W O RESEARCH METHODS IN SPECIFIC CONTENT AREAS8 CLINICAL FORENSIC PSYCHOLOGY 189 Kevin S. Douglas, Randy K. Otto, and Randy Borum9 PSYCHOTHERAPY OUTCOME RESEARCH 213 Evelyn S. Behar and Thomas D. Borkovec xvii

xviii Contents10 HEALTH PSYCHOLOGY 241 Timothy W. Smith11 ANIMAL LEARNING 271 Russell M. Church12 NEUROPSYCHOLOGY 289 Russell M. Bauer, Elizabeth C. Leritz, and Dawn Bowers13 PROGRAM EVALUATION 323 Melvin M. Mark PA RT T H R E E MEASUREMENT ISSUES14 MOOD MEASUREMENT: CURRENT STATUS AND FUTURE DIRECTIONS 351 David Watson and Jatin Vaidya15 MEASURING PERSONALITY AND PSYCHOPATHOLOGY 377 Leslie C. Morey16 THE CIRCUMPLEX MODEL: METHODS AND RESEARCH APPLICATIONS 407 Michael B. Gurtman and Aaron L. Pincus17 ITEM RESPONSE THEORY AND MEASURING ABILITIES 429 Karen M. Schmidt and Susan E. Embretson18 GROWTH CURVE ANALYSIS IN CONTEMPORARY PSYCHOLOGICAL RESEARCH 447 John J. McArdle and John R. Nesselroade PA RT F O U R DATA ANALYSIS METHODS19 MULTIPLE LINEAR REGRESSION 483 Leona S. Aiken, Stephen G. West, and Steven C. Pitts20 LOGISTIC REGRESSION 509 Alfred DeMaris21 META-ANALYSIS 533 Frank L. Schmidt and John E. Hunter22 SURVIVAL ANALYSIS 555 Judith D. Singer and John B. Willett23 TIME SERIES ANALYSIS 581 Wayne F. Velicer and Joseph L. Fava

Contents xix24 STRUCTURAL EQUATION MODELING 607 Jodie B. Ullman and Peter M. Bentler25 ORDINAL ANALYSIS OF BEHAVIORAL DATA 635 Jeffrey D. Long, Du Feng, and Norman Cliff26 LATENT CLASS AND LATENT TRANSITION ANALYSIS 663 Stephanie T. Lanza, Brian P. Flaherty, and Linda M. CollinsAuthor Index 687Subject Index 703

ContributorsLeona S. Aiken, PhD Norman Cliff, PhDDepartment of Psychology Professor of Psychology EmeritusArizona State University University of Southern CaliforniaTempe, Arizona Los Angeles, CaliforniaRussell M. Bauer, PhD Linda M. Collins, PhDDepartment of Clinical and Health Psychology The Methodology CenterUniversity of Florida Pennsylvania State UniversityGainesville, Florida University Park, PennsylvaniaEvelyn S. Behar, MS Patricio E. Cumsille, PhDDepartment of Psychology Escuela de PsicologiaPennsylvania State University Universidad Católica de ChileUniversity Park, Pennsylvania Santiago, ChileJohn T. Behrens, PhD Alfred DeMaris, PhDCisco Networking Academy Program Department of SociologyCisco Systems, Inc. Bowling Green State UniversityPhoenix, Arizona Bowling Green, OhioPeter M. Bentler, PhD Kevin S. Douglas, PhD, LLBDepartment of Psychology Department of Mental Health Law & PolicyUniversity of California Florida Mental Health InstituteLos Angeles, California University of South Florida Tampa, FloridaThomas D. Borkovec, PhDDepartment of Psychology Du Feng, PhDPennsylvania State University Human Development and Family StudiesUniversity Park, Pennsylvania Texas Tech University Lubbock, TexasRandy Borum, PsyDDepartment of Mental Health Law & Policy Elvira Elek-Fisk, PhDFlorida Mental Health Institute The Methodology CenterUniversity of South Florida Pennsylvania State UniversityTampa, Florida University Park, PennsylvaniaDawn Bowers, PhD Susan E. Embretson, PhDDepartment of Clinical and Health Psychology Department of PsychologyUniversity of Florida University of KansasGainesville, Florida Lawrence, KansasRussell M. Church, PhD Joseph L. Fava, PhDDepartment of Psychology Cancer Prevention Research CenterBrown University University of Rhode IslandProvidence, Rhode Island Kingston, Rhode Island xxi

xxii ContributorsLinda S. Fidell, PhD Melvin M. Mark, PhDDepartment of Psychology Department of PsychologyCalifornia State University Pennsylvania State UniversityNorthridge, California University Park, PennsylvaniaBrian P. Flaherty, MS John J. McArdle, PhDThe Methodology Center Department of PsychologyPennsylvania State University University of VirginiaUniversity Park, Pennsylvania Charlottesville, VirginiaRichard L. Gorsuch, PhD Glenn W. Milligan, PhDGraduate School of Psychology Department of Management SciencesFuller Theological Seminary Ohio State UniversityPasadena, California Columbus, OhioJohn W. Graham, PhD Leslie C. Morey, PhDDepartment of Biobehavioral Health Department of PsychologyPennsylvania State University Texas A&M UniversityUniversity Park, Pennsylvania College Station, TexasMichael B. Gurtman, PhD John R. Nesselroade, PhDDepartment of Psychology Department of PsychologyUniversity of Wisconsin-Parkside University of VirginiaKenosha, Wisconsin Charlottesville, VirginiaStephen C. Hirtle, PhD Randy K. Otto, PhDSchool of Information Sciences Department of Mental Health Law & PolicyUniversity of Pittsburgh Florida Mental Health InstitutePittsburgh, Pennsylvania University of South Florida Tampa, FloridaJohn E. Hunter, PhD Aaron L. Pincus, PhDDepartment of Psychology Department of PsychologyMichigan State University Pennsylvania State UniversityEast Lansing, Michigan University Park, PennsylvaniaRoger E. Kirk, PhD Steven C. Pitts, PhDDepartment of Psychology and Neuroscience Department of PsychologyBaylor University University of Maryland, Baltimore CountyWaco, Texas Baltimore, MarylandStephanie T. Lanza, MS Karen M. Schmidt, PhDThe Methodology Center Department of PsychologyPennsylvania State University University of VirginiaUniversity Park, Pennsylvania Charlottesville, VirginiaElizabeth C. Leritz, MS Frank L. Schmidt, PhDDepartment of Clinical and Health Psychology Department of Management and OrganizationUniversity of Florida University of IowaGainesville, Florida Iowa City, IowaJeffrey D. Long, PhD Judith D. Singer, PhDDepartment of Educational Psychology Graduate School of EducationUniversity of Minnesota Harvard UniversityMinneapolis, Minnesota Cambridge, Massachusetts

Contributors xxiiiTimothy W. Smith, PhD David Watson, PhDDepartment of Psychology Department of PsychologyUniversity of Utah University of IowaSalt Lake City, Utah Iowa City, IowaBarbara G. Tabachnick, PhD Stephen G. West, PhDDepartment of Psychology Department of PsychologyCalifornia State University Arizona State UniversityNorthridge, California Tempe, ArizonaJodie B. Ullman, PhD Rand R. Wilcox, PhDDepartment of Psychology Department of PsychologyCalifornia State University University of Southern CaliforniaSan Bernadino, California Los Angeles, CaliforniaJatin Vaidya John B. Willett, PhDDepartment of Psychology Graduate School of EducationUniversity of Iowa Harvard UniversityIowa City, Iowa Cambridge, MassachusettsWayne F. Velicer, PhD Chong-ho Yu, PhDCancer Prevention Research Center Cisco Networking Academy ProgramUniversity of Rhode Island Cisco Systems, Inc.Kingston, Rhode Island Chandler, Arizona

PA R T O N E FOUNDATIONS OF RESEARCH ISSUES: STUDY DESIGN, DATA MANAGEMENT,DATA REDUCTION, AND DATA SYNTHESIS

CHAPTER 1Experimental DesignROGER E. KIRKSOME BASIC EXPERIMENTAL DESIGN CONCEPTS 3 FACTORIAL DESIGNS WITH CONFOUNDING 21THREE BUILDING BLOCK DESIGNS 4 Split-Plot Factorial Design 21 Completely Randomized Design 4 Confounded Factorial Designs 24 Randomized Block Design 6 Fractional Factorial Designs 25 Latin Square Design 9 HIERARCHICAL DESIGNS 27CLASSIFICATION OF EXPERIMENTAL DESIGNS 10 Hierarchical Designs With One orFACTORIAL DESIGNS 11 Two Nested Treatments 27 Completely Randomized Factorial Design 11 Hierarchical Design With Crossed Alternative Models 14 and Nested Treatments 28 Randomized Block Factorial Design 19 EXPERIMENTAL DESIGNS WITH A COVARIATE 29 REFERENCES 31SOME BASIC EXPERIMENTAL about (a) one or more parameters of a population or (b) theDESIGN CONCEPTS functional form of a population. Statistical hypotheses are rarely identical to scientific hypotheses—they areExperimental design is concerned with the skillful interroga- testable formulations of scientific hypotheses.tion of nature. Unfortunately, nature is reluctant to reveal 2. Determination of the experimental conditions (independenther secrets. Joan Fisher Box (1978) observed in her autobiog- variable) to be manipulated, the measurement (dependentraphy of her father, Ronald A. Fisher, “Far from behaving variable) to be recorded, and the extraneous conditionsconsistently, however, Nature appears vacillating, coy, and (nuisance variables) that must be controlled.ambiguous in her answers” (p. 140). Her most effective 3. Specification of the number of participants required andtool for confusing researchers is variability—in particular, the population from which they will be sampled.variability among participants or experimental units. But 4. Specification of the procedure for assigning the partici-two can play the variability game. By comparing the variabil- pants to the experimental conditions.ity among participants treated differently to the variability 5. Determination of the statistical analysis that will beamong participants treated alike, researchers can make in- performed.formed choices between competing hypotheses in scienceand technology. In short, an experimental design identifies the independent, We must never underestimate nature—she is a formidable dependent, and nuisance variables and indicates the way infoe. Carefully designed and executed experiments are re- which the randomization and statistical aspects of an experi-quired to learn her secrets. An experimental design is a plan ment are to be carried out.for assigning participants to experimental conditions and thestatistical analysis associated with the plan (Kirk, 1995, p. 1).The design of an experiment involves a number of inter- Analysis of Variancerelated activities: Analysis of variance (ANOVA) is a useful tool for under-1. Formulation of statistical hypotheses that are germane to the standing the variability in designed experiments. The seminal scientific hypothesis. A statistical hypothesis is a statement ideas for both ANOVA and experimental design can be traced 3

4 Experimental Designto Ronald A. Fisher, a statistician who worked at the Rotham- Fisher popularized two other principles of good experi-sted Experimental Station. According to Box (1978, p. 100), mentation: replication and local control or blocking. Replica-Fisher developed the basic ideas ofANOVAbetween 1919 and tion is the observation of two or more participants under1925. The first hint of what was to come appeared in a 1918 identical experimental conditions. Fisher observed that repli-paper in which Fisher partitioned the total variance of a human cation enables a researcher to estimate error effects andattribute into portions attributed to heredity, environment, and obtain a more precise estimate of treatment effects. Blocking,other factors. The analysis of variance table for a two-treat- on the other hand, is an experimental procedure for isolatingment factorial design appeared in a 1923 paper published with variation attributable to a nuisance variable. As the nameM. A. Mackenzie (Fisher & Mackenzie, 1923). Fisher referred suggests, nuisance variables are undesired sources of varia-to the table as a convenient way of arranging the arithmetic. In tion that can affect the dependent variable. There are many1924 Fisher (1925) introduced the Latin square design in con- sources of nuisance variation. Differences among partici-nection with a forest nursery experiment. The publication in pants comprise one source. Other sources include variation1925 of his classic textbook Statistical Methods for Research in the presentation of instructions to participants, changes inWorkers and a short paper the following year (Fisher, 1926) environmental conditions, and the effects of fatigue andpresented all the essential ideas of analysis of variance. The learning when participants are observed several times. Threetextbook (Fisher, 1925, pp. 244–249) included a table of the experimental approaches are used to deal with nuisancecritical values of the ANOVA test statistic in terms of a func- variables:tion called z, where z = 1 (ln ␴2 2 ˆ Treatment − ln ␴2 ). The statis- ˆ Error 2 2tics ␴Treatment and ␴Error denote, respectively, treatment and ˆ ˆ 1. Holding the variable constant.error variance. A more convenient form of Fisher’s z table that 2. Assigning participants randomly to the treatment levels sodid not require looking up log values was developed by that known and unsuspected sources of variation amongGeorge Snedecor (1934). His critical values are expressed in the participants are distributed over the entire experimentterms of the function F = ␴2 ˆ Treatment /␴2 ˆ Error that is obtained and do not affect just one or a limited number of treatmentdirectly from the ANOVA calculations. He named it F in honor levels.of Fisher. Fisher’s field of experimentation—agriculture— 3. Including the nuisance variable as one of the factors in thewas a fortunate choice because results had immediate applica- experiment.tion with assessable economic value, because simplifyingassumptions such as normality and independence of errors The last experimental approach uses local control or blockingwere usually tenable, and because the cost of conducting to isolate variation attributable to the nuisance variable soexperiments was modest. that it does not appear in estimates of treatment and error effects. A statistical approach also can be used to deal withThree Principles of Good Experimental Design nuisance variables. The approach is called analysis of covariance and is described in the last section of this chapter.The publication of Fisher’s Statistical Methods for Research The three principles that Fisher vigorously championed—Workers and his 1935 The Design of Experiments gradually randomization, replication, and local control—remain theled to the acceptance of what today is considered to be the cornerstones of good experimental design.cornerstone of good experimental design: randomization.It is hard to imagine the hostility that greeted the suggestionthat participants or experimental units should be randomly THREE BUILDING BLOCK DESIGNSassigned to treatment levels. Before Fisher’s work, mostresearchers used systematic schemes, not subject to the laws Completely Randomized Designof chance, to assign participants. According to Fisher, ran-dom assignment has several purposes. It helps to distribute One of the simplest experimental designs is the randomizationthe idiosyncratic characteristics of participants over the treat- and analysis plan that is used with a t statistic for independentment levels so that they do not selectively bias the outcome of samples. Consider an experiment to compare the effectivenessthe experiment. Also, random assignment permits the com- of two diets for obese teenagers. The independent variable isputation of an unbiased estimate of error effects—those the two kinds of diets; the dependent variable is the amount ofeffects not attributable to the manipulation of the independent weight loss two months after going on a diet. For notationalvariable—and it helps to ensure that the error effects are convenience, the two diets are called treatment A. The levelsstatistically independent. of treatment A corresponding to the specific diets are denoted

Three Building Block Designs 5by the lowercase letter a and a subscript: a1 denotes one dietand a2 denotes the other. A particular but unspecified level oftreatment A is denoted by aj, where j ranges over the values 1and 2. The amount of weight loss in pounds 2 months afterparticipant i went on diet j is denoted by Yij. The null and alternative hypotheses for the weight-lossexperiment are, respectively, H0: ␮1 − ␮2 = 0 H1: ␮1 − ␮2 = 0 ,where ␮1 and ␮2 denote the mean weight loss of the respec-tive populations. Assume that 30 girls who want to loseweight are available to participate in the experiment. Theresearcher assigns n = 15 girls to each of the p = 2 diets so Figure 1.2 Layout for a completely randomized design (CR-3 design).that each of the (np)!/(n!) p = 155,117,520 possible assign- Forty-five girls are randomly assigned to three levels of treatment A with thements has the same probability. This is accomplished by restriction that 15 girls are assigned to each level. The mean weight loss innumbering the girls from 1 to 30 and drawing numbers from pounds for the girls in treatment levels a1, a2, and a3 is denoted by Y ·1 , Y ·2 , and Y ·3 , respectively.a random numbers table. The first 15 numbers drawn between1 and 30 are assigned to treatment level a1; the remaining 15numbers are assigned to a2. The layout for this experiment is three diets. The null and alternative hypotheses for theshown in Figure 1.1. The girls who were assigned to treat- experiment are, respectively,ment level a1 are called Group1; those assigned to treatmentlevel a2 are called Group2. The mean weight losses of the two H0: ␮1 = ␮2 = ␮3groups of girls are denoted by Y ·1 and Y ·2 . H1: ␮ j = ␮ j for some j and j . The t independent-samples design involves randomlyassigning participants to two levels of a treatment. A com- Assume that 45 girls who want to lose weight are available topletely randomized design, which is described next, extends participate in the experiment. The girls are randomly as-this design strategy to two or more treatment levels. The com- signed to the three diets with the restriction that 15 girls arepletely randomized design is denoted by the letters CR-p, assigned to each diet. The layout for the experiment is shownwhere CR stands for “completely randomized” and p is the in Figure 1.2. A comparison of the layout in this figure withnumber of levels of the treatment. that in Figure 1.1 for a t independent-samples design reveals Again, consider the weight-loss experiment and suppose that they are the same except that the completely randomizedthat the researcher wants to evaluate the effectiveness of design has three treatment levels. The t independent-samples design can be thought of as a special case of a completely randomized design. When p is equal to two, the layouts and randomization plans for the designs are identical. Thus far I have identified the null hypothesis that the researcher wants to test, ␮1 = ␮2 = ␮3 , and described the manner in which the participants are assigned to the three treatment levels. In the following paragraphs I discuss the composite nature of an observation, describe the classical model equation for a CR-p design, and examine the meaning of the terms treatment effect and error effect. An observation, which is a measure of the dependent variable, can be thought of as a composite that reflects the effects of the (a) independent variable, (b) individual charac-Figure 1.1 Layout for a t independent-samples design. Thirty girls are ran- teristics of the participant or experimental unit, (c) chancedomly assigned to two levels of treatment A with the restriction that 15 girlsare assigned to each level. The mean weight loss in pounds for the girls in fluctuations in the participant’s performance, (d) measure-treatment levels a1 and a2 is denoted by Y ·1 and Y ·2 , respectively. ment and recording errors that occur during data collection,

6 Experimental Designand (e) any other nuisance variables such as environmental level are different because the error effects, ␧i( j)s, for theconditions that have not been controlled. Consider the weight observations are different. Recall that error effects reflect idio-loss of the fifth participant in treatment level a2. Suppose that syncratic characteristics of the participants—those character-two months after beginning the diet this participant has lost istics that differ from one participant to another—and any13 pounds (Y52 = 13). What factors have affected the value of other variables that have not been controlled. Researchers at-Y52? One factor is the effectiveness of the diet. Other factors tempt to minimize the size of error effects by holding sourcesare her weight prior to starting the diet, the degree to which of variation that might contribute to the error effects constantshe stayed on the diet, and the amount she exercised during and by the judicial choice of an experimental design. Designsthe two-month trial, to mention only a few. In summary, Y52 is that are described next permit a researcher to isolate and re-a composite that reflects (a) the effects of treatment level a2, move some sources of variation that would ordinarily be in-(b) effects unique to the participant, (c) effects attributable to cluded in the error effects.chance fluctuations in the participant’s behavior, (d) errors inmeasuring and recording the participant’s weight loss, and(e) any other effects that have not been controlled. Our con- Randomized Block Designjectures about Y52 or any of the other 44 observations can be The two designs just described use independent samples. Twoexpressed more formally by a model equation. The classical samples are independent if, for example, a researcher ran-model equation for the weight-loss experiment is domly samples from two populations or randomly assigns participants to p groups. Dependent samples, on the other hand, Yi j = ␮ + ␣ j + ␧i( j) (i = 1, . . . , n; j = 1, . . . , p), can be obtained by any of the following procedures.where 1. Observe each participant under each treatment level in the experiment—that is, obtain repeated measures on the Yi j is the weight loss for participant i in treatment participants. level aj. 2. Form sets of participants who are similar with respect to ␮ is the grand mean of the three weight-loss popula- a variable that is correlated with the dependent variable. tion means. This procedure is called participant matching. ␣j is the treatment effect for population j and is equal to 3. Obtain sets of identical twins or littermates in which case ␮ j − ␮. It reflects the effects of diet aj. the participants have similar genetic characteristics. ␧i( j) is the within-groups error effect associated with Yi j 4. Obtain participants who are matched by mutual selection, and is equal to Yi j − ␮ − ␣ j . It reflects all effects for example, husband and wife pairs or business partners. not attributable to treatment level aj. The notation i( j) indicates that the ith participant appears only in In the behavioral and social sciences, the participants are treatment level j. Participant i is said to be nested often people whose aptitudes and experiences differ markedly. within the jth treatment level. Nesting is discussed Individual differences are inevitable, but it is often possible in the section titled “Hierarchical Designs.” to isolate or partition out a portion of these effects so that they do not appear in estimates of the error effects. One designAccording to the equation for this completely randomized for accomplishing this is the design used with a t statistic fordesign, each observation is the sum of three parameters dependent samples. As the name suggests, the design uses␮, ␣ j , and ␧i( j) . The values of the parameters in the equation dependent samples. A t dependent-samples design also uses aare unknown but can be estimated from sample data. more complex randomization and analysis plan than does a t The meanings of the terms grand mean, ␮, and treatment independent-samples design. However, the added complexityeffect, ␣ j , in the model equation seem fairly clear; the mean- is often accompanied by greater power—a point that I will de-ing of error effect, ␧i( j) , requires a bit more explanation. Why velop later in connection with a randomized block design.do observations, Yi j s, in the same treatment level vary from Let’s reconsider the weight-loss experiment. It is reason-one participant to the next? This variation must be due to dif- able to assume that ease of losing weight is related to theferences among the participants and to other uncontrolled amount by which a girl is overweight. The design of the exper-variables because the parameters ␮ and ␣ j in the model equa- iment can be improved by isolating this nuisance variable.tion are constants for all participants in the same treatment Suppose that instead of randomly assigning 30 participants tolevel. To put it another way, observations in the same treatment the treatment levels, the researcher formed pairs of participants

Three Building Block Designs 7 participants who have been exposed to only one treatment level. Some writers reserve the designation randomized block design for this latter case. They refer to a design with repeated measurements in which the order of administration of the treatment levels is randomized independently for each participant as a subjects-by-treatments design. A design with repeated measurements in which the order of administration of the treatment levels is the same for all participants is referred to as a subject-by-trials design. I use the designationFigure 1.3 Layout for a t dependent-samples design. Each block contains randomized block design for all three cases.two girls who are overweight by about the same amount. The two girls in a Of the four ways of obtaining dependent samples, the useblock are randomly assigned to the treatment levels. The mean weight loss in of repeated measures on the participants typically results inpounds for the girls in treatment levels a1 and a2 is denoted by Y ·1 and Y ·2 ,respectively. the greatest homogeneity within the blocks. However, if repeated measures are used, the effects of one treatment level should dissipate before the participant is observed under an-so that prior to going on a diet the participants in each pair are other treatment level. Otherwise the subsequent observationsoverweight by about the same amount. The participants in each will reflect the cumulative effects of the preceding treatmentpair constitute a block or set of matched participants. A simple levels. There is no such restriction, of course, if carryover ef-way to form blocks of matched participants is to rank them fects such as learning or fatigue are the researcher’s principalfrom least to most overweight. The participants ranked 1 and 2 interest. If blocks are composed of identical twins or litter-are assigned to block one, those ranked 3 and 4 are assigned to mates, it is assumed that the performance of participants hav-block two, and so on. In this example, 15 blocks of dependent ing identical or similar heredities will be more homogeneoussamples can be formed from the 30 participants. After all of the than the performance of participants having dissimilar hered-blocks have been formed, the two participants in each block ities. If blocks are composed of participants who are matchedare randomly assigned to the two diets. The layout for this ex- by mutual selection (e.g., husband and wife pairs or businessperiment is shown in Figure 1.3. If the researcher’s hunch is partners), a researcher should ascertain that the participantscorrect that ease in losing weight is related to the amount by in a block are in fact more homogeneous with respect to thewhich a girl is overweight, this design should result in a more dependent variable than are unmatched participants. A hus-powerful test of the null hypothesis, ␮·1 − ␮·2 = 0, than would band and wife often have similar political attitudes; the cou-a t test for independent samples. As we will see, the increased ple is less likely to have similar mechanical aptitudes.power results from isolating the nuisance variable (the amount Suppose that in the weight-loss experiment the researcherby which the girls are overweight) so that it does not appear in wants to evaluate the effectiveness of three diets, denotedthe estimate of the error effects. by a1, a2, and a3. The researcher suspects that ease of losing Earlier we saw that the layout and randomization proce- weight is related to the amount by which a girl is overweight.dures for a t independent-samples design and a completely If a sample of 45 girls is available, the blocking procedurerandomized design are the same except that a completely ran- described in connection with a t dependent-samples designdomized design can have more than two treatment levels. can be used to form 15 blocks of participants. The three par-The same comparison can be drawn between a t dependent- ticipants in a block are matched with respect to the nuisancesamples design and a randomized block design. A random- variable, the amount by which a girl is overweight. The lay-ized block design is denoted by the letters RB-p, where RB out for this experiment is shown in Figure 1.4. A comparisonstands for “randomized block” and p is the number of levels of the layout in this figure with that in Figure 1.3 for a tof the treatment. The four procedures for obtaining depen- dependent-samples design reveals that they are the same ex-dent samples that were described earlier can be used to form cept that the randomized block design has p = 3 treatmentthe blocks in a randomized block design. The procedure that levels. When p = 2, the layouts and randomization plans foris used does not affect the computation of significance tests, the designs are identical. In this and later examples, I assumebut the procedure does affect the interpretation of the results. that all of the treatment levels and blocks of interest are rep-The results of an experiment with repeated measures general- resented in the experiment. In other words, the treatment lev-ize to a population of participants who have been exposed to els and blocks represent fixed effects. A discussion of the caseall of the treatment levels. However, the results of an experi- in which either the treatment levels or blocks or both are ran-ment with matched participants generalize to a population of domly sampled from a population of levels, the mixed and

8 Experimental Design ␧i j is the residual error effect associated with Yi j and is equal to Yi j − ␮ − ␣ j − ␲i . It reflects all effects not attributable to treatment level aj and Blocki. According to the model equation for this randomized block design, each observation is the sum of four parameters: ␮,␣ j , ␲i , and ␧i j . A residual error effect is that portion of an observation that remains after the grand mean, treatment effect, and block effect have been subtracted from it; that is, ␧i j = Yi j − ␮ − ␣ j − ␲i . The sum of the squared errorFigure 1.4 Layout for a randomized block design (RB-3 design). Each effects for this randomized block design,block contains three girls who are overweight by about the same amount.The three girls in a block are randomly assigned to the treatment levels. The ␧i2j = (Yi j − ␮ − ␣ j − ␲i )2 ,mean weight loss in pounds for the girls in treatment levels a1, a2, and a3 isdenoted by Y ·1 , Y ·2 , and Y ·3 , respectively. The mean weight loss for thegirls in Block1, Block2, . . . , Block15 is denoted by Y 1· , Y 2· , . . . , Y 15· , will be smaller than the sum for the completely randomizedrespectively. design, ␧i( j) = 2 (Yi j − ␮ − ␣ j )2 ,random effects cases, is beyond the scope of this chapter. Thereader is referred to Kirk (1995, pp. 256–257, 265–268). if ␲i2 is not equal to zero for one or more blocks. This idea is A randomized block design enables a researcher to test illustrated in Figure 1.5, where the total sum of squares andtwo null hypotheses. degrees of freedom for the two designs are partitioned. The F H0: ␮·1 = ␮·2 = ␮·3 statistic that is used to test the null hypothesis can be thought (Treatment population means are equal.) of as a ratio of error and treatment effects, H0: ␮1· = ␮2· = · · · = ␮15· f (error effects) + f (treatment effects) F= (Block population means are equal.) f (error effects)The second hypothesis, which is usually of little interest, where f ( ) denotes a function of the effects in parentheses. Itstates that the population weight-loss means for the 15 levels is apparent from an examination of this ratio that the smallerof the nuisance variable are equal. The researcher expects a the sum of the squared error effects, the larger the F statistictest of this null hypothesis to be significant. If the nuisance and, hence, the greater the probability of rejecting a false nullvariable represented by the blocks does not account for an ap-preciable proportion of the total variation in the experiment,little has been gained by isolating the effects of the variable.Before exploring this point, I describe the model equation foran RB-p design. The classical model equation for the weight-loss experi- SSWGment is p(n Ϫ 1) ϭ 42 Yi j = ␮ + ␣ j + ␲i + ␧i j (i = 1, . . . , n; j = 1, . . . , p),where SSRES Yi j is the weight loss for the participant in Block i and (n Ϫ 1)(p Ϫ 1) ϭ 28 treatment level aj. Figure 1.5 Partition of the total sum of squares (SSTOTAL) and degrees of ␮ is the grand mean of the three weight-loss popula- freedom (np − 1 = 44) for CR-3 and RB-3 designs. The treatment and tion means. within-groups sums of squares are denoted by, respectively, SSA and SSWG. The block and residual sums of squares are denoted by, respectively, SSBL ␣j is the treatment effect for population j and is equal to and SSRES. The shaded rectangles indicate the sums of squares that are used ␮· j − ␮. It reflects the effect of diet aj. to compute the error variance for each design: MSWG = SSWG/ p(n − 1) ␲i is the block effect for population i and is equal to and MSRES = SSRES/(n − 1)( p − 1). If the nuisance variable (SSBL) in the randomized block design accounts for an appreciable portion of the total sum ␮i· − ␮. It reflects the effect of the nuisance variable of squares, the design will have a smaller error variance and, hence, greater in Blocki. power than the completely randomized design.

Three Building Block Designs 9hypothesis. Thus, by isolating a nuisance variable that ac-counts for an appreciable portion of the total variation in arandomized block design, a researcher is rewarded with amore powerful test of a false null hypothesis. As we have seen, blocking with respect to the nuisancevariable (the amount by which the girls are overweight)enables the researcher to isolate this variable and remove itfrom the error effects. But what if the nuisance variable Figure 1.6 Three-by-three Latin square, where aj denotes one of thedoesn’t account for any of the variation in the experiment? In j = 1, . . . , p levels of treatment A; bk denotes one of the k = 1, . . . , p levels of nuisance variable B; and cl denotes one of the l = 1, . . . , p levels of nui-other words, what if all of the block effects in the experiment sance variable C. Each level of treatment A appears once in each row andare equal to zero? In this unlikely case, the sum of the squared once in each column as required for a Latin square.error effects for the randomized block and completely ran-domized designs will be equal. In this case, the randomized Latin square: b1 is less than 15 pounds, b2 is 15 to 25 pounds,block design will be less powerful than the completely ran- and b3 is more than 25 pounds. The advantage of being able todomized design because its error variance, the denominator isolate two nuisance variables comes at a price. The ran-of the F statistic, has n − 1 fewer degrees of freedom than domization procedures for a Latin square design are morethe error variance for the completely randomized design. It complex than those for a randomized block design. Also, theshould be obvious that the nuisance variable should be se- number of rows and columns of a Latin square must eachlected with care. The larger the correlation between the nui- equal the number of treatment levels, which is three in the ex-sance variable and the dependent variable, the more likely it ample. This requirement can be very restrictive. For example,is that the block effects will account for an appreciable it was necessary to restrict the continuous variable of theproportion of the total variation in the experiment. amount by which girls are overweight to only three levels. The layout of the LS-3 design is shown in Figure 1.7.Latin Square DesignThe Latin square design described in this section derives itsname from an ancient puzzle that was concerned with thenumber of different ways that Latin letters can be arranged ina square matrix so that each letter appears once in each rowand once in each column. An example of a 3 × 3 Latin squareis shown in Figure 1.6. In this ﬁgure I have used the letter awith subscripts in place of Latin letters. The Latin square de-sign is denoted by the letters LS-p, where LS stands for“Latin square” and p is the number of levels of the treatment.A Latin square design enables a researcher to isolate the ef-fects of not one but two nuisance variables. The levels of onenuisance variable are assigned to the rows of the square; thelevels of the other nuisance variable are assigned to thecolumns. The levels of the treatment are assigned to the cellsof the square. Let’s return to the weight-loss experiment. With a Latinsquare design the researcher can isolate the effects of theamount by which girls are overweight and the effects of a sec-ond nuisance variable, for example, genetic predisposition tobe overweight. A rough measure of the second nuisance vari- Figure 1.7 Layout for a Latin square design (LS-3 design) that is based onable can be obtained by asking a girl’s parents whether they the Latin square in Figure 1.6. Treatment A represents three kinds of diets;were overweight as teenagers: c1 denotes neither parent over- nuisance variable B represents amount by which the girls are overweight;weight, c2 denotes one parent overweight, and c3 denotes both and nuisance variable C represents genetic predisposition to be overweight.parents overweight. This nuisance variable can be assigned to The girls in Group1, for example, received diet a1, were less than ﬁfteen pounds overweight (b1), and neither parent had been overweight as athe columns of the Latin square. Three levels of the amount by teenager (c1). The mean weight loss in pounds for the girls in the nine groupswhich girls are overweight can be assigned to the rows of the is denoted by Y ·111 , Y ·123 , . . . , Y ·331 .

10 Experimental Design The design in Figure 1.7 enables the researcher to test if the combined effects of ␤2 , ␥l2 , and k ␧2 are jklthree null hypotheses: greater than ␲i2 . The benefits of isolating two nuisance variables are a smaller error variance and increased power. H0: ␮1·· = ␮2·· = ␮3·· Thus far I have described three of the simplest experimen- (Treatment population means are equal.) tal designs: the completely randomized design, randomized H0: ␮·1· = ␮·2· = ␮·3· block design, and Latin square design. The three designs are (Row population means are equal.) called building block designs because complex experimental H0: ␮··1 = ␮··2 = ␮··3 designs can be constructed by combining two or more of these (Column population means are equal.) simple designs (Kirk, 1995, p. 40). Furthermore, the randomization procedures, data analysis, and model assumptions forThe first hypothesis states that the population means for the complex designs represent extensions of those for the threethree diets are equal. The second and third hypotheses make building block designs. The three designs provide the organi-similar assertions about the population means for the two zational structure for the design nomenclature and classifica-nuisance variables. Tests of these nuisance variables are ex- tion scheme that is described next.pected to be significant. As discussed earlier, if the nuisancevariables do not account for an appreciable proportion of thetotal variation in the experiment, little has been gained by iso- CLASSIFICATION OF EXPERIMENTAL DESIGNSlating the effects of the variables. The classical model equation for this version of the A classification scheme for experimental designs is given inweight-loss experiment is Table 1.1. The designs in the category systematic designs do not use random assignment of participants or experimental Yi jkl = ␮ + ␣ j + ␤k + ␥l + ␧ jkl + ␧i( jkl) units and are of historical interest only. According to Leonard (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , p; l = 1, . . . , p), and Clark (1939), agricultural field research employing systematic designs on a practical scale dates back to 1834. Overwhere the last 80 years systematic designs have fallen into disuse because designs employing random assignment are more likely Yi jkl is the weight loss for the ith participant in treat- to provide valid estimates of treatment and error effects and ment level aj, row bk, and column cl. can be analyzed using the powerful tools of statistical infer- ␣j is the treatment effect for population j and is equal ence such as analysis of variance. Experimental designs using to ␮ j·· − ␮. It reflects the effect of diet aj. random assignment are called randomized designs. The ran- ␤k is the row effect for population k and is equal domized designs in Table 1.1 are subdivided into categories to ␮·k· − ␮. It reflects the effect of nuisance vari- based on (a) the number of treatments, (b) whether participants able bk. are assigned to relatively homogeneous blocks prior to random ␥l is the column effect for population l and is equal assignment, (c) presence or absence of confounding, (d) use of to ␮··l − ␮. It reflects the effects of nuisance vari- crossed or nested treatments, and (e) use of a covariate. able cl. The letters p and q in the abbreviated designations denote ␧ jkl is the residual effect that is equal to ␮ jkl − ␮ j·· − the number of levels of treatments A and B, respectively. If a ␮·k· − ␮··l + 2␮. design includes a third and fourth treatment, say treatments C ␧i( jkl) is the within-cell error effect associated with Yijkl and D, the number of their levels is denoted by r and t, and is equal to Yi jkl − ␮ − ␣ j − ␤k − ␥l − ␧ jkl . respectively. In general, the designation for designs with two or more treatments includes the letters CR, RB, or LS toAccording to the model equation for this Latin square design, indicate the building block design. The letter F or H is addedeach observation is the sum of six parameters: ␮, ␣ j , ␤k , to the designation to indicate that the design is, respectively, a␥l , ␧ jkl , and ␧i( jkl) . The sum of the squared within-cell error factorial design or a hierarchical design. For example, the F ineffects for the Latin square design, the designation CRF-pq indicates that it is a factorial design; the CR and pq indicate that the design was constructed by ␧i( jkl) = 2 (Yi jkl − ␮ − ␣ j −␤k − ␥l − ␧ jkl )2 , combining two completely randomized designs with p and q treatment levels. The letters CF, PF, FF, and AC are added towill be smaller than the sum for the randomized block design, the designation if the design is, respectively, a confounded factorial design, partially confounded factorial design, frac- ␧i2j = (Yi j − ␮ − ␣ j − ␲i )2 , tional factorial design, or analysis of covariance design.

Factorial Designs 11TABLE 1.1 Classification of Experimental Designs Abbreviated AbbreviatedExperimental Design Designationa Experimental Design Designationa I. Systematic Designs (selected examples). b. Randomized block completely confounded RBCF-pk 1. Beavan’s chessboard design. factorial design. 2. Beavan’s half-drill strip design. c. Randomized block partially confounded RBPF-pk 3. Diagonal square design. factorial design. 4. Knut Vik square design. 4. Designs with treatment-interaction confounding. II. Randomized Designs With One Treatment. a. Completely randomized fractional CRFF-pk−i A. Experimental units randomly assigned to factorial design. treatment levels. b. Graeco-Latin square fractional factorial design. GLSFF-pk 1. Completely randomized design. CR-p c. Latin square fractional factorial design. LSFF-pk B. Experimental units assigned to relatively d. Randomized block fractional factorial design. RBFF-pk−i homogeneous blocks or groups prior to B. Hierarchical designs: designs in which one or random assignment. more treatments are nested. 1. Balanced incomplete block design. BIB-p 1. Designs with complete nesting. 2. Cross-over design. CO-p a. Completely randomized hierarchical design. CRH-pq(A) 3. Generalized randomized block design. GRB-p b. Randomized block hierarchical design. RBH-pq(A) 4. Graeco-Latin square design. GLS-p 2. Designs with partial nesting. 5. Hyper-Graeco-Latin square design. HGLS-p a. Completely randomized partial CRPH-pq(A)r 6. Latin square design. LS-p hierarchical design. 7. Lattice balanced incomplete block design. LBIB-p b. Randomized block partial hierarchical design. RBPH-pq(A)r 8. Lattice partially balanced incomplete LPBIB-p c. Split-plot partial hierarchical design. SPH-p·qr(B) block design. IV. Randomized Designs With One or More Covariates. 9. Lattice unbalanced incomplete block design. LUBIB-p A. Designs that include a covariate have 10. Partially balanced incomplete block design. PBIB-p the letters AC added to the abbreviated 11. Randomized block design. RB-p designation as in the following examples. 12. Youden square design. YBIB-p 1. Completely randomized analysis of covariance CRAC-pIII. Randomized Designs With Two or More Treatments. design. A. Factorial designs: designs in which all treatments 2. Completely randomized factorial analysis CRFAC-pq are crossed. of covariance design. 1. Designs without confounding. 3. Latin square analysis of covariance design. LSAC-p a. Completely randomized factorial design. CRF-pq 4. Randomized block analysis of covariance design. RBAC-p b. Generalized randomized block factorial design. GRBF-pq 5. Split-plot factorial analysis of covariance design. SPFAC-p·q c. Randomized block factorial design. RBF-pq V. Miscellaneous Designs (select examples). 2. Design with group-treatment confounding. 1. Solomon four-group design. a. Split-plot factorial design. SPF- p·q 2. Interrupted time-series design. 3. Designs with group-interaction confounding. a. Latin square confounded factorial design. LSCF-pkaThe abbreviated designations are discussed later.Three of these designs are described later. Because of space choose. Because of the wide variety of designs available, it islimitations, I cannot describe all of the designs in Table 1.1. important to identify them clearly in research reports. OneI will focus on those designs that are potentially the most often sees statements such as “a two-treatment factorial de-useful in the behavioral and social sciences. sign was used.” It should be evident that a more precise It is apparent from Table 1.1 that a wide array of designs description is required. This description could refer to 10 ofis available to researchers. Unfortunately, there is no univer- the 11 factorial designs in Table 1.1.sally accepted designation for the various designs—some Thus far, the discussion has been limited to designs withdesigns have as many as five different names. For example, one treatment and one or two nuisance variables. In the fol-the completely randomized design has been called a one-way lowing sections I describe designs with two or more treat-classification design, single-factor design, randomized group ments that are constructed by combining several buildingdesign, simple randomized design, and single variable exper- block designs.iment. Also, a variety of design classification schemes havebeen proposed. The classification scheme in Table 1.1 owes FACTORIAL DESIGNSmuch to Cochran and Cox (1957, chaps. 4–13) and Federer(1955, pp. 11–12). Completely Randomized Factorial Design A quick perusal of Table 1.1 reveals why researcherssometimes have difficulty selecting an appropriate experi- Factorial designs differ from those described previously inmental design—there are a lot of designs from which to that two or more treatments can be evaluated simultaneously

12 Experimental Designin an experiment. The simplest factorial design from thestandpoint of randomization, data analysis, and model as-sumptions is based on a completely randomized design and,hence, is called a completely randomized factorial design. Atwo-treatment completely randomized factorial design is de-noted by the letters CRF-pq, where p and q denote the num-ber of levels, respectively, of treatments A and B. In the weight-loss experiment, a researcher might be inter-ested in knowing also whether walking on a treadmill for20 minutes a day would contribute to losing weight, as wellas whether the difference between the effects of walking ornot walking on the treadmill would be the same for each ofthe three diets. To answer these additional questions, a re-searcher can use a two-treatment completely randomized fac-torial design. Let treatment A consist of the three diets (a1, a2,and a3) and treatment B consist of no exercise on the tread-mill (b1) and exercise for 20 minutes a day on the treadmill(b2). This design is a CRF-32 design, where 3 is the numberof levels of treatment A and 2 is the number of levels of treat-ment B. The layout for the design is obtained by combiningthe treatment levels of a CR-3 design with those of a CR-2design so that each treatment level of the CR-3 design ap- Figure 1.8 Layout for a two-treatment completely randomized factorialpears once with each level of the CR-2 design and vice versa. design (CRF-32 design). Thirty girls are randomly assigned to six combinations of treatments A and B with the restriction that five girls are assigned toThe resulting design has 3 × 2 = 6 treatment combinations each combination. The mean weight loss in pounds for girls in the six groupsas follows: a1b1, a1b2, a2b1, a2b2, a3b1, a3b2. When treatment is denoted by Y ·11 , Y ·12 , . . . , Y ·32 .levels are combined in this way, the treatments are said to becrossed. The use of crossed treatments is a characteristic of ␧i( jk) is the within-cell error effect associated with Yi jkall factorial designs. The layout of the design with 30 girls and is equal to Yi jk − ␮ − ␣ j − ␤k − (␣␤) jk . Itrandomly assigned to the six treatment combinations is reflects all effects not attributable to treatmentshown in Figure 1.8. level aj, treatment level bk, and the interaction of aj The classical model equation for the weight-loss experi- and bk.ment is The CRF-32 design enables a researcher to test three null Yi jk = ␮ + ␣ j + ␤k + (␣␤) jk + ␧i( jk) hypotheses: (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , q), H0: ␮1· = ␮2· = ␮3· (Treatment A population means are equal.)where H0: ␮·1 = ␮·2 Yi jk is the weight loss for participant i in treatment (Treatment B population means are equal.) combination aj bk. H0: ␮ jk − ␮ jk − ␮ j k + ␮ j k = 0 for all j and k ␮ is the grand mean of the six weight-loss popula- (All A × B interaction effects equal zero.) tion means. The last hypothesis is unique to factorial designs. It states that ␣j is the treatment effect for population aj and is the joint effects (interaction) of treatments A and B are equal equal to ␮ j· − ␮. It reflects the effect of diet aj. to zero for all combinations of the two treatments. Two treat- ␤k is the treatment effect for population bk and is ments are said to interact if any difference in the dependent equal to ␮·k − ␮. It reflects the effects of exercise variable for one treatment is different at two or more levels of condition bk. the other treatment. (␣␤) jk is the interaction effect for populations aj and bk Thirty girls are available to participate in the weight-loss ex- and is equal to ␮ jk − ␮ j· − ␮·k − ␮. Interaction periment and have been randomly assigned to the six treatment effects are discussed later. combinations with the restriction that five girls are assigned to

Factorial Designs 13TABLE 1.2 Weight-Loss Data for the Diet (aj) and Exercise TABLE 1.4 Analysis of Variance for the Weight-Loss DataConditions (bk) Source SS df MS F pa1b1 a1b2 a2b1 a2b2 a3b1 a3b2 Treatment A (Diet) 131.6667 2 65.8334 4.25 .026 7 7 9 10 15 13 Treatment B (Exercise) 67.5000 1 67.5000 4.35 .04813 14 4 5 10 16 A×B 35.0000 2 17.5000 1.13 .340 9 11 7 7 12 20 Within cell 372.0000 24 15.5000 5 4 14 15 5 19 1 9 11 13 8 12 Total 606.1667 29each combination. The data, weight loss for each girl, are given the hypotheses are false. As John Tukey (1991) wrote, “thein Table 1.2. A descriptive summary of the data—sample effects of A and B are always different—in some decimalmeans and standard deviations—is given in Table 1.3. place—for any A and B. Thus asking ‘Are the effects differ- An examination of Table 1.3 suggests that diet a3 resulted ent?’ is foolish” (p. 100). Furthermore, rejection of a nullin more weight loss than did the other diets and 20 minutes a hypothesis tells us nothing about the size of the treatmentday on the treadmill was beneficial. The analysis of variance effects or whether they are important or large enough to befor the weight-loss data is summarized in Table 1.4, which useful—that is, their practical significance. In spite of numer-shows that the null hypotheses for treatments A and B can be ous criticisms of null hypothesis significance testing, re-rejected. We know that at least one contrast or difference searchers continue to focus on null hypotheses and p values.among the diet population means is not equal to zero. Also, The focus should be on the data and on what the data tell thefrom Tables 1.3 and 1.4 we know that 20 minutes a day on researcher about the scientific hypothesis. This is not a newthe treadmill resulted in greater weight loss than did the idea. It was originally touched on by Karl Pearson in 1901no-exercise condition. The A × B interaction test is not and more explicitly by Fisher in 1925. Fisher (1925) pro-significant. When two treatments interact, a graph in which posed that researchers supplement null hypothesis signifi-treatment-combination population means are connected by cance tests with measures of strength of association. Sincelines will always reveal at least two nonparallel lines for one then over 40 supplementary measures of effect magnitudeor more segments of the lines. The nonsignificant interac- have been proposed (Kirk, 1996). The majority of the mea-tion test in Table 1.4 tells us that there is no reason for be- sures fall into one of two categories: measures of strength oflieving that the population difference in weight loss between association and measures of effect size (typically, standard-the treadmill and no-treadmill conditions is different for the ized mean differences). Hays (1963) introduced a measurethree diets. If the interaction had been significant, our interest of strength of association that can assist a researcher in as-would have shifted from interpreting the tests of treatments A sessing the importance or usefulness of a treatment: omegaand B to understanding the nature of the interaction. Proce- squared, ␻2 . Omega squared estimates the proportion of the ˆdures for interpreting interactions are described by Kirk population variance in the dependent variable accounted(1995, pp. 370–372, 377–389). for by a treatment. For experiments with several treatments, as in the weight-loss experiment, partial omega squared isStatistical Significance Versus Practical Significance computed. For example, the proportion of variance in the dependent variable, Y, accounted for by treatment A eliminat-The rejection of the null hypotheses for the diet and exercise ing treatment B and the A × B interaction is denoted bytreatments is not very informative. We know in advance that ␻2 |A·B, AB . Similarly, ␻2 |B·A, AB denotes the proportion of the ˆY ˆY variance accounted for by treatment B eliminating treatmentTABLE 1.3 Descriptive Summary of the Weight-Loss Data: Means A and the A × B interaction. For the weight-loss experiment,(Y) and Standard Deviations (S) the partial omega squareds for treatments A and B are, Mean respectively, Standard Diet a1 Diet a2 Diet a3 DeviationNo treadmill Y ·11 = 7.0 Y ·21 = 9.0 Y ·31 = 10.0 Y ··1 = 8.7 S·11 = 4.0 S·21 = 3.4 S·31 = 3.4 S··1 = 3.8 ( p − 1)(FA − 1) exercise (b1) ␻2 |A·B, AB = ˆYTreadmill Y ·12 = 9.0 Y ·22 = 10.0 Y ·32 = 16.0 Y ··2 = 11.7 ( p − 1)(FA − 1) + npq exercise (b2) S·12 = 3.4 S·22 = 3.7 S·32 = 3.2 S··2 = 4.6 Y ·1· = 8.0 Y ·2· = 9.5 Y ·3· = 13.0 (3 − 1)(4.247 − 1) S·1· = 3.8 S·2· = 3.6 S·3· = 4.4 = = 0.18 (3 − 1)(4.247 − 1) + (5)(3)(2)

14 Experimental Design (q − 1)(FB − 1) procedures for computing sums of squares assume that all ␻2 |B·A, AB = ˆY (q − 1)(FB − 1) + npq cell ns in multitreatment experiments are equal. If the cell ns are not equal, some researchers use one of the following pro- (2 − 1)(4.376 − 1) = = 0.10. cedures to obtain approximate tests of null hypotheses: (a) es- (2 − 1)(4.376 − 1) + (5)(3)(2) timate the missing observations under the assumption that the treatments do not interact, (b) randomly set aside data to re-Following Cohen’s (1988, pp. 284–288) guidelines for inter- duce all cell ns to the same size, and (c) use an unweighted-preting omega squared, means analysis. The latter approach consists of performing .010 is a small association an ANOVA on the cell means and then multiplying the sums .059 is a medium association of squares by the harmonic mean of the cell ns. None of these procedures is entirely satisfactory. Fortunately, exact solu- .138 is a large association, tions to the unequal cell n problem exist. Two solutions thatwe conclude that the diets accounted for a large proportion are described next are based on a regression model and a cellof the population variance in weight loss. This is consistent means model. Unlike the classical model approach, thewith our perception of the differences between the weight- regression and cell means model approaches require a com-loss means for the three diets: girls on diet a3 lost five more puter and software for manipulating matrices.pounds than did those on a1. Certainly, any girl who is anx- Suppose that halfway through the weight-loss experimentious to lose weight would want to be on diet a3. Likewise, the the third participant in treatment combination a2 b2 (Y322 = 7)medium association between the exercise conditions and moved to another area of the country and dropped out of theweight loss is practically significant: Walking on the tread- experiment. The loss of this participant resulted in unequalmill resulted in a mean weight loss of 3 pounds. Based on cell ns. Cell a2b2 has four participants; the other cells have fiveTukey’s HSD statistic, 95% confidence intervals for the three participants. The analysis of the weight-loss data using thepairwise contrasts among the diet means are regression model is described next. −5.9 < ␮·1 − ␮·2 < 2.9 Regression Model −9.4 < ␮·1 − ␮·3 < −0.6 A qualitative regression model equation with h − 1 = −7.9 < ␮·2 − ␮·3 < 0.9. ( p − 1) + (q − 1) + ( p − 1)(q − 1) = 5 independent vari-Because the confidence interval for ␮·1 − ␮·3 does not con- ables (Xi1, Xi2, . . ., Xi2 Xi3) and h = 6 parameters (␤0 , ␤1 , . . . ,tain 0, we can be confident that diet a3 is superior to diet a1. ␤5 ),Hedges’s (1981) effect size for the difference between diets A effects B effects A× B effectsa1 and a3 is Yi = ␤0 + ␤1 X i 1 + ␤2 X i 2 + ␤3 X i 3 + ␤4 X i 1 X i 3 + ␤5 X i 2 X i 3 + ei , Y ··1 − Y ··2 |8.0 − 13.0| g= = = 1.27, can be formulated so that tests of selected parameters of the σPooled ˆ 3.937 regression model provide tests of null hypotheses for A, B,a large effect. and A × B in the weight-loss experiment. Tests of the fol- Unfortunately, there is no statistic that measures practical lowing null hypotheses for this regression model are of par-significance. The determination of whether results are impor- ticular interest:tant or useful must be made by the researcher. However, con-fidence intervals and measures of effect magnitude can help H0: ␤1 = ␤2 = 0the researcher make this decision. If our discipline is to H0: ␤3 = 0progress as it should, researchers must look beyond signifi- H0: ␤4 = ␤5 = 0cance tests and p values and focus on what their data tell themabout the phenomenon under investigation. For a fuller In order for tests of these null hypotheses to provide testsdiscussion of this point, see Kirk (2001). of ANOVA null hypotheses, it is necessary to establish a correspondence between the five independent variables ofAlternative Models the regression model equation and ( p − 1) + (q − 1) + ( p − 1)(q − 1) = 5 treatment and interaction effects of theThus far, I have described the classical model equation for CRF-32 design. One way to establish this correspondence isseveral experimental designs. This model and associated to code the independent variables of the regression model as

Factorial Designs 15follows: F statistics for testing hypotheses for selected regression 1, if an observation is in a1 parameters are obtained by dividing a regression mean square, X i1 = −1, if an observation is in a3 MSR, by an error mean square, MSE, where MSR = SSR/dfreg 0, otherwise and MSE = SSE/dferror. The regression sum of squares, SSR, 1, if an observation is in a2 that reflects the contribution of independent variables X1 and X i2 = −1, if an observation is in a3 X2 over and above the contribution of X3, X1X3, and X2X3 is 0, otherwise given by the difference between two error sums of squares, SSE, as follows: 1, if an observation is in b1 X i3 = −1, if an observation is in b2 A B A×B product of coded values SSR( X 1 X 2 | X 3 X 1 X 3 X 2 X 3 ) X i1 X i3 = associated with a1 and b1 B A×B A B A×B product of coded values X i2 X i3 = = SSE( X 3 X 1 X 3 X 2 X 3 ) – SSE( X 1 X 2 X 3 X 1 X 3 X 2 X 3 ) associated with a2 and b1This coding scheme, which is called effect coding, produced An error sum of squares is given bythe X matrix in Table 1.5. The y vector in Table 1.5 containsweight-loss observations for the six treatment combinations. SSE( ) = y y − [(Xi Xi )−1 (Xi y)] (Xi y),The first column vector, x0, in the X matrix contains ones; thesecond through the sixth column vectors contain coded where the Xi matrix contains the first column, x0, of X andvalues for Xi1, Xi2, . . . , Xi2Xi3. To save space, only a portion the columns corresponding the independent variables con-of the 29 rows of X and y are shown. As mentioned earlier, tained in SSE( ). For example, the X matrix used in comput-observation Y322 is missing. Hence, each of the treatment ing SSE(X3 X1X3 X2X3) contains four columns: x0, x3, x1x3,combinations contains five observations except for a2b2, and x2x3. The regression sum of squares corresponding towhich contains four. SSA in ANOVA is A B A×BTABLE 1.5 Data Vector, y, and X Matrix for the Regression Model SSR( X 1 X 2 | X 3 X 1 X 3 X 2 X 3 ) y X 29×1 29×6 B A×B A B A×B A B A× B = SSE( X 3 X 1 X 3 X 2 X 3 ) – SSE( X 1 X 2 X 3 X 1 X 3 X 2 X 3 ) Ά Ά Ά x0 x1 x2 x3 x1x3 x2x3 Ά 7 . 1 . 1 . 0 . 1 . 1 . 0 . = 488.1538 − 360.7500 = 127.4038a1b1 . . . . . . . . . . . . . . 1 1 1 0 1 1 0 with p − 1 = 2 degrees of freedom. This sum of squares −1 −1 is used in testing the regression null hypothesis H0: ␤1 = Ά 7 1 1 0 0 . . . . . . . . . . . . . .a1b2 . . . . . . . ␤2 = 0. Because of the correspondence between the regres- 9 1 1 0 −1 −1 0 sion and ANOVA parameters, a test of this regression null hypothesis is equivalent to testing the ANOVA null hypothe- Ά 9 1 0 1 1 0 1 . . . . . . . . . . . . . .a2b1 . . . . . . . sis for treatment A. 11 1 0 1 1 0 1 The regression sum of squares corresponding to SSB in −1 −1 ANOVA is Ά 10 1 0 1 0 . . . . . . . . . . . . . .a2b2 . . . . . . . B A A×B 13 1 0 1 −1 0 −1 SSR( X 3 | X 1 X 2 X 1 X 3 X 2 X 3 ) −1 −1 −1 −1 Ά 15 1 1 . . . . . . . . . . . . . . A A×B A B A×Ba3b1 . . . . . . . 8 1 −1 −1 1 −1 −1 = SSE( X 1 X 2 X 1 X 3 X 2 X 3 ) – SSE( X 1 X 2 X 3 X 1 X 3 X 2 X 3 ) Ά 13 1 −1 −1 −1 1 1 . . . . . . . . . . . . . . = 436.8000 − 360.7500 = 76.0500a3b2 . . . . . . . 12 1 −1 −1 −1 1 1 with q − 1 = 1 degree of freedom.

16 Experimental Design The regression sum of squares corresponding to SSA × B focuses on the grand mean, treatment effects, and interactionin ANOVA is effects. The cell means model equation for the CRF-pq design, A×B A B Yi jk = ␮ jk + ␧i( jk) SSR( X 1 X 3 X 2 X 3 | X 1 X 2 X 3 ) (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , q), A B A B A×B = SSE( X 1 X 2 X 3 ) − SSE( X 1 X 2 X 3 X 1 X 3 X 2 X 3 ) focuses on cell means, where ␮ jk denotes the mean in cell aj and bk. Although I described the classical model ﬁrst, this is not the order in which the models evolved historically. = 388.5385 − 360.7500 = 27.7885 According to Urquhart, Weeks, and Henderson (1973), Fisher’s early development of ANOVA was conceptualizedwith ( p − 1)(q − 1) = 2 degrees of freedom. by his colleagues in terms of cell means. It was not until later The regression error sum of squares corresponding to that cell means were given a linear structure in terms of theSSWCELL in ANOVA is grand mean and model effects, that is, ␮ jk = ␮ + ␣ j + ␤k + (␣␤) jk . The classical model equation for a CRF-pq design A B A×B uses four parameters, ␮ + ␣ j + ␤k + (␣␤) jk , to represent SSE( X 1 X 2 X 3 X 1 X 3 X 2 X 3 ) = 360.7500 one parameter, ␮ jk . Because of this structure, the classical model is overparameterized. For example, the expectation ofwith N − h = 29 − 6 = 23 degrees of freedom. the classical model equation for the weight-loss experiment The total sum of squares is contains 12 parameters: ␮, ␣1, ␣2, ␣3, ␤1, ␤2, (␣␤)11, (␣␤)12, (␣␤)21, (␣␤)22, (␣␤)31, (␣␤)32. However, there are only six SSTO = y y − y JyN −1 = 595.7931, cells means from which to estimate the 12 parameters. When there are missing cells in multitreatment designs, a researcherwhere J is a 29 × 29 matrix of ones and N = 29, the number is faced with the question of which parameters or parametricof weight-loss observations. The total sum of squares has functions are estimable. For a discussion of this and otherN − 1 = 28 degrees of freedom. The analysis of the weight- problems, see Hocking (1985), Hocking and Speed (1975),loss data is summarized in Table 1.6. The null hypotheses Searle (1987), and Timm (1975).␤1 = ␤2 = 0 and ␤3 = 0 can be rejected. Hence, indepen- The cell means model avoids the problems associated withdent variables X1 or X2 as well as X3 contribute to predicting overparameterization. A population mean can be estimatedthe dependent variable. As we see in the next section, the for each cell that contains one or more observations. Thus,F statistics in Table 1.6 are identical to the ANOVA F statis- the model is fully parameterized. Unlike the classical model,tics for the cell means model. the cell means model does not impose a structure on the analysis of data. Consequently, the model can be used to test hypotheses about any linear combination of population cellCell Means Model means. It is up to the researcher to decide which tests are meaningful or useful based on the original research hypothe-The classical model equation for a CRF-pq design, ses, the way the experiment was conducted, and the data that are available. Yi jk = ␮ + ␣ j + ␤k + (␣␤) jk + ␧i( jk) I will use the weight-loss data in Table 1.2 to illustrate the (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , q), computational procedures for the cell means model. Again, TABLE 1.6 Analysis of Variance for the Weight-Loss Data (Observation Y322 is missing) Source SS df MS F p X1 X2 | X3 X1X3 X2X3 127.4038 p−1=2 63.7019 4.06 .031 X3 | X1 X2 X1X3 X2X3 76.0500 q −1=1 76.0500 4.85 .038 X1X3 X2X3 | X1 X2 X3 27.7885 ( p − 1)(q − 1) = 2 13.8943 0.89 .426 Error 360.7500 N − h = 23 15.6848 Total 595.7931 N − 1 = 28

Factorial Designs 17we will assume that observation Y322 is missing. The null TABLE 1.7 Data Vector, y, and X Matrix for the Cell Means Modelhypothesis for treatment A is y X 29×1 29×6 x1 x2 x3 x4 x5 x6 H0: ␮1· = ␮2· = ␮3· . Ά 7 1 0 0 0 0 0 . . . . . . . . . . . . . . a1b1 . . . . . . .An equivalent null hypothesis that is used with the cell means 1 1 0 0 0 0 0model is Ά 7 0 1 0 0 0 0 . . . . . . . . . . . . . . H0: ␮1· − ␮2· = 0 a1b2 . . . . . . . (1.1) 9 0 1 0 0 0 0 ␮2· − ␮3· = 0. Ά 9 0 0 1 0 0 0 . . . . . . . . . . . . . .In terms of cell means, this hypothesis can be expressed as a2b1 . . . . . . . 11 0 0 1 0 0 0 ␮ + ␮12 ␮ + ␮22 H0: 11 − 21 =0 Ά 10 0 0 0 1 0 0 2 2 (1.2) . . . . . . . a2b2 . . . . . . . . . . . . . . ␮21 + ␮22 ␮ + ␮32 − 31 = 0, 13 0 0 0 1 0 0 2 2 Ά 15 0 0 0 0 1 0 . . . . . . . . . . . . . . a3b1where ␮1· = (␮11 + ␮12 )/2, ␮2· = (␮21 + ␮22 )/2, and so . . . . . . . 8 0 0 0 0 1 0on. In matrix notation, the null hypothesis is Ά 13 0 0 0 0 0 1 . . . . . . . . . . . . . . CA ␮ 0 a3b2 . . . . . . . ( p−1) × h h×1 ( p−1)×1   12 0 0 0 0 0 1 ␮11  ␮12  1 1 1 −1 −1   0 0  ␮21  0 H0:  = , 2 0 0 1 1 −1 −1  ␮22  0   experiment is given in Table 1.7. The structural matrix is ␮31 ␮32 coded as follows:where p is the number of levels of treatment A and h is the 1, if an observation is in a1 b1number of cell means. In order for the null hypothesis x1 = 0, otherwiseCA ␮ = 0 to be testable, the CA matrix must be of full rowrank. This means that each row of CA must be linearly 1, if an observation is in a1 b2 x2 =independent of every other row. The maximum number of 0, otherwisesuch rows is p − 1, which is why it is necessary to express thenull hypothesis as Equation 1.1 or 1.2. An estimator of the null 1, if an observation is in a2 b1 x3 =hypothesis, CA ␮ − 0, is incorporated in the formula for com- 0, otherwiseputing a sum of squares. For example, the estimator appears . . .as CA ␮ − 0 in the formula for the treatment A sum of squares ˆ 1, if an observation is in a3 b2 −1 −1 x6 = SSA = (CA ␮ − 0) [CA (X X) C A ] (CA ␮ − 0), ˆ ˆ (1.3) 0, otherwisewhere ␮ is a vector of sample cell means. Equation 1.3 sim- ˆ For the weight-loss data, the sum of squares for treatmentplifies to A is SSA = (CA ␮) [CA (X X)−1 C A ]−1 (CA ␮) ˆ ˆ SSA = (CA ␮) [CA (X X)−1 C A ]−1 (C A ␮) = 127.4038 ˆ ˆbecause 0 is a vector of zeros. In the formula, CA is a with p − 1 = 2 degrees of freedom.coefficient matrix that defines the null hypothesis, ␮ = ˆ The null hypothesis for treatment B is[(X X)−1 (X y)] = [Y ·11 , Y ·12 · · · Y ·23 ] , and X is a struc-tural matrix. The structural matrix for the weight-loss H0: ␮·1 = ␮·2 .

18 Experimental DesignAn equivalent null hypothesis that is used with the cell meansmodel is H0: ␮·1 − ␮·2 = 0.In terms of cell means, this hypothesis is expressed as ␮11 + ␮21 + ␮31 ␮ + ␮22 + ␮32 H0: − 12 = 0. Figure 1.9 Two interaction terms of the form ␮ jk − ␮ jk − ␮ j k + ␮ j k are 3 3 obtained from the crossed lines by subtracting the two ␮ijs connected by a dashed line from the two ␮ijs connected by a solid line.In matrix notation, the null hypothesis is The sum of squares for the A × B interaction is CB ␮ 0 (q−1)×h (q−1)×1  h×1  ␮11 SSA × B = (CA×B ␮) [CA ×B (X X)−1 C A×B ]−1 (CA×B ␮) ˆ ˆ  ␮12  = 27.7885   1 ␮  H0: [ 1 −1 1 −1 1 −1 ]  21  = [0], 3  ␮22    with ( p − 1)(q − 1) = 2 degrees of freedom. ␮31 ␮32 The within-cell sum of squares is SSWCELL = y y − ␮ (X y) = 360.7500, ˆwhere q is the number of levels of treatment B and h is thenumber of cell means. The sum of squares for treatment where y is the vector of weight-loss observations:B is [7 13 9 . . . 12]. The within-cell sum of squares has N − h = 29 − 6 = 23 degrees of freedom. SSB = (CB ␮) [CB (X X)−1 C B ]−1 (CB ␮) = 76.0500 ˆ ˆ The total sum of squares iswith q − 1 = 1 degree of freedom. SSTO = y y − y JyN −1 = 595.7931, The null hypothesis for the A × B interaction is where J is a 29 × 29 matrix of ones and N = 29, the number H0: ␮ jk − ␮ jk − ␮ j k + ␮ j k = 0 for all j and k. of weight-loss observations. The total sum of squares has N − 1 = 28 degrees of freedom. The analysis of the weight-loss data is summarized inFor the weight-loss data, the interaction null hypothesis is Table 1.8. The F statistics in Table 1.8 are identical to those in Table 1.6, where the regression model was used. H0: ␮11 − ␮12 − ␮21 + ␮22 = 0 The cell means model is extremely versatile. It can be ␮21 − ␮22 − ␮31 + ␮32 = 0 used when observations are missing and when entire cells are missing. It allows a researcher to test hypotheses aboutThe two rows of the null hypothesis correspond to the two any linear combination of population cell means. It has ansets of means connected by crossed lines in Figure 1.9. In important advantage over the regression model. With the cellmatrix notation, the null hypothesis is means model, there is never any ambiguity about the hypothesis that is tested because an estimator of the null hypothesis, CA×B ␮ 0 C ␮ − 0, appears in the formula for a sum of squares. Lack of ˆ ( p−1)(q−1)×h ( p−1)(q−1)×1  h×1  space prevents a discussion of the many other advantages of ␮11  ␮12  the model; the reader is referred to Kirk (1995, pp. 289–301,   413–431). However, before leaving the subject, the model 1 −1 −1 1 0 0  ␮21  0 H0 :   = . 0 0 1 −1 −1 1  ␮22  0 will be used to test a null hypothesis for weighted means.   ␮31 Occasionally, researchers collect data in which the sample ␮32 sizes are proportional to the population sizes. This might

Factorial Designs 19 TABLE 1.8 Analysis of Variance for the Weight-Loss Data (Observation Y322 is missing) Source SS df MS F p Treatment A (Diet) 127.4038 p−1=2 63.7019 4.06 .031 Treatment B (Exercise) 76.0500 q −1=1 76.0500 4.85 .038 A×B 27.7885 ( p − 1)(q − 1) = 2 13.8943 0.89 .426 Within cell 360.7500 N − h = 23 15.6848 Total 595.7931 N − 1 = 28occur, for example, in survey research. When cell ns are where MSWCELL is obtained from Table 1.8. The null hy-unequal, a researcher has a choice between computing un- pothesis is rejected. This is another example of the versatilityweighted means or weighted means. Unweighted means are of the cell means model. A researcher can test hypothesessimple averages of cell means. These are the means that were about any linear combination of population cell means.used in the previous analyses. Weighted means are weighted In most research situations, sample sizes are not propor-averages of cell means in which the weights are the sample tional to population sizes. Unless a researcher has a com-cell sizes, n jk . Consider again the weight-loss data in which pelling reason to weight the sample means proportional to theobservation Y322 is missing. Unweighted and weighted sam- sample sizes, unweighted means should be used.ple means for treatment level a2 where observation Y322 ismissing are, respectively, Randomized Block Factorial Design ␮21 + ␮22 ˆ ˆ 9.00 + 10.75 Next I describe a factorial design that is constructed from two ␮2· = ˆ = = 9.88 q 2 randomized block designs. The design is called a randomized n 21 ␮21 + n 22 ␮22 ˆ ˆ 5(9.00) + 4(10.75) block factorial design and is denoted by RBF-pq. The RBF-pq ˆ ␮2· = = = 9.78; n j· 9 design is obtained by combining the levels of an RB-p design with those of an RB-q design so that each level of the RB-pnj. is the number of observations in the jth level of treatment design appears once with each level of the RB-q design andA. The null hypothesis using weighted cell means for treat- vice versa. The design uses the blocking technique describedment A is in connection with an RB-p design to isolate variation attributable to a nuisance variable while simultaneously evaluating n 11 ␮11 + n 12 ␮12 n 21 ␮21 + n 22 ␮22 two or more treatments and associated interactions. H0: − =0 In discussing the weight-loss experiment, I hypothesized n 1· n 2· that ease of losing weight is related to the amount by which a n 21 ␮21 + n 22 ␮22 n 31 ␮31 + n 32 ␮32 − = 0. girl is overweight. If the hypothesis is correct, a researcher n 2· n 3· can improve on the CRF-32 design by isolating this nuisance variable. Suppose that instead of randomly assigning 30 girlsThe coefﬁcient matrix for computing SSA is to the six treatment combinations in the diet experiment, the 5 5 −5 −4 0 0 researcher formed blocks of six girls such that the girls in a 10 10 9 9 CA = , block are overweight by about the same amount. One way to 0 0 5 9 4 9 − 10 5 − 10 5 form the blocks is to rank the girls from the least to the most overweight. The six least overweight girls are assigned towhere the entries in CA are ±n jk /n j. and zero. The sum of block 1. The next six girls are assigned to block 2 and so on.squares and mean square for treatment A are, respectively, In this example, ﬁve blocks of dependent samples can be formed from the 30 participants. Once the girls have been SSA = (CA ␮) [CA (X X)−1 C A ]−1 (CA ␮) = 128.2375 ˆ ˆ assigned to the blocks, the girls in each block are randomly MSA = SSA/( p − 1) = 146.3556/(3 − 1) = 64.1188. assigned to the six treatment combinations. The layout for this experiment is shown in Figure 1.10.The F statistic and p value for treatment A are The classical model equation for the experiment is MSA 64.1188 Yi jk = ␮ + ␲i + ␣ j + ␤k + (␣␤) jk + (␣␤␲) jki F= = = 4.09 p = .030, MSWCELL 15.6848 (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , q),

20 Experimental Design Figure 1.10 Layout for a two-treatment randomized block factorial design (RBF-32 design). Each block contains six girls who are overweight by about the same amount. The girls in a block are randomly assigned to the six treatment combinations.where H0: ␮··1 = ␮··2 (Treatment B population means are equal.) Yi jk is the weight loss for the participant in Blocki H0: ␮· jk − ␮· jk − ␮· j k + ␮· j k = 0 for all j and k and treatment combination ajbk. (All A × B interaction effects equal zero.) ␮ is the grand mean of the six weight-loss population means. The hypothesis that the block population means are equal ␲i is the block effect for population i and is equal is of little interest because the blocks represent different to ␮i·· − ␮. It reflects the effect of the nuisance amounts by which the girls are overweight. variable in Blocki. The data for the RBF-32 design are shown in Table 1.9. The same data were analyzed earlier using a CRF-32 ␣j is the treatment effect for population aj and is design. Each block in Table 1.9 contains six girls who at equal to ␮· j· − ␮. It reflects the effect of diet aj. the beginning of the experiment were overweight by about ␤k is the treatment effect for population bk and is the same amount. The ANOVA for these data is given in equal to ␮··k − ␮. It reflects the effects of exer- Table 1.10. A comparison of Table 1.10 with Table 1.4 re- cise condition bk. veals that the RBF-32 design is more powerful than the (␣␤) jk is the interaction effect for populations aj and bk CRF-32 design. Consider, for example, treatment A. The and is equal to ␮· jk − ␮· j· − ␮··k − ␮. F statistic for the randomized block factorial design is (␣␤␲) jki is the residual error effect for treatment combi- F(2, 20) = 8.09, p = .003; the F for the completely random- nation ajbk and Blocki. ized factorial design is F(2, 24) = 4.25, p = .026. The randomized block factorial design is more powerful becauseThe design enables a researcher to test four null hypotheses: the nuisance variable—the amount by which participants are overweight— has been removed from the residual error H0: ␮1·· = ␮2·· = · · · = ␮5·· variance. A schematic partition of the total sum of squares (Block population means are equal.) and degrees of freedom for the two designs is shown in Figure 1.11. It is apparent from Figure 1.11 that the H0: ␮·1· = ␮·2· = ␮·3· SSRESIDUAL will always be smaller than the SSWCELL if (Treatment A population means are equal.) TABLE 1.10 Analysis of Variance for the Weight-Loss DataTABLE 1.9 Weight-Loss Data for the Diet (aj) and Exercise Source SS df MS F pConditions (bk) Blocks 209.3333 4 52.3333 6.43 .002 a1b1 a1b2 a2b1 a2b2 a3b1 a3b2 Treatments 234.1667 5Block1 5 4 7 5 8 13 Treatment A (Diet) 131.6667 2 65.8334 8.09 .003Block2 7 7 4 7 5 16 Treatment B (Exercise) 67.5000 1 67.5000 8.30 .009Block3 1 14 9 13 10 12 A×B 35.0000 2 17.5000 2.15 .142Block4 9 9 11 15 12 20 Residual 162.6667 20 8.1333Block5 13 11 14 10 15 19 Total 606.1667 29

Factorial Designs with Confounding 21 SSWCELL ϭ 372.0 pq(n Ϫ 1) ϭ 24 SSWCELL ϭ 372.0 pq(n Ϫ 1) ϭ 24 SSRES ϭ 162.7 (n Ϫ 1)(pq Ϫ 1) ϭ 20 Figure 1.11 Schematic partition of the total sum of squares and degrees of freedom for CRF-32 and RBF-32 designs. The shaded rectangles indicate the sums of squares that are used to compute the error variance for each design: MSWCELL = SSWCELL/ pq(n − 1) and MSRES = SSRES/(n − 1)( pq − 1). If the nuisance variable (SSBL) in the RBF-32 design accounts for an appreciable portion of the total sum of squares, the design will have a smaller error variance and, hence, greater power than the CRF-32 design.the SSBLOCKS is greater than zero. The larger the SS- mentation in which the levels of, say, treatment A areBLOCKS in a randomized block factorial design are, the applied to relatively large plots of land—the whole plots.greater the reduction in the SSRESIDUAL. The whole plots are then split or subdivided, and the levels of treatment B are applied to the subplots within each whole plot. A two-treatment split-plot factorial design is constructedFACTORIAL DESIGNS WITH CONFOUNDING by combining two building block designs: a completely randomized design having p levels of treatment A and a random-Split-Plot Factorial Design ized block design having q levels of treatment B. The assign-As we have just seen, an important advantage of a random- ment of participants to the treatment combinations is carriedized block factorial design relative to a completely random- out in two stages. Consider the weight-loss experiment again.ized factorial design is greater power. However, if either p or Suppose that we ranked the 30 participants from least to mostq in a two-treatment randomized block factorial design is overweight. The participants ranked 1 and 2 are assigned tomoderately large, the number of treatment combinations in block 1, those ranked 3 and 4 are assigned to block 2, andeach block can be prohibitively large. For example, an RBF- so on. This procedure produces 15 blocks each containing two45 design has blocks of size 4 × 5 = 20. Obtaining blocks girls who are similar with respect to being overweight. In thewith 20 matched participants or observing each participant ﬁrst stage of randomization the 15 blocks of girls are randomly20 times is generally not feasible. In the late 1920s Ronald assigned to the three levels of treatment A with ﬁve blocks inA. Fisher and Frank Yates addressed the problem of prohib- each level. In the second stage of randomization the two girlsitively large block sizes by developing confounding schemes in each block are randomly assigned to the two levels of treat-in which only a portion of the treatment combinations in an ment B. An exception to this randomization procedure must beexperiment are assigned to each block. Their work was made when treatment B is a temporal variable such as succes-extended in the 1940s by David J. Finney (1945, 1946) and sive learning trials or periods of time. Trial 2, for example, can-Oscar Kempthorne (1947). One design that achieves a re- not occur before Trial 1.duction in block size is the two-treatment split-plot factorial The layout for a split-plot factorial design with threedesign. The term split-plot comes from agricultural experi- levels of treatment A and two levels of treatment B is

22 Experimental Design Figure 1.12 Layout for a two-treatment split-plot factorial design (SPF-3·2 design). The 3n blocks are randomly assigned to the p = 3 levels of treatment A with the restriction that n blocks are assigned to each level of A. The n blocks assigned to each level of treatment A constitute a group of blocks. In the second stage of randomization, the two matched participants in a block are randomly assigned to the q = 2 levels of treatment B.shown in Figure 1.12. Treatment A is called a between-blocks treatment A; a different and usually much smaller errortreatment; B is a within-blocks treatment. The designation variance, MSRESIDUAL, is used to test treatment B and thefor a two-treatment split-plot factorial design is SPF- p·q. A × B interaction. As a result, the power of the tests for BThe p preceding the dot denotes the number of levels of the and the A × B interaction is greater than that for A. Hence,between-blocks treatment; the q after the dot denotes the a split-plot factorial design is a good design choice if anumber of levels of the within-blocks treatment. Hence, researcher is more interested in treatment B and the A × Bthe design in Figure 1.12 is an SPF-3·2 design. interaction than in treatment A. When both treatments and An RBF-32 design contains 3 × 2 = 6 treatment combi- the A × B interaction are of equal interest, a randomizednations and has blocks of size six. The SPF-3·2 design in block factorial design is a better choice if the larger blockFigure 1.12 contains the same six treatment combinations, size is acceptable. If a large block size is not acceptable andbut the block size is only two. The advantage of the split- the researcher is primarily interested in treatments A and B,plot factorial—smaller block size—is achieved by con- an alternative design choice is the confounded factorialfounding groups of blocks with treatment A. Consider the design. This design, which is described later, achieves asample means Y ·1· , Y ·2· , and Y ·3· in Figure 1.12. The differ- reduction in block size by confounding groups of blocksences among the means reﬂect the differences among the with the A × B interaction. As a result, tests of treatmentsthree groups as well as the differences among the three A and B are more powerful than the test of the A × Blevels of treatment A. To put it another way, we cannot tell interaction.how much of the differences among the three sample means Earlier, an RBF-32 design was used for the weight-lossis attributable to the differences among Group1, Group2, and experiment because the researcher was interested in tests ofGroup3 and how much is attributable to the differences treatments A and B and the A × B interaction. For purposesamong treatments levels a1, a2, and a3. For this reason, the of comparison, I analyze the same weight-loss data as ifthree groups and treatment A are said to be completely an SPF-3·2 design had been used even though, as we willconfounded. see, this is not a good design choice. But ﬁrst I describe the The use of confounding to reduce the block size in an classical model equation for a two-treatment split-plot facto-SPF- p·q design involves a tradeoff that needs to be made rial design.explicit. The RBF-32 design uses the same error variance, The classical model equation for the weight-loss experi-MSRESIDUAL, to test hypotheses for treatments A and B ment isand the A × B interaction. The two-treatment split-plotfactorial design, however, uses two error variances. Yi jk = ␮ + ␣ j + ␲i( j) + ␤k + (␣␤) jk + (␤␲)ki( j)MSBLOCKS within A, denoted by MSBL(A), is used to test (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , q),

Factorial Designs with Confounding 23where TABLE 1.11 Weight-Loss Data for the Diet (aj) and Exercise Conditions (bk) Yi jk is the weight loss for the participant in Block i( j) Treatment Treatment and treatment combination aj bk. Level b1 Level b2 Ά ␮ is the grand mean of the six weight-loss popula- Block1 5 4 Block2 7 7 tion means. Group1 a1 Block3 1 14 ␣j is the treatment effect for population aj and is Block4 9 9 equal to ␮· j· − ␮. It reﬂects the effect of diet aj. Block5 13 11 Ά ␲i( j) is the block effect for population i and is equal to Block6 7 5 Block7 4 7 ␮i j· − ␮· j· . The block effect is nested within aj. Group2 a2 Block8 9 13 ␤k is the treatment effect for population bk and is Block9 11 15 equal to ␮··k − ␮. It reﬂects the effects of exer- Block10 14 10 Ά cise condition bk. Block11 8 13 Block12 5 16 (␣␤) jk is the interaction effect for populations aj and bk Group3 a3 Block13 10 12 and is equal to ␮· jk − ␮· j· − ␮··k + ␮. Block14 12 20 Block15 15 19 (␤␲)ki( j) is the residual error effect for treatment level bk and Blocki( j) and is equal to Yi jk − ␮ − ␣ j − ␲i( j) − ␤k − (␣␤) jk . effectiveness of the three diets, the SPF-3·2 design is a poor The design enables a researcher to test three null hypotheses: choice. However, the SPF-3·2 design fares somewhat better if one’s primary interests are in treatment B and the A × B H0: ␮·1· = ␮·2· = ␮·3· interaction: (Treatment A population means are equal.) Treatment B H0: ␮··1 = ␮··2 67.5000/1 67.5000 (Treatment B population means are equal.) SPF-3·2 design F= = = 6.23 p = .028 130.0000/12 10.8333 H0: ␮· jk − ␮· jk − ␮· j k + ␮· j k = 0 for all j and k 67.5000/1 67.5000 CRF-32 design F= = = 4.35 p = .048 (All A × B interaction effects equal zero.) 372.0000/24 15.5000 67.5000/1 67.5000 RBF-32 design F= = = 8.30 p = .009 The weight-loss data from Tables 1.2 and 1.9 are recasts in 162.6667/20 8.1333the form of an SPF-3·2 design in Table 1.11. The ANOVA A × B interactionfor these data is given in Table 1.12. The null hypothesis for 35.0000/2 17.5000treatment B can be rejected. However, the null hypothesis SPF-3·2 design F= = = 1.62 p = .239 130.0000/12 10.8333for treatment A and the A × B interaction cannot be rejected. 35.0000/2 17.5000The denominator of the F statistic for treatment A CRF-32 design F= = = 1.13 p = .340 372.0000/24 15.5000[MSBL(A) = 20.1667] is almost twice as large as the de- 35.0000/2 17.5000nominator for the tests of B and A × B (MSRES = 10.8333). RBF-32 design F= = = 2.15 p = .142 162.6667/20 8.1333A feeling for the relative power of the test of treatment A forthe SPF-3·2, CRF-32, and RBF-32 designs can be obtainedby comparing their F statistics and p values: TABLE 1.12 Analysis of Variance for the Weight-Loss Data Source SS df MS F p Treatment A 1. Between blocks 373.6667 14 131.6667/2 65.8334 2. Treatment A (Diet) 131.6667 2 65.8334 [2͞3]a 3.26 .074 SPF-3·2 design F= = = 3.26 p = .074 3. Blocks within A 242.0000 12 20.1667 242.0000/12 20.1667 4. Within blocks 232.5000 15 131.6667/2 65.8334 CRF-32 design F= = = 4.25 p = .026 5. Treatment B 67.5000 1 67.5000 [5͞7] 6.23 .028 372.0000/24 15.5000 (Exercise) 131.6667/2 65.8334 6. A × B 35.0000 2 17.5000 [6͞7] 1.62 .239 RBF-32 design F= = = 8.09 p = .003 162.6667/20 8.1333 7. Residual 130.0000 12 10.8333 8. Total 606.1667 29For testing treatment A, the SPF-3·2 design is the least a The fraction [2͞3] indicates that the F statistic was obtained by dividing thepowerful. Clearly, if one’s primary interest is in the mean square in row two by the mean square in row three.

24 Experimental Design The SPF-3·2 design is the ﬁrst design I have described A and B. The RBF-pq design is a better choice if blocks of sizethat involves two different building block designs: a CR-p p × q are acceptable. If this block size is too large, an alterna-design and an RB-q design. Also, it is the ﬁrst design that tive choice is a two-treatment confounded factorial design.has two error variances: one for testing the between-blocks This design confounds an interaction with groups of blocks.effects and another for testing the within-blocks effects. A As a result, the test of the interaction is less powerful than testsweighted average of the two error variances is equal to of treatments A and B. Confounded factorial designs are con-MSWCELL in a CRF-pq design, where the weights are the structed from either a randomized block design or a Latindegrees of freedom of the two error variances. This can be square design. The designs are denoted by, respectively,shown using the mean squares from Tables 1.4 and 1.12: RBCF-pk and LSCF-pk, where RB and LS identify the building block design, C indicates that the interaction is completely p(n − 1)MSBL(A) + p(n − 1)(q − 1)MSRESIDUAL confounded with groups of blocks, F indicates a factorial de- p(n − 1) + p(n − 1)(q − 1) sign, and pk indicates that the design has k treatments each having p levels. The simplest randomized block confounded = MSWCELL factorial design has two treatments with two levels each. Con- sider the RBCF-22 design in Figure 1.14. The A × B interac- 3(5 − 1)20.1667 + 3(5 − 1)(2 − 1)10.8333 tion is completely confounded with Group1 and Group2, as I = 15.5000 3(5 − 1) + 3(5 − 1)(2 − 1) will now show. An interaction effect for treatments A and B has the general form ␮ jk − ␮ jk − ␮ j k + ␮ j k . Let ␮i jkzA schematic partition of the total sum of squares and degrees denote the population mean for the ith block, jth level of A,of freedom for the CRF-32 and SPF-3·2 designs is shown in kth level of B, and zth group. For the design in Figure 1.14,Figure 1.13. the A × B interaction effect is ␮·111 − ␮·122 − ␮·212 + ␮·221Confounded Factorial Designs orAs we have seen, an SPF-p·q design is not the best designchoice if a researcher’s primary interest is in testing treatments (␮·111 + ␮·221 ) − (␮·122 + ␮·212 ). SSWCELL ϭ 372.0 pq(n Ϫ 1) ϭ 24 SSBL(A) ϭ 242.0 p(n Ϫ 1) ϭ 12 SSRES ϭ 130.0 p(n Ϫ 1)(q Ϫ 1) ϭ 12 Figure 1.13 Schematic partition of the total sum of squares and degrees of freedom for CRF-32 and SPF-3·2 designs. The shaded rectangles indicate the sums of squares that are used to compute the error variance for each design. The SPF-3·2 design has two error variances: MSBL(A) = SSBL(A)/ p(n − 1) is used to test treatment A; MSRES = SSRES/ p(n − 1)(q − 1) is used to test treatment B and the A × B interaction. The within-blocks error variance, MSRES, is usually much smaller than the between-blocks error variance, MSBL(A). As a result, tests of treatment B and the A × B interaction are more powerful than the test of treatment A.

Factorial Designs with Confounding 25 Figure 1.14 Layout for a two-treatment randomized block confounded factorial design (RBCF-22 design). A score in the ith block, jth level of treatment A, kth level of treatment B, and zth group is denoted by Yijkz.The difference between the effects of Group1 and Group2, Fractional Factorial Designs (␮·111 − ␮·221 ) − (␮·122 + ␮·212 ), Two kinds of confounding have been described thus far: group-treatment confounding in an SPF- p·q design andinvolves the same contrast among means as the A × B inter- group-interaction confounding in an RBCF-pk design. A thirdaction effect. Hence, the two sets of effects are completely form of confounding, treatment-interaction confounding, isconfounded because we cannot determine how much of the used in a fractional factorial design. This kind of confoundingdifference (␮·111 + ␮·221 ) − (␮·122 + ␮·212 ) is attributable to reduces the number of treatment combinations that must bethe A × B interaction and how much is attributable to the included in a multitreatment experiment to some fraction—difference between Group1 and Group2. , , , , , 1 1 1 1 1 2 3 4 8 9 and so on—of the total number of treatment The RBCF-pk design, like the SPF- p·q design, has two combinations. A CRF-22222 design has 32 treatment combi-error variances: one for testing the between-blocks effects nations. By using a 1 or 1 fractional factorial design, the 2 4and a different and usually much smaller error variance for number of treatment combinations that must be includedtesting the within-blocks effects. In the RBCF-pk design, in the experiment can be reduced to, respectively,treatments A and B are within-block treatments and are eval- 1 2 (32) = 16 or 1 (32) = 8. 4uated with greater power than the A × B interaction that is a The theory of fractional factorial designs was developedbetween-block component. Researchers need to understand for 2k and 3k designs by Finney (1945, 1946) and extended bythe tradeoff that is required when a treatment or interaction is Kempthorne (1947) to designs of the type pk, where p is aconfounded with groups to reduce the size of blocks. The prime number that denotes the number of levels of eachpower of the test of the confounded effects is generally less treatment and k denotes the number of treatments. Fractionalthan the power of tests of the unconfounded effects. Hence, if factorial designs are most useful for pilot experiments andpossible, researchers should avoid confounding effects that exploratory research situations that permit follow-up experi-are the major focus of an experiment. Sometimes, however, ments to be performed. Thus, a large number of treatments,confounding is necessary to obtain a reasonable block size. If typically six or more, can be investigated efﬁciently in anthe power of the confounded effects is not acceptable, the initial experiment, with subsequent experiments designed topower always can be increased by using a larger number of focus on the most promising independent variables.blocks. Fractional factorial designs have much in common with One of the characteristics of the designs that have been confounded factorial designs. The latter designs achieve adescribed so far is that all of the treatment combinations reduction in the number of treatment combinations that mustappear in the experiment. The fractional factorial design that be included in a block. Fractional factorial designs achieve ais described next does not share this characteristic. As the reduction in the number of treatment combinations in the ex-name suggests, a fractional factorial design includes only a periment. The reduction in the size of an experiment comes atfraction of the treatment combinations of a complete factorial a price, however. Considerable ambiguity may exist in inter-design. preting the results of an experiment when the design includes

26 Experimental Designonly one half or one third of the treatment combinations.Ambiguity occurs because two or more names can be givento each sum of squares. For example, a sum of squares mightbe attributed to the effects of treatment A and the BCDE in-teraction. The two or more names given to the same sum ofsquares are called aliases. In a one-half fractional factorialdesign, all sums of squares have two aliases. In a one-thirdfractional factorial design, all sums of squares have threealiases, and so on. Treatments are customarily aliased withhigher-order interactions that are assumed to equal zero. Thishelps to minimize but does not eliminate ambiguity in inter-preting the outcome of an experiment. Fractional factorial designs are constructed from com-pletely randomized, randomized block, and Latin square de-signs and denoted by, respectively, CRFF-pk–1, RBFF-pk–1,and LSFF-pk. Let’s examine the designation CRFF-25–1. The Figure 1.15 Layout for a three-treatment completely randomized fractional factorial design (CRFF-23−1 design). A score for the ith participant inletters CR indicate that the building block design is a com- treatment combination aj bk cl is denoted by Yi jkl . The 4n participants are ran-pletely randomized design; FF indicates that it is a fractional domly assigned to the treatment combinations with the restriction that n par-factorial design; and 25 indicates that each of the five treat- ticipants are assigned to each combination. The mean for the participants in the four groups is denoted by Y ·111 , Y ·122 , Y ·212 , and Y ·221 .ments has two levels. The −1 in 25−1 indicates that the designis a one-half fraction of a complete 25 factorial design. Thisfollows because the designation for a one-half fraction of a 25 A × C interaction are two names for another source of varia-factorial design can be written as 1 25 = 2−1 25 = 25−1 . A 2 tion, as are A × B and C. Hence, the F statisticsone-fourth fraction of a 25 factorial design is denoted byCRFF- p5−2 because 1 25 = 22 25 = 2−2 25 = 25−2 . 1 MSA MSB × C 4 F= and F = To conserve space, I describe a small CRFF-23−1 design. MSWCELL MSWCELLA fractional factorial design with only three treatments is un-realistic, but the small size simplifies the presentation. The test the same sources of variation. If F = MSA/MSWCELL islayout for the design is shown in Figure 1.15. On close in- significant, a researcher does not know whether it is becausespection of Figure 1.15, it is apparent that the CRFF-23−1 de- treatment A is significant, the B × C interaction is signifi-sign contains the four treatment combinations of a CRF-22 cant, or both.design. For example, if we ignore treatment C, the design in At this point you are probably wondering why anyoneFigure 1.15 has the following combinations of treatments A would use such a design—after all, experiments are supposedand B: a1b1, a1b2, a2b1, and a2b2. The correspondence be- to help us resolve ambiguity, not create it. In defense of frac-tween the treatment combinations of the CRF-22 and CRFF- tional factorial designs, recall that they are typically used in23−1 designs suggests a way to compute sums of squares for exploratory research situations where a researcher is inter-the latter design—ignore treatment C and analyze the data as ested in six or more treatments. In addition, it is customary toif they came from a CRF-22 design. limit all treatments to either two or three levels, thereby in- Earlier, I observed that all sums of squares in a one-half creasing the likelihood that higher order interactions arefractional factorial design have two aliases. It can be shown small relative to treatments and lower order interactions.(see Kirk, 1995, pp. 667–670) that the alias pattern for the Under these conditions, if a source of variation labeled treat-design in Figure 1.15 is as follows: ment A and its alias, the BCDEF interaction, is significant, it is reasonable to assume that the significance is probably dueAlias (Name) Alias (Alternative name) to the treatment rather than the interaction. Continuing the defense, a fractional factorial design can A B ×C B A×C dramatically decrease the number of treatment combinations A×B C that must be run in an experiment. Consider a researcher who is interested in determining whether any of six treat-The labels—treatment A and the B × C interaction—are two ments having two levels each is significant. An experimentnames for the same source of variation. Similarly, B and the with six treatments and two participants assigned to each

Hierarchical Designs 27treatment combination would have 64 combinations and re- A.” A hierarchical design has at least one nested treatment;quire 2 × 64 = 128 participants. By using a one-fourth frac- the remaining treatments are either nested or crossed.tional factorial design, CRFF-26−2 design, the researcher canreduce the number of treatment combinations in the exper- Hierarchical Designs With One oriment from 64 to 16 and the number of participants from Two Nested Treatments128 to 32. Suppose that the researcher ran the 16 treatmentcombinations and found that none of the F statistics in the Hierarchical designs are constructed from completelyfractional factorial design is significant. The researcher has randomized or randomized block designs. A two-treatmentanswered the research questions with one fourth of the effort. hierarchical design that is constructed from CR-p and CR-qOn the other hand, suppose that F statistics for treatments C designs is denoted by CRH-pq(A), where pq(A) indicates thatand E and associated aliases are significant. The researcher the design has p levels of treatment A and q levels ofhas eliminated four treatments (A, B, D, F), their aliases, and treatment B that are nested in treatment A. A comparison ofcertain other interactions from further consideration. The nested and crossed treatments for a CRH-24(A) design and aresearcher can then follow up with a small experiment to CRF 22 design is shown in Figure 1.16. Experiments withdetermine which aliases are responsible for the significant one or more nested treatments are well suited to research inF statistics. education, industry, and the behavioral and medical sciences. In summary, the main advantage of a fractional factorial Consider an example from education in which two ap-design is that it enables a researcher to investigate efficiently proaches to introducing long division (treatments levels a1a large number of treatments in an initial experiment, with and a2) are to be evaluated. Four schools (treatments levelssubsequent experiments designed to focus on the most b1, . . . , b4 ) are randomly assigned to the two levels of treat-promising lines of investigation or to clarify the interpreta- ment A, and eight teachers (treatment levels c1, . . . , c8) aretion of the original analysis. Many researchers would con- randomly assigned to the four schools. Hence, this is a three-sider ambiguity in interpreting the outcome of the initial treatment CRH-24(A)8(AB) design: schools, treatment B(A),experiment a small price to pay for the reduction in experi- are nested in treatment A and teachers, treatment C(AB), aremental effort. nested in both A and B. A diagram of the nesting of treatments The description of confounding in a fractional factorial for this design is shown in Figure 1.17.design completes a cycle. I began the cycle by describing A second example is from medical science. A researchergroup-treatment confounding in a split-plot factorial design. wants to compare the efficacy of a new drug denoted by a1I then described group-interaction confounding in a con- with the currently used drug denoted by a2. Four hospitals,founded factorial design, and, finally, treatment-interaction treatment B(A), are available to participate in the experiment.confounding in a fractional factorial design. The three forms Because expensive equipment is needed to monitor the sideof confounding achieve either a reduction in the size of a effects of the new drug, it was decided to use the new drug inblock or the size of an experiment. As we have seen, con- two of the four hospitals and the current drug in the other twofounding always involves a tradeoff. The price we pay for re- hospitals. The hospitals are randomly assigned to the drugducing the size of a block or an experiment is lower power in conditions with the restriction that two hospitals are assignedtesting a treatment or interaction or ambiguity in interpreting to each drug. Patients are randomly assigned to the hospitals.the outcome of an experiment. In the next section I describe Panel A of Figure 1.16 illustrates the nesting of treatment Bhierarchical designs in which one or more treatments are within treatment A.nested.HIERARCHICAL DESIGNSAll of the multitreatment designs that have been discussedso far have had crossed treatments. Treatments A and B arecrossed, for example, if each level of treatment B appearsonce with each level of treatment A and vice versa. Treatment Figure 1.16 Comparison of designs with nested and crossed treatments. In panel A, treatment B(A) is nested in treatment A because b1 and b2 appearB is nested in treatment A if each level of treatment B appears only with a1 while b3 and b4 appear only with a2. In panel B, treatments A andwith only one level of treatment A. The nesting of treatment B are crossed because each level of treatment B appears once and only onceB within treatment A is denoted by B(A) and is read “B within with each level of treatment A and vice versa.

28 Experimental Design Notice that because treatment B(A) is nested in treatment A, the model equation does not contain an A × B interaction term. This design enables a researcher to test two null hypotheses: H0: ␮1· = ␮2· (Treatment A population means are equal.) H0: ␮11 = ␮12 or ␮23 = ␮24Figure 1.17 Diagram of a three-treatment completely randomized hierar- (Treatment B(A) population means are equal.)chical design (CRH-24(A)8(AB) design). The four schools, b1, . . . , b4, arenested in the two approaches to introducing long division, treatment A. Theeight teachers, c1, . . . , c8, are nested in the schools and teaching approaches. If the second null hypothesis is rejected, the researcher canStudents are randomly assigned to the pq( j) r( jk) = (2)(2)(2) = 8 treatment conclude that the dependent variable is not the same for thecombinations with the restriction that n students are assigned to each populations represented by hospitals b1 and b2, that the de-combination. pendent variable is not the same for the populations represented by hospitals b3 and b4, or both. However, the test of treatment B(A) does not address the question of whether, for As is often the case, the nested treatments in the drug and example, ␮11 = ␮23 because hospitals b1 and b3 were as-educational examples resemble nuisance variables. The re- signed to different levels of treatment A.searcher in the drug example probably would not conductthe experiment just to find out whether the dependent vari-able is different for the two hospitals assigned to drug a1 or Hierarchical Design With Crossed andthe hospitals assigned to a2. The important question for the Nested Treatmentsresearcher is whether the new drug is more effective than the In the educational example, treatments B(A) and C(AB) werecurrently used drug. Similarly, the educational researcher both nested treatments. Hierarchical designs with three orwants to know whether one approach to teaching long divi- more treatments can have both nested and crossed treatments.sion is better than the other. The researcher might be inter- Consider the partial hierarchical design shown in Figure 1.18.ested in knowing whether some schools or teachers perform The classical model equation for this design isbetter than others, but this is not the primary focus of the re-search. The distinction between a treatment and a nuisance Yi jkl = ␮ + ␣ j + ␤k + ␥l(k) + (␣␤) jk + (␣␥) jl(k) + ␧i( jkl)variable is in the mind of the researcher—one researcher’snuisance variable can be another researcher’s treatment. (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , q; l = 1, . . . , r), The classical model equation for the drug experiment is where Yi jk = ␮ + ␣ j + ␤k( j) + ␧i( jk) Yi jkl is an observation for participant i in treatment (i = 1, . . . , n; j = 1, . . . , p; k = 1, . . . , q), levels aj, bk, and cl(k). ␮ is the grand mean of the population means.where ␣j is the treatment effect for population aj and is Yi jk is an observation for participant i in treatment lev- equal to ␮ j·· − ␮. els aj and bk( j). ␤k is the treatment effect for population bk and is ␮ is the grand mean of the population means. equal to ␮·k· − ␮. ␣j is the treatment effect for population aj and is ␥l(k) is the treatment effect for population cl(k) and is equal to ␮ j. − ␮. It reflects the effect of drug aj. equal to ␮·kl − ␮·k· . ␤k( j) is the treatment effect for population bk( j) and is (␣␤) jk is the interaction effect for populations aj and bk equal to ␮ jk − ␮ j. . It reflects the effects of hospi- and is equal to ␮ jk· − ␮ jk· − ␮ j k· + ␮ j k· . tal bk( j) that is nested in aj. (␣␥) jl(k) is the interaction effect for populations aj and ␧i( jk) is the within-cell error effect associated with Yi jk cl(k) and is equal to ␮ jkl − ␮ jkl − ␮ j kl + ␮ j kl . and is equal to Yi jk − ␮ − ␣ j − ␤k( j) . It reflects ␧i( jkl) is the within-cell error effect associated with all effects not attributable to treatment levels aj Yi jkl and is equal to Yi jkl − ␮ − ␣ j − ␤k − and bk( j). ␥l(k) − (␣␤) jk − (␣␥) jl(k) .

Experimental Designs with a Covariate 29 blocks, and refinement of techniques for measuring a dependent variable. In this section, I describe an alternative approach to reducing error variance and minimizing the effects of nuisance variables. The approach is called analysis of covariance (ANCOVA) and combines regression analysis and analysis of variance. Analysis of covariance involves measuring one or more concomitant variables (also called covariates) in addition toFigure 1.18 Diagram of a three-treatment completely randomized partialhierarchical design (CRPH-pqr(B) design). The letter P in the designation the dependent variable. The concomitant variable representsstands for “partial” and indicates that not all of the treatments are nested. In a source of variation that was not controlled in the experi-this example, treatments A and B are crossed; treatment C(B) is nested in ment and one that is believed to affect the dependent variable.treatment B because c1 and c2 appear only with b1 while c3 and c4 appear onlywith b2. Treatment C(B) is crossed with treatment A because each level of Analysis of covariance enables a researcher to (a) removetreatment C(B) appears once and only once with each level of treatment A that portion of the dependent-variable error variance that isand vice versa. predictable from a knowledge of the concomitant variable, thereby increasing power, and (b) adjust the dependent variable so that it is free of the linear effects attributable to theNotice that because treatment C(B) is nested in treatment B, concomitant variable, thereby reducing bias.the model equation does not contain B × C and A × B × C Consider an experiment with two treatment levels a1 and a2.interaction terms. The dependent variable is denoted by Yi j , the concomitant This design enables a researcher to test five null variable by X i j . The relationship between X and Y for a1 and a2hypotheses: might look like that shown in Figure 1.19. Each participant in the experiment contributes one data point to the figure as de- H0: ␮1·· = ␮2·· termined by his or her X i j and Yi j scores. The points form two (Treatment A population means are equal.) scatter plots—one for each treatment level. These scatter plots H0: ␮·1· = ␮·2· are represented in Figure 1.19 by ellipses. Through each ellip- (Treatment B population means are equal.) sis a line has been drawn representing the regression of Y on X. H0: ␮·11 = ␮·12 or ␮·23 = ␮·24 In the typical ANCOVA model it is assumed that each regres- (Treatment C(B) population means are equal.) sion line is a straight line and that the lines have the same slope. H0: ␮ jk − ␮ jk · − ␮ j k· + ␮ j k · = 0 for all j and k The size of the error variance in ANOVA is determined by the (All A × B interaction effects equal zero.) dispersion of the marginal distributions (see Figure 1.19). The size of the error variance in ANCOVA is determined by the H0: ␮ jkl − ␮ jkl − ␮ j kl + ␮ j kl = 0 for all j, k, and l (All A × C(B) interaction effects equal zero.)If the last null hypothesis is rejected, the researcher knowsthat treatments A and C interact at one or more levels of treat-ment B. Lack of space prevents me from describing other partialhierarchical designs with different combinations of crossedand nested treatments. The interested reader is referredto the extensive treatment of these designs in Kirk (1995,chap. 11).EXPERIMENTAL DESIGNS WITH A COVARIATE Figure 1.19 Scatter plots showing the relationship between the dependentThe emphasis so far has been on designs that use experimen- variable, Y, and concomitant variable, X, for the two treatment levels. Thetal control to reduce error variance and minimize the effects size of the error variance in ANOVA is determined by the dispersion of the marginal distributions. The size of the error variance in ANCOVA is deter-of nuisance variables. Experimental control can take vari- mined by the dispersion of the conditional distributions. The higher the cor-ous forms such as random assignment of participants to treat- relation between X and Y is, the greater the reduction in the error variancement levels, stratification of participants into homogeneous due to using analysis of covariance.

30 Experimental Design Figure 1.20 Analysis of covariance adjusts the concomitant-variable means, X ·1 and X ·2 , so that they equal the concomitant-variable grand mean, X ·· . When the concomitant-variable means differ, the absolute difference between adjusted means for the dependent variable, |Y adj·1 − Y adj·2 |, can be less than that between unadjusted means, |Y ·1 − Y ·2 |, as in panels A and B, or larger, as in panel C.dispersion of the conditional distributions (see Figure 1.19). groups differ. Random assignment is the best safeguardThe higher the correlation between X and Y, in general, the against unanticipated nuisance variables. In the long run,narrower are the ellipses and the greater is the reduction in the over many replications of an experiment, random assignmenterror variance due to using analysis of covariance. will result in groups that are, at the time of assignment, simi- Figure 1.19 depicts the case in which the concomitant- lar on all nuisance variables.variable means, X ·1 and X ·2, are equal. If participants are A second situation in which analysis of covariance is oftenrandomly assigned to treatment levels, in the long run the con- used is when it becomes apparent that even though randomcomitant-variable means should be equal. However, if random assignment was used, the participants were not equivalent onassignment is not used, differences among the means can be some relevant variable at the beginning of the experiment.sizable, as in Figure 1.20. This ﬁgure illustrates what happens For example, in an experiment designed to evaluate the ef-to the dependent variable means when they are adjusted for dif- fects of different drugs on stimulus generalization in rats, theferences in the concomitant-variable means. In panels A and B researcher might discover that the amount of stimulus gener-the absolute difference between adjusted dependent-variable alization is related to the number of trials required to estab-means |Y adj·1 − Y adj·2 | is smaller than that between unadjusted lish a stable bar-pressing response. Analysis of covariancemeans |Y ·1 − Y ·2 |. In panel C the absolute difference between can be used to adjust the generalization scores for differencesadjusted means is larger than that between unadjusted means. among the groups in learning ability. Analysis of covariance is often used in three kinds of Analysis of covariance is useful in yet another researchresearch situations. One situation involves the use of intact situation in which differences in a relevant nuisance variablegroups with unequal concomitant-variable means and is com- occur during the course of an experiment. Consider the ex-mon in educational and industrial research. Analysis of periment to evaluate two approaches toward introducing longcovariance statistically equates the intact groups so that their division that was described earlier. It is likely that the dailyconcomitant variable means are equal. Unfortunately, a re- schedules of the eight classrooms provided more study peri-searcher can never be sure that the concomitant variable used ods for students in some classes than in others. It would befor the adjustment represents the only nuisance variable or difﬁcult to control experimentally the amount of time avail-the most important nuisance variable on which the intact able for studying long division. However, each student could

References 31record the amount of time spent studying long division. If test Cohen, J. (1988). Statistical power analysis for the behavioralscores on long division were related to amount of study time, sciences (2nd ed.). Hillsdale, NJ: Erlbaum.analysis of covariance could be used to adjust the scores for Federer, W. T. (1955). Experimental design: Theory and applica-differences in this nuisance variable. tion. New York: Macmillan. Statistical control and experimental control are not mutu- Finney, D. J. (1945). The fractional replication of factorial arrange-ally exclusive approaches for reducing error variance and ments. Annals of Eugenics, 12, 291–301.minimizing the effects of nuisance variables. It may be con- Finney, D. J. (1946). Recent developments in the design of fieldvenient to control some variables by experimental control experiments. III. Fractional replication. Journal of Agriculturaland others by statistical control. In general, experimental Science, 36, 184–191.control involves fewer assumptions than does statistical Fisher, R. A. (1918). The correlation between relatives on thecontrol. However, experimental control requires more infor- supposition of Mendelian inheritance. Transactions of the Royalmation about the participants before beginning an experi- Society of Edinburgh, 52, 399–433.ment. Once data collection has begun, it is too late to assign Fisher, R. A. (1925). Statistical methods for research workers.participants randomly to treatment levels or form blocks of Edinburgh: Oliver & Boyd.dependent participants. The advantage of statistical control is Fisher, R. A. (1926). The arrangement of field experiments. Journalthat it can be used after data collection has begun. Its disad- of the Ministry of Agriculture, 33, 503–513.vantage is that it involves a number of assumptions such as a Fisher, R. A. (1935). The design of experiments. Edinburgh: Oliverlinear relationship between the dependent and concomitant & Boyd.variables and equal within-groups regression coefficients that Fisher, R. A., & Mackenzie, W. A. (1923). Studies in crop variation.may prove untenable in a particular experiment. II. The manurial response of different potato varieties. Journal of In this chapter I have given a short introduction to those Agricultural Science, 13, 311–320.experimental designs that are potentially the most useful in Harris, R. J. (1994). ANOVA: An analysis of variance primer. Itasca,the behavioral and social sciences. For a full discussion of the IL: Peacock.designs, the reader is referred to the many excellent books on Hays, W. L. (1963). Statistics. Fort Worth, TX: Holt, Rinehart andexperimental design: Bogartz (1994), Cobb (1998), Harris Winston.(1994), Keppel (1991), Kirk (1995), Maxwell and Delaney Hedges, L. V. (1981). Distributional theory for Glass’s estimator(1990), and Winer, Brown, and Michels (1991). Experimental of effect size and related estimators. Journal of Educationaldesigns differ in a number of ways: (a) randomization proce- Statistics, 6, 107–128.dures, (b) number of treatments, (c) use of independent sam- Hocking, R. R. (1985). The analysis of linear models. Pacific Grove,ples or dependent samples with blocking, (d) use of crossed CA: Brooks/Cole.and nested treatments, (e) presence of confounding, and (f) Hocking, R. R., & Speed, F. M. (1975). A full rank analysis of someuse of covariates. Researchers have many design decisions to linear model problems. Journal of the American Statisticalmake. I have tried to make the researcher’s task easier by em- Association, 70, 706–712.phasizing two related themes throughout the chapter. First, Kempthorne, O. (1947). A simple approach to confounding andcomplex designs are constructed from three simple building fractional replication in factorial experiments. Biometrika, 34,block designs. Second, complex designs share similar lay- 255–272.outs, randomization procedures, and assumptions with their Keppel, G. (1991). Design and analysis: A researcher’s handbookbuilding block designs. (3rd ed.). Englewood Cliffs, NJ: Prentice-Hall. Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole.REFERENCES Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56,Bogartz, R. S. (1994). An introduction to the analysis of variance. 746–759. Westport, CT: Praeger. Kirk, R. E. (2001). Promoting good statistical practices: SomeBox, J. F. (1978). R. A. Fisher: The life of a scientist. New York: suggestions. Educational and Psychological Measurement, 61, Wiley. 213–218.Cobb, G. W. (1998). Introduction to design and analysis of experi- Leonard, W. H., & Clark, A. G. (1939). Field plot techniques. ments. New York: Springer-Verlag. Minneapolis, MN: Burgess.Cochran, W. G., & Cox, G. M. (1957). Experimental designs (2nd ed.). Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments New York: Wiley. and analyzing data. Belmont, CA: Wadsworth.

32 Experimental DesignPearson, K. (1901). On the correlation of characters not quantita- Tukey, J. W. (1991). The philosophy of multiple comparisons. tively measurable. Philosophical Transactions of the Royal Soci- Statistical Science, 6, 100–116. ety of London, 195, 1–47. Urquhart, N. S., Weeks, D. L., & Henderson, C. R. (1973). Estima-Searle, S. R. (1987). Linear models for unbalanced data. New York: tion associated with linear models: A revisitation. Communica- Wiley. tions in Statistics, 1, 303–330.Snedecor, G. W. (1934). Analysis of variance and covariance. Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statisti- Ames, IA: Iowa State University. cal principles in experimental design (3rd ed.). New York:Timm, N. H. (1975). Multivariate analysis with applications in McGraw-Hill. education and psychology. Paciﬁc Grove, CA: Brooks/Cole.

CHAPTER 2Exploratory Data AnalysisJOHN T. BEHRENS AND CHONG-HO YUHISTORY, RATIONALE, AND PHILOSOPHY OF EDA 34 Resistance 55 John Tukey and the Origins of EDA 34 Summary 57 Rationale and General Tenets 34 CURRENT COMPUTING AND FUTURE DIRECTIONS 57 Abduction as the Logic of EDA 38 Computing for EDA 57 Summary 42 Future Directions 58HALLMARKS OF EDA 42 Summary 60 Revelation 42 CONCLUSION 60 Residuals and Models 48 REFERENCES 61 Reexpression 52Quantitative research methods in the twentieth century were graphical techniques and conceptual frameworks mentionedmarked by the explosive growth of small-sample statistics in the report, are rooted in the quantitative tradition of ex-and the expansion of breadth and complexity of models for ploratory data analysis (EDA). Exploratory data analysis is astatistical hypothesis testing. The close of the century, how- well-established tradition based primarily on the philosophi-ever, was marked primarily with a frustration over the limita- cal and methodological work of John Tukey. Although Tukeytions of common statistical methods and frustration with their is clearly recognized as the father of EDA in statistical cir-inappropriate or ineffective use (Cohen, 1994). Responding cles, most psychologists are familiar only with small aspectsto the confusion that emerged in the psychological commu- of his work, such as that in the area of multiple-comparisonnity, the American Psychological Association convened a procedures. Although Tukey worked in mathematical statis-task force on statistical inference that published a report tics throughout his life, the middle of his career brought(Wilkinson & Task Force, 1999) recommending best prac- dissatisfaction with the value of many statistical tools for un-tices in the area of method, results, and discussion. Among derstanding the complexities of real-world data. Moreover,the recommendations in the area of conducting and reporting Tukey fought what he perceived as an imbalance in effortsresults, the task force suggested researchers undertake a clus- aimed at understanding data from a hypothesis-testing orter of activities to supplement common statistical test proce- confirmatory data analysis (CDA) mode while neglectingdures with the aim of developing a detailed knowledge of the techniques that would aid in understanding of data moredata, an intimacy with the many layers of patterns that occur, broadly. To fill this gap and promote service to the scientificand a knowledge of the implications of these patterns for sub- community, as well as balance to the statistical community,sequent testing. Tukey developed and implemented the processes and philos- Unbeknownst to many psychological researchers, the gen- ophy of exploratory data analysis to be discussed shortly. Toeral goals recommended by the task force, as well as specific introduce the reader to this tradition, the chapter is divided into four parts. First, the background, rationale, and philoso-This work was completed while Dr. Behrens was on leave from phy of EDA are presented. Second, a brief tour of the EDAArizona State University, Division of Psychology in Education. He toolbox is presented. The third section discusses computerwould like to thank the staff and administration of the department software and future directions for EDA. The chapter endsfor their support. with a summary and conclusion. 33

34 Exploratory Data AnalysisHISTORY, RATIONALE, AND PHILOSOPHY Behrens, 1997a, 2000), the work of EDA is slowly filteringOF EDA into daily practice through software packages and through the impact of a generation of statisticians who have beenJohn Tukey and the Origins of EDA trained under the influence of Tukey and his students. For example, although highly graphical data analysis was rare inThe tradition of EDA was begun and nurtured by John Tukey the 1970s, the current reliance on computer display screensand his students through his many years at Princeton Univer- has led statistical graphics to hold a central role in datasity and Bell Laboratories. As a young academic, Tukey was analysis as recommended in common software packagesa prodigious author and formidable mathematical statistician. (e.g., Norusis, 2000; Wilkinson, 2001). Tukey’s work in-He received his PhD in mathematics from Princeton at the spired entire paradigms of statistical methods, including re-age of 25 and at 35 reached the rank of full professor at the gression graphics (Cook & Weisberg, 1994), robustnesssame institution (Brillinger, Fernholz, & Morgenthaler, studies (e.g. Wilcox, 1997, 2001), and computer graphics for1997). A sense of Tukey’s breadth and impact can be gleaned statistical use (Scott, 1992; Wilkinson, 1999).from examination of the eight volumes of his collected Despite these advances in the application of EDA-likeworks. Volumes 1 and 2 (Brillinger, 1984, 1985) highlight his technique, statistical training remains largely focused on spe-contributions to time-series analysis (especially through cific techniques with less than optimal emphasis on philo-spectral decomposition). Volumes 3 (Jones, 1986a) and 4 sophical and heuristic foundations (cf. Aiken, West, Sechrest,(Jones, 1986b) address Philosophy and Principles of Data & Reno, 1990). To prepare the reader for appropriate appli-Analysis, and volume 5 is devoted to graphics (Cleveland, cation of the techniques discussed later, we first turn to a1988). Volume 6 (Mallows, 1990) covers miscellaneous treatment of the logical and philosophical foundations ofmathematical statistics, whereas volumes 7 (Cox, 1992) and EDA.8 (Braun, 1994) cover factorial and analysis of variance(ANOVA) and multiple comparisons, respectively. Moremay appear at a future date because Tukey remained an ac- Rationale and General Tenetstive researcher and writer until his death in July of 2000. It’s all about the World In addition to the many papers in his collected works,Tukey authored and coauthored numerous books. In the EDA Exploratory data analysis is an approach to learning fromliterature his central work is Exploratory Data Analysis data (Tukey & Wilk, 1966/1986) aimed at understanding the(Tukey, 1977), whereas Data Analysis and Regression: A world and aiding the scientific process. Although these maySecond Course (Mosteller & Tukey, 1977) is equally com- not be “fighting words” among psychologists and psycholog-pelling. Three volumes edited by Hoaglin, Mosteller, and ical methodologists, they were for Tukey as he first raised hisTukey (1983, 1985, 1991) complete the foundational corpus concerns with the statistical community.of EDA. Brillinger, Fernholz, and Morgenthaler (1997) pro- Tukey’s emphasis on the scientific context of data analysisvide a Festschrift for Tukey based on writings of his students leads to a view of data analysis as a scientific endeavor usingat the time of his 80th birthday in 1995. the tools of mathematics, rather than a mathematical en- As Tukey became increasingly involved in the application deavor that may have value for some real-world applications.of statistics to solve real-world problems, he developed his A number of changes to standard statistical practice are im-own tradition of values and themes that emphasized flexibil- plied in this view. First, the statistician cannot serve as anity, exploration, and a deep connection to scientific goals and aloof high priest who swoops down to sanctify a set of proce-methods. He referred to his work as data analysis rather than dures and decisions (Salsburg, 1985). Data analysts andstatistics because he believed the appropriate scientific work scientists (not mutually exclusive categories) must work in-associated with data was often much broader than the work teractively in a cyclical process of pattern extraction (mathe-that was followed by the traditional statistical community. matics) and pattern interpretation (science). Neither canTukey did not seek to supplant statistics; rather, he sought function without the other. This view has implications for theto supplement traditional statistics by restoring balance to organization of academic departments and organization ofwhat he considered an extreme emphasis on hypothesis test- graduate training.ing at the expense of the use of a broader set of tools and Second, because the effort of data analysis is to under-conceptualizations. stand data in all circumstances, the role of probability models Although most psychologists are unaware of the specific relates primarily to confirmatory aspects of the scientificproposals Tukey made for EDA (but see Tukey, 1969; process. This leaves a wide swath of the scientific processes

History, Rationale, and Philosophy of EDA 35for which researchers are left to use nonprobabilistic methods not reject probabilistic approaches, but rather considers themsuch as statistical graphics. This emphasis is based on the within a larger context of the many tools and ideas that bearfact that in many stages of research the working questions are on scientific work.not probabilistic. When probabilistic methods are applied, A central idea in the EDA corpus is the goal of developingthere are layers of assumptions which themselves need to be a detailed and accurate mental model that provides a sense ofassessed in nonprobabilistic ways to avoid an unending loop intimacy with the nuances of the data. Such an experience as-of assumptions. Contrasting classical statistics with data sists both in constructing scientific hypotheses and buildinganalysis, Tukey (1972/1986a) wrote, “I shall stick to ‘data mathematical representations that allow more formal confir-analysis’ in part to indicate that we can take probability seri- matory methods to be used later. All of these issues argueously, or leave it alone, as may from time to time be appro- against the routine use of statistical procedures and the lock-priate or necessary” (p. 755). step application of decision rules. Tukey (1969) saw the com- The probabilistic approaches taken in most confirmatory monplace use of statistical tests to prove “truth” as a socialwork may lead to different practices than the nonprobabilistic process of “sanctification.” In this approach, the codifying ofapproaches that are more common to working in the ex- specific actions to be undertaken on all data obscures theploratory mode. For example, a number of researchers have individual nature of the data, removes the analyst fromlooked at the issue of deleting outliers from reaction time the details of the world held in the data, and impedes the de-data. From a probabilistic view this problem is addressed by velopment of intimacy and the construction of a detailedsimulating distributions of numbers that approximate the mental model of the world.shape commonly found in reaction time data. Next, extreme Consider the story of the researcher who sought to analyzevalues are omitted using various rules, and observation is the ratings of university faculty of various ethnicities accord-made of the impact of such adjustments on long-run decision ing to the ethnicities of the students providing the ratings. Tomaking in the Monte Carlo setting. As one would expect in make sure the job was done properly, the researcher con-such simulations, estimates are often biased, leading the re- tacted the statistical consulting center and spoke with the di-searcher to conclude that deleting outliers is inappropriate. rector. After a brief description of the problem, it was clear to Working from the exploratory point of view, the data ana- the consultant that this situation required a series of two-waylyst would bring to bear the scientific knowledge he or she ANOVAs of rating value across levels of teacher ethnicityhas about the empirical generating process of the data—for and student ethnicity. A graduate student was assigned toexample, the psychological process of comparison. Using compute the ANOVAs using commonly available statisticalthis as the primary guideline, outliers are considered observa- software, and both the researcher and consultant were quitetions whose value is such that it is likely they are the result of pleased with the resulting list of p values and binary signifi-nonexperimental attributes. If common sense and previous cance decisions.data and theory suggest the reaction times should be less than In this true story, as in many, it was unfortunate that the3 s, extreme values such as 6 or 10 s are most likely the result discussion focused primarily on choosing a statistical modelof other generating processes such as distraction or lack of (ANOVA) to fit the design, rather than being a balanced dis-compliance. If this is the case, then a failure to exclude cussion of the need for a broader understanding of the data.extreme values is itself a form of biasing and is deemed When the researcher later sought help in answering a re-inappropriate. viewer’s question, a simple calculation of cell frequencies These divergent conclusions arise from approaching the revealed that the scarcity of students and faculty in many mi-problem with different assumptions. From a probabilistic nority groups led to a situation in which almost half of theview, the question is likely to be formulated as If the underly- cells in the analysis were empty. In addition, many cells thating process has a distribution of X and I exclude data from it, were not empty had remarkably few data points to estimateis the result biased in the long run? On the other hand, the their means. In many ways the original conclusions from theexploratory question addressed is Given that I do not know analysis were more incorrect than correct.the underlying distribution is X, what do I know about the The error in this situation occurred because a series of un-processes that may help me decide if extreme values are from spoken assumptions propagated throughout the data analysisthe same process as the rest of the data? In this way EDA processes. Both the researcher and the director were con-emphasizes the importance of bringing relevant scientific cerned primarily with the testing of hypotheses rather thanknowledge to bear in the data-analytic situation rather than with developing a rich understanding of the data. Because ofdepending solely on probabilistic conceptualizations of the this, the statistician failed to consider some basic assump-phenomenon under study. As with all techniques, EDA does tions (such as the availability of data) and focused too much

36 Exploratory Data Analysison an abstract conceptualization of the design. It was third stage. . . . It is at this stage . . . that we require our bestsuch lockstep application of general rules (factorial means statistical techniques” (Tukey, 1973/1986b, p. 795). As a re-between groups implies ANOVA) that Tukey sought to searcher moves through these stages, he or she moves fromdiscourage. hypothesis generation to hypothesis testing and from pattern Unfortunately, it is not unusual that authors of papers pub- identification to pattern confirmation.lished in refereed journals neglect detailed examination of Whereas CDA is more ambitious in developing proba-data. This leads to inferior mental models of the phenomenon bilistic assessments of theoretical claims, the flexibility andand impedes the necessary assessment of whether the data bottom-up nature of EDA allows for broader application. Inconform to parametric test assumptions. For example, after many cases an appropriate model of parametric statistics mayreviewing more than 400 large data sets, Micceri (1989) be unavailable for full CDA, while the simpler techniques offound that the great majority of data collected in behavioral EDA may be of use. Under such a circumstance the maximsciences do not follow univariate normal distributions. should be followed that “[f]ar better an approximate answerBreckler (1990) reviewed 72 articles in personality and social to the right question, which is often vague, than an exactpsychology journals and found that only 19% acknowledged answer to the wrong question, which can always be madethe assumption of multivariate normality, and fewer than precise” (Tukey, 1962/1986c, p. 407).10% considered whether this assumption had been met. Re-viewing articles in 17 journals, Keselman et al. (1998) found Summarization and the Loss of Informationthat researchers rarely verified that statistical assumptionswere satisfied and typically used analyses that were nonro- Behrens and Smith (1996) characterize the nature of databust to assumption violations. These authors noted that many analysis as being oriented toward generating summary andof these types of problems could be detected by the applica- extracting gist. When faced with numerous pieces of data, thetion of techniques from the EDA tradition. goal of the analyst is to construct a terse yet rich mathematical description of the data. This is analogous to the summa-Detective Work and the Continuum of Method rization process that occurs in natural language processing. After reading a long book, one does not recall every individ-In working to learn about the world, EDA holds several com- ual word, but rather remembers major themes and prototypi-plementary goals: to find the unexpected, to avoid being cal events. In a similar way, the data analyst and researchfooled, and to develop rich descriptions. The primary analogy consumer want to come away with a useable and parsimo-used by Tukey to communicate these goals is that of the data nious description rather than a long list of data. An essentialanalyst as detective. Detective work is held up as a valuable concept associated with summarization is that every sum-analogy because the process is essentially exploratory and mary represents a loss of information. When some aspects ofinteractive; it involves an iterative process of generating data are brought to the foreground, other aspects are sent tohypotheses and looking for fit between the facts and the ten- the background.tative theory or theories; and the process is messy and replic-able only at the heuristic level. Detective work also provides Algebra Lies, So You Need Graphicsa solid analogy for EDA because it is essentially a bottom-upprocess of hypothesis formulation and data collection. Anscombe (1973) described a data set of numbers, each mea- Tukey (e.g., 1972/1986a, 1973/1986b) did not consider sured on the scales of x and y. He described the data as hav-methodology as a bifurcation between EDA and CDA, but ing a mean of 9 and standard deviation of 3.3 in x and a meanconsidered quantitative methods to be applied in stages of of 7.5 and standard deviation of 2.03 in y. The data were fit byexploratory, rough confirmatory, and confirmatory data ordinary least squares (OLS) criteria to have a slope of .5, ananalyses. In this view EDA was aimed at the initial goals of intercept of 3, and a correlation of .83. This allows the tersehypothesis generation and pattern detection, following the and easily interpretable summary for the data in the formdetective analogy. Rough CDA is sometimes equated with y = 3 + .5(x) + error. As a thought experiment, we encour-null-hypothesis significance testing or the use of estimation age the reader to try to visualize a scatter plot that depictsprocedures such as confidence intervals with the aim to an- such data.swer the question, “With what accuracy are the appearances If you imagined a scatter plot similar to that shown inalready found to be believed?” (Tukey, 1973/1986b, p. 794). Figure 2.1, then you are quite correct, because this representsWith regard to strict confirmatory analysis Tukey notes, the data Anscombe provided that met the descriptive statis-“When the results of the second stage is marginal, we need a tics we described previously. This, however, is only a small

History, Rationale, and Philosophy of EDA 37 14 starting point and bin widths produces the displays seen in Figure 2.4. 12 In sum, all data analysis is a process of summarization. This process leads to a focus on some aspects of data while taking focus off of other aspects. Conscious of these is- 10 sues, the exploratory analyst always seeks multiple representations of the data and always holds a position of skepticism 8 toward any single characterization of data. Y1 6 The Importance of Models Apart from the cognitive aspects of statistical information just 4 discussed, there is an additional epistemic layer of meaning that must be dealt with. As George Box (1979) wrote: “All 2 models are wrong but some are useful” (p. 202). An important aspect of EDA is the process of model specification and testing 0 with a focus on the value and pattern of misfit or residuals. Al- 0 2 4 6 8 10 12 14 16 18 20 X1 though some psychologists are familiar with this view from their experience with regression graphics or diagnostics, manyFigure 2.1 Plot of bivariate normal version of Anscombe data. individuals fail to see their statistical work as model building. In the EDA view, all statistical work can be considered model building. The simple t test is a model of mean differencepart of the story, for if you imagined the data to have the as a function of group membership considered in terms ofshape shown in panel A of Figure 2.2 then you are also cor- sampling fluctuation. Regression analyses attempt to modelrect. If you imagined the pattern in panels B or C of Figure criteria values as a function of predictor variables, whereas2.2, you are also correct because all the patterns shown in analysis of variance (ANOVA) models means and variances ofFigures 2.1 and 2.2 conform to the same algebraic summary dependent variables as a function of categorical variables.statistics given by Anscombe. Although this example speaks Unaware of the options, many individuals fail to considerto the weakness of overdependence on algebraic representa- the wide range of model variants that are available. In re-tions alone, it points to the larger issue that all summarization gression, for example, the “continuous” dependent variableleads to a loss of information. may be highly continuous or marginally continuous. If the dependent variable is binary, then a logistic regression is appropriate and a multilevel categorical dependent variable canGraphics Lie, So You Need Algebra likewise be fit (Hosmer & Lemeshow, 2000). The closely re-Although graphics are a mainstay of EDA, graphics are not lated variant of Poisson regression exists for counts, and pro-immune from this general principle. Consider the follow- bit and Tobit variants also can be used.ing data set: 1,1,2,2,3,3,4,4,5,5,5,5,6,6,6,6,6,6,7,7,7,7,8,8, The application of models in these instances is central to9,9,10,10,11,11. Entering this data into a standard statistics EDA. Different models will have different assumptions andpackage produces the display presented in Figure 2.3. As the often describe the data well in one way, but fail in others. Forreader can see, a slight skew is evident that may not be de- example, in the world of item response theory, there is oftentected in the listing of numbers themselves. It is important to great consternation regarding the choice of models to use.consider the computational model that underlies this graphic. One-parameter models may misfit the data in some way, butIt consists of a mapping of bar height to frequency and bar have the desirable properties of sufficiency and consistentwidth to bin width in the frequency table. Bin width refers to ordering of individuals. Two- and three-parameter modelsthe size of the interval in which numbers are aggregated when generally fit the data more tightly but without the conceptualdetermining frequencies. (The term bandwidth is similarly advantages of the one-parameter model. From an EDA pointused in many domains, including nonparametric smoothing; of view, each model is “correct” in some respect insofar ascf. Härdle, 1991). Different bin widths and starting points for each brings some value and loses others. Depending on thebins will lead to different tables, and hence, different graph- exact scientific or practical need, the decision maker mayics. Using the same data with different combinations of bin choose to emphasize one set of values or another.

38 Exploratory Data Analysis(A) 14 (B) 14 12 12 10 10 8 8 Y2 Y3 6 6 4 4 2 2 0 0 0 2 4 6 8 10 12 14 16 18 20 0 2 4 6 8 10 12 14 16 18 20 X2 X3 (C) 14 12 10 8 Y4 6 4 2 0 0 2 4 6 8 10 12 14 16 18 20 X4Figure 2.2 Additional data sets with same algebraic summaries as the data in Figure 2.1, with varying patterns and model ﬁt. Regardless of individual decisions about models, the most important issues are that one realizes (a) that there is always model being used (even if implicitly), (b) that for real-world data, there is no such thing as the perfect model, and (c) that 5 the way in which the model is wrong tells us something about the phenomenon. Abduction as the Logic of EDA 0 Because of the rich mathematical foundation of CDA, many 0 2 4 6 8 10 12 X1 researchers assume that the complete philosophical basis for inferences from CDA have been worked out. Interestingly,Figure 2.3 Histogram of small data set revealing slight skew. this is not the case. Fisher (1935, 1955) considered his

History, Rationale, and Philosophy of EDA 39 (A) (B) 4 5 2 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 0.0 1.9 3.8 5.7 7.6 9.5 11.4 X1 X1 (C) (D) 6 4 5 2 0 0 0.00 2.05 4.10 6.15 8.20 10.25 1.0 2.5 4.0 5.5 7.0 8.5 10.0 11.5 X1 X1 Figure 2.4 Additional histograms of the same data depicted in Figure 2.3 with varying appearances as a function of bin width and bin starting value.approach to significance testing an as implementation of “in- operative; abduction merely suggests that something may be”ductive inference” and argued that all knowledge is gained (vol. 5, p. 171). Put another way: Abduction plays the role ofin this way. Neyman and Pearson (1928, 1933a, 1933b), on generating new ideas or hypotheses; deduction functionsthe other hand, developed the concepts of power, type II as evaluating the hypotheses; and induction justifies theerror, and confidence intervals, to which Fisher objected hypotheses with empirical data (Staat, 1993).(Gigerenzer, 1987, 1993; Lehmann, 1993). Neyman argued Deduction involves drawing logical consequences fromthat only deductive inference was possible in statistics, as premises. The conclusion is true given that the premises areshown in the hypothesis testing tradition he developed. true also (Peirce, 1868). For instance,Others argue that classical statistics involves both logicalmodes, given that the hypothesis is generated deductively First premise: All As are Bs (True).and data are compared against the hypothesis inductively Second premise: C is A (True).(Lindsey, 1996). Conclusion: Therefore, C is B (True). Where, then, does this leave EDA? Because Tukey wasprimarily a mathematician and statistician, there has been lit- Deductive logic confines the conclusion to a dichotomoustle explicit work on the logical foundations of EDA from a answer (true-false). A typical example is the rejection or fail-formal philosophical viewpoint. A firm basis for understand- ure of rejection of the null hypothesis. To be specific, the for-ing EDA, however, can be found in the concept of abduction mulated hypothesis is regarded as the first premise. When theproposed by the American philosopher Charles Sanders data (the second premise) conform to the hypothesis, the con-Peirce. Peirce, whose name is pronounced “pers,” was a tour clusion must assert that the first premise is true.de force in American philosophy as the originator of modern Some have argued that deduction is incomplete becausesemiotics, an accomplished logician in logic of probability, we cannot logically prove all the premises are true. Russelland the originator of pragmatism that was popularized by and Whitehead (1910) attempted to develop a self-sufficientJames and Dewey. Peirce (1934/1960) explained the three logical-mathematical system. In their view, not only canlogical processes by arguing, “Deduction proves some- mathematics be reduced to logic, but also logic is the founda-thing must be. Induction shows that something actually is tion of mathematics. However, Gödel (1947/1986) found that

40 Exploratory Data Analysisit is impossible to have such a self-contained system. Any remains overlooked among many texts of logic and researchlower order theorem or premise needs a higher order theorem methodology, while gaining ground in the areas of artificialor premise for substantiation, and it goes on and on; no sys- intelligence and probabilistic computing (e.g., Josephson &tem can be complete and consistent at the same time. Build- Josephson, 1994; Schum, 1994). However, as logic is divideding on this argument, Kline (1980) held that mathematics had into formal types of reasoning (symbolic logic) and informaldeveloped illogically with false proof and slips in reasoning. types (critical thinking), abduction is represented as informalThus, he argued that deductive proof from self-evident prin- logic. Therefore, unlike deduction and induction, abductionciples in mathematics is an “intellectual tragedy” (p. 3) and a is a type of critical thinking rather than a formalism captured“grand illusion” (p. 4). by symbolic logic. The following example illustrates the For Peirce, inductive logic is based upon the notion that function of abduction, though illustrated with symbols forprobability is the relative frequency in the long run and that a simplification:general law can be concluded based on numerous cases. Forexample, The surprising phenomenon, X, is observed. A1, A2, A3 . . . A100 are B. Among hypotheses A, B, and C, A is capable of explain- ing X. A1, A2, A3 . . . A100 are C. Hence, there is a reason to pursue A. Therefore, B is C. Hume (1777/1912) argued that things are inconclusive by At first glance, abduction may appear as no more thaninduction because in infinity there are always new cases and an educated guess among existing hypotheses. Thagard andnew evidence. Induction can be justified if and only if in- Shelley (1997) addressed this concern. They argued that uni-stances of which we have no experience resemble those of fying conceptions are an important part of abduction, and itwhich we have experience. Thus, the problem of induction is would be unfortunate if our understanding of abduction werealso known as “the skeptical problem about the future” limited to more mundane cases where hypotheses are simply(Hacking, 1975). Take the previous argument as an example. assembled. Abduction does not occur in the context of a fixedIf A101 is not B, the statement “B is C” will be refuted. We language, since the formation of new hypotheses often goesnever know when a line predicting future events will turn flat, hand in hand with the development of new theoretical termsgo down, or go up. Even inductive reasoning using numerous such as quark and gene. Indeed, Peirce (1934/1960) empha-accurate data and high-power computing can go wrong, sized that abduction is the only logical operation that intro-because predictions are made only under certain specified duces new ideas.conditions (Samuelson, 1967). Although abduction is viewed as a kind of “creative intu- Induction suggests the possible outcome in relation to ition” for idea generation and fact explanation (Hoffmann,events in the long run. This is not definable for an individual 1997), it is dangerous to look at abduction as impulsiveevent. To make a judgment for a single event based on prob- thinking and hasty judgment. In The Fixation of Belief,ability, such as saying that someone’s chance of surviving Peirce explicitly disregarded the tenacity of intuition as thesurgery is 75%, is nonsensical. In actuality, the patient will source of knowledge. Peirce strongly criticized his contem-either live or die. In a single event, not only is the probability poraries’ confusion of propositions and assertions. Proposi-indefinable, but also the explanatory power is absent. Induc- tions can be affirmed or denied while assertions are finaltion yields a general statement that explains the event of ob- judgments (Hilpinen, 1992). The objective of abduction is toserving, but not the facts observed. Josephson and Josephson determine which hypothesis or proposition to test, not which(1994) gave this example: one to adopt or assert (Sullivan, 1991). In EDA, after observing some surprising facts, we exploit Suppose I choose a ball at random (arbitrarily) from a large hat them and check the predicted values against the observed containing colored balls. The ball I choose is red. Does the fact values and residuals (Behrens, 1997a). Although there may that all of the balls in the hat are red explain why this particular be more than one convincing pattern, we “abduct” only those ball is red? No. . . “All A’s are B’s” cannot explain why “this A is that are more plausible for subsequent confirmatory experi- a B” because it does not say anything about how its being an A is connected with its being a B. (p. 20) mentation. Since experimentation is hypothesis driven and EDA is data driven, the logic behind each of them is quite The function of abduction is to look for a pattern in a different. The abductive reasoning of EDA goes from datasurprising phenomenon and suggest a plausible hypothesis to hypotheses, whereas inductive reasoning of experimenta-(Peirce, 1878). Despite the long history of abduction, it tion goes from hypothesis to expected data. In fact, closely

History, Rationale, and Philosophy of EDA 41(and unknowingly) following Tukey (1969), Shank (1991), For most individuals it is clear that a new discovery isJosephson and Josephson (1994), and Ottens and Shank valuable and should be well documented and collected.(1995) related abductive reasoning to detective work. Detec- The entomologist’s failure to have hypothesized it does nottives collect related “facts” about people and circumstances. impugn its uniqueness, and indeed many great scientificThese facts are actually shrewd guesses or hypotheses based conclusions have started with unanticipated findingson their keen powers of observation. (Beveridge, 1950). Should the entomologist worry about In short, abduction can be interpreted as observing the Type I error? Since Type I error concerns long-run error inworld with appropriate categories, which arise from the inter- decision making based on levels of specific cutoff values innal structure of meanings. Abduction in EDA means that specific distributions, that precise interpretation does notthe analyst neither exhausts all possibilities nor makes hasty seem to matter much here. If she makes an inference aboutdecisions. Researchers must be well equipped with proper this finding then she should consider the probabilistic basiscategories in order to sort out the invariant features and pat- for such an inference, but nevertheless the butterfly shouldterns of phenomena. Quantitative research, in this sense, is be collected. Finally, should she be concerned that this find-not number crunching, but a thoughtful way of peeling back ing will contaminate her original hypothesis? Clearly shelayers of meaning in data. should continue her travel and look for the evidence concerning her initial hypothesis on the mountaintop. If theExploration, Discovery, and Hypothesis Testing new butterfly contradicts the existing hypothesis, then the entomologist has more data to deal with and additionalMany researchers steeped in confirmatory procedures have complexity that should not be ignored. If she is concernedappropriately learned that true hypothesis tests require true about changing her hypothesis in midstream to match thehypotheses and that unexpected results should not be treated new data, then she has confused hypothesis generation andwith the same deference as hypothesized results. A corollary hypothesis testing. With regard to any new theories, sheis that one should keep clear what has been hypothesized must create additional predictions to be tested in a differentin a research study and not modify a hypothesis to match location.data. This is certainly true and is an essential aspect of con-firmatory inference. In some researchers, however, a neuro- EDA and Exploratory Statisticssis develops that extends the avoidance of hypothesis-modification based on knowledge of the data to an EDA and exploratory statistics (ES) have the same ex-avoidance of intimacy with the data altogether. Sometimes ploratory goals; thus, the question sometimes arises as tothis neurosis is exacerbated by the fear that every piece of whether ES is simply a subset of EDA or EDA is a subset ofknowledge has an amount of Type I error associated with it ES. Because EDA is primarily an epistemological lens andand, therefore, the more we know about the data the higher ES is generally presented in terms of a collection of tech-our Type I error. The key to appropriately balancing ex- niques, a more appropriate question is Can ES be conductedploratory and confirmatory work is to keep clear what has from an EDA point of view? To this question we can answerbeen hypothesized in advance and what is being “discov- yes. Furthermore, EDA is a conceptual lens, and most re-ered” for the first time. Discoveries are important, but do search procedures can be undertaken with an EDA slant. Fornot count as confirmations. example, if one is conducting an “exploratory” factor analy- After years of extensive fieldwork an entomologist devel- sis without graphing of data, examination of residuals, orops a prediction that butterflies with a certain pattern of spot- attention to specific patterns of raw data underlying the cor-ting should exist on top of a particular mountain, and sets off relations that are central to the analysis, then little seems to befor the mountaintop. Clearly, if the entomologist finds such consistent with the EDA approach. On the other hand, abutterflies there will be evidence in support of her theory; clearly probabilistic analysis can be well augmented by plotsotherwise, there is an absence of evidence. On her way to the of data on a number of dimensions (Härdle, Klinke, &mountain she traverses a jungle in which she encounters a Turlach, 1995; Scott, 1992), attention to residual patterns in apreviously unknown species of butterflies with quite unantic- number of dimensions, and the use of detailed diagnosticsipated spottings. How does she handle this? Should she that point to patterns of fits and misfits. Regardless ofignore the butterfly because she has not hypothesized it? the software or specific statistical procedures used, suchShould she ignore it because it may simply be a misleading activity would clearly be considered EDA and ES. ES doesType I error? Should she ignore it because she may change not necessarily imply EDA, but ES can be conducted as EDAher original hypothesis to say she has really hypothesized this if the conceptual and procedural hallmarks of EDA arejungle butterfly? employed.

42 Exploratory Data AnalysisSummary Another psychological advantage of graphics is that it allows for a parsimonious representation of the data. The factsExploratory data analysis is a rich data-analytic tradition de- that are easily derivable from the image include all the indi-veloped to aid practical issues of data analysis. It recom- vidual values, the relative position of each data point to everymends a data-driven approach to gaining intimacy with one’s other, shape-based characterizations of the bivariate distribu-data and the phenomenon under study. This approach follows tion, and the relationship between the data and the proposedthe analogy of the detective looking for clues to develop regression line. After some practice, the trained eye can eas-hunches and perhaps seek a grand jury. This counters the ily discern and describe the marginal distributions as well asmore formal and ambitious goals of confirmatory data analy- the distribution of residuals. The construction of a text-basedsis, which seeks to obtain and present formal evidence in representation of all of this information would require an ex-order to gain a conviction. EDA is recommended as a com- tensive set of text-based descriptors. In short, visual imagesplement to confirmatory methods, and in no way seeks to re- of the type shown here exploit the visual-spatial memory sys-place or eliminate them. Indeed, effective researchers should tem to support efficient pattern recognition (Garner, 1974),incorporate the best aspects of all approaches as needed. problem solving (Larkin & Simon, 1987), and the construction of appropriate mental models (Bauer & Johnson-Laird, 1993).HALLMARKS OF EDA Tukey’s early work and concomitant advances in computing have led to an explosion in graphical methods over theIn this section, the techniques and attitudes that are standard last three decades. Numerous authors, including Tufte (1990,aspects of EDA are discussed. The tools described here are 1997, 1983/2001) and Wainer (1997; Wainer & Velleman,only recommendations that have worked to allow researchers 2001) have worked to popularize data-based graphics.to reach the underlying goals of EDA. However, it is the un- William Cleveland has had a large impact on the statisticalderlying goals that should be sought, not the particular tech- community with his empirical studies of the use of graphicsniques. Following Hoaglin, Mosteller, and Tukey (1983), we (Cleveland & McGill, 1984), the initiation of cognitive mod-discuss these tools under the four Rs of EDA: revelation, els of graph perception (Cleveland, 1985), and his applica-residuals, reexpression, and resistance. tion of these principles to statistical graphics (especially Cleveland, 1993). Wilkinson (1993, 1994, 1999) made substantial contributions to the study of proper use of statisticalRevelation graphics, and has recently provided a comprehensive volumeGraphics are the primary tool for the exploratory data ana- regarding graphics in software and statistical analysislyst. The most widely cited reason for this is Tukey’s (1977) (Wilkinson, 1999) that is required reading for anyone inter-statement that “The greatest value of a picture is when it ested in the field. Kosslyn (1994) provided a discussion offorces us to notice what we never expected to see” (p. vi). In numerous potential rules for graph construction from a psy-many ways, the graphics in Figures 2.1 and 2.2 illustrate all chological perspective, and Lewandowsky and Behrensthe rationale of graphics in EDA. First, even though the alge- (1999) provide a recent review of cognitive aspects of statis-braic summaries are “sufficient statistics,” they are sufficient tical graphs and maps.for only the very limited purpose of summarizing particularaspects of the data. For specifying the exact form of the data Graphics Made for EDAwithout additional assumptions regarding distributionalshapes, the summary statistics are not only “insufficient” During the emergence of the EDA tradition, Tukey developedbut are downright dangerous. Second, the indeterminacy of a large number of graphical tools, some of which have be-the algebra calls us to fill in the details with possibly unten- come commonplace, others of which have had little visibilityable assumptions. In the Anscombe data-thought experiment, outside specialized applications. It is important to rememberparticipants almost universally imagine the data to be of the that at the time of their original construction, much of whatcanonical form shown in Figure 2.1. In the absence of a skep- Tukey sought to do was to support quick summarization andtical mind and in the light of the history of statistics textbooks analysis when data were available, and the analysis was tothat are focused on mathematical idealizations at the expense occur by hand.of real-world patterns, many psychologists have developed Perhaps the best known of Tukey’s graphical devicesschemas and mental models (Johnson-Laird, 1983) that lead for EDA is the box-and-whisker plot, otherwise called theto erroneous inferences. box-plot. The box-plot is a graph based on a five-number

Hallmarks of EDA 43summary of a distribution of data; these numbers are the me- (A)dian, the ﬁrst and second hinges, and either the lowest andhighest number or a similar measure of range number arrived 1at by separating very extreme values. The median is equal tothe 50th percentile of the distribution. The hinges are eitherequal to or very close to the 25th and 75th percentiles— Effect sizealthough they are found using a simpler rank-based formula 0for computation. To construct a box-plot, a scale is drawn,and a box is placed on the scale with one end of the box indi-cating the scale value of the lower hinge (25th percentile) andthe other end of the box occurring at the scale position of the –1upper hinge (75th percentile). An additional line is drawn inthe middle to indicate the scale value of the median. Thescale-value difference between the two hinges is called eitherthe hinge spread or the interquartile range (often abbreviated – 2 5 6 7 10IQR), and in a normal distribution corresponds to approxi- Presentation speedmately 0.67 standard deviations on each side of the mean. (B) Rules for the construction of the “whisker” portion of thedisplay vary. In the most common version, lines are extendedalong the scale from the hinges to the farthest data value in 1each direction up to 1.5 hinge spreads. If there are data pastthat point, the whiskers extend to the farthest data point priorto the 1.5 hinge-spread cutoff. Data points beyond thewhiskers are usually identiﬁed individually to bring attention Effect size 0to their extremeness and potential characterization asoutliers. An example of multiple box-plots is presented in Fig-ure 2.5, panel A, with individual raw data values presented inpanel B for comparison. These graphics depict the distribu- –1tions of effect sizes from the meta-analysis of social memoryconducted by Stangor and McMillan (1992). The categoricalvariable on the horizontal axis is the length of stimulus pre-sentation in seconds. The continuous variable on the verticalaxis is the size of the effect for each study included in the – 2 5 6 7 10meta-analysis. As the reader may see, the box-plots provide a Presentation speedcompact description of each distribution and allow relatively Figure 2.5 Panel A consists of multiple box-plots of effect sizes in socialeasy comparison of both the level and spread of each distrib- memory meta-analysis, organized by presentation speed (from Stangor & McMillan, 1992). Panel B depicts the same data by plotting individualution. The distribution farthest to the left represents all the values in a dot-plot.studies for which no presentation speed is reported. Therange is approximately from −2 to +2, with a medianslightly below zero. The second box-plot depicts the distrib- When two box-plots are compared, the analyst is under-ution of effect sizes from studies that used a 2-s presentation taking the graphical analog of the t test. Displays with addi-speed. It is the highest distribution of all, with some positive tional boxes, as shown here, are analogous to the analysis ofskew. The median of this distribution is higher than the variance: Group-level measures of central tendency are dis-75th percentile of the remaining distributions, indicating a played relative to the amount of within-group variability inclear trend toward larger values. The median is also higher the data.than the 75th percentile of the 6- and 10-s studies. The stud- Although the box-plots are very useful and informative inies with presentation times of 6 s show very little variance their current state, working in the exploratory mode raiseswith the exception of two outliers, which are indicated additional issues. First, how might we be fooled by these dis-separately. plays? The answer to this is that there are times when the

44 Exploratory Data Analysis(A) Berk (1994) recommended the overplotting of a dot-plot (as 10 seen in the lower panel of Figure 2.6) on top of the box-plot so that two levels of detail can be seen simultaneously. 8 Regardless of the exact implementation used, users must be 6 wary that software packages vary on the algorithms used to calculate their five-number summaries, and they may not 4 be looking at the summaries one expects (Frigge, Hoaglin, & Iglewicz, 1989). 2 1 2 group Interactive Graphics(B) 11 00 11 0000000 Although much can be gained by modifying the static ap- 10 00 10 pearance of plots such as the box-plot, substantial gains in 9 00 9 data analysis can be made in computerized environments 8 00 8 00000000 by using interactive graphics with brushing and linking 7 0000 7 (Cleveland & McGill, 1988). Interactive graphics are 6 000000 6 graphic displays that respond to the brushing (selection) ac- 5 0000 5 4 00 4 00000000 tion of a pointing device (such as a mouse) by modifying the 3 00 3 graphics in real time. Linking is the connection of values on 2 00 2 the same observation through brushing or selecting across 1 00 1 0000000 different graphics. Highlighting an observation in one display + + + + + + + + (say, a scatter plot) causes the value of the same observation 1 2 to appear highlighted in another display (say, a histogram) as group well. In this way, an analyst working to analyze one graphicFigure 2.6 Panel A is two box-plots with identical summary statistics. can quickly see how information in that graphic relates to in-Panel B depicts the underlying data used to make panel A, illustrating the formation in another graphic. For example, in Figure 2.5 thepossibility that different data may produce identical graphs. conditional level of each distribution varies greatly. An analyst may wonder if this is primarily from the categoricalfive-number summaries are the same for different distribu- variables listed on the horizontal axis, or if there are othertions. This leads to a case in which the box-plots look identi- variables that may also covary with these medians. One pos-cal yet the data differ in structure. Consider the top panel of sibility is that different research laboratories tend to use dif-Figure 2.6, in which each of two box-plots is identical, indi- ferent speeds and therefore, that laboratory covaries withcating identical values of the five-number summary. The speed.lower panel of Figure 2.6 depicts the underlying raw data val- Prior to the advent of interactive graphics, one would stopues that vary in form. Distortions can also occur if there are the analysis in order to look in a table to determine whichvery few levels of the variable being measured, because it data came from which laboratory. Such a process could eas-will cause many data points to have the same value. In some ily become tedious and distracting from the main task. Usingcases the appearance of a plot may be distorted if the hinges a program that has high graphical interactivity, in this caseare equal to the minimum or maximum, because no whiskers Data Desk (Data Description, 1997), highlighting the data ofappear. interest in one graphical display highlights the same data in A second point of interest concerns how we can learn other graphical displays or in the variable listings. To accom-more from the data by enhancing the box-plot. Toward this plish this in Data Desk we simply turn off the box-plots usingend, recommendations abound. Tufte (1983/2001) recom- the pull-down menu at the top of the graph window (therebymended the omission of the box (an idea not well supported changing panel A of Figure 2.5 to panel B), indicate “Showby empirical data; Stock & Behrens, 1991). Other suggested identifying text” from a menu at the top of the screen, clickimprovements include indicating the sample size of the sub- on the identifying variable of interest (study name), and high-set below the box (e.g., Becker, Chambers, & Wilks, 1988), light observations to be linked. The final outcome of theseadding confidence intervals around the median in the box, or few quick hand movements is presented in Figure 2.7. Heredistorting the box shape to account both for sample size and the graphics reveal some unexpected results. Eight ofthe confidence intervals (McGill, Tukey, & Larsen, 1978). the nine effect sizes in this group come from only two studies

Hallmarks of EDA 45 measured (continuous or nearly continuous) data, the scatter Stangor&Ruble, 1989b, 10years plot is the common fundamental graphic. Here variations abound, as well. Whereas the scatter plot is often used to un- 1 Stangor&Ruble, 1989b, overall derstand two dimensions of data, when faced with high- Stangor&Ruble, 1989b, 6years dimensional data, one often uses a matrix of scatter plots to Stangor&Ruble, 1989b, 4years see the multidimensionality from multiple bivariate views. Signore&Ruble, 1984, Study2 An example of a scatter plot matrix is presented in Figure 2.8.Effect size Signorello&Liben, 1984, Study1, kindergarten The data portrayed here are a subset of the data that Stangor 0 Signorello&Liben, 1984, Study1, overall Signorello&Liben, 1984, Study1, and McMillan (1992) used in a weighted least-squares re- Bargh&Thein, 1985, controlled gression analysis of the effect sizes. The plot can be thought of as a graphical correlation matrix. Where we would have placed the value of the correlation, we instead put the graph- –1 ical bivariate display. For each individual scatter plot, one can identify the variable on the vertical axis by looking at the variable named at the far left of the row. The horizontal axis is identified by looking down the column of the matrix to the variable identified at the bottom of the scatter plot. For exam- – 2 5 6 7 10 Presentation speed ple, the plot in the upper right corner has “N” (sample size) on the vertical axis and “con / incon” on the horizontal axis,Figure 2.7 Dot-plot with identifying text obtained by selecting andlinking. indicating the ratio of congruent to incongruent stimuli. The top of the “con / incon” label is hidden due to plot size in the plot in the lower right corner. The plots in the diagonal cellsand the studies are highly stratified by effect size. When the are normal-probability plots whose interpretation is dis-eight data points of the 7-s effects are circled, the study cussed below.names indicate that all these effects come from a single study. In this situation, as is often the case, the scatter plot matrix does an excellent job of revealing unexpected structure. For many of the bivariate relationships there is great departureMoving Up the Path of Dimensionality from bivariate normality. Of particular concern is the combi-Whereas box-plots are often considered univariate graphics nation of high skew in the congruent-incongruent ratio andbecause they are often used to display a single variable, our the floor effect in the targets and traits variables. These issuessimple example has demonstrated that the box-plot can easily lead to L-shaped distributions that will present a clear chal-function in three variables. In this case the variables are pre- lenge to any continuous linear model. Outliers and combina-sentation speed, effect size, and study origin. In highly inter- tions of missing data should also be considered carefully. Ofactive environments, however, additional variables are easily particular note in these data is that the higher level of theadded. For example, in Data Desk, palettes available on the dummy-coded delay variable exists in only two observations,desktop allow one to choose the shape or color of symbols for but one of those observations has no matching data on manyindividual data points. This is accomplished through select- variables and thus functions as a single point. In a multi-ing the data points of interest by circling the points on the variate situation such as regression analysis, this is quitegraphic and clicking on the desired shape or color. Symbol problematic because the estimation of the relationship of thiscoloring can also be accomplished automatically by using a variable with all others rests precariously on the value of thepull-down menu that indicates a desire to color symbols by single point. Error at this point will thereby be propagatedthe value of a specific variable. In our meta-analysis data, through the system of partial correlations used to estimatecoloring the data points by sample size and varying the shape regression effects.to indicate the value of another categorical variable of inter- Plots with multiple straight lines indicate the 0 and 1 lev-est may aid in finding unanticipated patterns. In this way, els of dummy coding. A number of additional dummy-codedwe would have created a rather usable yet complex five- variables subjected to simultaneous regression by Stangordimensional graphic representation of the data. and McMillan (1992) were omitted because the number of For combinations of categorical and measured data, the plots became too large to present here clearly. Earlier ver-box-plot and corresponding dot-plot provide an excellent sions of this matrix revealed additional unusual values thatstarting point. For analyses that focus more heavily on were traced back to the present authors’ transcription process

46 Exploratory Data Analysis Figure 2.8 Scatter plot matrix of meta-analysis data from Stangor and McMillan (1992).and have been since corrected. In this case, the graphics To provide a quick and provisional evaluation of thisrevealed structure and avoided error. possibility, we created a histogram of the target variables, selected those bins in the graphic that represent low levels ofGoing Deeper targets, and chose a unique color and symbol for the observations that had just been selected. From here, one can simplyAlthough the scatter plot matrix is valuable and informa- click on the pull-down menu on any scatter plot and choosetive, it is important that the reader recognize that a series of “Add color regression lines.” Because the observationstwo-dimensional views is not as informative as a three- have been colored by low and high levels of the target vari-dimensional view. For example, when Stangor and McMillan able, the plots will be supplemented with regression lines be-computed a simultaneous regression model, the variables in- tween independent variables and the effect size–dependentdicating the number of targets and traits used in each study variable separately for low and high levels of targets, as dis-reversed the direction of their slope, compared with their played in Figure 2.9.simple correlations. Although a classical “suppressor” inter- Moving across the second row of Figure 2.9 (which corre-pretation was given, the exploratory analyst may wonder sponds to the response variable), ﬁrst we see two regressionwhether the simple constant and linear functions used lines with low identical slopes indicating little relationship be-to model these data were appropriate. One possibility is tween task and effect, which is constant across levels of target.that the targets variable mediates other relationships. For The delay variable in the next column shows a similar pattern,example, it may be the case that some variables are highly whereas the next three variables show small indications of in-related to effect size for certain levels of target, but have teraction. The interaction effect is very clear in the relation-different relationships with effect size at other levels of ship between effect size and the congruent-incongruent ratiotargets. in the rightmost column. This relationship is positive for

Hallmarks of EDA 47 Figure 2.9 View of computer screen using Data Desk software for selecting, brushing, and linking across multiple plots. Several plots have been enhanced with multiple regression lines that vary by subsets of selected data.observations with high numbers of targets, but negative for Working in an exploratory mode, these experiences sug-low numbers of targets. Unfortunately, in failing to recognize gest we step back and ask a more general question about thethis pattern, one may use a model with no interactions. In such meta-analytic data: In what way does the size and availabilitya case the positive slope observations are averaged with the of effects vary across the variety of study characteristics? Tonegative slope observations to create an estimate of 0 slope. begin to get such a view of the data, one may ﬁnd three-This would typically lead the data analyst to conclude that no dimensional plots to be useful. A graphic created using a non-relationship exists at all, when in fact a clear story exists just linear smoother for the effect size of each study as a functionbelow the surface (one variable down!). of the number of targets and presentation speed is presented Although the graphics employed so far have been helpful, in panel A of Figure 2.10. The general shape is similar to thewe have essentially used numerous low-dimensional views “saddle” shape that characterizes a two-way interaction inof the data to try to develop a multidimensional conceptual- continuous regression models (Aiken & West, 1991). Theization. This is analogous to the way many researchers graphic also reveals that little empirical work has been un-develop regression models as a list of variables that are “re- dertaken with high presentation speed and a low number oflated” or “not related” to the dependent variable, and then targets, so it is difﬁcult to assess the veracity of the smooth-consider them altogether. Our brushing and coding of the ing function given the lack of data in that area. At a mini-scatter plot matrix has shown that this is a dangerous mum, it suggests that future research should be conducted toapproach because “related” is usually operationalized as “lin- assess those combinations of study characteristics. Panel B ofearly related”—an assumption that is often unwarranted. Figure 2.10 shows an alternate representation of the data withMoreover, in multidimensional space, variables may be a traditional linear surface function that is designed to pro-related in one part of the space but not in the other. vide a single additive prediction across all the data.

48 Exploratory Data Analysis(A) 2 The simple graphics we have used here provide an excellent start at peeling away the layers of meaning that reside in these data. If nothing else is clear, the data are more complex 1 than can be easily described by a simple linear model in multiple dimensions. The theoretical concerns of Anscombe Effect size 0 (1973) have proven to be realistic after all. Despite the inte- rocular effect provided by these graphics, some readers will assure themselves that such difficulties appear primarily in Ϫ1 data from meta-analyses and that the data they work with will not be so problematic. Unfortunately this is not often the case, and there is a cottage industry among EDA proponents 10 1 1 of reanalyzing published data with simple graphics to show 9 Pr rich structure that was overlooked in original work. 8 es 7 en ta 00 6 1 tio 0 90 5 n 70 8 sp 0 60 4 ee 40 5 et Residuals and Models 3 d 0 30 Targ 2 0 2 0 1 1 In the EDA tradition, the second R stands for residual, yet(B) 2 this word signifies not simply a mathematical definition, but a foundational philosophy about the nature of data analysis. Throughout Tukey’s writings, the theme of DATA = FIT + 1 RESIDUALS is repeated over and over, often in graphical Effect size analog: DATA = SMOOTH + ROUGH. This simple for- 0 mula reminds us that our primary focus is on the development of compact descriptions of the world and that these Ϫ1 descriptions will never be perfect; thus there will always be some misfit between our model and the data, and this misfit occurs with every observation having a residual. 10 1 1 This view counters implicit assumptions that often arise in 9 Pr statistical training. First, many students acquire an unfortu- 8 es 7 en 01 00 6 ta 80 9 nate belief that “error” has an ontological status equivalent to 5 tio 0 70 n 0 6 4 sp 40 5 et “noise that can be ignored” and consequently believe the 3 ee 0 30 Targ 2 d 0 2 results of a model-fitting procedure (such as least squares 0 1 1 regression) is the “true” model that should be followed. SuchFigure 2.10 Panel A is a nonlinear surface estimate of the interaction of a view fails to emphasize the fact that the residuals are sim-presentation speed, number of targets, and effect size in the Stangor andMcMillan (1992) meta-analysis data. Panel B is a completely linear predic- ply a byproduct of the model used, and that different modelstion surface as would be obtained using common least squares regression. will lead to different patterns of residuals. As we saw in the previous section, different three-dimensional models provide different degrees of hugging the data, and hence, different For another impression of the data, we can take a series of amounts of residual. Second, in EDA the analyst focuses onslices of the three-dimensional data cube and lay the slices the size and pattern of individual residuals and subsets ofout side by side. Such an arrangement is possible using a gen- residuals. A curve that remains in a residual plot indicates theeral framework called a trellis plot (Becker, Cleveland, & model has failed to describe the curve. Multiple modes re-Shyu, 1996; Clark, Cleveland, Denby, & Liu, 1999) as maining in the residuals likewise suggest that a pattern hasimplemented in Splus (Insightful, 2001) and shown in Fig- been missed. On the other hand, if students are taught toure 2.11. The three panels show progressive slices of the data focus on only the gross summary of the sums of squares, theywith linear regression lines overlaid. As the reader can see, will also miss much of the detail in the pattern that is affordedthe plots correspond to three slices of the three-dimensional by a careful look at residuals. For example, as indicated bycube shown in Figure 2.10, panel A, with changes in the re- the common r among the Anscombe (1973) data sets, all fourgression line matching different portions of the “hills” in the data sets have the same sums-of-squares residual, but dra-data. matically different patterns of residuals.

Hallmarks of EDA 49 3 6 9 TARGET: 1.0 to 16.5 TARGET: 18.0 to 121.0 2 Ϫ1 Effect size TARGET: 1.0 to 1.0 TARGET: 1.0 to 1.0 2 Ϫ1 3 6 9 Presentation speed Figure 2.11 Plot of presentation speed by effect size at different ranges of number of targets using a trellis display. Data are identical with those presented in Figure 2.10. This emphasis on residuals leads to an emphasis on an iter- subsequent examination of the data, R. A. Fisher (1936)ative process of model building: A tentative model is tried questioned the validity of Mendel’s reported results, arguingbased on a best guess (or cursory summary statistics), residu- that Mendel’s data seemed “too good to be true.” Using chi-als are examined, the model is modified, and residuals are square tests of association, Fisher found that Mendel’s resultsreexamined over and over again. This process has some were so close to the predicted model that residuals of the sizeresemblance to forward variable selection in multiple regres- reported would be expected by chance less than once insion; however, the trained analyst examines the data in great 10,000 times if the model were true.detail at each step and is thereby careful to avoid the errors that Reviewing this and similar historical anomalies, Pressare easily made by automated procedures (cf. Henderson & and Tanur (2001) argue that the problem is caused by theVelleman, 1981). Tukey (1977) wrote, “Recognition of the unchecked subjectivity of scientists who had the confirma-iterative character of the relationship of exposing and tion of specific models in mind. This can be thought of assummarizing makes it clear that there is usually much value in having a weak sense of residuals and an overemphasis onfitting, even if what is fitted is neither believed nor satisfacto- working for dichotomous answers. Even when residuals ex-rily close” (p. 7). isted, some researchers tended to embrace the model for fear The emphasis on examining the size and pattern of resid- that by admitting any inconsistency, the entire model woulduals is a fundamental aspect of scientific work. Before this be rejected. Stated bluntly, those scientists had too muchnotion was firmly established, the history of science was re- focus on the notion of DATA = MODEL. Gould (1996) pro-plete with stories of individuals that failed to consider misfit vides a detailed history of how such model-confirmation bi-carefully. For example, Gregor Mendel (1822–1884), who is ases and overlooked residuals led to centuries of unfortunateconsidered the founder of modern genetics, established the categorization of humans.notion that physical properties of species are subject to hered-ity. In accumulating evidence for his views, Mendel con- The Two-Way Fitducted a fertilization experiment in which he followed sev-eral generations of axial and terminal flowers to observe how To illustrate the generality of the model-residual view ofspecific genes carried from one generation to another. On EDA, we will consider the extremely useful and flexible

50 Exploratory Data Analysismodel of the two-way fit introduced by Tukey (1977) and original data value in the data matrix. After this simple firstMosteller and Tukey (1977). The two-way fit is obtained by pass, we have a new table in which each cell is a residual anditeratively estimating row effects and column effects and the data from the original table are equal to the column effectusing the sum of those estimates to create predicted (model or plus the cell residual. The row effects are estimated next byfit) cell values and their corresponding residuals. The cycles calculating the median value of residuals in each row andare repeated with effects adjusted on each cycle to improve subtracting the cell values (first-stage residuals) from thesethe model and reduce residuals until additional adjustments medians. The row effects are generally placed in the marginprovide no improvement. This procedure can be applied di- of the table and the new residuals replace the residuals fromrectly to data with two-way structures. More complicated the previous stage. A similar calculation occurs on the row ofstructures can be modeled by multiple two-way structures. In medians that represents the column effects; the median ofthis way, the general approach can subsume such approaches the column effects becomes the estimate of the overall oras the measures of central tendency in the ANOVA model, the “grand” effect and the estimates of the column effects areratios in the log-linear model, and person and item parameter likewise adjusted through subtraction.estimates of the one-parameter item response theory model. This process is repeated iteratively until continued calcu- Consider the data presented in Table 2.1. It represents av- lation of effects and residuals provides no improvement. Theerage effect sizes for each of a series of univariate analyses result of such a summary is provided in Table 2.2. Hereconducted by Stangor and McMillan (1992). Such a display we see an overall effect of −.02 as well as a characterizationis a common way to communicate summary statistics. From of each row and column. It is clear which columns arean exploratory point of view, however, we would like to see low, medium, and high, and likewise which rows stand out.if some underlying structure or pattern can be discerned. Re- Each cell in the original table can be reconstructed usingviewing the table, it is easy to notice that some values are the formula Data = grand effect + row effect + columnnegative and some positive, and that the large number of effect + residual. For example, the memorization task for theϪ2.6 is a good bit larger than most of the other numbers bias condition can be recreated using the model of 1.01 =which are between 0 and +/− 1.0. −0.02 + 0.69 + 0.14 − 0.02. To suggest an initial structure with a two-way fit we cal- To form a visual impression of these values, Tukey (e.g.,culate column effects by calculating the median of each col- Mosteller & Tukey, 1977) recommended a two-way fit plotumn. The median of each column then becomes the model for such as that shown in Figure 2.12, panel A. In this figure,that column, and we subtract that initial model estimatefrom the raw data value to obtain a residual that replaces the TABLE 2.2 Two-Way Decomposition of Average Effect Sizes by Dependent Variable and Study Characteristic. From Stangor andTABLE 2.1 Average Effect Sizes by Dependent Variable and Study McMillan (1992).Characteristic. From Stangor and McMillan (1992). Variable Recall Recognition Bias Row EffectVariable Recall Recognition Bias Strength of expectationsStrength of expectations a. Experimental session 0.00 0.43 0.00 −0.35 a. Experimental session −0.37 −0.47 0.32 b. Existing 0.08 −0.51 0.00 0.26 b. Existing 0.32 −0.8 0.93 Content of the stimuliContent of the stimuli c. Behaviors −0.18 0.46 0.00 −0.00 c. Behaviors −0.21 −0.1 0.66 d. Traits 0.00 −2.34 0.58 0.73 d. Traits 0.71 −2.16 1.98 Type of behavioral inconsistencyType of behavioral inconsistency e. Evaluative and descriptive 0.00 0.90 −0.13 −0.25 e. Evaluative and descriptive −0.27 0.1 0.29 f. Descriptive only 0.20 −0.17 0.00 0.19 f. Descriptive only 0.36 −0.54 0.85 Type of targetType of target g. Individual 0.00 −0.29 0.67 −0.30 g. Individual −0.32 −1.14 1.04 h. Group 0.07 0.00 −0.51 0.17 h. Group 0.22 −0.38 0.33 Processing goalProcessing goal i. From impressions −0.34 0.84 0.00 −0.10 i. From impressions −0.46 0.19 0.57 j. Memorize 0.00 −0.30 0.20 0.14 j. Memorize 0.12 −0.71 1.01 Interpolated taskInterpolated task k. No −0.37 0.30 0.00 −0.05 k. No −0.44 −0.30 0.62 l. Yes 0.00 −0.79 0.00 0.08 l. Yes 0.06 −1.26 0.75 Type of delayType of delay m. Within single session −0.07 0.00 0.25 −0.10 m. Within single session −0.19 −0.65 0.82 n. Separate session 0.00 0.52 −0.01 0.00 n. Separate session −0.02 −0.03 0.66 Column effects 0.00 −0.53 0.69 −0.02

Hallmarks of EDA 51(A) bias when we first characterized row d, it was with a focus on the large −2.16 value, which is the largest (negative) value in the 1.0 table. This graphic, however, suggests more needs to be considered. Reexamining the data for that row we see that not recall only does that row have the largest negative value, but also 0.5 two of the largest positive values. All together, we end up with a strong positive row effect. For individual cells, how- recognition ever, the effect may be low or high, depending on the column. d 0.0 Because the grand, row, and column effects represent the model, an assessment of that model requires an examination b of where the model fits and does not fit. The residuals for h f Ϫ0.5 lj these models are presented in the center of Table 2.2. Exami- oc k in nation of these values reveals that the extreme value observed e g in the raw data remains extreme in this model, and that this a value is not simply the result of combining row and column(B) 2 effects. A graphical analog to the residuals can also be provided, as shown in Figure 2.12, panel B. In this diagram, lines are drawn from the value of the predicted values (the inter- bias section of row and column lines), downward or upward to the actual data value. The length of each line thereby indicates 1 the size of the residual. Clearly, the size of the trait residual recall for recognition tasks dwarfs the size of all other effects and residuals in the data. Other patterns of residuals may provide additional information about the data because they tell us recognition d what departs from a standard description. 0 In this example we used simple raw residuals. In other ap- b plications, the actual value of residuals may be modified in a j h f ol c number of ways. One common method is to report residuals ink reexpressed as normal-deviates in the distribution of residu- e ag als. This approach, often used in structural equation analysis, Ϫ1 can help identify the locations of the extremes, but hides the scale values of the error. In the highly developed area of regression diagnostics, residuals may be adjusted for the size of the leverage associated with the value of the criterion variable (studentized residuals) or calculated using a model that Ϫ2 obtained predicted values without the presence of the obser-Figure 2.12 Panel A is a plot of main effects of a two-way fit. The height vation in the model (externally studentized). This preventsof the intersection of lines represents the sum of row, column, and grand extreme values from distorting the model to the point thateffects and the corresponding predicted value for that cell. Panel B shows an aberrant value leads to a small residual, as displayed inmain effects of two-way fit with additional lines extending from predicted toactual values to highlight the location and size of residuals. panel C of Figure 2.2. As illustrated previously, complementary to the notion of patterns of residuals and meaning in individual residuals is thethere is a line for each row and column effect in the model. emphasis on mathematical models of effects that provide richFor each row and column effect, the height of the intersection and parsimonious description. This view is very much in lineof the two lines is equal to the value of the predicted value in with the recently emerging view of the importance of effectthe model (overall effect + column effect + row effect). This sizes suggested by Glass (1976) and renewed by Cohen (1994)plot clearly portrays the separation of the bias, recall, and and the APA Task Force on Statistical Inference (Wilkinson,recognition conditions, shows the clear separation of row d 1999). EDA reminds us that at the same time we focus on ef-(trait levels), and displays a cluster of common small effects fects as a description of the data, we must also focus on the sizefor rows a, g, and e. For us, this view was surprising because and pattern of misfits between effects and the data.

52 Exploratory Data AnalysisReexpression expression for positively skewed data is the logarithmic transformation. For ratios of counts, the most common rec-The data examined previously remind us that data often come ommendation is to “fold” the counts around a midpointto the exploratory data analyst in messy and nonstandard (usually .5) so that equal fractions equal 0. This generallyways. This should not be unexpected, given the common as- means using P/1 − P, where P is the proportion of the totalsumption that the data distributions are either always well be- that the count comprises. A second step is to take the log ofhaved, or that statistical techniques are sufficiently robust this folded fraction to create a “flog” equal to log(P/1 − P).that we can ignore any deviations that might arise, and there- In more common parlance, this is a logit that serves as thefore skip detailed examination. In fact, it is quite often the basis for logistic regression, survival, or event-history analy-case that insufficient attention has been paid to scaling issues sis, and measurement via item response theory. Additionalin advance, and it is not until the failure of confirmatory techniques recommend that balances should generally be leftmethods that a careful examination of scaling is undertaken alone whereas grades and ranks should be treated much like(if at all). In the exploratory mode, however, appropriate scal- counted fractions (see, e.g., Mosteller & Tukey, 1977).ing is considered one of the fundamental activities and is Although reexpression is a long-standing practice in thecalled reexpression. Although mathematically equivalent to statistical community, going back at least to Fisher’s (1921)what is called transformation in other traditions, reexpres- construction of the r to z transformation, only recently has itssion is so named to reflect the idea that the numerical changes use become more widespread in psychological literature. Inare aimed at appropriate scaling rather than radical change. fact, it often continues to arise more out of historic tradition Because reexpression requires an understanding of the un- than as the result of careful and comprehensive analysis.derlying meaning of the data that are being reexpressed, the Consider, for example, the subset of data from a word-EDA approach avoids using the common categorizations of recognition experiment recently reported by Paap, Johansen,data as nominal, ordinal, interval, and ratio that follow Chun, and Vonnahme (2000) and depicted in Figure 2.13.Stevens (e.g., 1951). Rather, Mosteller and Tukey (1977) The experiment reported in this paper concerns the percent-discussed broad classes of data as (a) amounts and counts; age of times participants correctly identify word pairs (%C)(b) balances (numbers that can be positive or negative with from a memory task as a function of the word pair’s correct-no bound); (c) counted fractions (ratios of counts); (d) ranks; incorrect confusability (CIC), percentage correct-letter dis-and (e) grades (nominal categories). tinctiveness (CD), number of neighbors (N), percentage of When dealing with common amounts and counts, Tukey friends in the lexical neighborhood (%F), number of highersuggested heuristics that hold that (a) data should often be frequency neighbors (H), log of frequency of the test wordreexpressed toward a Gaussian shape, and (b) an appro- (LTF), and log of frequency formed by incorrect alternativepriate reexpression can often be found by moving up or down (LAF).“the ladder of reexpression.” A Gaussian shape is sought As the reader may see, although the distributions associ-because this will generally move the data toward more equal- ated with the logarithmic reexpression are quite Gaussian, theinterval measurement through symmetry, will often stabilize variables that are not reexpressed differ quite a bit in thisvariance, and can quite often help linearize trend (Behrens, respect and lead to quite non-Gaussian bivariate distribu-1997a). In EDA, the term normal is avoided in favor of tions. The CIC variable is the most skewed. This leads toGaussian to avoid the connotation of prototypicality or social quite distorted correlations that would suffer from variancedesirability. compression. Distributional outliers in CIC are indicated The ladder of reexpression is a series of exponents one with X symbols, and regression lines on the %C against CICmay apply to original data that show considerable skew. Rec- scatter plot are present both for all data (lower line) and forognizing that the raw data exists in the form of X1, moving up the data with the outliers removed (sloped line).the ladder would consist of raising the data to X2 or X3. Because these authors have already reexpressed the twoMoving down the ladder suggests changing the data to the frequency (count) variables, it will be useful to reverse thescale of X1/2, −X−1/2, −X−1, −X−2, and so on. Because X0 is reexpression to see the original data, which are presentedequal to 1, this position on the ladder is generally replaced in Figure 2.14. The top panels show the histograms of thewith the reexpression of log10 (X). To choose an appropriate original raw data along with the quantile-quantile (QQ) plot,transformation, one moves up or down the ladder toward the which is often called the normal-probability plot in cases likebulk of the data. This means moving down the ladder for these because the ranks of the data are plotted against the zdistributions with positive skew and up the ladder for distrib- scores of corresponding ranks in a unit-normal distribution.utions with negative skew. By far the most common re- When the points of the QQ plot are straight, this reflects the

Hallmarks of EDA 53 Figure 2.13 Scatter plot matrix of reaction time data from Paap, Johansen, Chun, and Vonnahme (2000). Patterns in the matrix reflect skew and outliers in some marginal distributions as well as nonlinearity in bivariate distributions. Outliers are highlighted in all graphics simultaneously through brushing and linking.match between the empirical and theoretical distributions, aside, the variance for the underlying binomial distribution iswhich in this case is Gaussian. The data running along the largest around .5 and smallest near 0 and 1. As noted above,bottom of these displays reflect the numerous data values at this mathematical situation leads to a general rule for substi-the bottom of the scales for the original frequency data. Pan- tuting logits (a.k.a., flogs) for raw percentages. Accordingly,els E and F show the scatter plot of the raw data before reex- we move forward by converting each percentage into a pro-pression and the corresponding simple regression residuals, portion (using the sophisticated process of dividing by 100)which indicate that the spread of the error is approximately and constructing the logit for each proportion. The effect ofequal to the spread of the data. Although this logarithmic this reexpression on the %C and CIC variables is portrayed intransformation is quite appropriate, it was chosen based upon Figure 2.15. As the reader can see, the distributions arehistorical precedent with data of this type rather than on em- greatly moved toward Gaussian, and the appearance of thepirical examination. Accordingly, in the view of EDA, the scatter plot changes dramatically.outcome is correct while the justification is lacking. The impact of this reexpression is considerable. Using the Turning to how remaining variables may be improved, we correlation coefficient in the original highly skewed andconsider the four percentage variables, especially the highly variance-unstable scale of percentages resulted in a measuredistorted CIC variable. Working directly with percentages of association of r = .014, suggesting no relationship be-can be quite misleading because differences in values are not tween these two values. However, when the scale value andequally spaced across the underlying continuum. For exam- corresponding variance are adjusted using the logistic reex-ple, it is generally easier to move from an approval rating of pression, the measure of relationship is r = .775—a dramatic50 to 55% than it is to move from 90 to 95%. All content difference in impression and likely a dramatic effect on

54 Exploratory Data Analysis (A) 150 (B) 10000 7500 10ˆ LTF 100 5000 50 2500 0 2500 5000 7500 10000 Ϫ1.5 0 1.5 nscores 10ˆ LTF (C) 150 (D) 10000 7500 10ˆ LAF 100 5000 50 2500 0 2500 5000 7500 10000 Ϫ1.5 0 1.5 nscores 10ˆ LAF (E) (F) 10000 10000 residuals(10L) 7500 7500 10ˆ LTF 5000 5000 2500 2500 0 2500 5000 7500 2500 5000 7500 10ˆ LAF 10ˆ LAF Figure 2.14 Histograms, normal-probability plots, and scatter plots for reaction time data as they would appear without the logarithmic reexpression used by the original authors.theory development and testing. In a similar vein, Behrens than the simple count. When such a proportion is computed(1997a) demonstrated how failure to appropriately reexpress and converted to a logit, the logit-H variable becomes verysimilar data led Paap and Johansen (1994) to misinterpret the well behaved and leads to much clearer patterns of data andresults of a multiple regression analysis. As in this case, a residuals. The revised scatter plot matrix for these variables issimple plotting of the data reveals gross violations of distrib- presented in Figure 2.16. As the reader may see, a dramaticutional assumptions that can lead to wildly compressed cor- improvement in the distributional characteristics has beenrelations or related measures. obtained. The H variable is likewise of interest. Because it is a Although some researchers may reject the notion of reex-count, general heuristics suggest a square-root or logarithmic pression as “tinkering” with the data, our experience has beenreexpression. Such reexpressions, however, fail to improve that this view is primarily a result of lack of experience withthe situation substantially, so another course of action is re- the new scales. In fact, in many instances individuals oftenquired. Because the H variable is a count of high-frequency use scale reexpressions with little thought. For example,neighbors, and this number is bounded by the number of the common practice of using a proportion is seldom ques-neighbors that exist, a logical alternative is to consider H as tioned, nor is the more common reexpression to z scores. Inthe proportion of neighbors that are high frequency rather daily language many people have begun to use the log10

Hallmarks of EDA 55 (A) 25 (B) 20 1.2 Logit(%C) 15 0.8 10 0.4 5 0 Ϫ0.1 0.4 0.9 1.4 Ϫ1.5 0 1.5 Logit(%C) nscores (C) 25 (D) 0 20 Logit(CIC) Ϫ0.5 15 10 Ϫ1.0 5 Ϫ1.5 Ϫ1.8 Ϫ1.3 Ϫ0.8 Ϫ0.3 0.2 Ϫ1.5 0 1.5 Logit(CIC) nscores (E) 1.2 Logit(%C) 0.8 0.4 0 Ϫ1.5 Ϫ1.0 Ϫ0.5 0 Logit(CIC) Figure 2.15 Histograms, normal-probability plots, and scatter plot for percent correct (%C) and percent congruent-incongruent ratio (CIC) following logistic reexpression.reexpression of dollar amounts as “five-figure,” “six-figure,” mean has a smaller standard error than the median, and so isor “seven-figure.” Wainer (1977) demonstrated that the often an appropriate estimator for many confirmatory tests. On therecommended reexpression of 1͞X for reaction-time tasks other hand, the median is less affected by extreme scores orsimply changes the scale from seconds per decision (time) to other types of perturbations that may be unexpected or un-decisions per second (speed). Surely such tinkering can have known in the exploratory stages of research.great value when dramatic distributional improvements are In general, there are three primary strategies for improv-made and sufficient meaning is retained. ing resistance. The first is to use rank-based measures and absolute values, rather than measures based on sums (such asResistance the mean) or sums of squares (such as the variance). Instead, practitioners of EDA may use the tri-mean, which is the aver-Because a primary goal of using EDA is to avoid being age of Q1, Q3, and the median counted twice. For measuresfooled, resistance is an important aspect of using EDA tools. of spread, the interquartile range is the most common,Resistant methods are methods that are not easily affected by although the median absolute deviation (MAD) from theextreme or unusual data. This value is the basis for the gen- median is available as well. The second general resistance-eral preference for the median rather than the mean. The building strategy is to use a procedure that emphasizes more

56 Exploratory Data Analysis Figure 2.16 Scatter plot matrix of reaction time data after full set of reexpressions. When compared with the original data shown in Figure 2.13, this matrix reflects the improved ability to model the data using linear models.centrally located scores and that uses less weight for more simple use of a scatter plot matrix revealed three consistentextreme values. This category includes trimmed statistics in outliers in distributions of variables measured in dollars. Thewhich values past a certain point are weighted to 0 and first outlier was the observation associated with the District ofthereby dropped from any estimation procedures. Less dras- Columbia. How should this extreme value be approached? Iftic approaches include the use of the biweight, in which val- the original intention was state-level analysis, the outlier inues are weighted as an exponential function of their distance the data simply calls attention to the fact that the data were notfrom the median. A third approach is to reduce the scope of prescreened for non-states. Here the decision is easy: Reestab-the data one chooses to model on the basis of knowledge lish the scope of the project to focus on state-level data.about extreme scores and the processes they represent. The remaining two outliers were observations associated with the states of Hawaii and Alaska. These two states hadDealing with Outliers values that were up to four times higher than the next highest values from the set of all states. In many cases, the mean ofThe goal of EDA is to develop understandings and descrip- the data when all 50 states were included was markedly dif-tions of data. This work is always set in some context and ferent from the mean computed using values from only thealways presents itself with some assumptions about the scope contiguous 48 states. What should the data analyst do aboutof the work, even when these assumptions are unrecognized. this problem? Here again, the appropriate role of the ex-Consider, for example, the task described in Behrens and ploratory work has led to a scope-clarification process thatSmith (1996) of developing a model of state-level economic many data analysts encounter in the basic question Do I modelaspects of education in the United States. In this analysis, all the data poorly, or do I model a specific subset that I can

Current Computing and Future Directions 57describe well? Although this question needs to be answered Summaryon a case-by-case basis, in the situation described here there islittle doubt. Alaska and Hawaii should be set aside and the re- The tools described in this section are computational andsearcher should be content to construct a good model for the conceptual tools intended to guide the skilled and skeptical48 contiguous states. Furthermore, the researcher should note data analyst. These tools center on the four Rs of revelation,that he or she has empirical evidence that Alaska and Hawaii residuals, reexpression, and resistance. The discussionfollow different processes. This is a process of setting aside provided here provides only a glimpse into the range of tech-data and focusing scope. Clearly this process of examination niques and conceptualizations in the EDA tradition. In prac-and scope revision would need to be reported. tice, the essential elements of EDA center on the attitudes of In this case, the rationale is clear and the data have seman- flexibility, skepticism, and ingenuity, all employed with thetic clarity. In other cases, however, quite extreme values may goals of discovering patterns, avoiding errors, and develop-be found in data that are not simply victims of poor measure- ing descriptions.ment models (e.g., the end of a long tail awaiting logarithmicreexpression). Under these circumstances the fundamental CURRENT COMPUTING AND FUTUREquestion to ask is Do we know something about these obser- DIRECTIONSvations that suggest they come from a different process thanthe process we are seeking to understand? In experimentally Computing for EDAoriented psychology, rogue values could be caused by nu-merous unintended processes: failure to understand instruc- While EDA has benefited greatly from advances in statisticaltions (especially during opening trials), failure to follow the computing, users are left to find the necessary tools and ap-instructions, failure to pay attention to the task (especially propriate interfaces spread across a variety of statistical pack-during closing trials), or equipment or data transcription fail- ages. To date, the software most clearly designed for EDA isures. Under such circumstances, it is clear the data are not in Data Desk (Data Description, 1997). Data Desk is highly in-the domain of the phenomenon to be studied, and the data teractive and completely graphical. Graphical windows inshould be set aside and the situation noted. Data Desk do not act like “static” pieces of paper with In other cases, extreme values present themselves with lit- graphic images, but rather consist of “live” objects that re-tle auxiliary information to explain the reason for the extreme- spond to brushing, selecting, and linking. The philosophy isness. In such a situation we may first assess how much damage so thoroughly graphical and interactive that additional vari-the values create in the model by constructing the model with ables can be added to regression models by dragging icons ofall the data involved as well as with the questionable data the variables onto the regression sums-of-squares table.set aside. For example, Behrens (1997b) conducted a meta- When this occurs, the model is updated along with all graph-analysis of correlations between subscales of the White Racial ics associated with it (e.g., residual plots). Data Desk has aIdentity Attitude Scale (Helms, 1997). Initial review of the full range of EDA-oriented tools as well as many standardsdata suggested the distributions were not homogeneous and including multivariate analysis of variance (MANOVA) andthat some study results differed dramatically from the average. cluster analysis—all with graphical aids. Because of its out-To assess the effect of these extreme values, Behrens calcu- standing emphasis on EDA, Data Desk has been slower to de-lated the average correlations, first, using all the data, and sec- velop more comprehensive data management tools, as hasond, using a 20% trimmed mean. Results were consistent been the case with more popular tools like those offered byacross approaches, suggesting the data could remain or be set SAS, SPSS, and SYSTAT.aside with little impact on the inferences he was to make. What Both SAS (SAS Institute, 2001) and SPSS (SPSS, Inc.,would have happened if, on the other hand, the trimmed re- 2001) have improved their graphical interactivity in recentsults deviated from the full-data results? In such a case both years. The SAS Insight module allows numerous interactivesets of analysis should be conducted and reported and the dif- graphics, but linking and other forms of interaction are lessference between the two results considered as a measure of the comprehensive than those found in Data Desk. SYSTATeffect of the rogue values. The most important aspect in either serves as a leader in graphical and exploratory tools, placingcase is that a careful and detailed description of the full data, itself between the complete interactivity of Data Desk and thethe reduced data, and the impact of the rogue data be reported. more standard approach of SAS and SPSS.Unfortunately, the extremely terse and data-avoidant descrip- The S language and its recent incarnation as S-PLUStions of much research reporting is inconsistent with this (Insightful, 2001) has long been a standard for research inhighly descriptive approach. statistical graphics, although it appears also to be emerging as

58 Exploratory Data Analysisa more general standard for statistical research. Venables and computing, more and more statistical software packagesRipley’s 1999 book titled Modern Applied Statistics with incorporate graphical tools that utilize other spatial systems.S-PLUS (often called MASS) provides an outstanding intro- For example, several statistical packages implement the bi-duction to both the Splus language and many areas of modern plot (Gabriel, 1981; Gower & Hand, 1996), which combinesstatistical computing of which psychologists are often un- variable space and subject space (also known as vectoraware, including projection pursuit, spline-based regression, space). In subject space each subject becomes a dimension,classification and regression trees (CART), k-means cluster- and vectors are displayed according to the location of vari-ing, the application of neural networks for pattern classifica- ables mapping into this subject space. In addition, SYSTATtion, and spatial statistics. implements additional projections, including triangular dis- High-quality free software is also available on the Internet plays, in which both the Cartesian space and the barycentricin a number of forms. One consortium of statisticians has cre- space are used to display four-dimensional data.ated a shareware language, called R, that follows the same Interactive methods for traversing multivariate data havesyntax rules as S-PLUS and can therefore be used inter- been developed as well. Swayne, Cook, and Buja (1998)changeably. A number of computing endeavors have been developed the X-GOBI system, which combines the high-based on Luke Tierney’s XLISP-STAT system (Tierney, dimensional search techniques of projection pursuit1990), which is highly extensible and has a large object- (Friedman & Tukey, 1974) and grand tour (Asimov, 1985).oriented feature set. Most notable among the extensions is The grand tour strategy randomly manipulates projections forForrest Young’s (1996) ViSta (visual statistics) program, high-dimensional data using a biplot or similar plotting sys-which is also free on the Internet. tem, so that the user has an experience of touring “around” Despite the features of many of these tools, each comes three- (or higher) dimensional rotating displays. Projectionwith weaknesses as well as strengths. Therefore, the end user pursuit is a computing-intensive method that calculates anmust have continuing facility in several computing environ- “interestingness function” (usually based on nonnormality)ments and languages. The day of highly exchangeable data and develops search strategies over the multidimensional gra-and exchangeable interfaces is still far off. dient of this function. Grand tour can provide interesting views but may randomly generate noninteresting views forFuture Directions quite some time. Projection pursuit actively seeks interesting views but may get caught in local minima. By combining theseThe future of EDA is tightly bound to the technologies of two high-dimensional search strategies and building them intocomputing, statistics, and psychology that have supported its a highly interactive and visual system, these authors leveragedgrowth to date. Chief among these influences is the rise of the best aspects of several advanced exploratory technologies.network computing. Network computing will bring a numberof changes to data analysis in the years ahead because a net- Data Immersionwork allows data to be collected from throughout the world,allows the data to have extensive central or distributed stor- To deal with the increasing ability to collect large data sets,age, allows computing power a distributed or centralized lo- applications of EDA are likely to follow the leads developedcation, and allows distribution of results quickly around the in high-dimensional data visualization used for physicalworld (Behrens, Bauer, & Mislevy, 2001). With regard to the systems. For example, orbiting satellites send large quantitiesincrease in data acquisition, storage, and processing power, of data that are impossible to comprehend in an integratedthese changes will lead to increasing availability and need for way without special rendering. To address these issues,techniques to deal with large-scale data. With increasingly researchers at the National Aeronautics and Space Adminis-large data sets, data analysts will have difficulty gaining de- tration (NASA) have developed tools that generate images oftailed familiarity with the data and uncovering unusual pat- planetary surface features that are rendered in three-terns. Hopefully, the capacity for ingenuity and processing dimensional virtual reality engines. This software creates anpower will keep up with the increase in data availability. imaginary topology from the data and allows users to “fly” through the scenes. Although most psychologists are unlikely to see such hugeData Projections (literally astronomical!) quantities of data, desktop computersCurrently, most existing visualization tools are based upon can provide multimedia assistance for the creation of inter-variable space, in which data points are depicted within the active, three-dimensional scatter plots and allow the animationCartesian coordinates. With the advent of high-powered of multidimensional data (e.g., Yu & Behrens, 1995).

Current Computing and Future Directions 59Distributed Collaboration journals, under tight restrictions for graphics and space, have largely omitted the reporting of exploratory results or de-Because data analysis is a social process and groups of re- tailed graphics. Although a textual emphasis on reporting wassearchers often work together, EDA will also be aided by the necessary for economic reasons in previous centuries, the risedevelopment of computer–desktop sharing technologies. In- of network-based computing, interactive electronic informa-ternetworking technologies currently exist that allow individ- tion display, and hypertext documents supports the expansionuals to share their views of their computer screens so that of the values of EDA in scientific reporting. In a paper-basedreal-time collaboration can occur. As statistical packages be- medium, narrative development generally needs to follow acome more oriented toward serving the entire data-analytic linear development. On the other hand, in a hypertext envi-process, developers will consider the social aspects of data ronment the textual narrative can appear as traditionally im-analysis and build in remote data-, analysis-, image-, and plemented along with auxiliary graphics, detailed computerreport-sharing facilities. Such tools will help highly trained output, the raw data, and interactive computing—all at adata analysts interact with subject-matter experts in schools, second level of detail easily accessed (or ignored) throughclinics, and businesses. hypertext links. In this way, the rich media associated with EDA can complement the terse reporting format of theHypermedia Networks for Scientific Reporting American Psychological Association and other authoringWhile the natures of scientific inquiry, scientific philosophy, styles (Behrens, Dugan, & Franz, 1997).and scientific data analysis have changed dramatically in the To illustrate the possibility of such a document structur-last 300 years, it is notable that the reporting of scientific ing, Dugan and Behrens (1998) applied exploratory tech-results differs little from the largely text-based and tabular niques to reanalyze published data reported in hypertext onpresentations used in the eighteenth century. Modern print the World Wide Web. Figure 2.17 is an image of the interface Figure 2.17 Screen appearance of electronic document created from EDA perspective by Dugan and Behrens (1998).

60 Exploratory Data Analysisused by these authors. The pages were formatted so that the CONCLUSIONnames of variables were hyperlinked to graphics that ap-peared on a side frame, and the names of references were hy- Despite the need for a wide range of analytic tools, trainingperlinked to the references that appeared on the top frame of in psychological research has focused primarily on statisticalthe page. References to F tests or regression-results linked to methods that focus on confirmatory data analysis. Ex-large and well-formatted result listings, and the data were hy- ploratory data analysis (EDA) is a largely untaught andperlinked to the paper as well. overlooked tradition that has great potential to guard psy- While we wait for arrival of widespread hypertext in chologists against error and consequent embarrassment. Inscientific journals, personal Web sites for improving the re- the early stages of research, EDA is valuable to help find theporting of results can be used. For example, Helms (1997) unexpected, refine hypotheses, and appropriately plan futurecriticized the analysis of Behrens (1997b), which questioned work. In the later confirmatory stages, EDA is valuable tothe psychometric properties of a commonly used scale in the ensure that the researcher is not fooled by misleading aspectscounseling psychology racial-identity literature. Part of the of the confirmatory models or unexpected and anomalousconcern raised by Helms was that she expected large amounts data patterns.of skew in the data, and hence, likely violations of the statis- There are a number of missteps the reader can make whentical assumptions of the meta-analyses and confirmatory fac- faced with introductory materials about EDA. First, sometor analyses that Behrens (1997b) reported. In reply, Behrens readers may focus on certain aspects of tradition and see theirand Rowe (1997) noted that the underlying distributions had own activity in that area as compelling evidence that theybeen closely examined (following the EDA tradition) and are already conducting EDA. Chief among these aspects isthat the relevant histograms, normal-probability plots, scatter the use of graphics. By showing a broad range of graphics,plot matrices (with hyperlinks to close-up views), and the we sought to demonstrate to the reader that statistical graph-original data were all on the World Wide Web (Behrens & ics has become a specialization unto itself in the statisticsDugan, 1996). This supplemental graphical archive included literature, and that there is much to learn beyond what isa three-dimensional view of the data that could be navigated commonly taught in many introductory courses. Whereas theby users with Web browsers equipped with commonly avail- exploratory data analyst may use graphics, the use of graph-able virtual-reality viewers. Such archiving quickly moves ics alone does not make an exploratory data analyst.the discussion from impressions about possibilities regarding A second pitfall the reader should be careful to avoid is re-the data (which can be quite contentious) to a simple display jecting the relevance of the examples used in this chapter.and archiving of the data. Some might argue that the pathological patterns seen herein exist only in data from meta-analyses or reaction time exper-Summary iments, or in educational data. Our own work with many types of data sets, and in conversations with psychologists inEmerging tools for EDA will continue to build on develop- numerous specializations, suggests that these are not isolatedments in integration of statistical graphics and multivariate or bizarre data sets but are quite common patterns. The readerstatistics, as well as developments in computer interface de- is encouraged to reanalyze his or her own data using some ofsign and emerging architectures for collecting, storing, and the techniques provided here before making judgments aboutmoving large quantities of data. As computing power contin- the prevalence of messy data.ues to increase and computing costs decrease, researchers A third pitfall to avoid is overlooking the fact that em-will be exposed to increasingly user-friendly interfaces and bracing EDA may imply some confrontation with traditionalwill be offered tools for increasingly interactive analysis and values and behaviors. If EDA is added to the methodologyreporting. In the same way that creating histograms and scat- curriculum then other aspects may need to be deemphasized.ter plots is common practice with researchers now, the con- If new software is desired, changes in budgets may need tostruction of animated visualizations, high-dimensional plots, occur, with their associated social conflicts. Additionally,and hypertext reports is expected to be commonplace in the conflict may arise within the researcher as he or she worksyears ahead. To offset the common tendency to use new tools to balance the value of EDA for scientific advancementfor their own sake, the emergence of new technologies cre- while finding little explicit value for EDA in manuscriptates an increased demand for researchers to be trained in the preparation.conceptual foundations of EDA. At the same time, the emer- Psychological researchers address complex and difficultgence of new tools will open doors for answering new scien- problems that require the best set of methodological toolstific questions, thereby helping EDA evolve as well. available. We recommend EDA as a set of conceptual and

References 61computational tools to supplement confirmatory statistics, Berk, K. N. (1994). Data analysis with Student SYSTAT. Cambridge,and expect psychological research will increase in efficiency MA: Course Technology.and precision by its wider applications. Beveridge, W. I. B. (1950). The art of scientific investigation. New York: Vintage Books. Box, G. E. P. (1979). Robustness in scientific model building. In R. L. Launer & G. N. Wilkinson (Eds.), Robustness in StatisticsREFERENCES (pp. 201–236). New York: Academic Press.Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and Braun, H. I. (Ed.). (1994). The collected works of John W. Tukey: interpreting interactions. Newbury Park, CA: Sage. Vol 8. Multiple comparisons, 1948–1983. New York: Chapman & Hall.Aiken, L. S., West, S. G., Sechrest, L., & Reno, R. R. (1990). Graduate training in statistics, methodology, and measurement in Breckler, S. J. (1990). Application of covariance structure modeling psychology: A review of Ph.D. programs in North America. in psychology: Cause for concern? Psychological Bulletin, 107, American Psychologist, 45, 721–734. 260–273.Anscombe, F. J. (1973). Graphs in statistical analysis. The American Brillinger, D. R. (Ed.). (1984). The collected works of John Statistician, 27, 17–21. W. Tukey: Vol 1. Time series: 1949–1964. Monterey, CA: Wadsworth.Asimov, D. (1985). The grand tour: A tool for viewing multidimensional data. SIAM Journal of Statistical Computing, 6, 138–143. Brillinger, D. R. (Ed.). (1985). The collected works of John W. Tukey: Vol 2. Time series: 1965–1984. Monterey, CA: Wadsworth.Bauer, M. I., & Johnson-Laird, P. N. (1993). How diagrams can improve reasoning. Psychological Science, 4(6), pp. 372–378. Brillinger, D. R., Fernholz, L. T., & Morgenthaler, S. (1997). The practice of data analysis: Essays in honor of John W. Tukey.Becker, R. A., Chambers, J. M., & Wilks, A. (1988). The new Princeton, NJ: Princeton University Press. S language: A programming environment for data analysis and graphics. New York: CRC Press. Clark, L. A., Cleveland, W. S., Denby, L., & Liu, C. (1999). Competitive profiling displays: Multivariate graphs for customerBecker, R. A., Cleveland, W. S., & Shyu, M. J. (1996). The visual satisfaction survey data. Marketing Research, 11, 25–33. design and control of Trellis Display. Journal of Computational and Statistical Graphics, 5, 123–155. Cleveland, W. S. (1985). The elements of graphing data. Monterey, CA: Wadsworth.Behrens, J. T. (1997a). Principles and procedures of exploratory data analysis. Psychological Methods, 2, 131–160. Cleveland, W. S. (Ed.). (1988). The collected works of John W. Tukey: Vol. 5. Graphics. Belmont, CA: Wadsworth.Behrens, J. T. (1997b). Does the white racial identity attitude scale measure racial identity? Journal of Counseling Psychology, 44, Cleveland, W. S. (1993). Visualizing data. Summit, NJ: Hobart 3–12. Press.Behrens, J. T. (2000). Exploratory data analysis. In A. E. Kazdin Cleveland, W. S., & McGill, R. (1984). Graphical perception: The- (Ed.), Encyclopedia of psychology (Vol. 3, pp. 303–305). New ory, experimentation, and application to the development of York: Oxford University Press. graphical methods. Journal of the American Statistical Associa- tion, 79, 531–554.Behrens, J. T., Bauer, M., & Mislevy, R. (2001, April). Future prospects for assessment in the on-line world. Paper presented at Cleveland, W. S., & McGill, R. (1988). Dynamic graphics for the Annual Meeting of the American Educational Research statistics. Monterey, CA: Wadsworth. Association, Seattle, WA. Cohen, J. (1994). The earth is round (p < .05). American Psycholo-Behrens, J. T., & Dugan, J. G. (1996). A graphical tour of the gist, 49, 997–1003. White Racial Identity Attitude Scale data in hypertext and Cook, R. D., & Weisberg, S. (1994). An introduction to regression VRML. Retrieved from http://research.ed.asu.edu/reports/wrias. graphics. New York: Wiley. graphical.tour/graphtour.html. Cox, D. R. (Ed.). (1992). The collected works of John W. Tukey:Behrens, J. T., Dugan, J. T., & Franz, S. (1997, August). Improving Vol. 7. Factorial and ANOVA, 1949–1962. Monterey, CA: the reporting of research results using the World Wide Web. Wadsworth. Paper presented at the 106th Annual Convention of the American Data Description, Inc. (1997). Data Desk (version 6.1) [computer Psychological Association, Chicago, IL. software]. Ithaca, NY: Data Description.Behrens, J. T., & Rowe, W. (1997). Measuring White racial identity: Dugan, J. G., & Behrens, J. T. (1998, November 18). A Hypermedia A reply to Helms (1997). Journal of Counseling Psychology, 44, exploration of the classification problem in special education. 17–19. Current Issues in Education, 1, (6). Retrieved from http://Behrens, J. T., & Smith, M. L. (1996). Data and data analysis. In cie.ed.asu.edu/volume1/number6/. D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational Fisher, R. A. (1921). On the “probable error” of a coefficient of psychology (pp. 949–989). New York: Macmillan. correlation deduced from a small sample. Metron, 1, 3–32.

62 Exploratory Data AnalysisFisher, R. A. (1935). The logic of inductive inference. Journal of the Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (Eds.). (1985). Exploring Royal Statistical Society, 98, 39–82. data tables, trends, and shapes. Reading, MA: Addison-Wesley.Fisher, R. A. (1936). Has Mendel’s work been rediscovered? Annals Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (Eds.). (1991). Funda- of Science, 1, 115–117. mentals of exploratory analysis of variance. Reading, MA:Fisher, R. A. (1955). Statistical methods and scientiﬁc induction. Addison-Wesley. Journal of the Royal Statistical Society B, 17, 69–78. Hoffmann, M. (1997). Is there a logic of abduction? Paper pre-Friedman, J. H., & Tukey, J. W. (1974). A projection pursuit sented at the 6th congress of the International Association for algorithm for Exploratory Data Analysis. IEEE Transactions on Semiotic Studies, Guadalajara, Mexico. Computers, 23, 881–889. Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regres-Frigge, M., Hoaglin, D. C., & Iglewicz, B. (1989). Some implemen- sion. New York: Wiley. tations of the boxplot. The American Statistician, 43, 50–54. Hume, D. (1912). An enquiry concerning human understanding,Gabriel, K. R. (1971). The biplot graphical display of matrices with and selections from a treatise of human nature. Chicago: Open application to principal component analysis. Biometrics, 58, Court. (Original work published 1777) 453–467. Insightful. (2001). Splus 6 [Computer software]. Available atGarner, W. R. (1974). The processing of information and structure. http://www.insightful.com/. Potomac, MD: Erlbaum. Johnson-Laird, P. N. (1983). Mental models: Towards a cognitiveGigerenzer, G. (1987). Probabilistic thinking and the ﬁght against science of language, inference, and consciousness. Cambridge, subjectivity. In L. Kroger, G. Gigerenzer, & M. S. Morgan MA: Harvard University Press. (Eds.), The probabilistic revolution: Vol. 2. Ideas in the sciences Jones, L. V. (Ed.). (1986a). The collected works of John W. Tukey: (pp. 11–33). Cambridge, MA: MIT Press. Vol. 3. Philosophy and principles of data analysis: 1949–1964.Gigerenzer, G. (1993). The superego, the ego, and the id in statisti- Belmont, CA: Wadsworth. cal reasoning. In G. Keren & C. Lewis (Eds.), A handbook for Jones, L. V. (Ed.). (1986b). The collected works of John W. Tukey: data analysis in the behavioral sciences: Methodological issues Vol. 4. Philosophy and principles of data analysis (1965–1986). (pp. 311–339). Hillsdale, NJ: Erlbaum. Belmont, CA: Wadsworth.Glass, G. V (1976). Primary, secondary, and meta-analysis of Josephson, J. R., & Josephson, S. G. (1994). (Ed.). Abductive research. Educational Researcher, 5, 3–8. inference: Computation, philosophy, technology. Cambridge,Gödel, C. (1986). Collected works. New York: Oxford University England: Cambridge University Press. Press. (Original work published 1947) Keselman, H. J., Huberty, C., Lix, L. M., Olejnik, S., Cribbie, R. A.,Gould, S. J. (1996). The mismeasure of man (2nd ed.). New York: Donahue, B., Kowalchuk, R. K., Lowman, L. L., Petoskey, Norton. M. D., & Keselman, J. C. (1998). Statistical practices of educa-Gower, J. C., & Hand, D. J. (1996). Biplots. London: Chapman & tional researchers: An analysis of their ANOVA, MANOVA, and Hall. ANCOVA analyses. Review of Educational Research, 68,Hacking, I. (1975). The emergence of probability: A philosophical 350–386. study of early ideas about probability, induction and statistical Kline, M. (1980). Mathematics: The loss of certainty. New York: inference. New York: Cambridge University Press. Oxford University Press.Härdle, W. (1991). Smoothing techniques with implementation in S. Kosslyn, S. (1994). Elements of graph design. New York: W. H. New York: Springer-Verlag. Freeman.Härdle, W., Klinke, S., & Turlach, B. A. (1995). XploRe: An Larkin, J., & Simon, H. (1987). Why a diagram is (sometimes) interactive statistical computing environment. New York: worth a thousand words. Cognitive Science, 11, 65–99. Springer-Verlag. Lehmann, E. L. (1993). The Fisher, Neyman-Pearson theories ofHelms, J. E. (1997). Implications of Behrens for the validity of the testing hypotheses: One theory or two? Journal of the American White Racial Identity Attitude Scale. Journal of Counseling Statistical Association, 88, 1242–1249. Psychology, 44, 13–16. Lewandowsky, S., & Behrens, J. T. (1999). Statistical graphs andHenderson, H. V., & Velleman, P. F. (1981). Building multiple maps. In F. T. Durso, R. S. Nickerson, R. W. Schvaneveldt, S. T. regression models interactively. Biometrics, 37, 391–411. Dumais, D. S. Lindsay, & M. T. H. Chi (Eds.), The handbook ofHilpinen, R. (1992). On Peirce’s philosophical logic: Propositions applied cognition (pp. 514–549). New York: Wiley. and their objects. Transaction of the Charles S. Peirce Society, Lindsey, J. K. (1996). Parametric statistical inference. Oxford, 28, 467–488. England: Clarendon Press.Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (Eds.). (1983). Under- Mallows, C. L. (Ed.). (1990). The collected works of John W. standing robust and exploratory data analysis. Reading, MA: Tukey: Vol. 6. More mathematical: 1938–1984. Monterey, CA: Addison-Wesley. Wadsworth.

References 63McGill, R., Tukey, J. W., & Larsen, W. A. (1978). Variations of box Shank, G. (1991, October). Abduction: Teaching to the ground state plots. American Statistican, 32, 12–16. of cognition. Paper presented at the Bergamo Conference onMicceri, T. (1989). The unicorn, the normal curve, and other im- Curriculum Theory and Classroom Practice, Dayton, OH. probable creatures. Psychological Bulletin, 105, 156–166. SPSS, Inc. (2001). SYSTAT [Computer software]. Retrieved fromMosteller, F., & Tukey, J. W. (1977). Data analysis and http://www.spss.com. regression: A second course in statistics. Reading, MA: Staat, W. (1993). On abduction, deduction, induction and the cate- Addison-Wesley. gories. Transactions of the Charles S. Peirce Society, 29,Neyman, J., & Pearson, E. S. (1928). On the use and interpretation 225–237. of certain test criteria for purposes of statistical inference. Part I Stangor, C., & McMillan, D. (1992). Memory for expectancy- and II. Biometrika, 20, 174–240, 263–294. congruent and expectancy-incongruent information: A review ofNeyman, J., & Pearson, E. S. (1933a). The testing of statistical the social and social development literatures. Psychological hypotheses in relation to probabilities a priori. Proceedings of Bulletin, 111, 42–61. Cambridge Philosophical Society, 20, 492–510. Stevens, S. S. (1951). Mathematics, measurement, and psy-Neyman, J., & Pearson, E. S. (1933b). On the problem of the most chophysics. In S. S. Stevens (Ed.), Handbook of experimental efficient tests of statistical hypotheses. Philosophical Transac- psychology (pp. 1–49). New York: Wiley. tions of Royal Society; Series A, 231, 289–337. Stock, W. A., & Behrens, J. T. (1991). Box, line, and mid-gap plots:Norusis, M. J. (2000). SPSS 10.0 Guide to data analysis. New York: Effects of display characteristics on the accuracy and bias of Prentice Hall. estimates of whisker length. Journal of Educational Statistics,Ottens, J., & Shank, G. (1995). The role of abductive logic in un- 16, 1–20. derstanding and using advanced empathy. Counselor Education Sullivan, P. F. (1991). On falsification interpretation of Peirce. & Supervision, 34, 199–213. Transactions of the Charles S. Peirce Society, 27, 197–219.Paap, K. R., & Johansen, L. S. (1994). The case of the vanishing Swayne, D. F., Cook, D., & Buja, A. (1998). XGobi: Interactive frequency effect: A retest of the verification model. Journal of dynamic data visualization in the X Window System. Journal of Experimental Psychology: Human Perception and Performance, Computational and Graphical Statistics, 7(1), 113–130. 20, 1129–1157. Thagard, P., & Shelley, C. (1997). Abductive reasoning: Logic,Paap, K. R., Johansen, L. S., Chun, E., & Vonnahme, P. (2000). visual thinking, and coherence. Retrieved from http://cogsci. Neighborhood frequency does affect performance in the Reicher uwaterloo.ca/Articles/Pages/%7FAbductive.html. task: Encoding or decision? Journal of Experimental Psychol- Tierney, L. (1990). Lisp-Stat: An object-oriented environment ogy: Human, Perception, and Performance, 26, 1691–1720. for statistical computing and dynamic graphics. New York:Peirce, C. S. (1868). Some consequences of four incapacities. Wiley. Journal of Speculative Philosophy, 2, 140–157. Tufte, E. R. (1990). Envisioning information. Cheshire, CT:Peirce, C. S. (1878). How to make our ideas clear. Popular Science Graphics Press. Monthly, 12, 286–302. Tufte, E. R. (1997). Visual explanations: Images and quantities,Peirce, C. S. (1960). Collected papers of Charles Sanders Peirce. evidence and narrative. Cheshire, CT: Graphics Press. Cambridge, MA: Harvard University Press. (Original work Tufte, E. R. (2001). The visual display of quantitative information published 1934) (2nd ed.). Cheshire, CT: Graphics Press. (Original work pub-Press, S. J., & Tanur, J. M. (2001). The subjectivity of scientists and lished 1983) the Bayesian approach. New York: Wiley. Tukey, J. W. (1969). Analyzing data: Sanctification or detectiveRussell, B., & Whitehead, A. N. (1910). Principia mathematica. work? American Psychologist, 24, 83–91. Cambridge, MA: Cambridge University Press. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA:Salsburg, D. S. (1985). The religion of statistics as practiced in med- Addison-Wesley. ical journals. American Statistician, 39, 220–223. Tukey, J. W. (1986a). Data analysis, computation and mathematics.Samuelson, P. (1967). Economic forecast and science. In P. A. In L. V. Jones (Ed.), The collected works of John W. Tukey: Samuelson, J. R. Coleman, & F. Skidmore (Eds.), Reading in Vol. 4. Philosophy and principles of data analysis: 1965–1986 economics (pp. 124–129). New York: McGraw-Hill. (pp. 753–775). Pacific Grove, CA: Wadsworth. (Original workSAS Institute (2001). SAS/Insight [Computer software]. Retrieved published 1972) from http://www.sas.com. Tukey, J. W. (1986b). Exploratory data analysis as part of a largerSchum, D. A. (1994). The evidential foundations of probabilistic whole. In L. V. Jones (Ed.), The collected works of John W. reasoning. New York: Wiley. Tukey: Vol. 4. Philosophy and principles of data analysis:Scott, D. W. (1992). Multivariate density estimation: Theory, prac- 1965–1986 (pp. 793–803). Pacific Grove, CA: Wadsworth. tice, and visualization. New York: Wiley. (Original work published 1973)

64 Exploratory Data AnalysisTukey, J. W. (1986c). The future of data analysis. In L. V. Jones Wilkinson, L. (1993). Comments on W. S. Cleveland, a model for (Ed.), The collected works of John W. Tukey: Vol. 3. Philosophy studying display methods of statistical graphs, Journal of Com- and principles of data analysis: 1949–1964 (pp. 391–484). putational and Graphical Statistics, 2, 355–360. Pacific Grove, CA: Wadsworth. (Original work published 1962) Wilkinson, L. (1994). Less is more: Two- and three-dimensionalTukey, J. W., & Wilk, M. B. (1986). Data analysis and statistics: An graphs for data display. Behavior Research Methods, Instru- expository overview. In L. V. Jones (Ed.), The collected works of ments, & Computers, 26, 172–176. John W. Tukey: Vol. 4. Philosophy and principles of data analy- Wilkinson, L. (1999). The grammar of graphics. New York: sis: 1965–1986 (pp. 549–578). Pacific Grove, CA: Wadsworth. Springer. (Original work published 1966) Wilkinson, L. (2001). Graphics. In Systat User’s Guide. Chicago:Venables, W. N., & Ripley, B. D. (1999). Modern Applied Statistics SPSS Inc. with S-PLUS. (3rd ed.). New York: Springer-Verlag. Wilkinson, L., & Task Force on Statistical Inference, APA Board ofWainer, H. (1977). Speed vs. reaction time as a measure of cognitive Scientific Affairs. (1999). Statistical methods in psychology performance. Memory and Cognition, 5, 278–280. journals: Guidelines and explanations. American Psychologist,Wainer, H. (1997). Visual Revelations: Graphical tales of fate and 54, 594–604. deception from Napoleon Bonaparte to Ross Perot. Mahwah, Young, F. W. (1996). ViSta: The Visual Statistics System. Chapel NJ: Erlbaum. Hill, NC: UNC L.L. Thurstone Psychometric Laboratory Re-Wainer, J., & Velleman, P. (2001). Statistical graphs: Mapping the search Memorandum. pathways of science. Annual Review of Psychology, 52, Yu, C. H., & Behrens, J. T. (1995). Applications of scientific multi- 305–335. variate visualization to behavioral sciences. Behavior ResearchWilcox, R. (1997). Introduction to robust estimation and hypothesis Methods, Instruments, and Computers, 27, 264–271. testing. San Diego, CA: Academic Press.Wilcox, R. (2001). Fundamentals of modern statistical methods: Substantially improving power and accuracy. New York: Springer.

CHAPTER 3Power: Basics, Practical Problems, and Possible SolutionsRAND R. WILCOXA CONVENTIONAL POWER ANALYSIS 66 Trimmed Means 76FACTORS OVER WHICH WE HAVE LIMITED Inferences Based on a Trimmed Mean 77 CONTROL 67 Judging Sample Sizes When Using Robust Estimators 78 Student’s T Can Be Biased 68 Rank-Based Methods and Outliers 78SAMPLE SIZE AND POWER 71 REGRESSION 78 Choosing Sample Sizes Before Sampling Observations 71 Heteroscedasticity and Probability Coverage 79 Stein-Type Methods for Means 72 ROBUST REGRESSION ESTIMATORS AND POWER 80 Multiple Comparisons 74 The Theil-Sen Estimator 80DEALING WITH SKEWNESS, HETEROSCEDASTICITY, CORRELATION 82 AND OUTLIERS 74 Robust Correlations 82 Heteroscedasticity 74 CONCLUDING REMARKS 84 Skewness and the Bootstrap 74 REFERENCES 84 Dealing With Low Power Using Robust Estimators 75There are, of course, two major types of errors one might power of the statistical tests used will be adequate for thecommit when testing any hypothesis. The ﬁrst, called a Type I smallest effect deemed to be important. Under normality anderror, is rejecting when in fact the null hypothesis is true, and homoscedasticity (meaning that all J groups have a commonthe second is failing to reject when the null hypothesis is variance), exact control over the probability of a Type I errorfalse. Certainly it is undesirable to claim that groups differ or can be achieved with the classic ANOVA F test, as is wellthat there is an association between two variables when this is known. Moreover, there is a standard method for assessingfalse. But simultaneously, it is undesirable to fail to detect a power as well, which is described and illustrated later indifference or to detect an association that is real, particularly this chapter. In essence, based on a certain measure of the dif-if the difference has important practical implications. This ference among the population means, it is possible to deter-latter error, failing to reject when the null hypothesis is false, mine power exactly given the sample sizes and a choice foris called a Type II error, and the probability of rejecting when the probability of a Type I error. In particular, the adequacy ofthe null hypothesis is false is called power. The roots of mod- proposed sample sizes can be assessed by determining howern approaches to power date back two centuries to Laplace, much power they provide. Today, the term power analysiswho derived the frequentist approach to computing conﬁ- brings to mind this technique, so it is important to cover itdence intervals used today. And even before Laplace, the here.basic idea can be gleaned from the work of de Moivre, who However, a goal in this chapter is to take a broader look atderived the equation for the normal curve. power, paying particular attention to modern insights and ad- Consider, for example, the usual one-way analysis of vari- vances. A half century ago, the method for assessing powerance (ANOVA) design where the goal is to test the hypothe- mentioned in the previous paragraph was certainly reason-sis of equal means among J independent groups. That is, the able, but little was known about its properties when violatinggoal is to test the assumptions of normality and homoscedasticity. Indeed, there were some indications that assumptions could be vio- H0: ␮1 = · · · = ␮ J , lated with impunity, but during the ensuing years there havewhere ␮1 , . . . , ␮ J are the corresponding population means. been many insights regarding the consequences of violatingPower analyses are used to plan studies with the goal that the these assumptions that have serious practical implications. 65

66 Power: Basics, Practical Problems, and Possible SolutionsSo one of the goals here is to summarize why there are prac- its variance, but once a population of individuals has beentical concerns and how they might be addressed. The theme selected for study, and once the outcome (dependent) variablein this paper is that conventional methods have proven to be has been settled upon, the variance becomes an unknownuseful but that they are far from perfect and sometimes disas- state of nature that is now beyond our control. Other factorstrous. Moreover, our understanding of factors that are rele- over which we have partial control are ␣ (the probability of avant to power continues to grow, as does the collection of Type I error), the reliability of the measures being used, andstatistical tools for dealing with problems that have been dis- the sample sizes. Steps can be taken to improve reliability;covered. In addition, there is more to power than determining nevertheless, it remains an issue when dealing with power.adequate sample sizes. Generally, it is a complex problem As is well known, the choice for ␣ influences power, butthat requires a plethora of tools, and one goal is to describe typically there are limits on how large ␣ can be. And ofsome of the tools that might be used. course there are limits on how many observations one can A related issue is power when dealing with associations obtain.among two or more variables. Again, relatively simple meth- At this stage, factors that remain within our control in-ods are available under normality and homoscedasticity. For clude the estimator used (such as the mean vs. the median)example, when testing the hypothesis that Pearson’s correla- and the hypothesis-testing technique employed. It has longtion is zero, if in reality it is equal to .3, say, the sample size been known that under normality and homoscedasticity,can be determined so that the probability of rejecting is Student’s T test achieves relatively high power. However, aequal to .8, say, or any value deemed important. But when practical concern is that arbitrarily small departures from athese assumptions are violated, practical problems are even normal curve toward a heavy-tailed distribution can destroyworse relative to ANOVA. Of course, one could simply ig- power when working with any method based on means. Also,nore these problems, so a goal in this chapter is to explain skewness can contribute to this problem in a substantial way,why this strategy can be highly unsatisfactory and summarize and even under normality heteroscedasticity is yet anothersome of the modern methods that might be used instead. Sub- factor that can lower power. Increasing sample sizes is onestantial progress has been made, but it will be argued that way of dealing with these concerns, but as will be explained,even more needs to be done. restricting attention to this one approach can be relatively Generally, achieving high power, and even judging unsatisfactory.whether power will be high in a given situation, is an ex-tremely complex problem that has seen many major advancesin recent years. These advances include a better understand- A CONVENTIONAL POWER ANALYSISing of what affects power and how power might be maxi-mized. Consider, for example, the problem of comparing two Although the goal in this paper is to provide a broad perspec-groups in terms of some measure of location such as the tive on power, a description of a conventional power analysismean or median. A variety of factors affects power, and is first presented for readers unfamiliar with it. Consider twosome are well known whereas others are being found to be independent groups, assume normality and homoscedasticity,more important than was previously thought. A basic factor is and assume that the population means are to be comparedthe smallest difference between the groups deemed impor- with Student’s T test. If the means differ, we should reject, sotant, which is reflected by some type of effect size, examples the issue is the probability of rejecting as a function of theof which will be given. Certainly the variance associated sample sizes. For example, if the sample sizes are n 1 =with some outcome variable is well known to influence n 2 = 20, what is the power, and if the power is judged to bepower when making inferences based on sample means, and too low, how large must the sample sizes be to correct thisadditional factors influencing power are skewness, heavy problem? Typically, a researcher specifies a desired amounttailedness (roughly referring to situations where outliers are of power and consults specially designed tables or softwarecommon), and heteroscedasticity (unequal variances). that indicates the required sample size. Doing this requiresAchieving relatively high power, as well as understanding first specifying some difference between the means that isthe limitations of standard approaches to power, requires an judged to be important. A natural way of doing this is with theunderstanding of how these factors influence power, so difference between the means, ␮1 − ␮2 , but it is impossibleanother goal here is to address this issue. to determine the required sample size based on this approach Note that at best, there is a limited amount of control one (Dantzig, 1940). (Switching to a two-stage procedure, powercan exert over these factors. In some situations, the outcome can be addressed based on the difference between the means,variable of interest can be constructed in a way that influences as will be explained later.) If a standardized difference is used

Factors Over Which We Have Limited Control 67instead, namely, sample sizes fixed. More generally, power is related to the squared standard error of the measure of location being used. ␮1 − ␮2 = , For the sample mean, X, the squared standard error is the ␴ variance of the sample mean (if a study could be repeatedwhere ␴2 is the assumed common variance, this technical dif- infinitely many times), which isficulty is avoided. Cohen (1977) defined a large effect as some- ␴2thing that is visible to the naked eye and concluded that for two VAR(X) = , (3.1) nnormal distributions having a common variance, small,medium, and large effect sizes correspond to = .2, = .5, where n is the sample size. It is this connection with the vari-and = .8, respectively. Given that the probability of a Type I ance that wreaks havoc when using any method based onerror is ␣, Cohen provides tables for determining the required means.sample sizes (also see Kraemer & Thiemann, 1987). For ex- A classic illustration of why is based on a particular mixedample, with n 1 = n 2 = 20, = .8, and ␣ = .05, power is (or contaminated) normal distribution where with probability.68. Rather than specify a value for , one can plot a so-called .9 an observation is sampled from a standard normal distrib-power curve where power is plotted versus , given the sam- ution and otherwise sampling is from a normal distributionple sizes and ␣. An advantage of this approach is that it pro- having a standard deviation of 10. Figure 3.1 shows the stan-vides a more global sense of how power is related to based dard and mixed normal distributions. The mixed normal ison the sample sizes used. (For software, see Bornstein, 2000; said to have thick or heavy tails because its tails lie above theElashoff, 2000; O’Brien, 1998.) normal curve, which implies that unusually small or large An extension of this standard power analysis to more than values, called outliers, are more common when samplingtwo groups, still assuming normality and homoscedasticity, from the mixed normal versus the normal. As is evident, thehas been derived. That is, given ␣, the sample sizes corre- two distributions are very similar in a certain sense, but theresponding to J groups, and a difference among the means is a crucial difference: The standard normal has variance 1,deemed to be important, power can be computed. Assuming but the mixed normal has variance 10.9. This illustrates theequal sample sizes, now the difference among the means is well-known result that an arbitrarily small change in any dis-typically measured with tribution, including normal distributions as a special case, can cause the variance to become arbitrarily large. That is, ␴2 1 (␮ j − ␮)2 is extremely sensitive to the tails of a distribution. One impli- , cation is that arbitrarily small departures from normality can ␴ J result in low power (relative to other methods we might use)where again ␴2 is the assumed common variance. There are when comparing means.fundamental problems with this standard approach, not the To begin to appreciate that alternative estimators canleast of which is the interpretation of this last equation when make a practical difference in applied work, consider the me-dealing with nonnormal distributions. Some of these prob- dian versus the mean. Figure 3.2 shows a plot of 5,000 medi-lems arise under arbitrarily small departures from normality, ans and means, each based on 20 observations randomlyas will be illustrated. sampled from the mixed normal shown in Figure 3.1. Note that the medians are more tightly clustered around zero, the value being estimated, than are the means. That is, the me-FACTORS OVER WHICH WE HAVE dian has a much smaller standard error than the mean, whichLIMITED CONTROL can translate into more power. However, if observations are sampled from a standard normal distribution instead, the plotAchieving relatively high power requires, among other of the medians versus the means now appears as shown inthings, a more detailed understanding about how factors over Figure 3.3. That is, using medians can result in low powerwhich we have limited control are related to power so that the relative to using the mean (as well as other estimatorsrelative merits of factors within our control can be under- described later in this chapter).stood. Consider some population of individuals and suppose To provide an explicit illustration regarding the effect ofthat some outcome measure, X, has been chosen for study. nonnormality on power when using means, suppose that 25When working with means, it is well known that the variance observations are randomly sampled from each of two normalof a distribution, ␴2 , has a direct effect on power: The larger distributions both having variance 1, the first having mean 0␴2 happens to be, the lower power will be with ␣ and the and the second having mean 1. Applying Student’s T test with

68 Power: Basics, Practical Problems, and Possible Solutions Figure 3.1 A mixed normal and standard normal distribution. Despite the similarity, the mixed normal has variance 10.9, whereas the standard normal which has variance 1.␣ = .05, the probability of rejecting (power) is .96. But if found that improves upon the power of the McKean-Schradersampling is from mixed normals instead, with the difference method for medians when sampling from normal distribu-between means again 1, power is only .28. (A complication tions and continues to have relatively high power when sam-when discussing means vs. medians is that for skewed distri- pling from a heavy-tailed distribution such as the mixedbutions, each generally estimates different quantities, so it is normal. Such methods are available and are described later inpossible for means to have more power regardless of their this chapter.standard errors, and the reverse is true as well.) For the situation just described, if medians are compared Student’s T Can Be Biasedwith the method derived by McKean and Schrader (1984),power is approximately .8 when sampling from the normal To illustrate the effects of skewness on power when usingdistributions. So a practical issue is whether a method can be Student’s T, suppose that 20 observations are sampled from Figure 3.2 Distribution of the mean versus median when sampling from a mixed normal distribution.

Factors Over Which We Have Limited Control 69 Figure 3.3 Distribution of the mean versus median when sampling from a standard normal distribution.the (lognormal) distribution shown in Figure 3.4, which has a values for T generated on a computer. The symmetric curvemean of .4658. From basic principles, inferences about the is the distribution of T under normality. The main point heremean are based on is that the mean (or expected value) of T is not 0—it is approximately −.5. This might appear to be impossible because X −␮ T = √ , (3.2) under random sampling the expected value of the numerator s/ n of T, X − ␮, is 0, which might seem to suggest that T mustassuming T has a Student’s T distribution with n − 1 degrees have a mean of 0 as well. However, for nonnormal distribu-of freedom, where s is the sample standard deviation and ␮ is tions, X and s are dependent, and this dependence makes itthe population mean. In particular, the distribution of T is as- possible for the mean of T to differ from zero. (Gosset, whosumed to be symmetric about zero, but when sampling from derived Student’s T distribution, was aware of this issue.)an asymmetric distribution, this is not the case. For the situa- This property is important because it has practical implica-tion at hand, the distribution of T is given, approximately, by tions about power: Power can actually decrease as we movethe asymmetric curve shown in Figure 3.5, which is based on away from the null hypothesis. That is, situations arise where Figure 3.4 A lognormal distribution.

70 Power: Basics, Practical Problems, and Possible Solutions Figure 3.5 The ragged line is the plot of T values based on data generated from a lognormal distribution. The smooth symmetric curve is the distribution of T under normality.there is a higher probability of rejecting when the null hy- standard deviation from the null, power is approximately thepothesis is true versus situations where the the null hypothe- same as when the null hypothesis is true.sis is false. In technical terms, Student’s T test is biased. The central limit theorem implies that with a sufﬁciently To provide perspective, Figure 3.6 shows the power curve large sample size, the distribution of T will converge to a nor-of Student’s T with n = 20 and when ␦ is added to every ob- mal distribution. It is known that for a lognormal distributionservation. That is, when ␦ = 0, the null hypothesis is true; (which is a skewed relatively light-tailed distribution amongotherwise, the null hypothesis is false, and the difference be- the class of g-and-h distribution derived by Hoaglin, 1985),tween the true mean and the hypothesized value is ␦. In this even with 160 observations, there are practical problems withcase, power initially decreases as we move away from the obtaining accurate probability coverage and control over thenull hypothesis, but eventually it goes up (cf. Sawilowsky, probability of a Type I error. (Westfall & Young, 1993, noteKelley, Blair, & Markham, 1994). The value ␦ = .6 repre- that for a one-sided test, the actual probability of a Type Isents a departure from the null value of slightly more than error is .11 when testing at the .05 level.) With about 200one fourth of a standard deviation. That is, moving a quarter observations, these problems become negligible. But when Figure 3.6 Power curve of T when sampling from a lognormal distribution.

Sample Size and Power 71sampling from a skewed, heavy-tailed distribution, a sample reasons. First, an extreme outlier was removed. If this outliersize greater than 300 might be required. It remains unclear, is included, the approximation of the distribution of T de-however, how quickly practical problems with bias disappear parts in an even more dramatic manner from the assumptionas the sample size increases. that it is symmetric about zero. Second, studies of the small- The properties of the one-sample T test, when sampling sample properties of the bootstrap-t suggest that Figure 3.7from a skewed distribution, have implications about compar- underestimates the degree to which the actual distribution ofing two independent groups. To get a rough indication as to T is skewed.why, consider the sample mean from two independent groups,X 1 and X 2 . If the two groups have identical distributions andequal sample sizes are used, the difference between the means SAMPLE SIZE AND POWERhas a symmetric distribution, and problems with bias andType I errors substantially higher than the nominal level are Perhaps the most obvious method for controlling power isminimal. But when distributions differ in skewness, practical simply to adjust the sample size. This is relatively easy to doproblems arise because the distribution of X 1 − X 2 will when working with means and when sampling is from nor-be skewed as well. This is not to suggest, however, that bias is mal distributions, but such methods are fraught with peril.not an issue when sampling from symmetric distributions. Forexample, even when sampling from normal distributions, if Choosing Sample Sizes Before Sampling Observationsgroups have unequal variances, the ANOVA F test can bebiased (e.g., Wilcox, Charlin, & Thompson, 1986). First, consider how the sample sizes might be chosen prior to A possible criticism of the problems with Student’s T collecting data when comparing the means of two indepen-illustrated by Figures 3.5 and 3.6 is that in theory the actual dent normal distributions. A commonly used approach is todistribution of T can be substantially asymmetric, but can this characterize the difference between the groups in terms of aproblem occur in practice? Using data from various studies, standardized effect size:Wilcox (2001, in press) illustrated that the answer is yes. ␮1 − ␮2Consider, for example, data from a study conducted by = , ␴Pedersen, Miller, Putcha, and Yang (in press) where n = 104.Figure 3.7 shows an approximation of the distribution of T where by assumption the two groups have a common vari-based on resampling with replacement 104 values from ance, ␴2 . As mentioned, Cohen (1977) deﬁned a large effectthe original data, computing T, and repeating this process as something that is visible to the naked eye and concluded1,000 times. (That is, a bootstrap-t method was used, which is that for two normal distributions having a common variance,described in more detail later.) In fact, all indications are that small, medium, and large effect sizes correspond to = .2,problems with T are underestimated here for at least two = .5, and = .8, respectively. Given ␣, the sample sizes Figure 3.7 An approximation of the distribution of T based on data with n = 104.

72 Power: Basics, Practical Problems, and Possible Solutionscan be chosen so that for a given value of , power will be some specified constant. Further assume that the Type I errorequal to some specified value—assuming normality (e.g., probability is to be ␣ and that the goal is to have power atCohen, 1977; Kraemer & Thiemann, 1987). least 1 − ␤ when ␮ − ␮0 = ␦. For example, if the goal is to For example, with = 1, and ␣ = .05, and sample sizes test H0: ␮ = 6, it might be desired to have power equal to .8of 25, power will be equal to .96 when using Student’s T, as when in reality ␮ = 8. Here, 1 − ␤ = .8 and ␦ = 8 − 6 = 2.previously indicated. What this reflects is a solution to The issue is, given n randomly sampled observations, howchoosing sample sizes under the most optimistic circum- many additional observations, if any, are required to achievestances possible. In reality, when comparing means, power the desired amount of power. If no additional observationswill be at most .96, and a realistic possibility is that power is are required, power is sufficiently high based on the samplesubstantially lower than intended if Student’s T is used. As size used; otherwise, the sample size was too small. Stein’salready noted, an arbitrarily small departure from normality method proceeds as follows. Let t1−␤ and t␣ be the 1 − ␤ andcan mean that power will be close to zero. Yet another con- ␣ quantiles of Student’s T distribution with ν = n − 1 de-cern is that this approach ignores the effects of skewness and grees of freedom. (So if T has a Student’s T distribution withheteroscedasticity. ν = n − 1 degrees, P[T ≤ t1− ␤] = 1 − ␤.) Let Despite its negative properties, this approach to determin- 2ing sample sizes may have practical value. The reason is that ␦ d= .when comparing groups with a robust measure of location t1−␤ − t␣(described later) by design power will be approximatelyequal to methods based on means and when sampling from a Then the required sample size isnormal distribution. Unlike means, however, power remainsrelatively high when sampling from a heavy-tailed or asym- s2 N = max n, +1 ,metric distribution. So a crude approximation of the required dsample size when using a modern robust method might bebased on standard methods for choosing samples sizes when where the notation [s 2 /d] means that s 2 /d is computed andcomparing means. rounded down to the nearest integer. For example, if s = 21.4, ␦ = 20, 1 − ␤ = .9, ␣ = .01,and ν = 9, thenStein-Type Methods for Means 2 20 d= = 22.6, 1.383 − (−2.82)When some hypothesis is rejected, power is not an issue—theprobability of a Type II error is zero. But when we fail to re- soject, the issue becomes why. One possibility is that the nullhypothesis is true, but another possibility is that the null hy- N = max(10, [21.42 /22.6] + 1) = max(10, 21) = 21.pothesis is false and we failed to detect this. How might wedecide which of these two possibilities is more reasonable? If N = n, the sample size is adequate; but in the illustration,When working with means, one possibility is to employ what N − n = 21 − 10 = 11. That is, 11 additional observationsis called a Stein-type two-stage procedure. Given some data, are needed to achieve the desired amount of power. Withthese methods are aimed at determining how large the sample ␦ = 29, N = 10, and no additional observations are required.sizes should have been in order to achieve power equal to If the additional N − n observations can be obtained,some specified value. If few or no additional observations are H0: ␮ = ␮0 can be tested, but for technical reasons the obvi-required to achieve high power, this naturally provides some ous approach of applying Student’s T is not used. Rather, aassurance that power is reasonably high based on the number slight modification is applied that is based on the test statisticof observations available. Otherwise, the indication is thatpower is relatively low due to using a sample size that is too √ n(␮ − ␮0 ) ˆsmall. Moreover, if the additional observations needed to TA = , sachieve high power are acquired, there are methods for test-ing hypotheses that typically are different from the standard where ␮ is the mean of all N observations. You test hypothe- ˆmethods covered in an introductory statistics course. ses by treating Ts as having a Student’s T distribution with To describe Stein’s (1945) original method, consider a sin- ν = n − 1 degrees of freedom. What is peculiar about Stein’sgle variable X that is assumed to have a normal distribution method is that the sample variance based on all N observa-and suppose that the goal is to test H0: ␮ = ␮0 , where ␮0 is tions is not used. Instead, s, which is based on the original

Sample Size and Power 73n observations, is used. For a survey of related methods, in- to achieve power equal to 1 − ␤ can be obtained. Next,cluding techniques for controlling power when using the computeWilcoxon signed rank test or the Wilcoxon-Mann-Whitneytest, see Hewett and Spurrier (1983). (ν − 2)c b= , Stein’s method has been extended to the problem of com- νparing two or more groups. Included are both ANOVA and 1 √multiple comparison procedures. The extension to ANOVA, A1 = { 2z + 2z 2 + A(2b − J + 2)}, 2when comparing the means of J independent groups, wasderived by Bishop and Dudewicz (1978) and is applied as B1 = A2 − b, 1follows. Imagine that power is to be 1 − ␤ for some givenvalue of ␣ and ν−2 ␦ d= × . ν B1 ␦= (␮ j − ␮)2 , Then the required number of observations for the jth group iswhere ␮ = ␮ j /J. Further assume that nj observations s j2have been randomly sampled from the jth group, j = N j = max n j + 1, +1 . (3.3) d1, . . . , J. One goal is to determine how many additional ob-servations are required for the jth group to achieve the For technical reasons, the number of observations needed fordesired amount of power. the jth group, Nj, cannot be smaller than n j + 1. (The notation Let z be the 1 − ␤ quantile of the standard normal random [s j2 /d] means that s j2 /d is computed and then rounded downdistribution. For the jth group, let ν j = n j − 1. Compute to the nearest integer.) Software for applying this method can be found in Wilcox (in press). J ␯= 1 + 2, In the event the additional N j − n j observations can be ob- ␯j −2 tained from the jth group, exact control over both the Type I error probability and power can be achieved even when the (J − 1)ν groups have unequal variances—still assuming normality. In A= , ν−2 particular, for the jth group compute ν2 J −1 nj B= × , Tj = Xi j , J ν−2 i=1 3(J − 1) C= , Nj ν−4 Uj = Xi j , i=n j +1 J 2 − 2J + 3 D= ,   ν−2 n j (N j d − s j2 ) 1  , bj = 1+ E = B(C + D), Nj (N j − n j )s j2 4E − 2A2 M= , Tj {1 − (N j − n j )b j } E − A2 − 2A ˜ Xj = + bj Uj . nj A(M − 2) L= , The test statistic is M ˜ 1 ˜ ˜ C = L f, F= ( X j − X)2 , dwhere f is the 1 − ␣ quantile of an F distribution with L and whereM degrees of freedom. The quantity c is the critical value 1 ˜ X= ˜ Xj.used in the event that the additional observations needed J

74 Power: Basics, Practical Problems, and Possible Solutions ˜The hypothesis of equal means is rejected if F ≥ c and test, and even for inferential methods used in regression andpower will be at least 1 − ␤. when dealing with correlations. When comparing groups having identical distributions, homoscedastic methods perform well in terms of Type I errors, but when comparing groups thatMultiple Comparisons differ in some manner, there are general conditions underStein-type multiple comparisons procedures were derived by which these techniques are using the wrong standard error,Hochberg (1975) and Tamhane (1977). One crucial differ- which in turn can result in relatively lower power. For exam-ence from the Bishop-Dudewicz ANOVA is that direct con- ple, when using the two-sample Student’s T test, the assump-trol over power is no longer possible. Rather, these methods tion is that the distribution of the test statistic T approaches acontrol the length of the confidence intervals, which of standard normal distribution as the sample sizes increase. Incourse is related to power. When sample sizes are small, both particular, the variance of the test statistic is assumed tomethods require critical values based on the quantiles of what converge to one, but Cressie and Whitford (1986) describedis called a Studentized range statistic. Tables of these critical general conditions under which this is not true. In a similarvalues can be found in Wilcox (in press). For the details of manner, the Wilcoxon-Mann-Whitney test is derived underhow to use these methods, plus easy-to-use software, see the assumption that distributions are identical. When distribu-Wilcox (in press). tions differ, the wrong standard error is being used, which causes practical problems. Methods for dealing with heteroscedasticity have been derived by Fligner and PolicelloDEALING WITH SKEWNESS, (1981), Mee (1990), and Cliff (1994), as well as by BrunnerHETEROSCEDASTICITY, AND OUTLIERS and Munzel (1999). The techniques derived by Cliff and Brunner and Munzel are particularly interesting because theyAlthough Stein-type methods are derived assuming normal- include methods for dealing with tied values.ity, they deal with at least one problem that arises under non-normality. In particular, they have the ability of alerting us to Skewness and the Bootstraplow power due to outliers. When sampling from a heavy-tailed distribution, the sample variance will tend to be rela- The central limit theorem says that under random samplingtively large, which in turn will yield a large N when using and with a sufficiently large sample size, it can be assumedEquation 3.3. This is because the sample variance can be that the distribution of the sample mean is normal. Moreover,greatly inflated by even a single outlier. In modern terminol- Student’s T approaches a normal distribution as well, but aogy, the sample variance has a finite sample breakdown point practical concern is that it approaches a normal distributionof only 1/n, meaning that a single observation can make it more slowly than X does when sampling from a skewed dis-arbitrarily large. As a simple example, consider the values 8, tribution (e.g., Wilcox, 2001). The problem is serious enough8, 8, 8, 8, 8, 8, 8, 8, 8, 8. There is no variation among these that power is affected, as previously demonstrated.values, so s 2 = 0. If we increase the last value to 10, the sam- One approach is to replace Student’s T and its het-ple variance is s 2 = .36. Increasing the last observation to 12, eroscedastic analogs with a bootstrap-t method. This ap-s 2 = 1.45, and increasing it to 14, s 2 = 3.3. The point is that proach is motivated by two general results. First, the theoryeven though there is no variation among the bulk of the ob- indicates that problems with nonnormality will diminishservations, a single value can make the sample variance arbi- more rapidly than with more conventional methods. To pro-trarily large. In particular, outliers can substantially inflate s2, vide a rough idea of what this means, note that under nonnor-but what can be done about improving power based on the mality there will be some discrepancy between the actual andobservations available, and how might problems due to nominal level value for ␣, the probability of a Type I error.skewness, heteroscedasticity, and outliers be approached? When sample sizes are large, the rate at which conventional (heteroscedastic) methods converge to the correct nominal √ level is 1/ n. In contrast, methods based on the bootstrap-tHeteroscedasticity converge at the rate of 1/n—namely, faster. This does notToday, virtually all standard hypothesis-testing methods necessarily imply, however, that with small to moderate sam-taught in an introductory course have heteroscedastic analogs, ple sizes, problems with low power due to skewness will besummaries of which are given in Wilcox (in press). This is negligible with a bootstrap technique. In terms of Type I er-true for methods based on measures of location as well as for rors, for example, problems are often reduced considerably,rank-based techniques such as the Wilcoxon-Mann-Whitney but for skewed heavy-tailed distributions, problems can

Dealing With Skewness, Heteroscedasticity, and Outliers 75persist even with n = 300 when attention is restricted to VAR(X), the squared standard error of the sample mean,means. Nevertheless, the bootstrap-t offers a practical advan- with s 2 /n, where s 2 is the sample variance based on the ntage because when making inferences based on means, it observations left after outliers are discarded. However, theregenerally performs about as well as conventional techniques are two concerns with this approach. First, it results in usingand in some cases offers a distinct advantage. As for power

Handbook of psychology_vol_2_-_research_methods_in_psychology

by Forum Penelitian dan Evaluasi Pendidikan

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Handbook of psychology_vol_2_-_research_methods_in_psychology Document Transcript