HANDBOOK         of    PSYCHOLOGY           VOLUME 2RESEARCH METHODS IN PSYCHOLOGY          John A. Schinka          Wayne...
HANDBOOK     ofPSYCHOLOGY
HANDBOOK         of    PSYCHOLOGY           VOLUME 2RESEARCH METHODS IN PSYCHOLOGY          John A. Schinka          Wayne...
➇This book is printed on acid-free paper.Copyright © 2003 by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights rese...
Editorial BoardVolume 1                          Volume 5                             Volume 9History of Psychology       ...
My efforts in this work are proudly dedicated to Katherine, Christy, and John C. Schinka.                                 ...
Handbook of Psychology PrefacePsychology at the beginning of the twenty-first century has                   A second unifyi...
x   Handbook of Psychology Prefaceand for persons outside of psychology seeking informa-           valuable contributions ...
Volume PrefaceA scientific discipline is defined in many ways by the re-             computers to perform complex methods ...
xii   Volume Prefacecontrol. Because most of the recent growth involves research       to focus on the individual and mode...
Volume Preface   xiiiresult of some type of error. The last two chapters in this part,   findings that are of interest to p...
xiv   Volume Preface    Chapter 17, “Item Response Theory and Measuring Abili-        Velicer and Joseph L. Fava describe ...
Volume Preface   xvSteven Reise, Joseph L. Rogers, Joseph Rossi, James                or supported by differing approaches...
ContentsHandbook of Psychology Preface ix    Irving B. WeinerVolume Preface xi     John A. Schinka and Wayne F. VelicerCon...
xviii   Contents10       HEALTH PSYCHOLOGY 241         Timothy W. Smith11       ANIMAL LEARNING 271         Russell M. Chu...
Contents   xix24    STRUCTURAL EQUATION MODELING 607      Jodie B. Ullman and Peter M. Bentler25    ORDINAL ANALYSIS OF BE...
ContributorsLeona S. Aiken, PhD                                  Norman Cliff, PhDDepartment of Psychology                ...
xxii   ContributorsLinda S. Fidell, PhD                           Melvin M. Mark, PhDDepartment of Psychology             ...
Contributors   xxiiiTimothy W. Smith, PhD               David Watson, PhDDepartment of Psychology            Department of...
PA R T O N E FOUNDATIONS OF RESEARCH ISSUES: STUDY DESIGN, DATA MANAGEMENT,DATA REDUCTION, AND DATA SYNTHESIS
CHAPTER 1Experimental DesignROGER E. KIRKSOME BASIC EXPERIMENTAL DESIGN CONCEPTS 3                               FACTORIAL...
4   Experimental Designto Ronald A. Fisher, a statistician who worked at the Rotham-               Fisher popularized two ...
Three Building Block Designs            5by the lowercase letter a and a subscript: a1 denotes one dietand a2 denotes the ...
6   Experimental Designand (e) any other nuisance variables such as environmental              level are different because...
Three Building Block Designs   7                                                                                   partici...
8    Experimental Design                                                                                          ␧i j    ...
Three Building Block Designs             9hypothesis. Thus, by isolating a nuisance variable that ac-counts for an appreci...
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Handbook of psychology_vol_2_-_research_methods_in_psychology
Upcoming SlideShare
Loading in...5
×

Handbook of psychology_vol_2_-_research_methods_in_psychology

2,435

Published on

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,435
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
100
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Handbook of psychology_vol_2_-_research_methods_in_psychology

  1. 1. HANDBOOK of PSYCHOLOGY VOLUME 2RESEARCH METHODS IN PSYCHOLOGY John A. Schinka Wayne F. Velicer Volume Editors Irving B. Weiner Editor-in-Chief John Wiley & Sons, Inc.
  2. 2. HANDBOOK ofPSYCHOLOGY
  3. 3. HANDBOOK of PSYCHOLOGY VOLUME 2RESEARCH METHODS IN PSYCHOLOGY John A. Schinka Wayne F. Velicer Volume Editors Irving B. Weiner Editor-in-Chief John Wiley & Sons, Inc.
  4. 4. ➇This book is printed on acid-free paper.Copyright © 2003 by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights reserved.Published simultaneously in Canada.No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic,mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 UnitedStates Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of theappropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400,fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to thePermissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008,e-mail: permcoordinator@wiley.com.Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, theymake no representations or warranties with respect to the accuracy or completeness of the contents of this book and specificallydisclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended bysales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation.You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit orany other commercial damages, including but not limited to special, incidental, consequential, or other damages.This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is soldwith the understanding that the publisher is not engaged in rendering professional services. If legal, accounting, medical,psychological or any other expert assistance is required, the services of a competent professional person should be sought.Designations used by companies to distinguish their products are often claimed as trademarks. In all instances where John Wiley &Sons, Inc. is aware of a claim, the product names appear in initial capital or all capital letters. Readers, however, should contact theappropriate companies for more complete information regarding trademarks and registration.For general information on our other products and services please contact our Customer Care Department within the U.S. at(800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available inelectronic books.Library of Congress Cataloging-in-Publication Data:Handbook of psychology / Irving B. Weiner, editor-in-chief. p. cm. Includes bibliographical references and indexes. Contents: v. 1. History of psychology / edited by Donald K. Freedheim — v. 2. Research methods in psychology / edited by John A. Schinka, Wayne F. Velicer — v. 3. Biological psychology / edited by Michela Gallagher, Randy J. Nelson — v. 4. Experimental psychology / edited by Alice F. Healy, Robert W. Proctor — v. 5. Personality and social psychology / edited by Theodore Millon, Melvin J. Lerner — v. 6. Developmental psychology / edited by Richard M. Lerner, M. Ann Easterbrooks, Jayanthi Mistry — v. 7. Educational psychology / edited by William M. Reynolds, Gloria E. Miller — v. 8. Clinical psychology / edited by George Stricker, Thomas A. Widiger — v. 9. Health psychology / edited by Arthur M. Nezu, Christine Maguth Nezu, Pamela A. Geller — v. 10. Assessment psychology / edited by John R. Graham, Jack A. Naglieri — v. 11. Forensic psychology / edited by Alan M. Goldstein — v. 12. Industrial and organizational psychology / edited by Walter C. Borman, Daniel R. Ilgen, Richard J. Klimoski. ISBN 0-471-17669-9 (set) — ISBN 0-471-38320-1 (cloth : alk. paper : v. 1) — ISBN 0-471-38513-1 (cloth : alk. paper : v. 2) — ISBN 0-471-38403-8 (cloth : alk. paper : v. 3) — ISBN 0-471-39262-6 (cloth : alk. paper : v. 4) — ISBN 0-471-38404-6 (cloth : alk. paper : v. 5) — ISBN 0-471-38405-4 (cloth : alk. paper : v. 6) — ISBN 0-471-38406-2 (cloth : alk. paper : v. 7) — ISBN 0-471-39263-4 (cloth : alk. paper : v. 8) — ISBN 0-471-38514-X (cloth : alk. paper : v. 9) — ISBN 0-471-38407-0 (cloth : alk. paper : v. 10) — ISBN 0-471-38321-X (cloth : alk. paper : v. 11) — ISBN 0-471-38408-9 (cloth : alk. paper : v. 12) 1. Psychology. I. Weiner, Irving B. BF121.H1955 2003 150—dc21 2002066380Printed in the United States of America.10 9 8 7 6 5 4 3 2 1
  5. 5. Editorial BoardVolume 1 Volume 5 Volume 9History of Psychology Personality and Social Psychology Health PsychologyDonald K. Freedheim, PhD Theodore Millon, PhD Arthur M. Nezu, PhDCase Western Reserve University Institute for Advanced Studies in Christine Maguth Nezu, PhDCleveland, Ohio Personology and Psychopathology Pamela A. Geller, PhD Coral Gables, Florida Drexel University Melvin J. Lerner, PhD Philadelphia, PennsylvaniaVolume 2 Florida Atlantic UniversityResearch Methods in Psychology Boca Raton, Florida Volume 10 Assessment PsychologyJohn A. Schinka, PhDUniversity of South Florida Volume 6 John R. Graham, PhDTampa, Florida Developmental Psychology Kent State University Richard M. Lerner, PhD Kent, OhioWayne F. Velicer, PhDUniversity of Rhode Island M. Ann Easterbrooks, PhD Jack A. Naglieri, PhDKingston, Rhode Island Jayanthi Mistry, PhD George Mason University Tufts University Fairfax, Virginia Medford, Massachusetts Volume 11Volume 3 Forensic PsychologyBiological Psychology Volume 7 Educational Psychology Alan M. Goldstein, PhDMichela Gallagher, PhD John Jay College of CriminalJohns Hopkins University William M. Reynolds, PhD Justice–CUNYBaltimore, Maryland Humboldt State University New York, New YorkRandy J. Nelson, PhD Arcata, CaliforniaOhio State University Gloria E. Miller, PhD Volume 12Columbus, Ohio University of Denver Industrial and Organizational Denver, Colorado Psychology Walter C. Borman, PhDVolume 4 Volume 8 University of South FloridaExperimental Psychology Clinical Psychology Tampa, FloridaAlice F. Healy, PhD George Stricker, PhD Daniel R. Ilgen, PhDUniversity of Colorado Adelphi University Michigan State UniversityBoulder, Colorado Garden City, New York East Lansing, MichiganRobert W. Proctor, PhD Thomas A. Widiger, PhD Richard J. Klimoski, PhDPurdue University University of Kentucky George Mason UniversityWest Lafayette, Indiana Lexington, Kentucky Fairfax, Virginia v
  6. 6. My efforts in this work are proudly dedicated to Katherine, Christy, and John C. Schinka. J. A. S. This work is dedicated to Sue, the perfect companion for life’s many journeys and the center of my personal universe. W. F. V.
  7. 7. Handbook of Psychology PrefacePsychology at the beginning of the twenty-first century has A second unifying thread in psychology is a commitmentbecome a highly diverse field of scientific study and applied to the development and utilization of research methodstechnology. Psychologists commonly regard their discipline suitable for collecting and analyzing behavioral data. Withas the science of behavior, and the American Psychological attention both to specific procedures and their applicationAssociation has formally designated 2000 to 2010 as the in particular settings, Volume 2 addresses research methods“Decade of Behavior.” The pursuits of behavioral scientists in psychology.range from the natural sciences to the social sciences and em- Volumes 3 through 7 of the Handbook present the sub-brace a wide variety of objects of investigation. Some psy- stantive content of psychological knowledge in five broadchologists have more in common with biologists than with areas of study: biological psychology (Volume 3), experi-most other psychologists, and some have more in common mental psychology (Volume 4), personality and social psy-with sociologists than with most of their psychological col- chology (Volume 5), developmental psychology (Volume 6),leagues. Some psychologists are interested primarily in the be- and educational psychology (Volume 7). Volumes 8 throughhavior of animals, some in the behavior of people, and others 12 address the application of psychological knowledge inin the behavior of organizations. These and other dimensions five broad areas of professional practice: clinical psychologyof difference among psychological scientists are matched by (Volume 8), health psychology (Volume 9), assessment psy-equal if not greater heterogeneity among psychological practi- chology (Volume 10), forensic psychology (Volume 11), andtioners, who currently apply a vast array of methods in many industrial and organizational psychology (Volume 12). Eachdifferent settings to achieve highly varied purposes. of these volumes reviews what is currently known in these Psychology has been rich in comprehensive encyclope- areas of study and application and identifies pertinent sourcesdias and in handbooks devoted to specific topics in the field. of information in the literature. Each discusses unresolved is-However, there has not previously been any single handbook sues and unanswered questions and proposes future direc-designed to cover the broad scope of psychological science tions in conceptualization, research, and practice. Each of theand practice. The present 12-volume Handbook of Psychol- volumes also reflects the investment of scientific psycholo-ogy was conceived to occupy this place in the literature. gists in practical applications of their findings and the atten-Leading national and international scholars and practitioners tion of applied psychologists to the scientific basis of theirhave collaborated to produce 297 authoritative and detailed methods.chapters covering all fundamental facets of the discipline, The Handbook of Psychology was prepared for the pur-and the Handbook has been organized to capture the breadth pose of educating and informing readers about the presentand diversity of psychology and to encompass interests and state of psychological knowledge and about anticipated ad-concerns shared by psychologists in all branches of the field. vances in behavioral science research and practice. With this Two unifying threads run through the science of behavior. purpose in mind, the individual Handbook volumes addressThe first is a common history rooted in conceptual and em- the needs and interests of three groups. First, for graduate stu-pirical approaches to understanding the nature of behavior. dents in behavioral science, the volumes provide advancedThe specific histories of all specialty areas in psychology instruction in the basic concepts and methods that define thetrace their origins to the formulations of the classical philoso- fields they cover, together with a review of current knowl-phers and the methodology of the early experimentalists, and edge, core literature, and likely future developments. Second,appreciation for the historical evolution of psychology in all in addition to serving as graduate textbooks, the volumesof its variations transcends individual identities as being one offer professional psychologists an opportunity to read andkind of psychologist or another. Accordingly, Volume 1 in contemplate the views of distinguished colleagues concern-the Handbook is devoted to the history of psychology as ing the central thrusts of research and leading edges of prac-it emerged in many areas of scientific study and applied tice in their respective fields. Third, for psychologists seekingtechnology. to become conversant with fields outside their own specialty ix
  8. 8. x Handbook of Psychology Prefaceand for persons outside of psychology seeking informa- valuable contributions to the literature. I would like finally totion about psychological matters, the Handbook volumes express my appreciation to the editorial staff of John Wileyserve as a reference source for expanding their knowledge and Sons for the opportunity to share in the development ofand directing them to additional sources in the literature. this project and its pursuit to fruition, most particularly to The preparation of this Handbook was made possible by Jennifer Simon, Senior Editor, and her two assistants, Marythe diligence and scholarly sophistication of the 25 volume Porterfield and Isabel Pratt. Without Jennifer’s vision of theeditors and co-editors who constituted the Editorial Board. Handbook and her keen judgment and unflagging support inAs Editor-in-Chief, I want to thank each of them for the plea- producing it, the occasion to write this preface would notsure of their collaboration in this project. I compliment them have arrived.for having recruited an outstanding cast of contributors totheir volumes and then working closely with these authors to IRVING B. WEINERachieve chapters that will stand each in their own right as Tampa, Florida
  9. 9. Volume PrefaceA scientific discipline is defined in many ways by the re- computers to perform complex methods of data analysis, in-search methods it employs. These methods can be said to rep- creased computer capacity allowing for more intense analysisresent the common language of the discipline’s researchers. of larger datasets, computer simulations that permit the eval-Consistent with the evolution of a lexicon, new research uation of procedures across a wide variety of situations, newmethods frequently arise from the development of new approaches to data analysis and statistical control, and ad-content areas. By every available measure—number of re- vances in companion sciences that opened pathways to thesearchers, number of publications, number of journals, num- exploration of behavior and created new areas of researchber of new subdisciplines—psychology has undergone a specialization and collaboration.tremendous growth over the last half-century. This growth is Consider the advances since the publication of thereflected in a parallel increase in the number of new research first edition of Kirk’s (1968) text on experimental design.methods available. At that time most studies were relatively small N experiments As we were planning and editing this volume, we dis- that were conducted in psychology laboratories. Research ac-cussed on many occasions the extent to which psychology tivity has subsequently exploded in applied and clinicaland the available research methods have become increasing areas, with a proliferation of new journals largely dedicatedcomplex over the course of our careers. When our generation to quasi-experimental studies and studies in the natural envi-of researchers began their careers in the late 1960s and early ronment (e.g., in neuropsychology and health psychology).1970s, experimental design was largely limited to simple Techniques such as polymerase chain reaction allow psychol-between-group designs, and data analysis was dominated by ogists to test specific genes as risk candidates for behaviorala single method, the analysis of variance. A few other ap- disorders. These studies rely on statistical procedures that areproaches were employed, but by a limited number of re- still largely ignored by many researchers (e.g., logistic re-searchers. Multivariate statistics had been developed, but gression, structural equation modeling). Brain imagingmultiple regression analysis was the only method that was procedures such as magnetic resonance imaging, magnetoen-applied with any frequency. Factor analysis was used almost cephalography, and positron-emission tomography provideexclusively as a method in scale development. Classical test cognitive psychologists and neuropsychologists the opportu-theory was the basis of most psychological and educational nity to study cortical activity on-line. Clinical trials involvingmeasures. Analysis of data from studies that did not meet behavioral interventions applied to large, representative sam-either the design or measurement assumptions required for an ples are commonplace in health psychology. Research em-analysis of variance was covered for most researchers by a ploying each of these procedures requires not only highlysingle book on nonparametric statistics by Siegel (1956). As specific and rigorous research methods, but also speciala review of the contents of this volume illustrates, the choice methods for handling and analyzing extremely large volumesof experimental and analytic methods available to the of data. Even in more traditional areas of research that con-present-day researcher is much broader. It would be fair to tinue to rely on group experimental designs, issues of mea-say that the researcher in the 1960s had to formulate research suring practical significance, determination of sample sizequestions to fit the available methods. Currently, there are re- and power, and procedures for handling nuisance variablessearch methods available to address most research questions. are now important concerns. Not surprisingly, the third edi- In the history of science, an explosion of knowledge is tion of Kirk’s (1995) text has grown in page length by 60%.usually the result of an advance in technology, new theoreti- Our review of these trends leads to several conclusions,cal models, or unexpected empirical findings. Advances in which are reflected in the selection of topics covered by theresearch methods have occurred as the result of all three fac- chapters in this volume. Six features appear to characterizetors, typically in an interactive manner. Some of the specific the evolution in research methodology in psychology.factors include advances in instrumentation and measure- First, there has been a focus on the development of proce-ment technology, the availability of inexpensive desktop dures that employ statistical control rather than experimental xi
  10. 10. xii Volume Prefacecontrol. Because most of the recent growth involves research to focus on the individual and model individual differences.in areas that preclude direct control of independent variables, This becomes increasingly important as we recognize that in-multivariate statistics and the development of methods such terventions do not affect everyone in exactly the same waysas path analysis and structural equation modeling have been and that interventions become more and more tailored to thecritical developments. The use of statistical control has al- individual.lowed psychology to move from the carefully controlled con- The text is organized into four parts. The first part, titledfines of the laboratory to the natural environment. “Foundations of Research,” addresses issues that are funda- Second, there has been an increasing focus on construct- mental to all behavioral science research. The focus is ondriven, or latent-variable, research. A construct is defined by study design, data management, data reduction, and data syn-multiple observed variables. Constructs can be viewed as thesis. The first chapter, “Experimental Design” by Roger E.more reliable and more generalizable than a single observed Kirk, provides an overview of the basic considerations thatvariable. Constructs serve to organize a large set of observed go into the design of a study. Once, a chapter on this topicvariables, resulting in parsimony. Constructs are also theoret- would have had to devote a great deal of attention to compu-ically based. This theory-based approach serves to guide tational procedures. The availability of computers permits astudy design, the choice of variables, the data analysis, and shift in focus to the conceptual rather than the computationalthe data interpretation. issues. The second chapter, “Exploratory Data Analysis” by Third, there has been an increasing emphasis on the de- John T. Behrens and Chong-ho Yu, reminds us of the funda-velopment of new measures and new measurement models. mental importance of looking at data in the most basic waysThis is not a new trend but an acceleration of an old trend. as a first step in any data analysis. In some ways this repre-The behavioral sciences have always placed the most empha- sents a “back to the future” chapter. Advances in computer-sis on the issue of measurement. With the movement of the based graphical methods have brought a great deal of sophis-field out of the laboratory combined with advances in tech- tication to this very basic first step.nology, the repertoire of measures, the quality of the mea- The third chapter, “Power: Basics, Practical Problems,sures, and the sophistication of the measurement models have and Possible Solutions” by Rand R. Wilcox, reflects the crit-all increased dramatically. ical change in focus for psychological research. Originally, Fourth, there is increasing recognition of the importance of the central focus of a test of significance was on controllingthe temporal dimension in understanding a broad range of psy- Type I error rates. The late Jacob Cohen emphasized that re-chological phenomena. We have become a more intervention- searchers should be equally concerned by Type II errors.oriented science, recognizing not only the complexity of This resulted in an emphasis on the careful planning of atreatment effects but also the importance of the change in pat- study and a concern with effect size and selecting the appro-terns of the effects over time. The effects of an intervention priate sample size. Wilcox updates and extends these con-may be very different at different points in time. New statisti- cepts. Chapter 4, “Methods for Handling Missing Data” bycal models for modeling temporal data have resulted. John W. Graham, Patricio E. Cumsille, and Elvira Elek-Fisk, Fifth, new methods of analysis have been developed describes the impressive statistical advances in addressingthat no longer require the assumption of a continuous, equal- the common practical problem of missing observations.interval, normally distributed variable. Previously, re- Previously, researchers had relied on a series of ad hoc pro-searchers had the choice between very simple but limited cedures, often resulting in very inaccurate estimates. The newmethods of data analysis that corresponded to the properties statistical procedures allow the researcher to articulate theof the measure or more complex sophisticated methods of assumptions about the reason the data is missing and makeanalysis that assumed, often inappropriately, that the measure very sophisticated estimates of the missing value based on allmet very rigid assumptions. New methods have been devel- the available information. This topic has taken on even moreoped for categorical, ordinal, or simply nonnormal variables importance with the increasing emphasis on longitudinalthat can perform an equally sophisticated analysis. studies and the inevitable problem of attrition. Sixth, the importance of individual differences is increas- The fifth chapter, “Preparatory Data Analysis” by Linda S.ingly emphasized in intervention studies. Psychology has Fidell and Barbara G. Tabachnick, describes methods of pre-always been interested in individual differences, but meth- processing data before the application of other methods ofods of data analysis have focused almost entirely on the rela- statistical analysis. Extreme values can distort the results oftionships between variables. Individuals were studied as the data analysis if not addressed. Diagnostic methods canmembers of groups, and individual differences served only to preprocess the data so that complex procedures are not un-inflate the error variance. New techniques permit researchers duly affected by a limited number of cases that often are the
  11. 11. Volume Preface xiiiresult of some type of error. The last two chapters in this part, findings that are of interest to psychologists in many fields.“Factor Analysis” by Richard L. Gorsuch and “Clustering The major goal of Chapter 11, “Animal Learning” by Russelland Classification Methods” by Glenn W. Milligan and M. Church, is to transfer what is fairly common knowledge inStephen C. Hirtle, describe two widely employed parsimony experimental animal psychology to investigators with limitedmethods. Factor analysis operates in the variable domain and exposure to this area of research. In Chapter 12, “Neuropsy-attempts to reduce a set of p observed variables to a smaller chology,” Russell M. Bauer, Elizabeth C. Leritz, and Dawnset of m factors. These factors, or latent variables, are more Bowers provide a discussion of neuropsychological inference,easily interpreted and thus facilitate interpretation. Cluster an overview of major approaches to neuropsychological re-analysis operates in the person domain and attempts to reduce search, and a review of newer techniques, including functionala set of N individuals to a set of k clusters. Cluster analysis neuroimaging, electrophysiology, magnetoencephalography,serves to explore the relationships among individuals and or- and reversible lesion methods. In each section, they describeganize the set of individuals into a limited number of sub- the conceptual basis of the technique, outline its strengths andtypes that share essential features. These methods are basic to weaknesses, and cite examples of how it has been used inthe development of construct-driven methods and the focus addressing conceptual problems in neuropsychology.on individual differences. Whatever their specialty area, when psychologists evalu- The second part, “Research Methods in Specific Content ate a program or policy, the question of impact is often at cen-Areas,” addresses research methods and issues as they apply ter stage. The last chapter in this part, “Program Evaluation”to specific content areas. Content areas were chosen in part to by Melvin M. Mark, focuses on key methods for estimatingparallel the other volumes of the Handbook. More important, the effects of policies and programs in the context of evalua-however, we attempted to sample content areas from a broad tion. Additionally, Mark addresses several noncausal formsspectrum of specialization with the hope that these chapters of program evaluation research that are infrequently ad-would provide insights into methodological concerns and dressed in methodological treatises.solutions that would generalize to other areas. Chapter 8, The third part is titled “Measurement Issues.” Advances in“Clinical Forensic Psychology” by Kevin S. Douglas, Randy measurement typically combine innovation in technologyK. Otto, and Randy Borum, addresses research methods and and progress in theory. As our measures become more so-issues that occur in assessment and treatment contexts. For phisticated, the areas of application also increase.each task that is unique to clinical forensic psychology Mood emerged as a seminal concept within psychologyresearch, they provide examples of the clinical challenges during the 1980s, and its prominence has continued unabatedconfronting the psychologist, identify problems faced when ever since. In Chapter 14, “Mood Measurement: Currentresearching the issues or constructs, and describe not only re- Status and Future Directions,” David Watson and Jatin Vaidyasearch strategies that have been employed but also their examine current research regarding the underlying structurestrengths and limitations. In Chapter 9, “Psychotherapy Out- of mood, describe and evaluate many of the most importantcome Research,” Evelyn S. Behar and Thomas D. Borkovec mood measures, and discuss several issues related to theaddress the methodological issues that need to be considered reliability and construct validity of mood measurement. Infor investigators to draw the strongest and most specific Chapter 15, “Measuring Personality and Psychopathology,”cause-and-effect conclusions about the active components of Leslie C. Morey uses objective self-report methods of mea-treatments, human behavior, and the effectiveness of thera- surement to illustrate contemporary procedures for scalepeutic interventions. development and validation, addressing issues critical to all The field of health psychology is largely defined by three measurement methods such as theoretical articulation, situa-topics: the role of behavior (e.g., smoking) in the develop- tional context, and the need for discriminant validity.ment and prevention of disease, the role of stress and emotion The appeal of circular models lies in the combination of aas psychobiological influences on disease, and psychological circle’s aesthetic (organizational) simplicity and its powerfulaspects of acute and chronic illness and medical care. Insight potential to describe data in uniquely compelling substantiveinto the methodological issues and solutions for research in and geometric ways, as has been demonstrated in describ-each of these topical areas is provided by Timothy W. Smith ing interpersonal behavior and occupational interests. Inin Chapter 10, “Health Psychology.” Chapter 16, “The Circumplex Model: Methods and Research At one time, most behavioral experimentation was con- Applications,” Michael B. Gurtman and Aaron L. Pincus dis-ducted by individuals whose training focused heavily on ani- cuss the application of the circumplex model to the descrip-mal research. Now many neuroscientists, trained in various tions of individuals, comparisons of groups, and evaluationsfields, conduct research in animal learning and publish of constructs and their measures.
  12. 12. xiv Volume Preface Chapter 17, “Item Response Theory and Measuring Abili- Velicer and Joseph L. Fava describe a method for studyingties” by Karen M. Schmidt and Susan E. Embretson, de- the change in a single individual over time. Instead of a sin-scribes the types of formal models that have been designed to gle observation on many subjects, this method relies on manyguide measure development. For many years, most tests of observations on a single subject. In many ways, this methodability and achievement have relied on classical test theory as is the prime exemplar of longitudinal research methods.a framework to guide both measure development and mea- Chapter 24, “Structural Equation Modeling” by Jodie B.sure evaluation. Item response theory updates this model in Ullman and Peter M. Bentler, describes a very generalmany important ways, permitting the development of a new method that combines three key themes: constructs or latentgeneration of measures of abilities and achievement that are variables, statistical control, and theory to guide data analy-particularly appropriate for a more interactive model of as- sis. First employed as an analytic method little more thansessment. The last chapter of this part, “Growth Curve Analy- 20 years ago, the method is now widely disseminated in thesis in Contemporary Psychological Research” by John J. behavioral sciences. Chapter 25, “Ordinal Analysis of Behav-McArdle and John R. Nesselroade, describes new quantita- ioral Data” by Jeffrey D. Long, Du Feng, and Norman Cliff,tive methods for the study of change in development psy- discusses the assumptions that underlie many of the widelychology. The methods permit the researcher to model a wide used statistical methods and describes a parallel series ofvariety of different patterns of developmental change over methods of analysis that only assume that the measure pro-time. vides ordinal information. The last chapter, “Latent Class and The final part, “Data Analysis Methods,” addresses statis- Latent Transition Analysis” by Stephanie L. Lanza, Brian P.tical procedures that have been developed recently and are Flaherty, and Linda M. Collins, describes a new method forstill not widely employed by many researchers. They are typ- analyzing change over time. It is particularly appropriateically dependent on the availability of high-speed computers when the change process can be conceptualized as a series ofand permit researchers to investigate novel and complex re- discrete states.search questions. Chapter 19, “Multiple Linear Regression” In completing this project, we realized that we were veryby Leona Aiken, Stephen G. West, and Steven C. Pitts, de- fortunate in several ways. Irving Weiner’s performance asscribes the advances in multiple linear regression that permit editor-in-chief was simply wonderful. He applied just the rightapplications of this very basic method to the analysis of com- mix of obsessive concern and responsive support to keep thingsplex data sets and the incorporation of conceptual models to on schedule. His comments on issues of emphasis, perspective,guide the analysis. The testing of theoretical predictions and and quality were insightful and inevitably on target.the identification of implementation problems are the two We continue to be impressed with the professionalism ofmajor foci of this chapter. Chapter 20, “Logistic Regression” the authors that we were able to recruit into this effort.by Alfred DeMaris, describes a parallel method to multiple Consistent with their reputations, these individuals deliv-regression analysis for categorical variables. The procedure ered chapters of exceptional quality, making our burdenhas been developed primarily outside of psychology and is pale in comparison to other editorial experiences. Because ofnow being used much more frequently to address psycholog- the length of the project, we shared many contributors’ical questions. Chapter 21, “Meta-Analysis” by Frank L. experiences-marriages, births, illnesses, family crises. A def-Schmidt and John E. Hunter, describes procedures that have inite plus for us has been the formation of new friendshipsbeen developed for the quantitative integration of research and professional liaisons.findings across multiple studies. Previously, research findings Our editorial tasks were also aided greatly by the generouswere integrated in narrative form and were subject to the bi- assistance of our reviewers, most of whom will be quicklyases of the reviewer. The method also focuses attention on the recognized by our readers for their own expertise in researchimportance of effect size estimation. methodology. We are pleased to thank James Algina, Phipps Chapter 22, “Survival Analysis” by Judith D. Singer and Arabie, Patti Barrows, Betsy Jane Becker, Lisa M. Brown,John B. Willett, describes a recently developed method for Barbara M. Byrne, William F. Chaplin, Pat Cohen, Patrick J.analyzing longitudinal data. One approach is to code whether Curren, Glenn Curtiss, Richard B. Darlington, Susanan event has occurred at a given occasion. By switching the Duncan, Brian Everitt, Kerry Evers, Ron Gironda, Lisafocus on the time to the occurrence of the event, a much more Harlow, Michael R. Harwell, Don Hedeker, David Charlespowerful and sophisticated analysis can be performed. Again, Howell, Lawrence J. Hubert, Bradley E. Huitema, Beththe development of this procedure has occurred largely out- Jenkins, Herbert W. Marsh, Rosemarie A. Martin, Scott E.side psychology but is being employed much more fre- Maxwell, Kevin R. Murphy, Gregory Norman, Daniel J.quently. In Chapter 23, “Time Series Analysis,” Wayne Ozer, Melanie Page, Mark D. Reckase, Charles S. Reichardt,
  13. 13. Volume Preface xvSteven Reise, Joseph L. Rogers, Joseph Rossi, James or supported by differing approaches. For flaws in the text,Rounds, Shlomo S. Sawilowsky, Ian Spence, James H. however, the usual rule applies: We assume all responsibility.Steiger, Xiaowu Sun, Randall C. Swaim, David Thissen,Bruce Thompson, Terence J. G. Tracey, Rod Vanderploeg, JOHN A. SCHINKAPaul F. Velleman, Howard Wainer, Douglas Williams, and WAYNE F. VELICERseveral anonymous reviewers for their thorough work andgood counsel. We finish this preface with a caveat. Readers will in- REFERENCESevitably discover several contradictions or disagreementsacross the chapter offerings. Inevitably, researchers in differ- Kirk, Roger E. (1968). Experimental design: Procedures for theent areas solve similar methodological problems in different behavioral sciences. Pacific Grove, CA: Brooks/Cole.ways. These differences are reflected in the offerings of this Kirk, Roger E. (1995). Experimental design: Procedures for thetext, and we have not attempted to mediate these differing behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole.viewpoints. Rather, we believe that the serious researcher Siegel, S. (1956). Nonparametric statistics for the behavioralwill welcome the opportunity to review solutions suggested sciences. New York: McGraw-Hill.
  14. 14. ContentsHandbook of Psychology Preface ix Irving B. WeinerVolume Preface xi John A. Schinka and Wayne F. VelicerContributors xxi PA RT O N E FOUNDATIONS OF RESEARCH ISSUES: STUDY DESIGN, DATA MANAGEMENT, DATA REDUCTION, AND DATA SYNTHESIS1 EXPERIMENTAL DESIGN 3 Roger E. Kirk2 EXPLORATORY DATA ANALYSIS 33 John T. Behrens and Chong-ho Yu3 POWER: BASICS, PRACTICAL PROBLEMS, AND POSSIBLE SOLUTIONS 65 Rand R. Wilcox4 METHODS FOR HANDLING MISSING DATA 87 John W. Graham, Patricio E. Cumsille, and Elvira Elek-Fisk5 PREPARATORY DATA ANALYSIS 115 Linda S. Fidell and Barbara G. Tabachnick6 FACTOR ANALYSIS 143 Richard L. Gorsuch7 CLUSTERING AND CLASSIFICATION METHODS 165 Glenn W. Milligan and Stephen C. Hirtle PA RT T W O RESEARCH METHODS IN SPECIFIC CONTENT AREAS8 CLINICAL FORENSIC PSYCHOLOGY 189 Kevin S. Douglas, Randy K. Otto, and Randy Borum9 PSYCHOTHERAPY OUTCOME RESEARCH 213 Evelyn S. Behar and Thomas D. Borkovec xvii
  15. 15. xviii Contents10 HEALTH PSYCHOLOGY 241 Timothy W. Smith11 ANIMAL LEARNING 271 Russell M. Church12 NEUROPSYCHOLOGY 289 Russell M. Bauer, Elizabeth C. Leritz, and Dawn Bowers13 PROGRAM EVALUATION 323 Melvin M. Mark PA RT T H R E E MEASUREMENT ISSUES14 MOOD MEASUREMENT: CURRENT STATUS AND FUTURE DIRECTIONS 351 David Watson and Jatin Vaidya15 MEASURING PERSONALITY AND PSYCHOPATHOLOGY 377 Leslie C. Morey16 THE CIRCUMPLEX MODEL: METHODS AND RESEARCH APPLICATIONS 407 Michael B. Gurtman and Aaron L. Pincus17 ITEM RESPONSE THEORY AND MEASURING ABILITIES 429 Karen M. Schmidt and Susan E. Embretson18 GROWTH CURVE ANALYSIS IN CONTEMPORARY PSYCHOLOGICAL RESEARCH 447 John J. McArdle and John R. Nesselroade PA RT F O U R DATA ANALYSIS METHODS19 MULTIPLE LINEAR REGRESSION 483 Leona S. Aiken, Stephen G. West, and Steven C. Pitts20 LOGISTIC REGRESSION 509 Alfred DeMaris21 META-ANALYSIS 533 Frank L. Schmidt and John E. Hunter22 SURVIVAL ANALYSIS 555 Judith D. Singer and John B. Willett23 TIME SERIES ANALYSIS 581 Wayne F. Velicer and Joseph L. Fava
  16. 16. Contents xix24 STRUCTURAL EQUATION MODELING 607 Jodie B. Ullman and Peter M. Bentler25 ORDINAL ANALYSIS OF BEHAVIORAL DATA 635 Jeffrey D. Long, Du Feng, and Norman Cliff26 LATENT CLASS AND LATENT TRANSITION ANALYSIS 663 Stephanie T. Lanza, Brian P. Flaherty, and Linda M. CollinsAuthor Index 687Subject Index 703
  17. 17. ContributorsLeona S. Aiken, PhD Norman Cliff, PhDDepartment of Psychology Professor of Psychology EmeritusArizona State University University of Southern CaliforniaTempe, Arizona Los Angeles, CaliforniaRussell M. Bauer, PhD Linda M. Collins, PhDDepartment of Clinical and Health Psychology The Methodology CenterUniversity of Florida Pennsylvania State UniversityGainesville, Florida University Park, PennsylvaniaEvelyn S. Behar, MS Patricio E. Cumsille, PhDDepartment of Psychology Escuela de PsicologiaPennsylvania State University Universidad Católica de ChileUniversity Park, Pennsylvania Santiago, ChileJohn T. Behrens, PhD Alfred DeMaris, PhDCisco Networking Academy Program Department of SociologyCisco Systems, Inc. Bowling Green State UniversityPhoenix, Arizona Bowling Green, OhioPeter M. Bentler, PhD Kevin S. Douglas, PhD, LLBDepartment of Psychology Department of Mental Health Law & PolicyUniversity of California Florida Mental Health InstituteLos Angeles, California University of South Florida Tampa, FloridaThomas D. Borkovec, PhDDepartment of Psychology Du Feng, PhDPennsylvania State University Human Development and Family StudiesUniversity Park, Pennsylvania Texas Tech University Lubbock, TexasRandy Borum, PsyDDepartment of Mental Health Law & Policy Elvira Elek-Fisk, PhDFlorida Mental Health Institute The Methodology CenterUniversity of South Florida Pennsylvania State UniversityTampa, Florida University Park, PennsylvaniaDawn Bowers, PhD Susan E. Embretson, PhDDepartment of Clinical and Health Psychology Department of PsychologyUniversity of Florida University of KansasGainesville, Florida Lawrence, KansasRussell M. Church, PhD Joseph L. Fava, PhDDepartment of Psychology Cancer Prevention Research CenterBrown University University of Rhode IslandProvidence, Rhode Island Kingston, Rhode Island xxi
  18. 18. xxii ContributorsLinda S. Fidell, PhD Melvin M. Mark, PhDDepartment of Psychology Department of PsychologyCalifornia State University Pennsylvania State UniversityNorthridge, California University Park, PennsylvaniaBrian P. Flaherty, MS John J. McArdle, PhDThe Methodology Center Department of PsychologyPennsylvania State University University of VirginiaUniversity Park, Pennsylvania Charlottesville, VirginiaRichard L. Gorsuch, PhD Glenn W. Milligan, PhDGraduate School of Psychology Department of Management SciencesFuller Theological Seminary Ohio State UniversityPasadena, California Columbus, OhioJohn W. Graham, PhD Leslie C. Morey, PhDDepartment of Biobehavioral Health Department of PsychologyPennsylvania State University Texas A&M UniversityUniversity Park, Pennsylvania College Station, TexasMichael B. Gurtman, PhD John R. Nesselroade, PhDDepartment of Psychology Department of PsychologyUniversity of Wisconsin-Parkside University of VirginiaKenosha, Wisconsin Charlottesville, VirginiaStephen C. Hirtle, PhD Randy K. Otto, PhDSchool of Information Sciences Department of Mental Health Law & PolicyUniversity of Pittsburgh Florida Mental Health InstitutePittsburgh, Pennsylvania University of South Florida Tampa, FloridaJohn E. Hunter, PhD Aaron L. Pincus, PhDDepartment of Psychology Department of PsychologyMichigan State University Pennsylvania State UniversityEast Lansing, Michigan University Park, PennsylvaniaRoger E. Kirk, PhD Steven C. Pitts, PhDDepartment of Psychology and Neuroscience Department of PsychologyBaylor University University of Maryland, Baltimore CountyWaco, Texas Baltimore, MarylandStephanie T. Lanza, MS Karen M. Schmidt, PhDThe Methodology Center Department of PsychologyPennsylvania State University University of VirginiaUniversity Park, Pennsylvania Charlottesville, VirginiaElizabeth C. Leritz, MS Frank L. Schmidt, PhDDepartment of Clinical and Health Psychology Department of Management and OrganizationUniversity of Florida University of IowaGainesville, Florida Iowa City, IowaJeffrey D. Long, PhD Judith D. Singer, PhDDepartment of Educational Psychology Graduate School of EducationUniversity of Minnesota Harvard UniversityMinneapolis, Minnesota Cambridge, Massachusetts
  19. 19. Contributors xxiiiTimothy W. Smith, PhD David Watson, PhDDepartment of Psychology Department of PsychologyUniversity of Utah University of IowaSalt Lake City, Utah Iowa City, IowaBarbara G. Tabachnick, PhD Stephen G. West, PhDDepartment of Psychology Department of PsychologyCalifornia State University Arizona State UniversityNorthridge, California Tempe, ArizonaJodie B. Ullman, PhD Rand R. Wilcox, PhDDepartment of Psychology Department of PsychologyCalifornia State University University of Southern CaliforniaSan Bernadino, California Los Angeles, CaliforniaJatin Vaidya John B. Willett, PhDDepartment of Psychology Graduate School of EducationUniversity of Iowa Harvard UniversityIowa City, Iowa Cambridge, MassachusettsWayne F. Velicer, PhD Chong-ho Yu, PhDCancer Prevention Research Center Cisco Networking Academy ProgramUniversity of Rhode Island Cisco Systems, Inc.Kingston, Rhode Island Chandler, Arizona
  20. 20. PA R T O N E FOUNDATIONS OF RESEARCH ISSUES: STUDY DESIGN, DATA MANAGEMENT,DATA REDUCTION, AND DATA SYNTHESIS
  21. 21. CHAPTER 1Experimental DesignROGER E. KIRKSOME BASIC EXPERIMENTAL DESIGN CONCEPTS 3 FACTORIAL DESIGNS WITH CONFOUNDING 21THREE BUILDING BLOCK DESIGNS 4 Split-Plot Factorial Design 21 Completely Randomized Design 4 Confounded Factorial Designs 24 Randomized Block Design 6 Fractional Factorial Designs 25 Latin Square Design 9 HIERARCHICAL DESIGNS 27CLASSIFICATION OF EXPERIMENTAL DESIGNS 10 Hierarchical Designs With One orFACTORIAL DESIGNS 11 Two Nested Treatments 27 Completely Randomized Factorial Design 11 Hierarchical Design With Crossed Alternative Models 14 and Nested Treatments 28 Randomized Block Factorial Design 19 EXPERIMENTAL DESIGNS WITH A COVARIATE 29 REFERENCES 31SOME BASIC EXPERIMENTAL about (a) one or more parameters of a population or (b) theDESIGN CONCEPTS functional form of a population. Statistical hypotheses are rarely identical to scientific hypotheses—they areExperimental design is concerned with the skillful interroga- testable formulations of scientific hypotheses.tion of nature. Unfortunately, nature is reluctant to reveal 2. Determination of the experimental conditions (independenther secrets. Joan Fisher Box (1978) observed in her autobiog- variable) to be manipulated, the measurement (dependentraphy of her father, Ronald A. Fisher, “Far from behaving variable) to be recorded, and the extraneous conditionsconsistently, however, Nature appears vacillating, coy, and (nuisance variables) that must be controlled.ambiguous in her answers” (p. 140). Her most effective 3. Specification of the number of participants required andtool for confusing researchers is variability—in particular, the population from which they will be sampled.variability among participants or experimental units. But 4. Specification of the procedure for assigning the partici-two can play the variability game. By comparing the variabil- pants to the experimental conditions.ity among participants treated differently to the variability 5. Determination of the statistical analysis that will beamong participants treated alike, researchers can make in- performed.formed choices between competing hypotheses in scienceand technology. In short, an experimental design identifies the independent, We must never underestimate nature—she is a formidable dependent, and nuisance variables and indicates the way infoe. Carefully designed and executed experiments are re- which the randomization and statistical aspects of an experi-quired to learn her secrets. An experimental design is a plan ment are to be carried out.for assigning participants to experimental conditions and thestatistical analysis associated with the plan (Kirk, 1995, p. 1).The design of an experiment involves a number of inter- Analysis of Variancerelated activities: Analysis of variance (ANOVA) is a useful tool for under-1. Formulation of statistical hypotheses that are germane to the standing the variability in designed experiments. The seminal scientific hypothesis. A statistical hypothesis is a statement ideas for both ANOVA and experimental design can be traced 3
  22. 22. 4 Experimental Designto Ronald A. Fisher, a statistician who worked at the Rotham- Fisher popularized two other principles of good experi-sted Experimental Station. According to Box (1978, p. 100), mentation: replication and local control or blocking. Replica-Fisher developed the basic ideas ofANOVAbetween 1919 and tion is the observation of two or more participants under1925. The first hint of what was to come appeared in a 1918 identical experimental conditions. Fisher observed that repli-paper in which Fisher partitioned the total variance of a human cation enables a researcher to estimate error effects andattribute into portions attributed to heredity, environment, and obtain a more precise estimate of treatment effects. Blocking,other factors. The analysis of variance table for a two-treat- on the other hand, is an experimental procedure for isolatingment factorial design appeared in a 1923 paper published with variation attributable to a nuisance variable. As the nameM. A. Mackenzie (Fisher & Mackenzie, 1923). Fisher referred suggests, nuisance variables are undesired sources of varia-to the table as a convenient way of arranging the arithmetic. In tion that can affect the dependent variable. There are many1924 Fisher (1925) introduced the Latin square design in con- sources of nuisance variation. Differences among partici-nection with a forest nursery experiment. The publication in pants comprise one source. Other sources include variation1925 of his classic textbook Statistical Methods for Research in the presentation of instructions to participants, changes inWorkers and a short paper the following year (Fisher, 1926) environmental conditions, and the effects of fatigue andpresented all the essential ideas of analysis of variance. The learning when participants are observed several times. Threetextbook (Fisher, 1925, pp. 244–249) included a table of the experimental approaches are used to deal with nuisancecritical values of the ANOVA test statistic in terms of a func- variables:tion called z, where z = 1 (ln ␴2 2 ˆ Treatment − ln ␴2 ). The statis- ˆ Error 2 2tics ␴Treatment and ␴Error denote, respectively, treatment and ˆ ˆ 1. Holding the variable constant.error variance. A more convenient form of Fisher’s z table that 2. Assigning participants randomly to the treatment levels sodid not require looking up log values was developed by that known and unsuspected sources of variation amongGeorge Snedecor (1934). His critical values are expressed in the participants are distributed over the entire experimentterms of the function F = ␴2 ˆ Treatment /␴2 ˆ Error that is obtained and do not affect just one or a limited number of treatmentdirectly from the ANOVA calculations. He named it F in honor levels.of Fisher. Fisher’s field of experimentation—agriculture— 3. Including the nuisance variable as one of the factors in thewas a fortunate choice because results had immediate applica- experiment.tion with assessable economic value, because simplifyingassumptions such as normality and independence of errors The last experimental approach uses local control or blockingwere usually tenable, and because the cost of conducting to isolate variation attributable to the nuisance variable soexperiments was modest. that it does not appear in estimates of treatment and error effects. A statistical approach also can be used to deal withThree Principles of Good Experimental Design nuisance variables. The approach is called analysis of covari- ance and is described in the last section of this chapter.The publication of Fisher’s Statistical Methods for Research The three principles that Fisher vigorously championed—Workers and his 1935 The Design of Experiments gradually randomization, replication, and local control—remain theled to the acceptance of what today is considered to be the cornerstones of good experimental design.cornerstone of good experimental design: randomization.It is hard to imagine the hostility that greeted the suggestionthat participants or experimental units should be randomly THREE BUILDING BLOCK DESIGNSassigned to treatment levels. Before Fisher’s work, mostresearchers used systematic schemes, not subject to the laws Completely Randomized Designof chance, to assign participants. According to Fisher, ran-dom assignment has several purposes. It helps to distribute One of the simplest experimental designs is the randomizationthe idiosyncratic characteristics of participants over the treat- and analysis plan that is used with a t statistic for independentment levels so that they do not selectively bias the outcome of samples. Consider an experiment to compare the effectivenessthe experiment. Also, random assignment permits the com- of two diets for obese teenagers. The independent variable isputation of an unbiased estimate of error effects—those the two kinds of diets; the dependent variable is the amount ofeffects not attributable to the manipulation of the independent weight loss two months after going on a diet. For notationalvariable—and it helps to ensure that the error effects are convenience, the two diets are called treatment A. The levelsstatistically independent. of treatment A corresponding to the specific diets are denoted
  23. 23. Three Building Block Designs 5by the lowercase letter a and a subscript: a1 denotes one dietand a2 denotes the other. A particular but unspecified level oftreatment A is denoted by aj, where j ranges over the values 1and 2. The amount of weight loss in pounds 2 months afterparticipant i went on diet j is denoted by Yij. The null and alternative hypotheses for the weight-lossexperiment are, respectively, H0: ␮1 − ␮2 = 0 H1: ␮1 − ␮2 = 0 ,where ␮1 and ␮2 denote the mean weight loss of the respec-tive populations. Assume that 30 girls who want to loseweight are available to participate in the experiment. Theresearcher assigns n = 15 girls to each of the p = 2 diets so Figure 1.2 Layout for a completely randomized design (CR-3 design).that each of the (np)!/(n!) p = 155,117,520 possible assign- Forty-five girls are randomly assigned to three levels of treatment A with thements has the same probability. This is accomplished by restriction that 15 girls are assigned to each level. The mean weight loss innumbering the girls from 1 to 30 and drawing numbers from pounds for the girls in treatment levels a1, a2, and a3 is denoted by Y ·1 , Y ·2 , and Y ·3 , respectively.a random numbers table. The first 15 numbers drawn between1 and 30 are assigned to treatment level a1; the remaining 15numbers are assigned to a2. The layout for this experiment is three diets. The null and alternative hypotheses for theshown in Figure 1.1. The girls who were assigned to treat- experiment are, respectively,ment level a1 are called Group1; those assigned to treatmentlevel a2 are called Group2. The mean weight losses of the two H0: ␮1 = ␮2 = ␮3groups of girls are denoted by Y ·1 and Y ·2 . H1: ␮ j = ␮ j for some j and j . The t independent-samples design involves randomlyassigning participants to two levels of a treatment. A com- Assume that 45 girls who want to lose weight are available topletely randomized design, which is described next, extends participate in the experiment. The girls are randomly as-this design strategy to two or more treatment levels. The com- signed to the three diets with the restriction that 15 girls arepletely randomized design is denoted by the letters CR-p, assigned to each diet. The layout for the experiment is shownwhere CR stands for “completely randomized” and p is the in Figure 1.2. A comparison of the layout in this figure withnumber of levels of the treatment. that in Figure 1.1 for a t independent-samples design reveals Again, consider the weight-loss experiment and suppose that they are the same except that the completely randomizedthat the researcher wants to evaluate the effectiveness of design has three treatment levels. The t independent-samples design can be thought of as a special case of a completely randomized design. When p is equal to two, the layouts and randomization plans for the designs are identical. Thus far I have identified the null hypothesis that the researcher wants to test, ␮1 = ␮2 = ␮3 , and described the manner in which the participants are assigned to the three treatment levels. In the following paragraphs I discuss the com- posite nature of an observation, describe the classical model equation for a CR-p design, and examine the meaning of the terms treatment effect and error effect. An observation, which is a measure of the dependent vari- able, can be thought of as a composite that reflects the effects of the (a) independent variable, (b) individual charac-Figure 1.1 Layout for a t independent-samples design. Thirty girls are ran- teristics of the participant or experimental unit, (c) chancedomly assigned to two levels of treatment A with the restriction that 15 girlsare assigned to each level. The mean weight loss in pounds for the girls in fluctuations in the participant’s performance, (d) measure-treatment levels a1 and a2 is denoted by Y ·1 and Y ·2 , respectively. ment and recording errors that occur during data collection,
  24. 24. 6 Experimental Designand (e) any other nuisance variables such as environmental level are different because the error effects, ␧i( j)s, for theconditions that have not been controlled. Consider the weight observations are different. Recall that error effects reflect idio-loss of the fifth participant in treatment level a2. Suppose that syncratic characteristics of the participants—those character-two months after beginning the diet this participant has lost istics that differ from one participant to another—and any13 pounds (Y52 = 13). What factors have affected the value of other variables that have not been controlled. Researchers at-Y52? One factor is the effectiveness of the diet. Other factors tempt to minimize the size of error effects by holding sourcesare her weight prior to starting the diet, the degree to which of variation that might contribute to the error effects constantshe stayed on the diet, and the amount she exercised during and by the judicial choice of an experimental design. Designsthe two-month trial, to mention only a few. In summary, Y52 is that are described next permit a researcher to isolate and re-a composite that reflects (a) the effects of treatment level a2, move some sources of variation that would ordinarily be in-(b) effects unique to the participant, (c) effects attributable to cluded in the error effects.chance fluctuations in the participant’s behavior, (d) errors inmeasuring and recording the participant’s weight loss, and(e) any other effects that have not been controlled. Our con- Randomized Block Designjectures about Y52 or any of the other 44 observations can be The two designs just described use independent samples. Twoexpressed more formally by a model equation. The classical samples are independent if, for example, a researcher ran-model equation for the weight-loss experiment is domly samples from two populations or randomly assigns par- ticipants to p groups. Dependent samples, on the other hand, Yi j = ␮ + ␣ j + ␧i( j) (i = 1, . . . , n; j = 1, . . . , p), can be obtained by any of the following procedures.where 1. Observe each participant under each treatment level in the experiment—that is, obtain repeated measures on the Yi j is the weight loss for participant i in treatment participants. level aj. 2. Form sets of participants who are similar with respect to ␮ is the grand mean of the three weight-loss popula- a variable that is correlated with the dependent variable. tion means. This procedure is called participant matching. ␣j is the treatment effect for population j and is equal to 3. Obtain sets of identical twins or littermates in which case ␮ j − ␮. It reflects the effects of diet aj. the participants have similar genetic characteristics. ␧i( j) is the within-groups error effect associated with Yi j 4. Obtain participants who are matched by mutual selection, and is equal to Yi j − ␮ − ␣ j . It reflects all effects for example, husband and wife pairs or business partners. not attributable to treatment level aj. The notation i( j) indicates that the ith participant appears only in In the behavioral and social sciences, the participants are treatment level j. Participant i is said to be nested often people whose aptitudes and experiences differ markedly. within the jth treatment level. Nesting is discussed Individual differences are inevitable, but it is often possible in the section titled “Hierarchical Designs.” to isolate or partition out a portion of these effects so that they do not appear in estimates of the error effects. One designAccording to the equation for this completely randomized for accomplishing this is the design used with a t statistic fordesign, each observation is the sum of three parameters dependent samples. As the name suggests, the design uses␮, ␣ j , and ␧i( j) . The values of the parameters in the equation dependent samples. A t dependent-samples design also uses aare unknown but can be estimated from sample data. more complex randomization and analysis plan than does a t The meanings of the terms grand mean, ␮, and treatment independent-samples design. However, the added complexityeffect, ␣ j , in the model equation seem fairly clear; the mean- is often accompanied by greater power—a point that I will de-ing of error effect, ␧i( j) , requires a bit more explanation. Why velop later in connection with a randomized block design.do observations, Yi j s, in the same treatment level vary from Let’s reconsider the weight-loss experiment. It is reason-one participant to the next? This variation must be due to dif- able to assume that ease of losing weight is related to theferences among the participants and to other uncontrolled amount by which a girl is overweight. The design of the exper-variables because the parameters ␮ and ␣ j in the model equa- iment can be improved by isolating this nuisance variable.tion are constants for all participants in the same treatment Suppose that instead of randomly assigning 30 participants tolevel. To put it another way, observations in the same treatment the treatment levels, the researcher formed pairs of participants
  25. 25. Three Building Block Designs 7 participants who have been exposed to only one treatment level. Some writers reserve the designation randomized block design for this latter case. They refer to a design with repeated measurements in which the order of administration of the treatment levels is randomized independently for each participant as a subjects-by-treatments design. A design with repeated measurements in which the order of administration of the treatment levels is the same for all participants is referred to as a subject-by-trials design. I use the designationFigure 1.3 Layout for a t dependent-samples design. Each block contains randomized block design for all three cases.two girls who are overweight by about the same amount. The two girls in a Of the four ways of obtaining dependent samples, the useblock are randomly assigned to the treatment levels. The mean weight loss in of repeated measures on the participants typically results inpounds for the girls in treatment levels a1 and a2 is denoted by Y ·1 and Y ·2 ,respectively. the greatest homogeneity within the blocks. However, if re- peated measures are used, the effects of one treatment level should dissipate before the participant is observed under an-so that prior to going on a diet the participants in each pair are other treatment level. Otherwise the subsequent observationsoverweight by about the same amount. The participants in each will reflect the cumulative effects of the preceding treatmentpair constitute a block or set of matched participants. A simple levels. There is no such restriction, of course, if carryover ef-way to form blocks of matched participants is to rank them fects such as learning or fatigue are the researcher’s principalfrom least to most overweight. The participants ranked 1 and 2 interest. If blocks are composed of identical twins or litter-are assigned to block one, those ranked 3 and 4 are assigned to mates, it is assumed that the performance of participants hav-block two, and so on. In this example, 15 blocks of dependent ing identical or similar heredities will be more homogeneoussamples can be formed from the 30 participants. After all of the than the performance of participants having dissimilar hered-blocks have been formed, the two participants in each block ities. If blocks are composed of participants who are matchedare randomly assigned to the two diets. The layout for this ex- by mutual selection (e.g., husband and wife pairs or businessperiment is shown in Figure 1.3. If the researcher’s hunch is partners), a researcher should ascertain that the participantscorrect that ease in losing weight is related to the amount by in a block are in fact more homogeneous with respect to thewhich a girl is overweight, this design should result in a more dependent variable than are unmatched participants. A hus-powerful test of the null hypothesis, ␮·1 − ␮·2 = 0, than would band and wife often have similar political attitudes; the cou-a t test for independent samples. As we will see, the increased ple is less likely to have similar mechanical aptitudes.power results from isolating the nuisance variable (the amount Suppose that in the weight-loss experiment the researcherby which the girls are overweight) so that it does not appear in wants to evaluate the effectiveness of three diets, denotedthe estimate of the error effects. by a1, a2, and a3. The researcher suspects that ease of losing Earlier we saw that the layout and randomization proce- weight is related to the amount by which a girl is overweight.dures for a t independent-samples design and a completely If a sample of 45 girls is available, the blocking procedurerandomized design are the same except that a completely ran- described in connection with a t dependent-samples designdomized design can have more than two treatment levels. can be used to form 15 blocks of participants. The three par-The same comparison can be drawn between a t dependent- ticipants in a block are matched with respect to the nuisancesamples design and a randomized block design. A random- variable, the amount by which a girl is overweight. The lay-ized block design is denoted by the letters RB-p, where RB out for this experiment is shown in Figure 1.4. A comparisonstands for “randomized block” and p is the number of levels of the layout in this figure with that in Figure 1.3 for a tof the treatment. The four procedures for obtaining depen- dependent-samples design reveals that they are the same ex-dent samples that were described earlier can be used to form cept that the randomized block design has p = 3 treatmentthe blocks in a randomized block design. The procedure that levels. When p = 2, the layouts and randomization plans foris used does not affect the computation of significance tests, the designs are identical. In this and later examples, I assumebut the procedure does affect the interpretation of the results. that all of the treatment levels and blocks of interest are rep-The results of an experiment with repeated measures general- resented in the experiment. In other words, the treatment lev-ize to a population of participants who have been exposed to els and blocks represent fixed effects. A discussion of the caseall of the treatment levels. However, the results of an experi- in which either the treatment levels or blocks or both are ran-ment with matched participants generalize to a population of domly sampled from a population of levels, the mixed and
  26. 26. 8 Experimental Design ␧i j is the residual error effect associated with Yi j and is equal to Yi j − ␮ − ␣ j − ␲i . It reflects all effects not attributable to treatment level aj and Blocki. According to the model equation for this randomized block design, each observation is the sum of four parameters: ␮,␣ j , ␲i , and ␧i j . A residual error effect is that portion of an observation that remains after the grand mean, treatment effect, and block effect have been subtracted from it; that is, ␧i j = Yi j − ␮ − ␣ j − ␲i . The sum of the squared errorFigure 1.4 Layout for a randomized block design (RB-3 design). Each effects for this randomized block design,block contains three girls who are overweight by about the same amount.The three girls in a block are randomly assigned to the treatment levels. The ␧i2j = (Yi j − ␮ − ␣ j − ␲i )2 ,mean weight loss in pounds for the girls in treatment levels a1, a2, and a3 isdenoted by Y ·1 , Y ·2 , and Y ·3 , respectively. The mean weight loss for thegirls in Block1, Block2, . . . , Block15 is denoted by Y 1· , Y 2· , . . . , Y 15· , will be smaller than the sum for the completely randomizedrespectively. design, ␧i( j) = 2 (Yi j − ␮ − ␣ j )2 ,random effects cases, is beyond the scope of this chapter. Thereader is referred to Kirk (1995, pp. 256–257, 265–268). if ␲i2 is not equal to zero for one or more blocks. This idea is A randomized block design enables a researcher to test illustrated in Figure 1.5, where the total sum of squares andtwo null hypotheses. degrees of freedom for the two designs are partitioned. The F H0: ␮·1 = ␮·2 = ␮·3 statistic that is used to test the null hypothesis can be thought (Treatment population means are equal.) of as a ratio of error and treatment effects, H0: ␮1· = ␮2· = · · · = ␮15· f (error effects) + f (treatment effects) F= (Block population means are equal.) f (error effects)The second hypothesis, which is usually of little interest, where f ( ) denotes a function of the effects in parentheses. Itstates that the population weight-loss means for the 15 levels is apparent from an examination of this ratio that the smallerof the nuisance variable are equal. The researcher expects a the sum of the squared error effects, the larger the F statistictest of this null hypothesis to be significant. If the nuisance and, hence, the greater the probability of rejecting a false nullvariable represented by the blocks does not account for an ap-preciable proportion of the total variation in the experiment,little has been gained by isolating the effects of the variable.Before exploring this point, I describe the model equation foran RB-p design. The classical model equation for the weight-loss experi- SSWGment is p(n Ϫ 1) ϭ 42 Yi j = ␮ + ␣ j + ␲i + ␧i j (i = 1, . . . , n; j = 1, . . . , p),where SSRES Yi j is the weight loss for the participant in Block i and (n Ϫ 1)(p Ϫ 1) ϭ 28 treatment level aj. Figure 1.5 Partition of the total sum of squares (SSTOTAL) and degrees of ␮ is the grand mean of the three weight-loss popula- freedom (np − 1 = 44) for CR-3 and RB-3 designs. The treatment and tion means. within-groups sums of squares are denoted by, respectively, SSA and SSWG. The block and residual sums of squares are denoted by, respectively, SSBL ␣j is the treatment effect for population j and is equal to and SSRES. The shaded rectangles indicate the sums of squares that are used ␮· j − ␮. It reflects the effect of diet aj. to compute the error variance for each design: MSWG = SSWG/ p(n − 1) ␲i is the block effect for population i and is equal to and MSRES = SSRES/(n − 1)( p − 1). If the nuisance variable (SSBL) in the randomized block design accounts for an appreciable portion of the total sum ␮i· − ␮. It reflects the effect of the nuisance variable of squares, the design will have a smaller error variance and, hence, greater in Blocki. power than the completely randomized design.
  27. 27. Three Building Block Designs 9hypothesis. Thus, by isolating a nuisance variable that ac-counts for an appreciable portion of the total variation in arandomized block design, a researcher is rewarded with amore powerful test of a false null hypothesis. As we have seen, blocking with respect to the nuisancevariable (the amount by which the girls are overweight)enables the researcher to isolate this variable and remove itfrom the error effects. But what if the nuisance variable Figure 1.6 Three-by-three Latin square, where aj denotes one of thedoesn’t account for any of the variation in the experiment? In j = 1, . . . , p levels of treatment A; bk denotes one of the k = 1, . . . , p levels of nuisance variable B; and cl denotes one of the l = 1, . . . , p levels of nui-other words, what if all of the block effects in the experiment sance variable C. Each level of treatment A appears once in each row andare equal to zero? In this unlikely case, the sum of the squared once in each column as required for a Latin square.error effects for the randomized block and completely ran-domized designs will be equal. In this case, the randomized Latin square: b1 is less than 15 pounds, b2 is 15 to 25 pounds,block design will be less powerful than the completely ran- and b3 is more than 25 pounds. The advantage of being able todomized design because its error variance, the denominator isolate two nuisance variables comes at a price. The ran-of the F statistic, has n − 1 fewer degrees of freedom than domization procedures for a Latin square design are morethe error variance for the completely randomized design. It complex than those for a randomized block design. Also, theshould be obvious that the nuisance variable should be se- number of rows and columns of a Latin square must eachlected with care. The larger the correlation between the nui- equal the number of treatment levels, which is three in the ex-sance variable and the dependent variable, the more likely it ample. This requirement can be very restrictive. For example,is that the block effects will account for an appreciable it was necessary to restrict the continuous variable of theproportion of the total variation in the experiment. amount by which girls are overweight to only three levels. The layout of the LS-3 design is shown in Figure 1.7.Latin Square DesignThe Latin square design described in this section derives itsname from an ancient puzzle that was concerned with thenumber of different ways that Latin letters can be arranged ina square matrix so that each letter appears once in each rowand once in each column. An example of a 3 × 3 Latin squareis shown in Figure 1.6. In this figure I have used the letter awith subscripts in place of Latin letters. The Latin square de-sign is denoted by the letters LS-p, where LS stands for“Latin square” and p is the number of levels of the treatment.A Latin square design enables a researcher to isolate the ef-fects of not one but two nuisance variables. The levels of onenuisance variable are assigned to the rows of the square; thelevels of the other nuisance variable are assigned to thecolumns. The levels of the treatment are assigned to the cellsof the square. Let’s return to the weight-loss experiment. With a Latinsquare design the researcher can isolate the effects of theamount by which girls are overweight and the effects of a sec-ond nuisance variable, for example, genetic predisposition tobe overweight. A rough measure of the second nuisance vari- Figure 1.7 Layout for a Latin square design (LS-3 design) that is based onable can be obtained by asking a girl’s parents whether they the Latin square in Figure 1.6. Treatment A represents three kinds of diets;were overweight as teenagers: c1 denotes neither parent over- nuisance variable B represents amount by which the girls are overweight;weight, c2 denotes one parent overweight, and c3 denotes both and nuisance variable C represents genetic predisposition to be overweight.parents overweight. This nuisance variable can be assigned to The girls in Group1, for example, received diet a1, were less than fifteen pounds overweight (b1), and neither parent had been overweight as athe columns of the Latin square. Three levels of the amount by teenager (c1). The mean weight loss in pounds for the girls in the nine groupswhich girls are overweight can be assigned to the rows of the is denoted by Y ·111 , Y ·123 , . . . , Y ·331 .