• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
American journal of psychotherapy 2013 vol 67 pp 23 -46 (2) by paul clement

American journal of psychotherapy 2013 vol 67 pp 23 -46 (2) by paul clement



Paul Clement's last study reporting outcomes from 45 years as a psychologist.

Paul Clement's last study reporting outcomes from 45 years as a psychologist.



Total Views
Views on SlideShare
Embed Views



1 Embed 54

https://twitter.com 54



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    American journal of psychotherapy 2013 vol 67 pp 23 -46 (2) by paul clement American journal of psychotherapy 2013 vol 67 pp 23 -46 (2) by paul clement Document Transcript

    • Practice-Based Evidence: 45 Years of Psychotherapy’s Effectiveness in a Private Practice PAUL CLEMENT, Ph.D., ABPP* Of 2,259 patients seen during 45 years of private practice, outcome data was produced for 1,599 cases. The mean (SD) number of sessions per case was 18.82 (29.89). The dropout rate was 18.76%. Of all treated cases with outcome data 4 (0.25%) were rated as Much Worse; 11 (0.69%), Worse; 497 (31.08%), No Change from Intake; 546 (34.15%), Improved; and 541 (33.83%), Much Improved. The mean (SD) pre-/post-treatment effect size (ES) was 1.90 (1.61), the median was 1.62, and the range was from Ϫ2.91 to ϩ15.22. Patients and parents of minors rated outcomes more positively than the therapist did. Outcome varied significantly across diagnostic cate- gories. There was a significant, positive relationship in length of treatment and outcome. The therapist’s effectiveness did not improve across the years. Years with the largest patient caseloads or the greatest proportion of patients with managed-care insurance tended to show the poorest outcomes.1 KEYWORDS: practice-based evidence; private practice; treatment effectiveness; treatment effect size; evidence-based practice INTRODUCTION About 60 years ago Eysenck (1952) reviewed the published research on adult psychotherapy and concluded, “The figures fail to support the hypothesis that psychotherapy facilitates recovery from neurotic disorder” (p. 323). Five years later Levitt (1957) reviewed the research on child and adolescent psychotherapy and reached a verdict similar to that of Eysenck: “. . . the results of the present study fail to support the view that psycho- therapy with ‘neurotic’ children is effective” (p. 195). These two articles triggered many criticisms from throughout the world, but these two *Mailing address: 719 Fremont Avenue, South Pasadena, CA 91030. e-mail: PaulWClement@aol.com 1 This paper is an update of “Outcomes from 40 Years of Psychotherapy in a Private Practice,” which was published in the American Journal of Psychotherapy, 62/3 AMERICAN JOURNAL OF PSYCHOTHERAPY, Vol. 67, No. 1, 2013 23
    • psychologists updated their literature reviews and came to the same conclusions made in their first articles (Eysenck, 1961; Levitt, 1963). In spite of such negative reviews of research on psychotherapy, psy- chotherapists continued to practice. On rare occasions they reported their results. For example, Heilbrunn (1966) evaluated her outcomes from 17 years of practicing psychoanalysis and psychoanalytic therapy and pub- lished her results in the American Journal of Psychotherapy. She claimed that 77 of 173 patients (i.e., 45%) improved; however, she excluded more than 80 patients seen for less than 20 sessions. When I read her paper a few months before becoming a licensed psychologist in California, I resolved to do something similar. That is the purpose of the present paper. Throughout the 1950s and 1960s researchers were responding to the challenges posed by Eysenck and Levitt. In 1970 Meltzoff and Kornreich reviewed that research. They concluded that well-designed and controlled research had demonstrated very positive outcomes from psychotherapy. They also reviewed research on characteristics of patients and therapists that contribute to positive treatment outcomes, and on patient-therapist relationship variables that make a difference. Although their review pro- vided encouragement to psychotherapists in all work settings, it did not reveal what kind of outcomes were obtained by therapists in private practice. It did not identify brief outcome measures suitable for repeated administrations to gauge patient change across time. Seven years later Smith and Glass (1977) introduced a quantitative approach for performing literature reviews of controlled treatment-out- come studies on adults. They called it “meta-analysis.” Three years later Smith, Glass, and Miller (1980) expanded and updated the previous review. Although many methodologists criticized their approach, many other psychologists (and researchers from many other disciplines) adopted and adapted meta-analysis for reviewing research findings. Many investi- gators have performed meta-analyses of controlled treatment-outcome research on psychotherapy for children and adolescents. In analyzing the findings from 27 meta-analyses of child, adolescent, and adult psychother- apy research, the mean and standard deviation effect size (ES) was 0.76 (0.24). 95% CI [0.66, 0.86] (Abbass, Kisley, & Kroenke, 2009; Anderson & Lambert, 1995; Bratton, Ray, Rhine, & Jones, 2005; Casey & Berman, 1985; Driessen, Cuijpers, de Matt, Abbass, de Jonghe, & Dekker, 2010; Erion, 2006; Fossum, Handegard, Martinussen, & Morch, 2008; Kazdin, Bass, Ayers, & Rodgers, 1990; Leichsenring, Rabung, & Leiging, 2004; Lewinsohn & Clarke, 1999; Maughan, Christiansen, Jenson, Olympia, & Clark, 2005; McCleod & Weisz, 2004; Messer & Abbass, 2010; Michael & AMERICAN JOURNAL OF PSYCHOTHERAPY 24
    • Crowley, 2002; Oei & Dingle, 2008; Olympia & Clark, 2005; Peleikis & Dahl, 2005; Reinecke, Ryan, & DuBois, 1998a, 1998b; Shadish et al., 1997; Shapiro & Shapiro, 1982; Smith & Glass, 1977; Smith et al., 1980; Smith, Bartz, & Richards, 2007; Stage & Quiroz, 1997; Weisz, McCarty, & Valeri, 2006; Weisz, Weiss, Alicke, & Klotz, 1987; Weisz, Weiss, Han, Granger, & Morton, 1995). I have used meta-analytic techniques as one of my basic ways of expressing magnitude of change within and across the patients of my private practice. My results have consistently surpassed the mean ESs given in the reviews listed above. The most common meta-analyses are reviews of the results of random controlled treatments or trials (RCTs) in which one or more groups of treated cases are compared to one or more control or contrast groups. In contrast, my meta-analyses have compared how a patient was functioning at intake to how much that patient has changed over time. Meta-analyses of such within-cases results have not appeared as frequently as those of RCTs, but they do exist. In analyzing the findings from 18 meta-analyses of within-patients-outcomes (pre-therapy versus post-therapy) research, the mean (SD) ES was 1.35 (0.39), 95% CI [1.16, 1.55] (Burlingame, Fuhriman, & Mosier, 2003; Clement, 2008; de Maat, de Jonghe, Schoevers, & Dekker, 2009; Driessen, Cuijpers de Maat, Abbass, de Jonghe, & Dekker, 2010; Friedman, Cardemil, Uebelacker, Beevers, Chestnut, & Miller, 2005; Hofman, Sawyer, Witt, & Oh, 2010; Huber, Henrich, & Klug, 2005; Kazdin & Whitley, 2006; Leichsenring & Leibing, 2003; Leichsenring, Rabung, & Leibing, 2004; Maughan, Christiansen, Jenson, Olympia, & Clark, 2005; Michael & Crowley, 2002; Minami, Wampold, Serlin, Kircher, & Brown, 2007; Norton & Philipp, 2008; Oei & Dingle, 2008; Stiles, Barkham, Connell, & Mellor-Clark, 2008; Stiles, Barkham, Mellor-Clark, & Connell, 2008; Stiles, Barkham, Twigg, Mellor-Clark, & Cooper, 2006). A one-way ANOVA on the mean ES of the 28 reviews of RCT research listed above and the 18 reviews of within-cases studies produced the following results: F (1, 43) ϭ 39.80, p ϭ 1.299E –007. Published meta- analyses of mean ESs from RCTs have greatly underestimated how much a given patient improves during a course of psychotherapy. Some of the meta-analyses carried out throughout the 1980s and early 1990s identified specific psychological treatments that were effective for treating particular disorders, for example, depression (Dobson, 1989; Steinbrueck, Maxwell, & Howard, 1983), generalized anxiety disorder (Gould, Otto, Pollack, & Yap, 1997), obsessive-compulsive disorder (Cox, Swinson, Morrison, & Lee, 1993; Christensen, Hadzi-Pavlovic, Andrews, Practice-Based Evidence from 45 Years 25
    • & Mattick, 1987). In response to such findings David Barlow, president of the American Psychological Association’s Division of Clinical Psychology, appointed a Task Force on Promotion and Dissemination of Psychological Procedures in 1993 (Sanderson & Woody, 1995a, 1995b). During the ensuing years many journal articles and books identified, described, and provided treatment manuals for such empirically supported treatments. But there were many protests against these lists. For many years at the annual conventions of the American Psychological Association (APA) there were debates about the appropriateness of such lists. Partly in response to these debates APA president Ron Levant created a Presidential Task Force on Evidence-Based Practice in 2005 to investi- gate the issues raised. That task force produced a report that ultimately became a policy statement of the association (APA Presidential Task Force, 2006). The task force concluded, “Evidence-based practice requires that psychologists recognize the strengths and limitations of evidence obtained from different types of research. Research has shown that the treatment method. . . . the individual psychologist. . . . the treatment relationship . . . .and the patient . . . . are all vital contributors to the success of psychological practice” (p. 275). The report called for the collection of effectiveness evidence to complement efficacy results from randomize controlled trials (RCTs). Unfortunately there is very little published effectiveness data gathered from the routine private practice of psychotherapy. References to “usual clinical care” have been misleading (e.g., Weisz, Jensen-Doss, & Hawley, 2006) because they overwhelmingly refer to findings from institutional settings, such as clinics, hospitals, and residential treatment centers. In contrast, according to the APA database on “Employment Characteristics of APA Members,” between 58% to 78% of psychologists who are employed full-time providing mental health services are in independent practice. In addition there are many more psychologists who are salaried by colleges, universities, hospitals, clinics, etc. who maintain part-time private practices. We know almost nothing about their treatment out- comes. The present article provides an exception. Most of the exceptions that do exist involve samples from an individual practice or from a group practice rather than presenting outcomes for all cases seen by an individual therapist or by a group of private practitioners. For example, Persons, Burns, and Perloff (1988) gave their results from treating 70 depressed adult patients in private practice using cognitive therapy. Similarly, Wise (2003) presented treatment outcomes for an intensive outpatient program with 225 patients. AMERICAN JOURNAL OF PSYCHOTHERAPY 26
    • There is another movement that complements empirically supported treatments and evidence-based practice. It is the call for practice-based evidence. Mellor-Clark, Barkham, Connell, and Evans (1999) gave an early example. Their article introduced the Clinical Outcomes in Routine Evaluation (CORE) information management system as well as empha- sized the importance of collecting practice-based evidence to complement that obtained through RCTs and other avenues. More recently Barkham, Hardy, and Mellor-Clark (2010) edited a book calling for and demonstrat- ing practice-based evidence. The present article is an example of practice- based evidence of the effectiveness of psychotherapy within a private practice. METHOD In general, I followed the methods described in several earlier articles (Clement, 1996, 1999, 2011), in which I used a five-level Global Estimate of Outcome (GEO) score as follows: 1 ϳ Much Worse at termination than at intake (i.e., the level of functioning was at least 50% worse than at intake), 2 ϳ Worse at termination than at intake (i.e., the level of functioning was 11% to 49% worse than at intake), 3 ϳ No Change since intake (i.e., the level of functioning at termination was within plus or minus 10% of what it had been at intake), 4 ϳ Better than at intake (i.e., the level of functioning at termination was 11% to 49% better than at intake) , and 5 ϳ Much Better than at intake (i.e., the level of functioning at termination was 50% or better than at intake)(cf., Clement, 1994). When ESs were available I converted to GEO scores as follows: If ES ϭ -1.50 or less, the GEO score was 1; if ES ϭ -0.51 to -1.49, the GEO score was 2; if ES ϭ -0.50 to ϩ0.50, the GEO score was 3; if ES ϭ ϩ0.51 to ϩ1.49, the GEO score was 4; and if ES ϭ ϩ1.50 or greater, the GEO score was 5 (cf., Clement, 1999, 2008, 2011). In the fall of 1988 I began reviewing all closed cases and assigned a GEO score based on all materials within the folder. For some cases, particularly children, observational data facilitated making a quantita- tive judgment. For a majority of cases such observational data did not exist. About 25 years ago I started using problem checklists to evaluate functioning at intake and at subsequent re-evaluations. Each checklist contains over 60 items. These checklists are available in Clement (1999). The patient or the parent of a minor patient rates each problem using a 10-point Scale of Functioning (SOF): 10 ϭ Excellent Functioning, 9 ϭ Good Functioning, 8 ϭ Slight Problem, 7 ϭ Some Problem, 6 ϭ Mod- Practice-Based Evidence from 45 Years 27
    • erate Problem, 5 ϭ Serious Problem, 4 ϭ Major Problem, 3 ϭ Unable to Function, 2 ϭ In Some Danger of Hurting Self or Others, and 1 ϭ In Persistent Danger of Hurting Self or Others. The mean (SD) number of items scored at intake has been 16.72 (11.13), 95% CI [15.58, 17.86], the median has been 14, and the range has been 1–62. I also calculate a Global Assessment of Functioning (GAF) score at intake by determining the mean of the SOF scores, subtracting the standard deviation of the SOF scores for the patient from the mean, and multiplying the result by 10. This approach to determining the GAF score at intake uses the quantitative ratings of the patient as described in the preceding paragraph to determine level of functioning. The more common practice is for the therapist to make an impressionistic estimate of the patient’s level of functioning. Using my method the mean (SD) GAF score at intake across all cases has been 44.42 (12.84), 95% CI [43.73, 45.11], the median has been 45, and the range has been 2 to 85. THERAPIST I am a Caucasian male of Western European descent. I grew up on a small farm north of Seattle. I started my career in 1965 at the Division of Medical Psychology, UCLA Department of Psychiatry. In 1967 I joined the faculty of the Graduate School of Psychology, Fuller Theological Seminary. Given my rural origins, I have always had a bias toward identifying what works to solve a given kind of problem and toward obtaining empirical evidence regarding how much the phenomenon in question has changed. I do not identify with any of the brand-name psychotherapies, for example, psychodynamic, client-centered, behavioral, cognitive, cognitive-behavioral, Gestalt, etc. Part of my approach has been to lean on theoretical models that have been supported by empirical research. In this spirit I spent the first 23 years of my career doing controlled treatment outcome studies of psychotherapy with children. In late 1988 I left academia and shifted from part-time to full-time private practice. I have always been interested in measuring treatment outcomes within my private practice. I have affirmed the empirically oriented movements including empirically supported treatments, evidence-based practice, and now practice-based evidence. PARTICIPANTS From July 1966 through July 2011 I had 2,258 intakes to my private practice. Of these 201 came only for psychological assessment, 386 (18.76%) dropped out without receiving any identified intervention, 40 cases AMERICAN JOURNAL OF PSYCHOTHERAPY 28
    • involved consultation without any intervention, 32 intervention cases had not yet produced outcome data, and 1,599 cases had produced outcome data. DIAGNOSES My career has spanned all editions of the Diagnostic and Statistical Manual (DSM) of the American Psychiatric Association; however, origi- nally I used DSM-III-R (American Psychiatric Association, 1987) to record diagnoses for patients seen during the first 22 years of my practice. Before performing the analyses for the present article I updated all diagnoses using DSM-IV-TR (American Psychiatric Association, 2000). In decreasing order of frequency the most common diagnoses were as follows: dysthymic disorder (n ϭ 468); adjustment disorders (n ϭ 403); attention-deficit/hyperactivity disorder, combined or predominantly hyperac- tive/impulsive (326); oppositional defiant disorder (283); generalized anxiety disorder (239); partner relational problem (220); major depression (165); social phobia (141); attention-deficit/hyperactivity disorder, predominantly inattentive type (121); no diagnosis (101); specific phobia (87); obsessive- compulsive disorder (85); panic disorder with and without agoraphobia (68); separation anxiety disorder (54); acute stress disorder and posttraumatic stress disorder (48); parent-child relational problem (41); sibling relational problem and relational problem NOS (40); bipolar disorders (38); academic problems (36); anxiety disorders NOS (23); body dysmorphic, hypochondriasis, or somatoform disorder (20); sexual disorders (16); Tourette’s disorder (15); schizophrenias and schizoaffective disorder (13); and autistic disorder and pervasive developmental disorder NOS (10). Of all patients seen I had recorded only one Axis I diagnosis for 66% of them, two Axis I diagnoses for 34%, three Axis I diagnoses for 13%, four Axis I diagnoses for 4%, and one Axis II diagnoses for 2.5%. I do not know how these numbers compare with trends in other life-span private practices. SESSIONS PER CASE Of the 1,599 treatment cases with outcome data the mean (SD) number of sessions per case was 18.82 (29.89), 95% CI [17.35, 20.29], the median was 10, and the range was 1-344. Three-quarters of my patients completed treatment within 20 sessions, and approximately 82% finished within 25 sessions; therefore, the majority of my practice has consisted of what the literature identifies as “short-term therapy” (e.g., Watkins, 2012). STORAGE AND RETRIEVAL OF DATA When I opened a new case, I entered essential data about the patient into an electronic database (Microsoft Access). The database includes Practice-Based Evidence from 45 Years 29
    • fields for identifying information on the patient, diagnoses, level of func- tioning at intake, and treatment outcome scores as well as other facts. All patients ever seen by me in my private practice were included. I could retrieve information from the database by running a query on one or more fields, such as age, sex, or diagnosis. Similarly, I could arrange the retrieved information by giving a sort command, for example, to list patients by age from the youngest to the oldest. Before analyzing any quantitative data I pasted the query from Access into Microsoft Excel. Statistical Analyses I copied raw data from Excel and used the StatMost statistical software package (1994) to perform statistical analyses. CANCELLATIONS AND NO-SHOWS Although I did not track cancellations and no-shows during much of my career, I have done so since January 1999. The mean (SD) percentage of cancelled and broken appointments per week has been 15.97 (0.07), 95% CI [15.44, 16.51]. RESULTS WHO MADE REFERRALS TO ME? In descending order of frequency the following have been my sources of referrals: psychologists 23.85%, patients 18.96%, managed care com- panies 18.52%, miscellaneous sources 15.36%, physicians 8.13%, school personnel 4.57%, unknown sources 2.66%, clergy members 2.58%, family members of the patient 1.87%, health care providers other than those mentioned elsewhere in this paragraph 1.87%, attorneys 1.02%, friends of the patient 0.27, social workers 0.27%, and patient him/herself 0.09%. HOW DID OUTCOME VARY BY THERAPIST ESTIMATE AND BY PATIENT (OR PARENT) RATING? For more than half of the 45 years in question, I did not calculate ESs; therefore, the GEO scores from those years were based on my assessment of all information within each patient’s record. In addition, since I started determining ESs many patients have drifted away from therapy without completing a termination interview. Obtaining self-ratings from patients after they stopped coming to sessions has been very difficult, so that there are many cases for which I have had to determine the GEO scores without the benefit of the patient’s self-rating. Only about 35% of treated patient provided self-ratings at termination using the Scale of Functioning and one of my checklists. For these patients I followed the procedures for trans- AMERICAN JOURNAL OF PSYCHOTHERAPY 30
    • forming an ES into a GEO score as described in the first paragraph of the Methods section above. I wondered who would evaluate my outcomes more favorably: Would I or the patients do so? My estimates were the basis of GEO scores for 1,041 cases. The mean (SD) of these ratings was 3.78 (0.80), 95% CI [3.73, 3.83]. The patient’s or parents’ ratings using the SOF and subsequent ESs were the basis of GEO scores for 558 cases. The mean (SD) of these ratings was 4.43 (0.73), 95% CI [4.37, 4.49]. An analysis of variance on these data produced the following results: F (1, 1,597) ϭ 260.14, p ϭ 0.0000. Patients and parents rated the outcomes more positively than I did. HOW MUCH DID PATIENTS IMPROVE? Percent Improved Out of all treated cases with outcome data 4 (0.25%) were Much Worse, 11 (0.69%) were Worse, 497 (31.08%) showed No Change from intake, 546 (34.15%) were Improved, and 541 (33.83%) were Much Improved, for an overall improvement rate of about 68%. Global Estimate of Outcome (GEO) Score The mean GEO (SD) score for 1,599 cases was 4.01 (0.84), 95% CI [3.97, 4.05], the median was 4.00, and the range was 1-5. Effect Size The mean (SD) ES for 558 cases at termination was 1.90 (1.61), 95% CI [1.77, 2.04], the median was 1.62, and the range was from –2.91 to ϩ15.22. Of all ESs 3.14% were negative. HOW DID OUTCOMES VARY BY DIAGNOSIS? Percent Improved Table 1 shows outcome by diagnosis with the best outcomes listed toward the top of the table and the worst outcomes listed toward the bottom. The table only includes diagnoses with at least 10 cases. For this list Chi Square was 99.18 with 23 df, p ϭ 0.0000. Effect Size Table 2 lists mean ES by diagnosis in descending order of effectiveness. The table only includes diagnoses with 10 or more cases. For this list F (1, 464) ϭ 1.38, p ϭ 0.1768. HOW DID OUTCOMES VARY BY PATIENT AGE? Percent Improved In decreasing order of effectiveness the results were as follows: age 0.5 (6 months?)-5 years, 85.33% improved (n ϭ 75); age 6 years-12years, Practice-Based Evidence from 45 Years 31
    • 71.59% improved (n ϭ 352); age 20 years-29 years, 70.11% (n ϭ 174); age 30 years-39 years, 68.04% (n ϭ 291); age 40 years-49 years, 65.61% (n ϭ 253); age 60 years–88years 63.64% (n ϭ 44); age 50 years-59 years 63.49% (n ϭ 126); and age 13 years-19 years, 63.41% (n ϭ 276). The Pearson correlation between the mid point of each age range and the percent improved was r ϭ -0.68, n ϭ 8, p ϭ 0.0609. Effect Size The mean ESs in descending order of magnitude by age group were as follows: age 60 years–88 years, 2.55 (3.54), 95% CI [0.66, 4.44], n ϭ 16; age 0.5–5 years, 2.34 (1.78), 95% CI [1.62, 3.06], n ϭ 26; age 50 years–59 years, 2.09 (1.76), 95% CI [1.59, 2.60], n ϭ 49; age 30 years–39 years, 2.05 (1.54), 95% CI [1.74, 2.35], n ϭ 102; age 20 years–29 years, 1.96 (1.58), Table I. IMPROVEMENT RATES IN DESCENDING ORDER BY DIAGNOSIS Diagnostic Group n % Improved Separation anxiety disorder 20 90.00 Encopresis and enuresis 16 87.50 Panic disorder without agoraphobia 15 86.67 No diagnosis on Axis I or II 28 85.71 Specific phobia 34 82.35 Eating disorders 15 80.00 Sexual disorders 10 80.00 Social phobia 60 76.67 Adjustment disorders 228 75.00 Generalized anxiety disorder 98 71.43 Sibling relational problem 21 71.43 Dysthymic disorder 221 70.14 A-D/HD, combined or hyperactive/impulsive 188 68.09 Oppositional defiant disorder 109 67.89 A-D/HD, predominantly inattentive type 48 64.58 Obsessive compulsive disorder 57 63.16 Anxiety disorder NOS 13 61.54 Parent-child relational problem 18 61.11 Major depressive disorder 99 60.61 Intermittent explosive disorder 12 58.33 Bipolar disorders 24 54.17 Partner relational problem 119 50.42 Panic disorder with agoraphobia 13 46.15 Conduct disorder 30 43.33 AMERICAN JOURNAL OF PSYCHOTHERAPY 32
    • 95% CI [1.56, 2.36], n ϭ 62; age 6 years 12 years, 1.83 (1.46), 95% CI [1.55, 2.10], n ϭ 110; age 40 years–49 years, 1.75 (1.56), 95% CI [1.45, 2.05], n ϭ 105; and age 13 years–19 years,, 1.66 (1.29), 95% CI [1.40, 1.92], n ϭ 99. The Pearson correlation between the mid point of each age range and the mean ES was r ϭ 0.42, n ϭ 8, p ϭ 0.3010. HOW DID OUTCOMES VARY BY SEX OF THE PATIENT? Percent Improved Of female patients 68.76% improved (n ϭ 653). Of all male patients 67.60 % improved (n ϭ 929). Although this is a slight difference, it has held up across the decades. Effect Size For all female patients the mean ES was 1.94 (1.73), 95% CI [1.71, 2.16], n ϭ 225. For all male patients the mean ES was 1.85 (1.15), 95% CI [1.69, 2.01], n ϭ 354. WAS THERE A RELATIONSHIP BETWEEN TREATMENT LENGTH AND OUTCOME? Percent Improved To answer this I ordered all treatment cases from the least number of sessions to the most number of sessions. Then I identified blocks of adjacent sessions. The first six blocks contained 49 cases each. The remaining 26 blocks contained 50 cases each. The first block had only one or two sessions per case with a median of two. The second block had a median of three sessions. The final block had a median of 125 sessions. Table II. MEAN ES IN DESCENDING MAGNITUDE BY DIAGNOSIS Diagnostic Group n ES (SD) 95% CI Major depressive disorder 46 2.44 (2.56) 1.68, 3.20 Specific phobia 10 2.19 (1.38) 1.19, 3.17 Generalized anxiety disorder 39 2.11 (1.97) 1.47, 2.75 Dysthymic disorder 102 2.01 (1.52) 1.71, 2.31 Adjustment disorders 83 1.93 (1.36) 1.64, 2.23 Social phobia 22 1.92 (2.09) 1.00, 2.85 Oppositional defiant disorder 37 1.90 (1.64) 1.35, 2.44 A-D/HD, combined or predominantly hyperactive/impulsive 59 1.81 (1.31) 1.47, 2.16 Bipolar disorders 12 1.70 (1.21) 0.93, 2.47 Partner relational problem 22 1.43 (1.46) 0.78, 2.07 Obsessive compulsive disorder 25 1.35 (1.31) 0.81, 1.89 A-D/HD, predominantly inattentive 20 1.26 (1.06) 0.77, 1.76 Practice-Based Evidence from 45 Years 33
    • Then I calculated the mean % improved within each block. Figure 1 presents the results in graphic form. The Pearson correlation between median sessions per block and percent improved was as follows: r ϭ 0.63, n ϭ 32, p ϭ 9.56E-05. Effect Size There was great variability in the improvement patients achieved with a given amount of therapy as measured by ES. The Pearson correlation of ES versus sessions-per-individual case was as follows: r ϭ 0.00, n ϭ 683. I did a second analysis similar to the one presented in the preceding paragraph. I created blocks of sessions from the fewest sessions to the most. The first eight blocks contained 28 entries. The remaining 17 blocks contained 27. The median number of sessions per block ranged from 3 for the first block to 115 for the last block on the right. Then I determined the mean ES within each of these 25 blocks. The Pearson correlation between median sessions and mean ES was as follows: r ϭ 0.07, n ϭ 25, p ϭ 0.7339. For some individual patients I obtained from two to five ESs across time. For such cases I determined the Spearman rank order correlation by comparing the order the ESs were obtained (i.e., first, second, etc.) with the magnitude of each ES within a patient (i.e., 1st ϭ smallest ES, 2nd ϭ next larger ES, etc.). The results were as follows: rSpearman ϭ 0.63, n ϭ 194, p ϭ 5.42E-23. Figure 1. PERCENT IMPROVED FOR BLOCKS OF SESSIONS ARRANGED FROM FEWEST (ON THE LEFT) TO MOST (ON THE RIGHT) 0.00 20.00 40.00 60.00 80.00 100.00 120.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 Blocks of Sessions (median of 2 sessions in block 1, median of 3 in block 2, to median of 125 in block 32) %ImprovedperBlock AMERICAN JOURNAL OF PSYCHOTHERAPY 34
    • WAS THERE A RELATIONSHIP BETWEEN TREATMENT FORMAT (MODALITY) AND OUTCOME? Percent Improved Figure 2 shows the mean percent improved for each of five treatment formats. A Chi Square performed on the depicted data was 18.24 with 4 df, p ϭ 0.001108. ES. Sufficient data to analyze outcome by treatment format were only available for three formats. The mean ES for individual therapy was 2.00 (1.72), 95% CI [1.82, 2.17], n ϭ 366. The mean ES for family therapy was 1.77 (1.41), 95% CI [1.56, 1.98], n ϭ 172. The mean ES for couples therapy was 1.37 (1.34), 95% CI [0.96, 1.79], n ϭ 42. An analysis of variance on these data produced the following results: F (2, 577) ϭ 3.44, p ϭ 0.0326. HAS MY THERAPEUTIC EFFECTIVENESS CHANGED OVER TIME? Percent improved The Pearson correlation between mean percent improved and year of my career was as follows: r ϭ -0.35, n ϭ 45, p ϭ 0.013. For the first 22 years of my career I maintained a part-time practice. For the more recent 23 years I shifted to full-time private practice. Given this fact I ran a Pearson correlation between the number of new cases opened per year versus the mean percent improved: r ϭ -0.31, n ϭ 45, p ϭ 0.0391: the Figure 2. MEAN OUTCOME BY TREATMENT FORMAT (MODALITY). CONSULTATION N ϭ 18, GROUP N ϭ 8, FAMILY N ϭ 482, NDIVIDUAL N ϭ 934, COUPLE N ϭ 158. 100.00 87.50 69.50 69.06 51.90 0.00 20.00 40.00 60.00 80.00 100.00 120.00 Consultation Group Family Individual Couple Mean%Improved Practice-Based Evidence from 45 Years 35
    • more cases seen in a given year, the poorer the outcome. I also determined the percentage of managed care cases seen each year and calculated a Pearson correlation between percentage of managed care cases and mean percent improved per year: r ϭ -0.32, n ϭ 45, p ϭ 0.0319: the more managed care cases seen in a given year, the poorer the outcome. I computed the Pearson correlation between year of my career and percent of all cases that were children (12 years old and younger): r ϭ -0.31, n ϭ 45, p ϭ 0.0375: as the years went by, a smaller proportion of my annual caseload consisted of children. Finally, I correlated percent improved each year versus percent of cases that were children each year: r ϭ 0.51, n ϭ 45, p ϭ 0.0003: the greater the proportion of my annual caseload consisting of children, the better my outcomes. Effect size I have only had 10 or more ESs per year for the most recent 20 years. I ran a Pearson correlation between the year of my practice and the mean ES within each year. The result was as follows: r ϭ -0.29, n ϭ 20, p ϭ 0.1661. DISCUSSION DISTINCTIVE FEATURES OF THE PRESENT REPORT The present article includes many unique features. It covers the lifespan with the patients ranging in age from 6 months to 88 years at intake. It spans all editions of the DSM through DSM-IV-TR. It includes a wide range of DSM diagnoses and shows differences in outcomes among these diagnoses. It provides quantitative analyses of treatment outcomes. It traverses 45 years of one psychotherapist’s private practice. It incorporates all cases of the practice, not just a sample. It takes into account an extraordinary number of cases. It demonstrates the importance of per- forming pre-/post-treatment analyses. It reveals outcomes from routine clinical practice. THERAPIST VERSUS PATIENT (OR PARENT) OUTCOME RATING According to Minami, Wampold, Serlin, Kircher, and Brown (2007) measures are considered high on reactivity if they are assessed by a clinician and low on reactivity if patients provide self-report data. Mea- sures that focus on symptoms or the targets of treatment are classified as high on specificity but when the measures cover global functioning they are considered low on specificity. These authors (2007) provided pre-/post- treatment ES benchmarks in the treatment of major depression. High- reactivity/high-specificity measures produced the largest ESs. Low-reac- AMERICAN JOURNAL OF PSYCHOTHERAPY 36
    • tivity/high-specificity measures produced intermediate ESs. And low- reactivity/low-specificity measures produced the smallest ESs. When my patients (or their parents) provided pre-treatment and post-treatment ratings on problem checklists, their ratings seemed to match Minami et al.’s definition of low-reactivity/high-specificity measures. When I pro- vided a global estimate of outcome (GEO) by reviewing all evidence within a case folder, these scores seemed to be examples of high-reactivity/low- specificity. The results provided by Minami et al. did not provide a clue as to where high-reactivity/low specificity results would fall in comparison to the three combinations that they did examine. My results were clear: the patients and their parents rated treatment outcomes more positively using low-reactivity/high-specificity measures than I did using high-reactivity/ low-specificity measures. DROPOUTS Patients who drop out of treatment before a mutually planned termi- nation date is a problem both in routine practice and controlled research. According to a recent review of 669 studies covering 83,834 patients, the dropout rate was 19.7%, 95% CI [18.70%, 20.70%] (Swift & Greenberg, 2012). In my private practice of all patients seeking treatment 386 (18.76%) dropped out after one, two, or three sessions without receiving any identified treatment. IMPROVEMENT Overall about 68% of my patients have improved, 31% have not improved, and 1% have gotten worse. These results are strikingly similar to those reported for 3,672 cases of practice-based evidence in the United Kingdom (Barkham et al., 2008). These researchers said that 67.50% had improved, 31.8% showed no reliable change, and 0.7% deteriorated. My mean ES at termination of 1.90 is probably more representative of outcomes in routine private practice than mean ESs reported from RCTs. As mentioned earlier Minami et al. (2007) determined benchmarks for treatment outcomes with adults presenting with major depression. For 846 “completers” plus “early terminators” the mean pre-/post-therapy ES using the Beck Depression Inventory (a low-reactivity/high-specificity measure) was 1.71. For 1,387 “completers” the mean ES was 1.86. This latter mean is very similar to my mean ES of 1.90 across all of my patients. My mean ES for major depression was 2.44. There are other findings from routine clinical practice to compare with my results. For example, Barkham, Mellor-Clark, Connell, Evans, Evans, Practice-Based Evidence from 45 Years 37
    • and Margison (2010) reported pre-/post-therapy ESs for 9,337 patients treated in various service settings within the UK. The mean of the five means that they published was 1.34. The grand mean falls between the mean ES of 1.24 for children with oppositional defiant disorder and of 1.47 for those with conduct disorder published by Kazdin and Whitley (2006) in their research program. These latter two means were also based on pre-/post-therapy comparisons. As indicated in the Discussion section of this article, such pre-/post-therapy ESs are consistently greater than between-groups ESs normally reported in meta-analyses of controlled treatment outcome research (e.g., Casey & Berman, 1985; Smith & Glass, 1977; Smith et al., 1980; Weisz et al., 1995). Hollon (1996) asserted, “[T]he available evidence appears to differ between the child and adult literatures, but in neither does [sic] outcomes observed in applied settings exceed those observed in controlled clinical trials” (p. 1028). In fact, the present mean ES clearly exceeds the mean ESs reported in published meta-analyses of RCTs on child, adolescent, and adult psychotherapies. Hollon did say, “I agree with Seligman that the best way to determine what goes on in actual clinical practice is to study it directly” (p. 1025), and Kazdin and Weisz (1998) declared, “[T]he mag- nitude of therapeutic change is an issue in need of much greater attention” (p. 30). The ES results reviewed above focused on magnitude of thera- peutic change. The typical RCT is not designed to determine magnitude of therapeutic change. Usually its purpose is to demonstrate a causal relationship between a particular treatment and its effect in comparison to a control or contrast condition (cf., Seligman, 1995). The resultant ESs show differences be- tween the treated and the control/contrast group. Most investigators employing RCT designs do not report pre-/post-treament ESs for their cases (Kazdin, Bass, Ayers, & Rodgers, 1990). Asay and Lambert (1999) warned, “Client outcome is principally determined by client variables and extratherapeutic factors rather than by the therapist or therapy” (p. 43); hence, the positive results presented above can be celebrated without claiming that I or the particular treat- ments that I used were primarily responsible for the measured gains. These same authors estimated that 15% of improvement comes from expectancy (placebo effects), 15% from techniques, 30% from the therapeutic rela- tionship, and 40% from extratherapeutic change. There is evidence that placebo effects may even be greater than the estimate of Asay and Lambert. For example, Glass and Kliegl (1983) gave an average ES of 0.56 from “placebo treatment.” Dush, Hirt, and Schroeder (1989) claimed, AMERICAN JOURNAL OF PSYCHOTHERAPY 38
    • “Collective results from other meta-analyses suggest an average placebo ES of about a third to a half of a standard deviation, in comparison with the ES of no treatment” (p. 100). Of all ESs obtained in my results, 3.14% were negative. In comparison, Dush et al. (1989) reported that 15% of the ESs in their meta-analysis of self-statement modification in children were negative. Smith and Glass (1977) said that 12% of the 833 ES measures that they calculated were negative. OUTCOME BY DIAGNOSIS The percent of my patients who showed improvement varied substan- tially across diagnoses. Although this finding is consistent with findings from some controlled research (e.g., Lambert & Archer, 2006), researchers tend to focus on one diagnostic group at a time. And therapists in private practice rarely publish their treatment outcomes; therefore, the present report, covering 45 years of one therapist’s private practice and all patients seen, is unique. Although there was quite a large range in my mean ESs across diagnoses, the large variance in ESs within diagnoses blurred the apparent differences. Nevertheless, these findings are distinctive in the published literature on outcomes of psychotherapy. OUTCOME BY PATIENT AGE There was no clear or consistent relationship between age groups of patients and treatment outcome. Regarding percent improved there was a negative but not quite statistically significant correlation, indicating that my younger patients improved more than my older ones. However, for mean ES, there was a positive but not statistically significant correlation. Looking at just the three youngest age groups (age 0.5 years–5 years, 6 years–12 years, and 13 years–19 years) there was consistency in the trends of the two measures of outcome: percent improved and ES. Preschoolers faired best. Children came in second. And teenagers had the poorest outcome. This trend has held up across my career. OUTCOME BY SEX OF PATIENT The slight but statistically nonsignificant advantage in outcome for females in my practice is consistent with previous research. For example, Casey and Berman (1985) found that the greater the proportion of males in a study, the smaller the ESs. And in their review of psychotherapy with children and adolescents, Weisz, Weiss, Han, Granger, and Morton (1995) reported that females faired better than males. Similarly, Smith et al. (1980) determined a small, negative correlation between the percent of Practice-Based Evidence from 45 Years 39
    • males in a study and ES. Thirty years later Parry, Castonguay, Borkovec, and Wolf (2010) reported that of 220 clients treated by 57 clinicians in the community, female clients showed more improvement than male clients. OUTCOME BY DURATION OF TREATMENT With the exception of patients who improved with just one or two sessions, the greater the number of treatments, the greater the percentage of patients who improved. This finding is consistent with that of Messer and Boals (1981). Clearly the dose-effect phenomenon is not always found. For example, in their meta-analysis of outcomes of psychotherapy with children Casey and Berman (1985) concluded that length of treatment was negatively related to mean ES: r ϭ -0.28 (p ϭ 0.02); whereas, Smith and Glass (1977) and Smith et al. (1980) in their meta-analyses of adult psychotherapy outcomes found no significant relationship between the duration of treatment and outcome. In discussing such findings Howard, Kopta, Krause, and Orlinsky (1986) warned, “. . .a between-study analysis . . . has no necessary implication for the relationship between duration and benefit within each study” (p. 159). Their article provided a quantitative model estimating the percentage of patients improved for specific amounts of psychotherapy. I calculated the Pearson correlation between their model and my outcome data at 4, 8, 13, 26, 52, and 104 sessions: r ϭ 0.92, n ϭ 6, p ϭ 0.0093. These findings are consistent with those of the Consumer Reports study (Seligman, 1995). Although there is no correlation across all cases between number of treatment sessions and mean ES, when multiple ESs are obtained on the same patient across time, there is a strong relationship between amount of treatment and outcome. This finding is also consistent with Howard’s dose-response model. OUTCOME BY TREATMENT FORMAT There is a significant difference in my outcomes based on treatment format (modality). Unfortunately I have not found an article by any other therapist or researcher that compared the five modalities compared in Figure 2. I am unsure whether the trends in my results would generalize to those of other therapists. Young (2007) retrospectively evaluated her outcomes from 55 years of practice as a psychoanalyst and psychoanalytic psychotherapist. During that period of time she had treated just 231 people. She said that she provided “classical analysis” to 22 patients (p. 316), “modified analysis” to 19 seen once or twice a week (p. 316), “analytically oriented psychotherapy” to 26 (p. 316), “insight-oriented supportive therapy” to 65 (p. 316), and “supportive therapy” to 47 (p. 316). She reported that her success rates ranged from 40.4% improved AMERICAN JOURNAL OF PSYCHOTHERAPY 40
    • with supportive non-insight oriented treatment to 94.5% improved with modified analysis. THERAPIST’S EFFECTIVENESS ACROSS 45 YEARS Measured by percent improved per year, not only have I failed to improve across the years, my outcomes have gotten worse across time. This trend was puzzling and troubling. Switching from part-time to full-time practice corresponded to a drop in effectiveness. This finding was similar to that reported by Borkovec, Echemendia, Ragusea, and Ruiz (2001); increases in therapists’ caseloads corresponded to lessened treatment outcomes. Parry et al. (2010) reported the same trend for 57 therapists in Pennsylvania. Similarly, increases in the proportion of my case load per year that had been referred by a managed care company corresponded with poorer outcomes. During the first 23 years of my career I was a full-time academic. My research, writing, and teaching during those years focused on children. Unhappily there was a significant drop in the percent of my caseload consisting of children across the 45 years in question. This was too bad, because the greater the proportion of my caseload consisting of children in a given year, the greater the % of my patients that had improved. The ES data did not show a significant change across time. Smith et al. (1980) concluded, “. . . there was no relationship between the years of experience of the therapists in a study and the magnitude of therapeutic effect produced in that study (r ϭ 0.00)” (p. 117); however, the average amount of experience of therapists in the studies reviewed was only three and a quarter years. Dawes (2008) claimed, however, that over 500 studies of psychotherapy outcome had shown no relationship between amount of experience in the psychotherapists and patient outcomes. One of my motives in starting to evaluate my treatment outcomes 24 years ago was to answer the question: How good am I as a psychothera- pist? Although how many years of experience a therapist has had does not predict treatment outcomes, there is growing evidence that therapists vary greatly in their effectiveness. In their large-scale study in the UK Stiles and Barkham (2012) found that improvement rates for individual therapists ranged from 23.5% for the least effective therapist to 95.6% for the most effective therapist. Seidel (2012) reported very similar ranges of effective- ness in 268 therapists who had each treated at least 30 patients in the US. I suspect that most therapists would be interested in discovering where they fall in such distributions from least effective to most effective but simultaneously afraid to find out. Practice-Based Evidence from 45 Years 41
    • CLOSING COMMENT Kazdin (2008) warned, “We do not benefit as a field from the accu- mulated practice of clinicians. . . .” (p. 157). I hope that the results I have provided in the present report will encourage my psychotherapy colleagues to evaluate their outcomes and to publish their results. In closing, I want to say, “I have shown you my practice-based evidence. Now you show me yours!” REFERENCES Abbass, A., Kisely, S., & Kroenke, K. (2009). Short-term psychodynamic psychotherapy for somatic disorders: Systematic review of meta-analysis of clinical trials. Psychotherapy and Psychosomat- ics, 78, 265-274. doi:10.1159/000228247 American Psychiatric Association. (1987). Diagnostic and statistical manual of mental disorders (3rd ed., rev.). Washington, D.C.: Author. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text revision). Washington, D.C.: Author. Anderson, E.M. & Lambert, M.J. (1995). Short-term dynamically oriented psychotherapy: A review and meta-analysis. Clinical Psychology Review, 15, 503-514. doi:10.1016/0272- 7358(95)00027-M APA Presidential Task Force. (2006). Evidence-based practice in psychology. American Psychologist, 61, 271-285. doi:10.1037/0003-066X.61.4.271 Asay, T.P., & Lambert, M.J. (1999). The empirical case for the common factors in therapy: Quantitative findings. In Hubble, M.A., Duncan, B.L., & Miller, S.D. (Eds.) (1999). The heart and soul of change: What works in therapy (pp. 23-55). Washington, D.C.: American Psycho- logical Association. doi:10.1037/11132-001 Barkham, M., Hardy, G.E., & Mellor-Clark, J. (Eds.). (2010). Developing and delivering practice-based evidence: A guide for the psychological therapies. West Sussex, UK: Wiley-Blackwell. doi: 10.1002/9780470687994 Barkham, M., Mellor-Clark, J., Connell, J., Evans, C., Evans, R., & Margison, F. (2010). Clinical Outcomes in Routine Evaluation (CORE)—the CORE measures and system: Measuring, monitoring and managing quality evaluation in the psychological therapies. In M. Barkham, G.E. Hardy, & J. Mellor-Clark (Eds.). Developing and delivering practice-based evidence: A guide for the psychological therapies. West Sussex, UK: Wiley-Blackwell. doi:10.1002/ 9780470687994.ch8 Barkham, M., Stiles, W.B., Connell, J., Twigg, E., Leach, C., Lucock, M., Angus, L. (2008). Effects of psychological therapies in randomized trials and practice-based studies. British Journal of Clinical Psychology, 47, 397-415. doi:10.1348/01446650X311713 Borcovec, T.D., Echemedia, R.J., Ragusea, S.A., & Ruiz, M. (2001). The Pennsylvania Practice Research Network and future possibilities for clinically meaningful and scientifically rigorous psychotherapy effectiveness research. Clinical Psychology: Science and Practice, 8, 155-167. doi: 10.1093/clipsy/8.2.155 Bratton, S.C., Ray, D., Rhine, T., & Jones, L. (2005). The efficacy of play therapy with children: A meta-analytic review of treatment outcomes. Professional Psychology: Research and Practice, 36, 376-390. doi: 10.1037/0735-7028.36.4.376 Burlingame, G.M., Fuhriman, A., & Mosier, J. (2003). The differential effectiveness of group psychotherapy: A meta-analytic perspective. Group Dynamics: Theory, Research, and Practice, 7, 3-12. doi: 10.1037/1089-2699.7.1.3 Casey, R.J., & Berman, J.S. (1985). The outcome of psychotherapy with children. Psychological Bulletin, 98, 388-400. doi:10.1037/0033-2909.98.2.388 Christensen, H., Hadzi-Pavlovic, D., Andrews, G., & Mattick, R. (1987). Behavior therapy and tricyclic medication in the treatment of obsessive-compulsive disorder: A quantitative review. Journal of Consulting and Clinical Psychology, 55, 701-711. doi: 10.1037/0022-006X.55.5.701 AMERICAN JOURNAL OF PSYCHOTHERAPY 42
    • Clement, P.W. (1994). Quantitative evaluation of 26 years of private practice. Professional Psychology: Research and Practice, 36, 105-113. doi: 10.1037/0735-7028.25.2.173 Clement, P.W. (1996). Evaluation in private practice. Clinical Psychology: Science and Practice, 3, 146-159. doi:10/1111/j.1468-2850.1996.tb00064.x Clement, P.W. (1999). Outcomes and incomes: How to evaluate, improve, and market your psychother- apy practice by measuring outcomes. New York: Guilford. Clement, P.W. (2008). Outcomes from 40 years of psychotherapy in a private practice. American Journal of Psychotherapy, 62, 215-239. Clement, P.W. (2011). Research in private practice: How to determine your effectiveness as a therapist. In J.C. Thomas & M. Hersen (Eds.). Understanding research in clinical and counseling psychology (2nd ed.)(pp. 375-398). New York: Taylor and Francis. Cox, B.J., Swinson, R.P., Morrison, B., & Lee, P.S. (1993). Clomipramine, fluoxetine, and behavior therapy in the treatment of obsessive-compulsive disorder: A meta-analysis. Journal of Behavior Therapy and Experimental Psychiatry, 24, 149-153. doi: 10.1016/0005-7916(93)90043-V Dawes, R.M. (2008). Psychotherapy: The myth of expertise. In S.O. Lilienfeld, J. Ruscio, & S.J. Lynn (Eds.). Navigating the mindfield: A user’s guide to distinguishing science from pseudoscience in mental health (pp. 311-344). Amherst, NY: Prometheus Books. de Maat, S., de Jonghe, F., Schoevers, R., & Dekker, J. (2009). The effectiveness of long-term psychoanalytic therapy: A systematic review of empirical studies. Harvard Review of Psychiatry, 17, 1-23. doi: 10.1080/10673220902742476 Dobson, K.S. (1989). A meta-analysis of the efficacy of cognitive therapy for depression. Journal of Consulting and Clinical Psychology, 57, 414-419. doi: 10.1037/0022-006X.57.3.414 Driessen, E., Cuijpers, P., de Maat, S.C.M., Abbass, A.A., de Jonghe, F., & Dekker, J.J.M. (2010). The efficacy of short-term psychodynamic psychotherapy for depression: A meta-analysis. Clinical Psychology Review, 30, 25-36. doi: 10.1016/j.cpr.2009.08.010 Dush, D.M., Hirt, M.L., & Schroeder, H.E. (1989). Self-statement modification in the treatment of child behavior disorders: A meta-analysis. Psychological Bulletin, 106, 97-106. doi:10.1037/ 0033-2909.106.1.97 Erion, J. (2006). Parent tutoring: A meta-analysis. Education & Treatment of Children, 29, 79-106. Eysenck, H.J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319-324. doi:10.1037/h0063633 Eysenck, H.J. (1961). The effects of psychotherapy. In H.J. Eysenck (Ed.), Handbook of abnormal psychology (pp. 697-725). New York: Basic Books. Fossum, S., Handegard, B.H., Martinussen, M., Morch, W.T. (2008). Psychosocial interventions for disruptive and aggressive behavior in children and adolescents: A meta-analysis. European Child & Adolescent Psychiatry, 17, 438-451. doi: 10.1007/s00787-008-0686-8 Friedman, M.A., Cardemil, E.V., Uebelacker, L.A., Beevers, C.G., Chestnut, C., & Miller, I.W. (2005). The GIFT program for major depression: Integrating group, individual, and family treatment. Journal of Psychotherapy Integration, 15, 147-168. doi: 10.1037/1053-0479.15.2.147 Glass, G.V., & Kliegl, R.M. (1983). An apology for research integration in the study of psychotherapy. Journal of Consulting and Clinical Psychology, 51, 28-41. doi:10.1037/0022-006X.51.1.28 Gould, R.A., Otto, M.W., Pollack, M.H., & Yap, L. (1997). Cognitive behavioral and pharmacological treatment of generalized anxiety disorder: A preliminary meta-analysis. Behavior Therapy, 28, 285-305. doi: 10.1016/S0005-7894(97)80048-2 Heilbrunn, G. (1966). Results with psychoanalytic therapy and professional commitment. American Journal of Psychotherapy, 20, 89-99. Hofmann, S.G., Sawyer, A.T., Witt, A.A., & Oh, D. (2010). The effect of mindfulness-based therapy on anxiety and depression: A meta-analytic review. Journal of Consulting and Clinical Psychol- ogy, 78, 169-183. doi: 10.1037/a0018555 Hollon, S.D. (1996). The efficacy and effectiveness of psychotherapy relative to medications. American Psychologist, 51, 1025-1030. doi:10.1037/0003-066X.51.10.1025 Howard, K.I., Kopta, S.M., Krause, M.S., & Orlinsky, D.E. (1986). The dose-effect relationship in psychotherapy. American Psychologist, 41, 159-164. doi:10.1037/0003-066X.41.2.159 Huber, D., Henrich, G., & Klug, G. (2005). The scales of psychological capacities: Measuring change in psychic structure. Psychotherapy Research, 15, 445-456. doi: 10.1080/10503300500091298 Kazdin, A.E. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical Practice-Based Evidence from 45 Years 43
    • research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 63, 146-159. doi:10.1037/0003-066X.63.3.146 Kazdin, A.E., Bass, D., Ayers, W.A., & Rodgers, A. (1990). Empirical and clinical focus of child and adolescent psychotherapy research. Journal of Consulting and Clinical Psychology, 58, 729-740. doi:10.1037/0022-006X.58.6.729 Kazdin, A.E., & Whiteley, M.K. (2006). Comorbidity, case complexity, and effects of evidence-based treatment for children referred for disruptive behavior. Journal of Consulting and Clinical Psychology, 74, 455-467. doi:10.1037/0022-006X.74.3.455 Kazdin, A.E., & Wiesz, J.R. (1998). Identifying and developing empirically supported child and adolescent treatments. Journal of Consulting and Clinical Psychology, 66, 19-36. doi:10.1037/ 0022-006X.66.1.19 Lambert, M.J., & Archer, A. (2006). Research findings on the effects of psychotherapy and their implications for practice. In C.D. Goodheart, A.E. Kazdin, & R.J. Sternberg (Eds.). Evidence- based psychotherapy: Where practice and research meet (pp. 111-130). Washington, DC: American Psychological Association. doi:10.1037/11423-005 Leichsenring, F. & Leibing, E. (2003). The effectiveness of psychodynamic therapy and cognitive behavior therapy in the treatment of personality disorders: A meta-analysis. American Journal of Psychiatry, 160, 1223-1232. doi: 10.1176/appi.ajp.160.7.1223 Leichsenring, F., Rabung, S., & Leibing, E. (2004). The efficacy of short-term psychodynamic psychotherapy in specific psychiatric disorders: A meta-analysis. Archives of General Psychiatry, 61, 1208-1261. doi: 10.1001/archpsyc.61.12.1208 Levitt, E.E. (1957). The results of psychotherapy with children: An evaluation. Journal of Consulting Psychology, 21, 189-196. doi:10.1037/h0039957 Levitt, E.E. (1963). Psychotherapy with children: A further evaluation. Behaviour Research and Therapy, 1, 45-51. doi:10.1016/0005-7967(63)90007-X Lewinsohn, P.M. & Clarke, G.N. (1999). Psychosocial treatments for adolescent depression. Clinical Psychology Review, 19, 329-342. doi: 10.1016/S0272-7358(98)00055-5 Maughan, D.R., Christainsen, E., Jenson, W.R., Olympia, D., & Clark, E. (2005). Behavioral parent training as a treatment for externalizing behaviors and disruptive behavior disorders: A meta-analysis. School Psychology Review, 34, 267-286. McCleod, B.D. & Weisz, J.R. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72, 235-251. doi: 10.1037/0022-0006X.72.2.235 Mellor-Clark, J., Barkham, M., Connell, J., & Evans, C. (1999). Practice-based evidence and stan- dardized evaluation: Informing the design of the CORE system. European Journal of Psycho- therapy, Counselling and Health, 2, 357-374. doi:10.1080/13642539908400818 Meltzoff, J., & Kornreich, M. (1970). Research in psychotherapy. New York: Atherton Press. Messer, S.B. & Abbass, A.A. (2010). Evidence-based psychodynamic therapy with personality disor- ders. In Magnavita, J.J. (Ed.) Evidence-based treatment of personality dysfunction: Principles, methods, and processes (pp. 79-111). Washington, DC: American Psychological Association. doi: 10.1037/12130-004 Messer, S.B., & Boals, G.F. (1981). Psychotherapy outcome in a university-based psychology training clinic. Professional Psychology, 12, 785-793. doi:10.1037/0735-7028.12.6.785 Michael, KD. & Crowley, S.L. (2002). How effective are treatments for child and adolescent depression? A meta-analytic review. Clinical Psychology Review, 22, 247-269. doi: 10.1016/ S0272-7358(01)00089-7 Minami, T., Wampold, B.E., Serlin, R.C., Kircher, J.C., & Brown, G.S. (2007). Benchmarks for psychotherapy efficacy in adult major depression. Journal of Consulting and Clinical Psychology, 75, 232-243. doi:10.1037/0022-006X.75.2.232 Norton, P.J. & Philipp, L.M. (2008). Transdiagnostic approaches to the treatment of anxiety disorders: A quantitative review. Psychotherapy: Theory, Research, Practice, Training, 45, 214-226. Oei, T.P.S. & Dingle, G. (2008). The effectiveness of group cognitive therapy for unipolar depressive disorders. Journal of Affective Disorders, 107, 5-21. doi: 10.1016/j.jad.2007.07.018 Parry, G., Castonguay, L.G., Borkovec, T.D., & Wolf, A.W. (2010). Practice research networks and psychological services research in the UK and USA. In M. Barkham, G.E. Hardy, & J. Mellor-Clark (Eds.). Devleoping and delivering practice-based evidence: A guide for the psycho- AMERICAN JOURNAL OF PSYCHOTHERAPY 44
    • logical therapies (pp. 311-325). West Sussex, UK: Wiley-Blackwell. doi:10.1002/ 9780470687994.ch12 Peleikis, D.E. & Dahl, A.A. (2005). Psychotherapy Research, 15, 304-315. doi: 10.1080/ 10503300500091835 Persons, J.B., Burns, D.D., & Perloff, J.M. (1988). Predictors of dropout and outcome in cognitive therapy in a private practice setting. Cognitive Therapy and Research, 12, 557-575. doi:10.1007/ BF01205010 Reinecke, M.A., Ryan, N.E., & DuBois, D.L. (1998a). Cognitive-behavioral therapy of depression and depressive symptoms during adolescence: A review and meta-analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 37, 26-34. doi: 10.1097/00004583-199801000-00012 Reinecke, M.A., Ryan, N.E., & DuBois, D.L. (1998b). “Meta-analysis of CBT for depression in adolescents”: Dr. Reinecke et al reply. Journal of the American Academy of Child & Adolescent Psychiatry, 37, 1006-1007. Sanderson, W.C., & Woody, S. (Eds.)(1995a). Manuals for empirically validated treatments. The Clinical Psychologist, 48(4), 7-11. Sanderson, W.C., & Woody, S. (Eds.)(1995b). Manuals for empirically validated treatments: A project of the Task Force on Psychological Interventions, Division of Clinical Psychology, American Psychological Association, 28 pp. http://psycnet.apa.org/psycextra/550792009-001.pdf Seidel, J.A. (2012, August). Feedback-informed treatment: The devil is in the details. In C.D. Goodheart (Chair), Practice-based evidence of psychotherapy’s effectiveness. Symposium con- ducted at the meeting of the American Psychological Association, Orlando, FL. Seligman, M.E.P. (1995). The effectiveness of psychotherapy: The Consumer Reports study. American Psychologist, 50, 965-974. doi:10.1037/0003-066X.50.12.965 Shadish, W.R., Matt, G.E., Navarro, A.N., Siegle, G., Crits-Christoph, P., Hazelrigg, M.D.,. . .Weiss, B. (1997). Evidence that therapy works in clinically representative conditions. Journal of Consulting and Clinical Psychology, 65, 355-365. doi: 10.1037/0022-006X.65.3.355 Shapiro, D.A. & Shapiro, D. (1982). Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychological Bulletin, 92, 581-607. doi: 10.1037/0033- 2909.92.3.581 Smith, T.B., Bartz, J., & Richards, P.S. (2007). Outcomes of religious and spiritual adaptations to psychotherapy: A meta-analytic review. Psychotherapy Research, 17, 643-655. doi: 10.1080/ 10503300701250347 Smith, M.L., & Glass, G.V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752-760. doi:10.1037/0003-066X.32.9.752 Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy. Baltimore: John Hopkins University Press. Stage, S.A. & Quiroz, D.R. (1997). A meta-analysis of interventions to decrease disruptive classroom behavior in education settings. School Psychology Review, 26, 333-368. StatMost statistical analysis and graphics: User’s handbook. (1994). Salt Lake City: DataMost Corpo- ration. Steinbrueck, S.M., Maxwell, S.E., & Howard, G.S. (1983). A meta-analysis of psychotherapy and drug therapy in the treatment of unipolar depression with adults. Journal of Consulting and Clinical Psychology, 51, 856-863. doi: 10.1037/0022-006X.51.6.856 Stiles, W.B., & Barkham, M. (2012, August). Practice-based evidence from the United Kingdom using the CORE system. In C.D. Goodheart (Chair), Practice-based evidence of psychotherapy’s effectiveness. Symposium conducted at the meeting of the American Psychological Association, Orlando, FL. Stiles, W.B., Barkham, M., Connell, J., & Mellor-Clark, J. (2008). Responsive regulation of treatment duration in routine practice in United Kingdom primary care settings: Replication in a larger sample. Journal of Consulting and Clinical Psychology, 76, 298-305. doi: 10.1037/0022- 006X.76.2.298 Stiles, W.B., Barkham, M., Mellor-Clark, J., & Connell, J. (2008). Effectiveness of cognitive-behav- ioural, person-centered, and psychodynamic therapies in UK primary-care routine practice: Replication in a larger sample. Psychological Medicine, 38, 677-688. doi: 10.1017/ S0033291707001511 Stiles, W.B., Barkham, M., Twigg, E., Mellor-Clark, J., & Cooper, M. (2006). Effectiveness of cognitive-behavioural, person-centered and psychodynamic therapies as practiced in UK Practice-Based Evidence from 45 Years 45
    • National Health Service settings. Psychological Medicine, 36, 555-566. doi: 10.1017/ S0033291706007136 Swift, J.K. & Greenberg, R.P. (2012). Premature discontinuation in adult psychotherapy: A meta- analysis. Journal of Consulting and Clinical Psychology, 80, 547-559. doi: 10.1037/a0028226 Watkins, C.E. Jr. (2012). The study of men in short-term and long-term psychodynamic psychother- apy: A brief research note. Psychoanalytic Psychology, 29, 429-439.doi: 10.1037/a0025183 Weisz, J.R., Jensen-Doss, A., & Hawley, K.M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist, 61, 671-689. doi:10.1037/0003-066X.61.7.671 Weisz, J.R., McCarty, C.A., & Valeri, S.M. (2006). Effects of psychotherapy for depression in children and adolescents: A meta-analysis. Psychological Bulletin, 132, 132-149. doi: 10.1037/0033- 2909.132.1.132 Weisz, J.R., Weiss, B., Alicke, M.D., & Klotz, M.L. (1987). Effectiveness of psychotherapy with children and adolescents: A meta-analysis for clinicians. Journal of Consulting and Clinical Psychology, 55, 542-549. doi: 10.1037/0022-006X.55.4.542 Weisz, J.R., Weiss, B., Han, S.S., Granger, D.A., & Morton, T. (1995). Effects of psychotherapy with children and adolescents revisited: A meta-analysis of treatment outcome studies. Psychological Bulletin, 117, 450-468. doi:10.1037/0033-2909.117.3.450 Wise, E.A. (2003). Psychotherapy outcome and satisfaction methods applied to intensive outpatient programming in a private practice setting. Psychotherapy: Theory, Research, Practice, Training, 40, 203-214. doi:10.1037/0033-3204.40.3.203 Young, B. (2007). The efficacy of psychoanalysis and the analytic therapies: Reflections of a psycho- analyst and her former patients. Journal of the American Academy of Psychoanalysis & Dynamic Psychiatry, 35, 313-334. doi:10.1521/jaap.2007.35.2.313b AMERICAN JOURNAL OF PSYCHOTHERAPY 46