The Consumer Reports study of psychotherapy: Invalid is invalid.

Abstract

The conclusion that Consumer Reports (CR) has provided empirical validation of the e f f e t i v e n e s s of p s y h o t h e r a p y (Seligman, 1995, p. 974) was reached after a bravura run across a minefield of methodological flaws, In Seligman's reanalysis of the CR (1994) survey findings, many threats to internal validity (Campbell & Stanley, 1969; Cook & Campbell, 1979) were found to apply to the CR data, but they were swiftly argued away. However, the arguing was not convincing, and the linchpin argument simply begged the question. We do not believe Seligman's dash scored the intended touchdown: No new conclusion about the effectiveness of psychotherapy was warranted. Seligman's (1995) collaboration with CR had the worthy aim of learning about the effectiveness of popular forms of psychotherapy--long-term dynamic, family, and eclectic (p. 966). These widely used forms, however, have eluded rigorous evaluation. In contrast, other forms of therapy (e.g., cognitive, interpersonal, and medication) have been assessed in efficacy studies (e.g., Frank & Spanier, 1995), many of which were well designed to minimize threats to validity. Thus, the Seligman reanalysis of the CR (1994) survey pertained exclusively to forms of therapy for which there was not already convincing evidence of effectiveness. In no sense, then, was the CR survey evidence characterizable as cumulative or corroborative with other evidence about psychotherapy effectiveness. Because it is enormously important to know more about the effectiveness of (Seligman, p. 966) therapies that are often sought out and experienced by clients, the rigor of the methodology should be commensurately stringent. Methodological criticisms do not mean that psychotherapy does not work; rather, they are offered in the conviction that the practice of psychology is too important to be based on unproven ideas (Dawes, 1994). One fatal flaw in the CR (1995) study was that the bulk of the mental health data came from only 4% of the originally contacted sample. It is easy to imagine that among the 96% nonrepliers to the mental health ques t ions--171,000 pe r sons there were many thousands who had tried some kinds of therapy with nil or unfavorable results and who also did not complete a lengthy questionnaire and mail it back to CR. Whether this was the case or not, the mountain of 171,000 nonrepliers appears to mandate utmost caution in interpreting the data from the foothill, the 7,000 who chose to reply regarding mental health issues, or the molehill, the subset of 2,900 (1.6% of the original survey sample!) who reported having seen some kind of mental health p r o v i d e r (of whom only 37% were psychologists). Seligman (1995) claimed that because all CR surveys consis t of self-report , nonblind procedures, it is far-fetched (p. 972) to believe that such procedures in the psychotherapy survey would be problematic. This assumes that the CR (1994) psychotherapy survey was psychologically akin to all other CR surveys, whereas, in fact, it was fundamentally different. While it is nonthreatening to state that a blender is lousy, it is somewhat aversive to indicate that one's current state of mental health is lousy, especially after one has invested considerable time, money, and emotional involvement in therapy. Thus, recipients may have been unwilling to rate their therapy experience negatively, because of its implications for their own self-worth. Client-satisfaction measures can reflect positive outcomes, whereas all objective criteria contradict these rosy reports (e.g., review of the CambridgeSommerville project by Kidder, Judd, & Smith, 1986, p, 411). In addition to this alternative motivational explanation for the results, a statistical artifact account is applicable. One glaring omission in Sel igman's (1995) own list of flaws was regression towards the mean, an effect independent of any psychological improvement. Simply stated, this threat to internal validity implies that when a sample is chosen on the basis of extreme scores on a measure, the second measurement is likely to be less extreme (e.g., Cook & Campbell, 1979). Conclusions drawn from a sample who originally felt very poor or fairly poor are suspect. It may tell nothing about therapy effectiveness when an improvement in well-being is noted the second time around. Although the minuscule percentage return was dismissed and regression ignored, a gamut of other threats to internal val idi ty--absence of control groups, retrospective self-reports, respondents' awareness of survey purpose, inadequate outcome measures, and so on-was argued away by Seligman (1995). What is so dismay ing is that the many c l a s s i a l invalidities (Cook & Campbell, 1979) were not only rationalized but were turned into assets. For example, the problem of nonrandom assignment of patients to treatments was actually deemed a virtue of the CR study because, the reader is told, there is no random assignment in the field.