Summary of Nontherapy Explanations and How They are Assessed in HSCED

Entry for 11 January 2009:

In revising the adjudicated Hermeneutic Single Case Efficiacy Design (HSCED) paper (see blog entry just previous to this one), I had to cut a lot of nice material, including a longish section summarizing the eight nontherapy explanations or validity threats that HSCED systematically evaluates in order to counter the therapist/researcher’s desire to see the client change (the first four explanations) and for therapy to be responsible for those changes (the last four explanations). However, it seemed a waste to just discard this material, so I’m including it here, lightly revised from the discarded section, in the hope that it may provide a useful summary:

1. Trivial or Negative Change. The first four nontherapy explanations assume that apparent client change is illusory or artifactual. To begin with, the apparent changes may be negative or trivial. On the one hand, clients might describe a change in highly qualified or ambivalent terms ("I guess I’m doing a bit better, sort of."), or they may describe changes in other people or their life circumstances. In the same way, changes on quantitative outcome measures may also fall into the trivial range (e.g., one point on the BDI). Alternatively, changes, both reported by clients and evident on outcome measures, might be negative, casting doubt on the overall effectiveness of the therapy, or even suggesting that it has been harmful. To evaluate for trivial or negative change, researchers can calculate reliable change index (RCI) values (Jacobson & Truax, 1991) for our three key measures (SCL-90-R, IIP, and PQ values taken from Barkham, Hardy, & Startup, 1996; Ogles, Lambert & Sawyer, 1995). (In other words, our definition of trivial change was change that did not exceed measurement error.) In addition, in order to assess for negative changes, the interviewer can ask the client to describe any negative changes which might have occurred over the course of therapy, and also asked them to rate the importance of each change (cf. Kazdin (1999), using a 5-point scale. In addition, the manner of the client's description can be examined for qualifiers and other forms of ambivalence.

2. Statistical Artifacts. Related to the possibility of trivial change is statistical error, including measurement error, regression to the mean, and experimentwise error (measurement error has already been discussed under the heading of trivial change). Regression to the mean occurs when measurements with less than perfect reliability are selected on the basis of their extreme values. This introduces bias that is not present when the measurement is later repeated, resulting in the second measurement taking a less extreme value, thus producing illusory change. This can be assessed by repeating the Personal Questionnaire prior to the client beginning therapy. In addition, the researchers can assess the duration of the client's problems. Experimentwise error is a function of carrying out multiple significance tests on change measures. When examining many measures for evidence of change, some apparently reliable differences may occur due to chance alone. For example, when three measures are used to evaluate the reliability of pre-post change, with the relaxed significance level used in HSCED, each measure has .2 probability of indicating change when none existed (Type I error). The solution used here was to require reliable change on two out of four measures (this corresponds to a probability of .18).

3. Relational Artifacts. Apparent client improvement may also reflect interpersonal dynamics, in particular, client efforts to impress the therapist or research staff with clinical distress at the beginning of treatment, coupled with the opposite emphasis at the end of therapy, intended to express gratitude or justify ending therapy. In order to determine the role of self-presentational interpersonal artifacts, researchers can look at the client’s Change Interview data, paying attention to nuance and style, using Bohart and Boyd's (1997) plausibility criteria of elaboration and discrimination, assessing for the presence or absence of specific details about what has changed and how the change came about (elaboration). In addition, researchers can look for vague, global descriptions vs. differentiated descriptions containing both positive and negative elements.

4. Expectancy Artifacts. Cultural or personal expectations ("scripts") or wishful thinking may also give rise to apparent client change. That is, clients may convince themselves and others that since they have been through therapy they must therefore have changed. Post-therapy qualitative accounts are particularly vulnerable to this sort of bias, but quantitative ratings are also susceptible. As with relational artifacts, expectancy effects can be assessed by examining the language the client uses to describe his or her experience, in particular looking for expectation-driven descriptions that rely on shared cultural schemas about the effects of therapy and are therefore limited to standard or clichéd phrases, such as "someone to talk to," or "insight into my problems." By contrast, descriptions that are idiosyncratic in their content or word choice are more believable. In addition, expectation-driven expressions can be expected to sound vague, intellectualized or distant from the client's experience, as opposed to detailed, careful, and self-reflective (cf. Bohart & Boyd, 1997). In addition, the client can be asked to rate each of their changes for the degree to which they expected vs. were surprised by it.

5. Self-help and Other Self-Correction Processes. The remaining nontherapy explanations all assume that change occurred, but that factors other than therapy were responsible. First, client internally-generated maturational processes or self-help efforts may be generally responsible for the observed changes. For example, the client may have entered therapy in a temporary state of distress which has reverted to normal functioning via the self-limiting nature of temporary crises or the person’s own problem-solving processes. Alternatively, the change could be a continuation of an ongoing developmental trend or self-help efforts independent of therapy. A general strategy for evaluating the final four nontherapy explanations is to ask the client. Similarly, the client can also be asked to assess how likely he or she feels the change would have been without therapy. Therapist process notes provide an efficient source of information about client self-help efforts, and can be used to account for shifts in PQ scores. Researchers can also looked for narratives of the client’s self-help efforts begun before or separate from therapy.

6. Extra-therapy life events include changes in relationships such as deaths, divorces, initiation of new relationships, marriages, births, and other relational crises, as well as the renegotiation of existing relationships. In addition, clients may change jobs, get fired from jobs, get promoted or take on new work responsibilities, change recreational activities, and so on. They may also include changes in quality of life due to physical injuries or illnesses or medical treatments. Extra-therapy events can contribute both positively and negatively to therapy outcome. It is also important to consider the bidirectional influence of therapy and life events on one another. The most obvious method for evaluating the causal influence of extra-therapy events is to ask the client. Researchers can use the Change Interview, to ask the client what they thought brought about their changes. (In this interview, the researcher asks about extra-therapy factors if the client fails to mention them and before asking about the influence of therapy, and also asks the client to estimate the likelihood that the change would have occurred without therapy.) In addition, researchers can look at therapist process notes to find out about extra-therapy events.

7. Psychobiological Causes. The next possibility is that credible improvement is present, but is due primarily to direct, unidirectional psychophysiological processes, especially psychotropic medications or herbal remedies and seasonal and endogenously-driven mood cycles. The most obvious approach to evaluating psychobiological factors is to keep track of medications, including changes and dose adjustments. Therefore, questions about medication and herbal remedies are included in the Change Interview. (Therapist process notes could also be used for this.)

8. Reactive Effects of Research. The final nontherapy explanation is the reactive effects of taking part in research. These include reactive research activities (e.g., PTSD assessment) that enhance (or interfere with) therapy, relation with the research staff, and enhanced sense of altruism. On the other hand, research activities can have negative effects on clients, especially if they are particularly difficult or time-consuming. Teasing out the reactive effects of research on client outcome can be difficult, but qualitative interviewing can help here as well. Thus, researchers can asked the client to talk about the effects the research had on them. A quantitative measure of the effects of research procedures can also be used.