On Pre-Post Effect Size Statistics

Entry for 27 July 2008:

Jess Owen, of Gannon University, has written to the SPR list server with an interesting question:

I was wondering if anyone had a good reference or just general thoughts for adjusting the effect size for single group pre-post study?

Currently, I have found a consistent theme in the literature to use the pre-SD in the equation. However, even with the pre-SD, I am finding some pretty large effect sizes (e.g., > 2). I would like to think that the intervention made a big difference, but I am better sure that this is inflated given the use of only an experimental group.

This question has statistical, design, and substantive implications.

First, in narrow statistical terms, it depends on what kind of significance test was used: If the client pre-post scores aren’t paired, then an independent samples t-test is used to detect whether the difference is statistically significant, and one divides the difference of pre- and post test by the pooled standard deviation (the geometric mean of pre and post standard deviations). This is the situation that Jess Owen is referring to. However, where pre-post client scores can be paired, a more powerful statistical test is available: the dependent t-test, which takes advantage of the pre-post (or test-retest) correlation to reduce error – a long as the pre-post correlation is .5 or higher. In meta-analysis, the pre-post correlation is generally not available, so one has to resort to rules of thumb estimation procedures (e.g., Smith, Glass & Miller, 1980; if it’s a standardized inventory measure given at a 4-month interval, assume r = .5). I personally don’t like having to use these estimation procedures, which is one reason why I don’t use effect sizes based on dependent t-tests.

Second, methodologically, it depends on how you are thinking of the pre-test. For example, I like to conceptualize the pre-test as representing the population of clients not in therapy, while the post-test represents the population of clients who have had therapy. This is the logic of talking about therapy from the point of view helping client moving from one population to another, e.g., the average client moves from the 50th to the 85th percentile of the untreated population, which is another way of saying that they improved by about one standard deviation unit. In this way, also, it makes sense to use the straightforward standardized different metric described above.

Third, substantively, it appears that uncontrolled pre-post effect sizes do tend to run larger than controlled effects between the post test scores of treated and untreated clients. Lipsey and Wilson (1993) reported this in their meta-meta-analysis. However, it’s never been clear to me how much their results reflect the use of the ESs based on dependent t-tests, or whether pre-post effects are just bigger than controlled post-only effects. Fortunately, it turns out that I can use the person-centred/experiential (PCE) therapy meta-analysis results to shed some light on this question: In this data set, clients seen in PCE therapy change about 1.0 sd (=a very large effect), while control participants change about .2 sd (a small effect); as a result, the controlled standardized gain statistic (i.e., the difference between treated and untreated participants) is about .8 sd (a large effect). In other words, the pre-post ES is a wee bit bigger than the controlled ES, and this is important to be aware of. This is of course why no-treatment control designs are used in first place, but the data show that in general such controls don’t do much except make our studies look more scientific. However, there are always exceptions; for example, some types of clients probably improve on their own more than .2 sd, while untreated distressed couples may get worse!

As a result of all three of these considerations, I prefer to use a simple standardized difference as a basic effect size for therapy outcome research: It’s more conservative (generally) than an effect size based on the dependent t-test; it’s easier to think about in population terms; and it’s closer rather than further away from a controlled ES. A final note: Psychotherapy is a wonderful thing, and perfectly capable of producing extremely large effects, including one on the order of 2 or 3 sd or even larger!

Lipsey, M.W., & Wilson, D.B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.

Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy. Baltimore: The Johns Hopkins University Press.