How Do you Audit a Qualitative Analysis?

Entry for 15 September 2008:

Whenever I talk about qualitative data analysis, I make a point of arguing that auditing has become a standard of good practice in general and for supervision of students in particular. However, I am then asked for recommendations for how to carry out an audit of a qualitative analysis, and find that there is actually very little written on the procedure.

Here is an attempt to spell out the process qualitative analysis auditing:

1. Prepare complete analysis for auditing. First, the analysis must be presented in an auditable fashion. This involves assembling, in outline form, all the categories and, under each category, all the meaning units that fall under it. If one is using computer-based qualitative analysis management software, then the program much be capable of displaying categories and data in this fashion. (I could be wrong, but it’s not clear to me how to do this in the latest version of NVIVO; comments on how to do this would be appreciated.)

Once the analysis has been presented in this fashion, it is turned over to the auditor, who in many cases is the research supervisor. (In my view, this is a major component of the duties of the supervisor of a qualitative masters or doctoral dissertation.)

Auditing consists of two main activities, which I will refer to as Fine Tooth Combing (FTC) and Category Structure Checking (CSC). Ideally, the two activities are carried out in parallel, because each of them depends to a certain extent on the other. (This is yet another example of the inevitable Hermeneutic Circle.)

2. Fine-Tooth Combing (FTC) begins by reading all the examples under a particular category very carefully. It is possible to audit on the basis of a sample of 2 or 3 instances per category, but it is really better to have all the examples, which also helps with Category Structure Checking. My practice – particularly with large meaning units – is to mark (underline or highlight) the parts that fit the category. (This is because a given meaning unit may contain information for several different categories.)

I’m particularly interested in looking for bits that don't fit the category because they may belong somewhere else or may be askew from what is going on with the rest of the category. I’m also looking for subgroups of meaning units under the category that indicate the need to add a layer of subcategories, as when two closely-related but slightly different kinds of ideas are being expressed within a category. Then I check the labelling of the category to make sure it fits my felt sense of the meaning units under it (an example of David Rennie's concept of using your stomach or gut feeling in qualitative analysis); I’ll revise the category label as needed to make it fit. At other times, I will elaborate the description of the category to bring out the richness of the meaning units within it.

3. Category Structure Checking (CSC) goes on, as I said, in parallel to FTC, and I typically check for several things when I do this: First, I make sure the categories have a parallel structure so they look like different aspects of a related aspect. For example, it might be nice for all the category names to refer to activities, or properties, or be in gerund verb form. Second, I look for for redundant/overlapping categories, which would need to be collapsed and possibly reconceptualized. This is the essence of what is called the Constant Comparison method in Grounded Theory Analysis. Third, I check for situations where there are too many categories (i.e., more than about 4) at the same level and place in the analysis (see my Qualitative Analysis Secrets blog entry from November 2007). If I find subgroups (and I usually do), I will typically suggest subcategory names. Fourth, I check to see if there is a logical order or temporal flow to the categories, out of which a mini-narrative or sequence could be constructed to make the analysis more lively or descriptive of some sort of process.

If effect, then, I propose various revisions to the analysis; however, I regard all the suggestions I make in this process to be tentative alternatives to what the original analyst has done, so I try to make it clear to them that they are free to change anything back. For this reason, and also to make it clear where I’ve proposed changes to the analysis, I turn on Track Changes at the beginning of the audit review process. After all, the original analyst knows their data better than anyone else, since its their study and they generally did the interviews on which the analysis is based. For me, it's enough that the analyst hears another way of understanding at their data.

In addition to providing quality control on qualitative anlaysis and helping analysts stay true to the their data, this sort of auditing is a very useful training device, as it conveys several important principles of good qualitative analysis, lessons that are very difficult to convey any other way. I can harp all day about checking data against categories, or about domain structure, or about the Rule of Four, but until you see what these things look like in your data, I don’t think the idea really sinks in.