Common Validity Threats and Suggested Solutions in Qualitative Research

Entry for 10 May 2008:

Paper presented at BACP Research Conference, Cardiff Bay, May 2008.

Validity Threats in Qualitative Research
• Cook & Campbell, 1979, and others have describe sets of validity threats for experimental studies
• Need similar framework for qualitative research
• Guidelines for reviewing qualitative research (e.g., Elliott, Fischer & Rennie, 1999)
• Here: Start by describing bad practice, then look at potential solutions

1. Closing down Qualitative Investigation from the Beginning
a. Problems:
• Attachment to own expectations or excessive professional socialization in psychology
• Starting with specific hypotheses (closed questions)
• Focusing on closed questions in the interview
• Selecting the data you like from the transcripts, ignoring things that don't fit your expectations
b. Solutions:
• Open up your research questions
• Reflect on your expectations and attachments
• Attempt to bracket your expectations and hopes

2. Qualitative Research Skill Deficits
a. Problems:
• Poor interviewing skills/personal qualities: Listening, empathy, compassion, technical process guiding skills
• Under-developed analytic skills: Empathy, vocabulary, concept formation, access to own experiencing, care/systematicness/ compulsiveness
b. Solutions:
• Practice & training
• Patience
• Switch to quantitative research

3. Methodolatry: Getting Stuck on a Particular Method
a. Problems:
• May not fit you or your topic/questions
• The seductions of brand names (GTA, CQR, IPA)
• Limitations of computer software
• Rigidities of common methods
• Arbitrary sample size requirements (too small or too large)
b. Solution:
• Use generic approach, adapt to own style, topic

4. Carelessness and Impatience
a. Problems:
•Failing to check your results by auditing, checking with informants
• Stopping data collection and analysis before you reach saturation
• Failing to extract rich conceptual structure from your categories (see flat, uninformative results)
• Failing to obtain credibility checks
• Group designs: Not sampling broadly enough to support general knowledge claims
• Single case studies: Not analyzing deeply enough to really understand the person
b. Solutions:
•Manage expectations: good qualitative research is harder (more demanding, time-consuming) than good quantitative research
•Allow enough time & energy to finish and check analyses
•Get adequate academic support

5. Flat, Uninformative Results
a. Problems:
•Often due to “going through the motions” without fully engaging with your data
• Claiming formal domains (topic areas) as substantive results
• The 37-category problem: too many unrelated but overlapping categories
• Giving up on analysis too soon
b. Solutions:
• “Stomach coding” (Rennie)
• Use of narrative structures or organize data
• Use of hierarchies of categories
• Constant comparison and the Rule of Four
• Make a picture or flowchart

6. Presentation Problems
a. Problems (from Elliott, Fischer & Rennie, 1999):
• Not owning your perspective
• Not describing the sample
• Abstract/ungrounded categories
• Disorganized/incoherent categories
• Not presenting the results in rich enough detail to allow readers to evaluate your categories for themselves
b. Solutions:
• Full reporting
• Provide narrative/visual model
• Provide rich case example

Conclusion: Realizing the Potential of Qualitative Research
•Need to reflect carefully at our research practice, make conscious efforts to improve