Week 13: Leveraging AI/LLM to assist in Qual Research
Guidance and Warnings
Overview
Today’s activities:
- Knowledge check
- Team/class shareout of reading regarding Use AI/LLM tools to assist in qualitative theme generation
- Articulate the limitations of automated qualitative analysis
Prepare
1. Watch
2. Read/Listen/Watch
- Pick something from Week 9 the ethics topics in Dr. D’s Intro to Data Science class
3. Read
Pick one paper from below and skim it enough to get a sense of what they did and what they found.
| Use | Example | What it found | Reading |
|---|---|---|---|
| First-pass inductive coding of interviews | GPT-3.5-Turbo on two open-access interview datasets (video game players, data science lecturers) | Recovered most of the main themes earlier researchers had found. It’s an exploratory study. | De Paoli, arXiv 2023; the journal version is Social Science Computer Review 2024 |
| Human and LLM building a codebook together | A human coder and ChatGPT discuss and refine codes until they agree, tested on survey data | Reported similar coding quality to human coders, with less labor and time | Dai, Xiong & Ku, Findings of EMNLP 2023 |
| Re-running a dataset someone already analyzed | ChatGPT on two previously analyzed datasets | Accurate, descriptive themes quickly, but weaker on subtle interpretive ones; the analyst needs to know the data (from secondary summaries) | Morgan (2023), IJQM |
| Deductive coding with a fixed codebook | An LLM applies predefined codes to transcripts | A study of how reliable ChatGPT is at this | Assessing the Reliability of LLMs for Deductive Qualitative Coding |
| “Conversing” with the data | Asking an LLM questions of your text, with disclosure of how it was used | Argues for stating when and how LLMs were used, and for watching bias and data security | Hayes (2025), IJQM |
| Checking bias on a contested topic | Australia’s Robodebt scandal as the case, plus a card toolkit for probing bias | The authors argue LLMs should augment human interpretation, not replace it | Khan et al. (2024), a pilot study; also Ashwin et al. (2025) on bias |
| Offline transcription | Whisper-based tools that run on your own computer | Keeps audio off vendor servers. Whisper needs a separate step to label speakers. | aTrain paper; SMU Libraries guide |
| Mapping the field | A systematic mapping study of LLMs in qualitative research | Prompt engineering and human oversight are critical; hallucinations and bias are the challenges | Barros et al. (2024) |
| The case against | Open letter saying reflexive thematic analysis is a human practice | Argues generative AI is not methodologically congruent with it, and objects on social and environmental justice grounds | Jowsey, Braun, Clarke, Lupton & Fine, SSRN preprint of the Qualitative Inquiry letter |