Cognitive interviewing is a commonly used questionnaire pretesting method designed to evaluate the cognitive properties of survey instruments as sources of potential measurement error. In a cognitive interview, the researcher administers a questionnaire, but the primary purpose is not the collection of survey data. The researcher instructs the cognitive interview participant to “think aloud” while determining his/her answers so that the researcher can capture and analyze how the participant cognitively processes each question. The researcher may administer concurrent or retrospective probes that ask the participant to report what he was thinking when the answer was determined, or how confident she is in her answer, or how he thinks other people might answer the question. Participants are encouraged to report when response categories are not applicable or when questions are confusing or difficult to understand.
Cognitive interview studies provide rich sources of data on questionnaire problems, but are labor and cost intensive due to time required to recruit and interview (often 1-2 hours) participants and to analyze the qualitative data provided in the interviews. For this reason, small convenience samples of 8-20 participants are typically used, recruited from a locally accessible population. Recent research suggests that the small samples of cognitive interview studies can be a problem. Blair & Conrad (2011) found that that a sample size of 10 can detect approximately half of the most severe problems and while detecting approximately 25% of the most subtle problems. Furthermore, easily recruited, locally available populations may provide a limited participant pool made up of professional respondents with limited cultural and socioeconomic variation.
A potential solution to cost and logistical challenges of cognitive interviews is the use of virtual communication technologies. We piloted a study of using Second Life and Skype to conduct cognitive interviews. Second Life may be well-suited to in-depth and cognitive interviewing methods, because it enables face-to-face-like communication with avatars, which when self-designed, can represent individuals’ selves more authentically than in real life (Taylor 2001). Additionally, the environment allows a simulation of face to face interaction with a community of people spread across the globe. Similarly, Skype allows face-to-face communication with participants via the computer all over the world, except that instead of an avatar, the participant uses video calling so that the researcher can see the participant’s facial expressions.
As a pilot test of using these two technologies as cognitive interview modes, we recently conducted 40 cognitive interviews across three modes: Second Life (using voice chat), Skype (using video conferencing), and in-person. Anecdotally, our researchers felt both modes were promising, for a variety of reasons:
- Logistically, virtual interviews in both Skype and Second Life were easier to schedule and conduct than in-person interviews.
- Participants were recruited from locations across the country, rather than near RTI’s offices and represented some populations that are hard to bring into a cognitive lab (extremely obese, homebound, employed more than 40 hours per week, Indian reservation resident).
- Cash incentives (in Second Life currency and Amazon gift cards) were easy to disburse electronically.
Preliminary results also show that both methods can result in reasonable quality cognitive interview data. Second Life interviews on average were longer (62 minutes compared to 44 minutes for Skype) but had more technical disruptions (5 minutes compared to nearly zero for Skype). Yet even controlling for technical problems, Second Life interviews were 50 minutes on average compared to 32 on average for Skype. We’re assuming a longer interview means more cognitive interview content, but a more detailed analysis of the content of the interviews will be required to confirm that.
As displayed in the table, looking at the analysis of our first four interviews, Skype interviews on average uncovered more problems (21 compared to 19 for Second Life). Preliminary evidence may suggest some variation in the types of problems uncovered. More comprehension and response problems were uncovered in Skype whereas more retrieval and judgment problems were uncovered in Second Life.
These findings are only preliminary, and I look forward to reporting the full analysis of all 40 interviews conducted (including some comparison interviews conducted in real life). But the results suggest that both while Skype may ultimately be more comparable to an in-person interview, both modes are viable for conducting cognitive interviews.
Blair, Johnny and Fredrick Conrad. 2011. “Sample Size for Cognitive Interviewing.” Public Opinion Quarterly, 74: 636-58.
Taylor, T. L. (2001). Living digitally: Embodiment in virtual worlds. In R. Schroeder (Ed.), The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments (pp. 40–61). London: Springer.