In 2021, we conducted a survey about an energy efficiency program sponsored by the city of Fort Collins. We implemented a novel, hybrid push-to-web door hanger method that we hoped would combine the attractive qualities of drop-off/pick-up and push-to-web. However, this strategy proved ineffective in survey recruitment.

The need for high-quality, real-time data has never presented itself as clearly as it did during the COVID-19 pandemic. Responding to the COVID-19 pandemic, from both a policy and a public health perspective, required timely, accurate data about the public's attitudes and behaviors from health surveillance, monitoring, and public opinion surveys. The uniqueness of the COVID-19 pandemic also created particular challenges for survey data collection, specifically, how to develop high quality survey questions on topics that had never been previously fielded. To account for this challenge, the National Center for Health Statistics adopted an iterative, two-component, mixed-method approach to question design and evaluation. The first, a cognitive interviewing study using virtual, online interviews was used to produce interpretative schemata of the response processes underlying the survey questions. The second, a two-round, mixed method survey using a statistically-sampled panel, was designed to further develop the interpretive schemata and to allow for detailed subgroup analyses. To increase the usefulness of the survey's second round, cognitive interview findings and results from the survey's first round were used to develop both open- and close-ended embedded probes. Taken together, the studies reveal the specific problems for question-design during such a novel, quickly-evolving event: 1) a lack of shared understanding of novel concepts and vocabulary, 2) the shifting reference period respondents use to think about attitudes and behaviors during a multi-year event, 3) the pervasive nature of the event that therefore frames how respondents conceptualize and process questions about unrelated topics. This iterative approach to understanding question-design problems not only allowed for the continuing improvement of COVID-19 survey items, going forward, it also provided a methodological foundation for question development for high quality, real-time data collection.

BACKGROUND: This article reports key findings of an evaluation of the AABB Uniform Donor History Questionnaire (a self-administered form completed before blood donation). The purpose of the study was to examine how respondents understand the questions and assess the nature of inaccurate responses. Another goal was to determine whether men who have sex with men (MSM) interpreted questions differently from non-MSM and whether questions were interpreted differently in various regions of the country. STUDY DESIGN AND METHODS: Cognitive interviewing was used for the study. This is a qualitative method that investigates how survey questions perform. It consists of semistructured interviews that explore whether respondents understand questions as intended and whether they can provide accurate answers. A total of 166 interviews were conducted. RESULTS: Respondents had an overwhelmingly similar understanding of the purpose of the questionnaire as assessing the safety of their blood for donation. This understanding framed respondents' interpretations such that each question was understood as asking the same thing; that is, "Is my blood safe to donate?" This interpretation did not vary among MSM versus non-MSM or by region. CONCLUSION: Respondents understood the questionnaire as assessing the safety of their blood. This interpretation served as the backdrop for the question-response process for each individual question. Specifically, rationale for answers was framed as much or more by the questionnaire's general purpose as by the specific topic of individual questions. This pattern of interpretation was the key factor responsible for both false-positive and false-negative response errors and did not vary by demographic, including in MSM.

OBJECTIVES: The aim of this study was to assess the construct validity of survey questions about the use of herbal and other nonvitamin/nonmineral dietary supplements. DESIGN: We conducted one-on-one, in-depth cognitive interviews with 32 respondents to test questions from the complementary and alternative medicine (CAM) supplement for the 2007 National Health Interview Survey. Respondents were sampled purposively according to their use of CAM. Interviewers probed respondents for their understanding of the questions, and analysis was guided by grounded theory, an approach that generates explanations of response error that are closely tied to the empirical data. RESULTS: We found two sources of misinterpretation of CAM questions. First, some respondents did not have any pre-established definition of what constitutes an herbal supplement while others had interpretations that did not match the intended definitions. These problems are common to many survey questions. However, a second finding is that misinterpretation also arose when respondents incorporated notions of self-concept into the act of taking "natural herbs," and answered based on their understanding of this image rather than on actual behavior. CONCLUSIONS: There are several sources of misinterpretation of CAM questions. One of the most important sources is whether or not the respondent has created a concept of self-image that includes the use of herbal supplements. Common questionnaire design techniques such as providing definitions to respondents will not help to eliminate misinterpretation due to self-image. We found that careful question wording that does not evoke definitions of self, combined with visual aids that narrow the focus of the questions, can lead to more accurate answers.