Health Education Research Advance Access originally published online on January 23, 2008
Health Education Research 2008 23(3):467-476; doi:10.1093/her/cym089
Applying cognitive response testing in message development and pre-testing
Health Communication Research Laboratory, Saint Louis University School of Public Health, Saint Louis, MO 63104, USA
* Correspondence to: C. Lapka. E-mail: lapkacm{at}slu.edu
 |
Abstract |
|---|
 Top Abstract IntroductionExamplesDiscussionConflict of interest statementAcknowledgementsReferences |
|---|
Pre-testing messages with audience members is a critical step in the creation of effective health information. Quantitative methods for message testing have limited effectiveness, as they cannot reveal complications with language and comprehension. Cognitive response testing (CRT), a form of qualitative research, allows the interviewer to probe for deeper understanding of comprehension and language by asking participants to paraphrase items, discuss thoughts or emotions that come to mind and offer suggestions for improvement. This study explores the usefulness of CRT in message development and testing, adding to the literature regarding qualitative methods in public health. CRT was employed to evaluate health messages on two topics—bioterrorism and influenza vaccination. This technique effectively identified message terminology and concepts that respondents found unfamiliar or confusing, providing the framework needed for message revision. Commonly misunderstood words were replaced and confusing concepts were explained in the revised messages, making pre-tested messages more likely to be appropriate for the intended audience. These findings are consistent with previous research that establishes the usefulness of CRT in the evaluation and development of health-related messages and surveys.
|
Introduction |
|---|
|
TopAbstractIntroductionExamplesDiscussionConflict of interest statementAcknowledgementsReferences |
|---|
Public health professionals often develop health promotion messages for the community. Messages must be audience appropriate and easily understood to produce the desired behavioral changes. Pre-testing messages and materials allow developers to determine the usefulness and effectiveness of the messages with the intended audience. Although working with advisory groups can add useful input when developing materials, only testing with members of the intended audience will determine what their reactions might be [1].
There are two basic research methodologies to test message effectiveness: quantitative and qualitative. Quantitative methodology uses research procedures aimed at collecting data to test theories of behavior; the emphasis is on generalizability, replicability and theory and effects testing [2]. This method is commonly used when assessing a program already in place or when the goal of the research is measurable, and is often conducted using a structured questionnaire containing predominantly closed-ended questions [1]. A survey typically measures antecedents suggested by existing research; however, these antecedent variables are typically the outgrowth of sociological or psychological theory and do not necessarily reflect the cultural and normative specifics of social life in diverse communities. Key determinants can be missed without qualitative research efforts to identify them [3].
In contrast, qualitative methodology uses procedures aimed at providing in-depth analysis of behavior through the collection of rich, narrative materials using a flexible research design. The emphasis is on understanding the phenomenon and generating theory, rather than generalizability [3, 4]. This methodology allows for a detailed understanding of the audience's knowledge of wording and language, tapping their experiences and beliefs [4, 5]. Qualitative methods such as participant observation, focus groups and in-depth interviews provide information that is grounded in program participants' own perspectives and not constrained by predetermined categories of analysis [2, 6]. Qualitative research is not without limitations, such as the lack of representativeness and limited generalizability of information; however, this loss of generalizability is typically deemed justifiable in exchange for the rich information gained [3].
A number of pre-testing methods have been identified in the literature to ensure that materials are comprehensible and effective [4]. Focus groups, commonly used in pre-testing, are a semi-structured group interview, typically conducted among 8–12 relatively homogeneous subjects moderated by a trained researcher [2]. They are socially oriented, cheap and quick and have face validity, but they can be difficult to organize and analyze, and their results can be unpredictable [7]. Likewise, in-depth interviews are face-to-face interviews in which a researcher asks a list of open-ended questions about a particular topic to an individual within the intended audience. In-depth interviews represent one of the richest sources of information available to researchers, but are often expensive and time consuming, and lack the structure to adequately ascertain participants' comprehension of the message [2, 8, 9].
Essentially, cognitive response testing (CRT) involves conducting a structured interview with an individual as he/she reads, interprets and answers an instrument item [10–12]. Studies dealing with cognitive responses generally include the following: the respondent is rehearsed in verbalizing or writing his thoughts, emotions and interpretations; responses are collected by tape recorder or in written form and responses are categorized on various criteria by the researchers [13]. Pre-testing using cognitive responses is beneficial in the generation of words, phrases and vernacular used by the intended audience so that appropriate language can be formulated into new messages [8].
CRT is developed from the cognitive response models made explicitly for monitoring communication effects. The premise of these models is that the spontaneous unstructured cognitive responses (thoughts) elicited by a communication message act as mediators of attitude change. Therefore, a precise understanding of communication impact is not possible without attention to the cognitive thought responses emitted by the receiver upon exposure to the communication [14]. The potential of CRT in message pre-testing, although largely unexplored, appears strong and warrants the attention of public health professionals [15]. This cognitive method of interviewing, developed from Wright's model, is variously termed cognitive testing, cognitive pre-testing and cognitive interviewing in the literature. We will refer to it as CRT to specify the importance of participants' cognitive responses in interpreting a message's effectiveness.
In public health literature, CRT is most often cited to improve the quality of survey data by reducing response error that occurs if questions are not understood or not interpreted in the intended manner [11, 16]. The purpose of CRT in survey development is to gain knowledge of participants' understanding and emotional response to survey questions [17, 18]. In survey development, the respondent is asked to verbalize thoughts, feelings, interpretations and ideas that come to mind while examining survey questions [19]. Although this method is useful for survey development with culturally diverse groups, the literature rarely documents CRT with regard to health message pre-testing [11, 17, 19–21]. We suggest that CRT is applicable for message pre-testing, given the common concern with respondents' comprehension and emotional response.
The overall aim of pre-testing messages using CRT is to understand how respondents perceive and interpret key words and phrases and to identify potential comprehension problems that may arise [22] (Table I). Respondents are also asked to suggest alternative wording to increase readability, comprehension and relevance of information. By probing for the most meaningful and relevant words, expressions and phrases, the audience is placed in the expert role and the interviewers gain valuable insight into subjects' cognitive processes and needs. The more relevant and understandable the message, the more likely it will be accepted [20].
|
This paper aims to add to the literature by describing a straightforward and effective method for assessing message comprehension as a part of message pre-testing. To allow readers to see CRT used in different content areas, we provide examples of projects in emergency risk communication and influenza vaccine promotion. In Study 1, CRT is used with diverse participants to pre-test messages about plague. Study 2 displays how CRT complements focus groups to assess responses of African Americans to messages about flu vaccine safety and efficacy. The purpose is not to provide instruction on collecting and analyzing data, but to illustrate how CRT can be used to assess comprehension of and emotional response to health promotion messages. CRT guidelines, which we followed, can be found in several references [5, 20, 22–24].
|
Examples |
|---|
|
TopAbstractIntroductionExamplesDiscussionConflict of interest statementAcknowledgementsReferences |
|---|
Study 1: The pre-event message development project
Method
The purpose of the Pre-Event Message Development Project (PEMDP) was to create educational materials that could be distributed in the event of a terrorist attack using biological, chemical or radiological agents. PEMDP was a cooperative project funded by the Centers for Disease Control and Prevention (CDC) from 2002 to 2006, with four schools of public health. All schools conducted formative focus groups with different audience segments including African American, Caucasian, Hispanic, American Indian/Alaskan Native and Asian/Pacific Islander groups. Findings from the formative research were used to create educational messages in print and broadcast formats. Messages were then pre-tested using CRT and focus groups.
In this paper, we present the findings for educational materials about plague created by researchers and writers at Saint Louis University. Each university's Institutional Review Board approved all research to ensure the protection of human participants. All final educational materials, reports and publications emerging from the PEMDP can be found at http://www.bt.cdc.gov/firsthours.
CRT was used to determine how people interpreted the terrorism-related information and emotionally responded when asked to read specific blocks of text within the messages. Based on expert review and earlier formative research, researchers selected message text for the CRT that was problematic, that was easily misinterpreted or that focused on higher risk audiences.
Male and female participants 18 years or older were recruited. Participants were told the interview would last between 30 and 60 min and they would receive $20 at the end of the interview. A sample of 30 participants were recruited to participate in the CRT. Interviews were audiotaped and transcribed. A note taker recorded non-verbal responses.
A structured script was followed for each CRT. Participants were shown nine specific sections of text. The moderator asked specific comprehension questions for each of the nine sections, as well as open-ended questions, allowing participants to list any other aspects of the message they found confusing (Table I). Table 1 shows one of the nine sections, including the moderator's questions and sample of participants' responses.
The comprehension questions asked participants to paraphrase or summarize the block of text they had just read. To measure comprehension, participants were asked to define words such as ‘antibiotics’ or ‘contagious.’ For the sections of the materials that addressed transmission processes, symptoms of illness or treatment, participants were asked if they were confident they could now protect themselves from getting plague, recognize symptoms or understand treatment.
To measure emotional response, participants were asked, ‘How did you feel after reading this section?’ and then asked to explain why they felt that way. Participants were then told to ‘Please tell me which parts of this section you did not understand or were not clear to you’. They were also asked to provide suggestions for improving the materials.
CRT transcripts were analyzed using consensus coding and a matrix process [25]. Transcripts were divided into sections by topics discussed in the interview. To see differences among the different population groups, a matrix was created for each population with the topic on the y-axis and each participant on the x-axis. Researchers read through transcripts of each interview and completed the matrix for each participant. Researchers then completed the population summary box by comparing participants' answers to questions for each topic. After each of the population summaries was analyzed, a second matrix was created using population groups on the x-axis, allowing researchers to compare summaries for the different groups, creating an overall summary of the findings (Table II). The summary also highlighted differences in each population group. Changes to the print materials were made based on the findings from the matrices.
|
Results
Overall, 30 cognitive response tests were completed at Saint Louis University. The participants were diverse: 22–86 years of age, mostly female (77%) and represented a number of different ethnic groups (30.0% African American, 26.7% Caucasian, 10% American Indian or Alaskan Native, 23.3% Latino/Hispanic and 10% Asian/Pacific Islander) (Table III).
|
For all population groups, a number of the terms and concepts were difficult for participants to understand. Comprehension is especially important should there be an outbreak of an infectious disease like plague, so that members of the general public can protect themselves. Participants were confused about transmission because they did not understand the terms, such as ‘respiratory droplets’ or ‘contagious’.
Recognizing symptoms and taking the correct actions based on those symptoms were also problematic. Not all people understood ‘going into shock,’ ‘abdominal pain’ and ‘respiratory failure’. Disease timelines were confusing, such as how long it takes to get symptoms, what period an infected person is contagious and when it is important that a person seeks medical attention. Also, participants did not have a good understanding of vaccination or antibiotics for the prevention and treatment of plague. The educational materials also included the scientific name for the bacterium that causes plague, Yersinia pestis, which most participants found confusing.
African Americans were concerned about bioterrorism because of a lack of resources for dealing with a potential outbreak. Reading about disease symptoms was frightening for some. Others found the information reassuring because they were learning additional information that was important and needed.
Some sections prompted more questions than they answered. Participants' questions highlighted the need for additional information. For instance, participants were concerned about a section on pets. Cats were mentioned as specific carriers, but the information did not provide additional animal recommendations. Participants wanted a complete list of animals that may carry plague.
CRT findings guided message revisions to promote improved comprehension and resulted in a number of changes to the materials. Disease timelines and transmissions were particularly troublesome for participants, so those were clarified. Words such as ‘contagious,’ ‘respiratory’ and ‘antibiotic’ were defined in text, and words that could be removed from the text without impairing comprehension, such as ‘naturally occurring’, were omitted. Additional information to help explain directives or encourage adherence, such as how to take care of one's pets, was included in the materials. It is critical that the public understands bioterrorism pre-event educational materials and reacts to them in an emotionally appropriate manner.
Study 2: Influenza vaccination project
Methods
With funding from the National Immunization Program at the CDC from 2004 to 2007, members of the study team developed messages to address older African Americans' safety and efficacy concerns about flu vaccination. Formative research using focus groups and key informant interviews highlighted common concerns and fears associated with the influenza vaccine. Based on the formative research, four different educational messages about influenza vaccination were developed on the following topics: ‘The Flu Shot Doesn't Give You the Flu’, ‘The Flu Shot is Safe’, ‘The Flu Shot Works’ and ‘Benefits and Risks’. Messages were pre-tested using CRT with African Americans aged 50 or older to assess participants' comprehension and emotional response to the messages. Table IV provides sample questions for focus groups, key informant interviews and CRT to illustrate how the methods are complementary in the development and pre-testing of health promotion messages. Focus groups and key informant interviews were used in the formative research, while the CRTs were used in message pre-testing. This project was approved by the Saint Louis University Institutional Review Board.
|
Ten cognitive response tests were completed with African American participants aged 50 or older ambivalent about the flu vaccine and willing to participate in one 60- to 90-min interview. Participants were screened to determine eligibility. Interviews were audiotaped and a note taker was present. A structured script was followed for each interview. The participants were shown nine sections with text excerpts from each of the four message topics discussed earlier. The moderator asked specific open-ended questions corresponding with each of the nine sections, designed to detect message comprehension, confusing terminology and concepts and emotional response to the materials (Table V).
|
A researcher transcribed audiotapes and notes from the interviews. The notes supplemented the transcripts, when needed. Interviews were not coded, due to the direct nature of the questions. Instead, they were analyzed using a matrix similar to the PEMDP's matrix. Learning domains were listed on one axis and participants on the other axis. Two researchers filled in the matrix by summarizing each participant's answers to the individual domain's scripted questions. After completing individual matrices, researchers met to reach a consensus on the interpretation of the interviews. These data were compiled into a summary report recommending how messages could be further improved.
Results
Ten cognitive response tests were completed. The participants were 56 to over 85 years old and mostly female (90%). Nine (90%) participants were African American; one (10%) was other (Table III).
Pre-tested messages were typical of Web-based CDC health information where information was presented in a ‘question and answer’ style. Participants read specific blocks of text excerpted from these messages and answered questions designed to elicit evidence of misunderstanding or confusion about the excerpt. CRT made researchers aware of sections of text, words and concepts that participants found particularly problematic.
The interviews determined that critical information was misinterpreted. Terms that confused participants included ‘killed virus’, ‘mildly allergic’, ‘antibodies’, ‘pure’ and ‘Guillain–Barré Syndrome’ (GBS), which was described as ‘muscle weakness and numbness of the arms and legs.’ GBS is a serious neurologic disorder where the body begins to attack its own peripheral nervous system and is typically characterized by progressive, symmetrical weakness in the legs and arms, with loss of reflexes [26]. GBS was described in one section of the messages because the influenza vaccine is not recommended for people with GBS. Most participants had never heard of this syndrome and were confused by its definition. When asked how participants felt after reading the section on GBS, common responses included ‘I felt like I was reading something I didn't know about’ and some felt the flu shot might be beneficial. Others felt better informed after reading the messages. Additionally, ‘milder case’ of the flu was frequently misinterpreted. Participants argued there is no such thing as a ‘mild case’ of the flu and that getting the flu is always serious.
Vaccination timelines were also confusing for participants. Although most understood the best time to get vaccinated, many could not describe how long it takes the flu shot to build immunity or how long the vaccination remains in the body. Many were also confused about the possibility of getting influenza during the 2 weeks following the vaccination. Even after reading the messages, most participants still believed the flu shot causes the flu.
Findings from CRT provided the framework for message revision. Commonly misunderstood terms were given definitions or explanations (Table VI). To provide clarity, the sentence ‘only a live virus can give people the flu’ was added in the revised message to better explain ‘killed virus’. Likewise, GBS was better defined in revised messages. Some words that were confusing for participants such as ‘injection site’ were replaced with words that seemed to cause less confusion such as ‘where the needle entered the skin’.
|
Difficult concepts such as vaccine timelines were clarified by providing even more specific information. Originally, the timeline was described in weeks: ‘During the next two weeks, antibodies develop in the body.’ The revised message contains the headers, ‘1–2 days after ...’, ‘1–14 days after ...’ or ‘after 14 days ...’ to better describe what happens in the days and weeks after the event of vaccination. Table VI shows how the message was revised following CRT.
|
Discussion |
|---|
|
TopAbstractIntroductionExamplesDiscussionConflict of interest statementAcknowledgementsReferences |
|---|
In this article, we described the gap in public health literature regarding the use of CRT in message development and pre-testing. We demonstrated the application of CRT to message pre-testing with diverse topics—influenza vaccination and bioterrorism education. CRT is easily applied to various subject matters and effectively highlights problematic terminology and phrases, allowing for beneficial and efficient message revisions. Ultimately, this method improves the probability that revised messages will include more concise definitions and succinct passages, boosting health message comprehension by the targeted community.
CRT encourages respondents of the targeted audience to think aloud as they read or report what went through their minds as they read or asks respondents to rate the comprehensibility of the messages [11, 12, 17, 27]. Therefore, messages pre-tested using CRT are more likely to be appropriate for the intended audience and not based solely on the message writer's assumptions about the community [28]. CRT complements less structured pre-testing methods, such as in-depth interviews and focus groups, not designed to reveal words and phrases within messages that are particularly difficult for respondents to comprehend [29].
CRT also provides valuable insight into respondents' cognitive process and informs the development of more effective educational messages [19]. Other benefits include the ability to clarify word definitions and to determine how respondents perceive specific concepts and phrases [16]. A versatile method, CRT has traditionally been used in survey development, but our findings demonstrate its comparable usefulness in message development and pre-testing.
Data collection using qualitative methodology, including CRT, may be more time consuming and costly than quantitative data collection. Even if the interview lasts less than an hour, protocol preparation, analysis and write-up require a great amount of time and improvements in the comprehension of messages can result in longer message text [19]. However, Carbone [21] determined that applying audience-based definitions, words and sentences—although longer in length—actually reduced respondent burden when compared with shorter (and more difficult) versions.
Participants are unfamiliar with CRT processes, such as reading education materials and responding to specific questions pertaining to the materials. Therefore, extra time must be allotted to explain the interview process. Not doing so created confusion and discontent for some participants. These limitations should be addressed in future studies by limiting the amount of text participants are asked to read and limiting the questions asked of participants.
Public health professionals often examine how study populations respond to and make behavioral changes based upon educational messages [20]. When messages are not easily understood, their effectiveness is diminished. To ensure messages are understood, focus groups have typically been arranged with the intended audience. A versatile method, CRT, has proven effective in survey development. Our results demonstrate CRT is also a straightforward, appropriate and useful method in health promotion message development and pre-testing. CRT can provide a clear assessment of issues related to the understanding of and emotional response to the messages, leading to content that is effective and appropriate for the targeted audience.
|
Conflict of interest statement |
|---|
|
TopAbstractIntroductionExamplesDiscussionConflict of interest statementAcknowledgementsReferences |
|---|
None declared.
|
Acknowledgements |
|---|
|
TopAbstractIntroductionExamplesDiscussionConflict of interest statementAcknowledgementsReferences |
|---|
We would like to thank Anthony Russo, Amanda Whitworth and Jennifer Rivers for their work on Study 1. Thanks to the clients and employees of the local community centers and retirement facilities that participated in this research. We would like to give special thanks to Katie Duggan at the School of Public Health and Edith Gary and Pascale Wortley at the CDC for their support and work on Study 2. We would also like to thank our PEMDP partners at the University of Alabama at Birmingham, University of California at Los Angeles and the University of Oklahoma. The PEMDP was funded by the CDC #A1104-21.23. Flu vaccine message testing research was funded by grant #6465 from the National Immunization Program at the CDC, via Special Interest Project 11, to the Prevention Research Center at the Saint Louis University School of Public Health.
|
References |
|---|
|
TopAbstractIntroductionExamplesDiscussionConflict of interest statementAcknowledgementsReferences |
|---|
1. U.S. Department of Health and Human Services. Making Health Communications Programs Work (2002) Bethesda, MD: National Cancer Institute.
2. Valente T. Evaluating Health Promotion Programs (2002) Oxford: Oxford University Press.
3. Maibach E, Parrott R, eds. Designing Health Messages: Approaches from Communication Theory and Public Health Practice (1995) Thousand Oaks, CA: Sage Publications.
4. Patton M. Qualitative Research and Evaluation Methods (2002) 3rd edn. Thousand Oaks, CA: Sage Publications.
5. Ingerstall-Dayton B. Combining Methodological Approaches in Social Support Research. Paper presented at: Annual Meeting of the American Psychological Association, August 1982, Washington, DC.
6. Secker J, Wimbush E, Watson J, et al. Qualitative methods in health promotion research: some criteria for quality. Health Educ J (1995) 54:74–87.
7. Krueger R. Focus Groups: A Practical Guide for Applied Research (1994) Newbury Park, CA: Sage Publications.
8. Rice R, Atkin C. Public Communication Campaigns (2001) 3rd edn. Thousand Oaks, CA: Sage Publications.
9. Andreasen A. Marketing Social Change: Changing Behavior to Promote Health, Social Development, and the Environment (1995) San Francisco, CA: Jossey-Bass.
10. Damaio T, Rothgeb J. Cognitive interviewing techniques: in the lab and in the field. In: Answering Questions: Methodology for Cognitive and Communicative Processes in Survey Research—Schwarz N, Sudman S, eds. (1996) San Francisco, CA: Jossey-Bass. 177–196.
11. Jobe J, Mingay D. Cognitive laboratory approach to designing questionnaires for surveys of the elderly. Public Health Rep (1990) 105(5):518–24.[Web of Science][Medline]
12. Tourangeau R, Rips L, Rasinksi K. The Psychology of Survey Response (2000) Cambridge, NY: Cambridge University Press.
13. Lutz R, Swasy J. Integrating cognitive structure and cognitive response approaches to monitoring communications effects. Adv Consum Res (1977) 4(1):363–71.
14. Wright P. On the direct monitoring of cognitive response to advertising. In: Buyer/Consumer Information Processing—Hughes G, Ray M, eds. (1974) Chapel Hill, NC: University of North Carolina Press. 220–48.
15. Olson J, Toy D, Dover P. Mediating effects of cognitive responses to advertising on cognitive structure. Adv Consum Res (1978) 5(1):72–8.
16. Alaimo K, Olson C, Frongillo E. Importance of cognitive testing for survey items: an example from food security questionnaires. J Nutr Educ (1999) 31(5):269–75.[Web of Science]
17. Sudman S, Bradburn N, Schwartz N. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology (1996) San Francisco, CA: Jossey-Bass Inc.
18. Forsyth B, Lessler J. Cognitive laboratory methods: a taxonomy. In: Measurement Errors in Surveys—Biemer P, Groves R, Lyberg L, Mathiowetz N, Sudman S, eds. (1991) New York: Wiley. 393–418.
19. Willis G. Cognitive Interviewing and Questionnaire Design: A Training Manual (1994) Hyattsville, MA: National Center for Health Statistics.
20. Carbone E, Campbell M, Honess-Morreale L. Use of cognitive interview techniques in the development of nutrition surveys and interactive nutrition messages for low-income populations. J Am Diet Assoc (2002) 102(5):690–6.[CrossRef][Web of Science][Medline]
21. Carbone E. LEARNSMART: The Application of Adult Learning Theories to Nutrition Education (1999) Chapel Hill, NC: University of North Carolina.
22. Gerber E, Wellens T. Perspectives on pretesting: "cognition" in the cognitive interview. Bull Methodol Sociol (1996) 55:18–39.
23. Whyte W. Learning from the Field (1984) Beverly Hills, CA: Sage Publications.
24. Woolley M, Bowen G, Bowen N. Cognitive pretesting and the developmental validity of child self-report instruments: theory and applications. Res Soc Work Pract (2004) 14(3):191–200.[Abstract]
25. Miles M, Huberman A. Qualitative Data Analysis: An Expanded Sourcebook (1994) 2nd edn. Thousand Oaks, CA: Sage Publications.
26. National Institute of Neurological Disorders and Stroke. Guillain–Barre Syndrome Fact Sheet. Available at: http://www.ninds.nih.gov/disorders/gbs/detail_gbs.htm#83123139 Accessed: 30 April 2007.
27. Foddy W. An empirical evaluation of in-depth probes used to pretest survey questions. Sociol Methods Res (1998) 27(1):103–33.[Abstract]
28. Farquhar S, Parker E, Schulz A. Application of qualitative methods in program planning for health promotion interventions. Health Promot Pract (2006) 7(2):234–42.
29. Pearson P. Integrating qualitative and quantitative data analysis. Nurse Res (1997) 4(3):69–80.
Received on June 7, 2007; accepted on November 8, 2007
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||