Health Education Research Advance Access published online on September 25, 2007
Health Education Research, doi:10.1093/her/cym050
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The quality of nutritional information available on popular websites: a content analysis
1 Department of Health Care and Epidemiology, University of British Columbia, Vancouver, V6T 1Z3, Canada
2 School of Journalism, University of British Columbia, Vancouver, V6T 1Z3, Canada
3 Present address: Faculty of Social Sciences University of Victoria, Victoria, BC, Canada
Correspondence to: * Correspondence to: A. Ostry. E-mail: Ostry{at}uvic.ca
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
The overall purpose of this study was to increase knowledge and understanding of the new informational landscape that is emerging on the Internet in relation to nutritional health content in order to provide policy makers with better communication and health promotion tools. We identified the sites most used by Canadians to access nutrition information and conducted content analyses to identify the sources of this nutritional information as well as its quality by systematic comparison with the main guidelines published in the Canada Food Guide. We found that commercial websites accounted for 80% of visits and time spent on seeking health and nutrition information. We also found uneven messaging about fruit and vegetable intake as well as consistent messaging undermining the ‘eat a variety of foods’ message, which is a central component of the Canada Food Guide. On the positive side, inappropriate or incongruent advice about salt, coffee and alcohol intake was virtually non-existent and advice congruent with the guide was found three times more often than incongruent advice. Finally, the site offering the best advice was a non-commercial government-based site. This site differed from the commercial sites not so much in its ability to deliver the ‘right’ advice but more in its ability to exclude articles with poor and misleading advice on their sites.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
The print and broadcast media have, historically, been the major sources of health and nutrition information for Canadians seeking guidance on health and nutrition [1–4]. While TV and print media have been and still are important sources of nutrition information, the media landscape is changing rapidly with expanding access to and use of the Internet as well as the recent proliferation of health and nutrition websites [4, 5]. Between 1998 and 2000, the number of online health users grew by 146%, substantially outpacing the total growth in Canadian Internet users (83%) [6]. By 2002, two-thirds of Canadian households had at least one member who used the Internet to search for medical or health-related information [7], while >80% of adult Canadian Internet users say they have consulted the World Wide Web's vast digital health library [8]. Not only are more Canadians accessing nutrition and health information online but also research suggests that they are changing their behaviour as a result, with more than one-third of men and women indicating altered eating and exercise habits as a result of advice obtained online [9].
Given that more people are looking to the Internet for guidance on nutritional matters, and people are increasingly reporting that this information impacts their health behaviours (notwithstanding the extent to which these self-reports may or may not mirror real behaviour change), it is important for health educators to know more about the quality of this nutritional information. This is particularly relevant given trends towards media globalization and the extent that many online health media editorial decision makers appear to be increasingly acting as purchasers of wholesale content from a heterogeneous mix of transnational information vendors [10]. As well, research on Internet providers suggests that website content is likely to be affected depending on whether the site is ‘not-for-profit’ or commercial [11]. And, the ability of non-profit websites to compete with commercial mega-portals in drawing audiences raises very real concerns about the long-term viability of non-commercial sources of health and nutrition information [11, 12]. This has particular relevance for health educators attempting to distribute scientifically based messages through non-profit and government-sponsored websites.
I find the above graph awkward—and could be better integrated with the following one.
As well, over the past decade, commercial sites that provide health and nutrition information have evolved under the pressures of intense globalization of the media, a growing public thirst for nutrition information and strong pressures from medical stakeholders such as physicians and pharmacists to be on the front lines of information provision. This has led to the creation of new specialized media health and nutrition information companies in North America, which are often affiliates of medical services companies engaged in providing support, such as technical, purchasing, software or disease databases to the medical or pharmaceutical industries. This apparent convergence between practitioners or companies supporting practitioners and health media is a new trend in the delivery of health information.
The critical factor in evaluating the potential for consumers to find useful high-quality information to guide their decision-making depends on the source and quality of the information provided. However, research into the quality of nutritional information on the web is limited.
Miles et al. [13] examined articles obtained from the first 50 websites from an Internet search using the terms ‘weight loss diets’, and compared their content with published clinical guidelines for obesity. They found that the ‘relevance and quality of the sites varied enormously’, with only three of these sites offering ‘sound dietary advice’. Most promoted dietary supplements or other ‘slimming’ aids', which were based on dubious physiological principles. In a similar comprehensive review of health information, Eysenbach et al. [14] reported that in 
70% of surveyed websites, the quality of health information ‘needed improvement’.
In the only nutrition quality study on the web by Canadian scholars [15], general key word searches were used to identify 167 different websites. These sites were subsequently analyzed according to the quality of nutrition information available. The study found that 45% of online nutrition information did not conform to the Canadian Guidelines for Healthy Eating.
This project is an audit of the type and quality of nutrition information available on the Internet sites most used by Canadians to obtain health and nutrition information. Its specific purpose was to determine the extent to which reporting on these widely used online health websites presents accurate information on healthy eating. We identified eight commonly used Internet health sites and then conducted an audit of these sites by systematically analyzing their content in relation to the Canada Food Guide [16]. The overall purpose of this study was to increase knowledge and understanding of the new informational landscape that is emerging on the Internet in relation to nutritional health content in order to provide policy makers with better communication and health promotion tools.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
Website selection
We identified the sites most used by Canadians to search for health and nutritional information by purchasing data from the only web-tracking service currently available in Canada, Media Metrix, which systematically follows the online navigational behaviour of a panel of 13 000 Internet users across the country [17]. The panel used by Media Metrix, although not randomly generated from the general population, by and large is representative of Canadian Internet users.
Using Media Metrix data for the 1-year period between October 2003 and November 2004, 50 health and nutrition sites were identified which accounted for >90% of the time spent at health websites by Canadians during this time. These were divided into non-commercial, commercial diet, commercial general health, commercial health channels of major news portals, commercial traditional print/broadcast media, commercial pharmaceutical industry produced or sponsored and ‘other’ commercial sites with non-diet and non-pharmaceutical health content.
Table I shows that non-commercial sites (consisting mainly of government or institution sponsored websites) accounted for approximately one-fifth of visits and hours among Canadians seeking health and nutrition information. Commercial sites accounted for 80% of visits and time spent by Canadians seeking health and nutritional information during the study period. Commercial diet sites accounted for the second highest proportion of visits (20.3%) and the highest proportion of hours (30.4%). Canadians do not appear to often visit or spend much time on commercial print/broadcast media sites or on pharmaceutical company-sponsored health sites.
|
From the list of the 50 most frequently visited websites, we selected a bilingual, government-sponsored health network, Canadian-Health-Network.com (CHN) because it is considered an impartial source of high-quality nutritional information by both consumers and experts and thus represents an unofficial ‘gold standard’ for the quality of nutritional information available online in Canada [18, 19].
We did not select any commercial print/broadcast media sites because there were only three of these in the top 50 most visited websites and these accounted for only 5% of visits. As well, we excluded pharmaceutical sites because these would likely be oriented towards drugs and health topics rather than nutritional topics and because these accounted for <10% of visits during the study period.
Instead, we focused on commercial health channels of news portals (as these accounted for 15% of visits) and commercial general health sites as these accounted for 20% of visits. Thus, we selected from commercial general health sites, commercial news portals with health channels and also selected one non-commercial public health site. These three site categories accounted for 60% of visits at the top 50 sites used by Canadians to assess health and nutrition information and half of all hours spent looking for nutrition and health information.
The eight websites chosen were as follows: About.com, Canadian-Health-Network.ca, Doctissimo.fr, iVillage.com (health and diet channels), MSN.com (health channel), ServiceVie.com, WebMD.com and Yahoo.com (health channel) (Table II).
|
Article sampling strategy
A preliminary investigation within our eight selected sites indicated that we needed to develop sampling strategies to fit each site depending on its combination of information architecture and editorial process. The type of architectural problems we encountered which entailed modification of our sampling strategy included extensive indexing within some sites, which resulted in the selection of too many navigational pages lacking usable nutrition information; the presence of menus and navigational headings containing our key words, which produced redundant and unusable results from searches, and the distribution of single articles across multiple web pages, which increased the likelihood of repeat sampling of a given article.
To make our original article selections, we used the search engine, Google.ca, because it has site-restricted capabilities and is the most widely used by Canadians (comScore qSearch 2004). We used broad keywords to keep article subjects as diverse as possible. Searches were conducted using the keywords: ‘nutrition’, ‘diet’ and ‘food’ (or ‘nutrition’, ‘aliment’ and ‘régime’ for French sites). The bilingual nature of this study required the selection of French search terms that were as equivalent as possible to their English counterparts. We tested several synonyms of our French keyword translations and compared the precision and recall of the web pages retrieved.
Because of variation in architectural and editorial processes and the varying impact on article relevancy due to use of broad keywords, we had to select a different number of original articles from each site in order to produce a relevant set of randomly selected articles at each site. Once the articles for each site were selected, they were printed. A bilingual research assistant scanned them and removed duplicates and navigational web pages or pages with <100 words. Database entries and recipes were also excluded to yield a pool of from 350 to 1000 raw articles with nutritional content only. Given that we obtained 350 articles from one site and given both that we desired to utilize all this material and that we wanted to have an equal number of articles from each website, we randomly selected 350 articles from each of the remaining seven websites for coding.
Content analysis
We developed a coding manual in order to systematically code nutritional and journalistic information for each randomly selected article.
Nutritional profile
We coded each article in terms of the extent to which the advice agreed, or disagreed with advice offered in the Canadian food guide.
Food guide agreement and disagreement.
In order to assess the nutritional accuracy of each article, the content was tested against a list of 12 basic Nutrition Recommendations for Canadians [16] (Table III). These were also chosen as they were used in the only other Canadian web content investigation of nutritional content [15]. To test the extent to which articles presented inaccurate information, conflicting recommendations were also coded. Articles were coded depending on whether they offered advice solely congruent with the Canada Food Guide (pure congruent advice), solely incongruent with the guidelines (pure incongruent advice), advice within the same article that was both congruent and incongruent with the guidelines (mixed advice) and no advice.
|
Using a pilot sample of articles, a nutritionist consultant trained the coders (in two intensive workshops) to apply the Health Canada Guidelines in order to recognize when content deviated from or concurred with them.
A 10% subsample of articles was randomly selected from each website to determine coding reliability. Two graduate students were trained to code the articles.
Reliability of coding was assessed using Cohen's kappa (
) [20]. Calculations were performed using SPSS Version 12.
Kappa scores ranged from 0.76 to 0.69 across the eight websites. Given that a score of 0.70 is acceptable, the reliability of coding was acceptable for seven sites and ‘borderline’ for the eighth site (About).
Journalistic practice
We included a section on journalistic practices to better understand how online journalists and editors select and produce nutrition information. We categorized the organizations responsible for content production, and the different kinds of sources cited within articles.
We also wanted to be able to attribute content quality to the type of provider to address a number of questions [1]. For example, were host websites themselves producing poor- or high-quality nutrition information? Or, are the emerging nutrition messages a patchwork of licensed content from several different sources? We anticipated that health organizations and professionals providing material would be of higher quality, but needed a way to determine the proportion of content on a website supplied by these kinds of organizations.
Readers evaluate the credibility of online information depending on its source. In general, news and reference sources are more trusted than commercial ones [21]. We examined the sources cited for each coded article. Finally, we also identified whether the articles had been written or reviewed by a health professional. A physicians' columnist is likely to be perceived as highly credible [22].
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
Of the 2770 articles coded, 858 (31.0%) offered advice purely congruent with that offered in the Canada Food Guide, 284 (10.3%) articles offered purely incongruent advice, 688 (33.9%) offered a mix of both congruent and incongruent advice and 940 (33.9%) had no nutritional information that could be coded in relation to the 12 guidelines.
Just <20% of the advice congruent with the Canada Food Guide was to promote achieving or maintaining a healthy body weight. Advice about promoting ‘variety’ in the diet, arguably one of the most important of the food guidelines, represented <15% of congruent advice in coded articles. Advice to reduce or moderate salt, coffee and alcohol similarly accounted for 15% of all congruent advice.
Another very important guideline advising consumers to choose dark green and orange vegetables and orange fruit accounted for <5% of congruent advice. This guideline and the guidelines promoting exclusive breastfeeding (accounting for <1% of congruent advice) were the least frequently mentioned in the coded articles.
Of the advice incongruent with the Canada Food Guide, about half was in relation to fruit and vegetable intake, about a quarter promoted exclusivity rather than variety (usually by promoting diets such as the Atkins diet) and just <20% of incongruent advice was in relation to the consumption of fat. Advice promoting the consumption of alcohol, salt or coffee intake was negligible (1.1% of all ‘incongruent’ advice) as was advice promoting formula for feeding infants (1.5%).
Table IV shows that except for the websites Doctissimo (52.9%) and Yahoo (43.1%), approximately one-third of articles contained no nutritional information that was codeable in relation to the 12 guidelines. From a quarter to one-third of articles on most websites contained mixed advice in relation to the 12 guidelines. Approximately one-third of articles on most sites contained purely congruent advice and from 10% to 15% presented purely incongruent advice.
|
Across the eight websites, CHN had the highest proportion of articles with purely congruent advice (37.4%) and the lowest proportion of articles with purely incongruent advice (2.6%). In contrast, About had the lowest proportion of articles with advice purely congruent with the 12 guidelines (22.1%) and the greatest proportion of purely incongruent articles (16.5%).
The main difference across the eight study websites was in where they obtained their nutrition articles. Thus, in order to generalize beyond these study sites, it is important to determine whether the differences in the quality of advice observed across these sites is driven by where and how they ‘source’ their nutrition articles.
A slightly greater proportion of articles congruent with guidelines was obtained from health channels (35.4% compared with the average of 31.1% from all sites). And, while 10% of articles presented advice purely incongruent with the 12 guidelines, this proportion was somewhat lower (6.6%) in articles obtained from health channels as was the proportion of articles with ‘mixed’ advice (17.7% on health channels versus an average of 25.1% across all sites).
In order to determine whether or not article writing or review by a health professional increased the likelihood of congruence with the 12 guidelines, the source of each articles was divided into reviewed or not reviewed for each channel. Within each channel, except for articles obtained through other sources, health professional authorship or review of produced articles with a greater proportion of congruent and a lesser proportion of incongruent advice, although the difference within all sources, except for health channels, was minor.
Articles authored or reviewed by a health professional and obtained through other sources were the least congruent in relation to the 12 guidelines. At the other extreme, articles authored or review by a health professional and obtained through ‘health’ channels were the most congruent in relation to the 12 guidelines.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
During the 1-year study period from the fall of 2003 to 2004, we determined in a sample of eight websites widely used by Canadians that
20% of visits and time spent by people seeking health and nutritional information were at non-commercial compared with 80% at commercial websites. We also found that the quality of nutritional advice obtained in articles available on our ‘gold’ standard non-commercial public health site (CHN) was consistently better than on the other seven commercial sites. We determined that the best of the eight websites sites differed from the worst, not so much in their ability to get the advice right, but more in their ability NOT to put articles with ‘bad’ advice on their sites. This likely indicates that the better sites employ more rigorous and better methods of article sourcing and editing which screen out low-quality articles.
Articles with the best advice came from designated health organizations with information online such as nih.gov, dietitians.ca, the Mayo Clinic, which was repurposed on study websites devoted solely to health and nutrition-related issues, which includes CHN. For example, our gold standard, CanadianHealthNetwork.ca, does not employ in-house writers or journalists but instead primarily relies on content originating from reputable medical and health sources such as the Dietitians of Canada. Articles written by health professionals, even those obtained from sites with an employed ‘in-house’ doctor or pharmacist, were the next-best source of advice. Examples of this kind of content creation model include WebMD and Doctissimo.fr.
In contrast, we found that the worst advice was found in articles reviewed or written by professionals or ‘specialists’ on general news channels (i.e. other sites), such as About.com, iVillage.com, MSN.com and Yahoo.com. These sites tended not to produce their own content and sometimes used material from industry-affiliated sites such as the Dairy Producers of Canada. For example, only 17.4% of articles of this type were purely congruent with advice from the CFG and 26.1% of articles of this type were purely incongruent. Indeed, these channels do not appear to employ the rigorous methods used by the channels devoted solely to health or sites that repurpose information from health organizations because they are one part of larger media organizations. They also appear to accept health information from a wider range of sources, which lowered congruence with the guidelines.
Health policy makers concerned with promoting better nutrition can take away several lessons from this research. First, the research makes clear that positive messages about the promotion of breastfeeding are non-existent in these websites. This is a concern and indicates that this particular guideline in Canada must be made more prominent on health and nutrition websites. Second, it is important to warn consumers about the impact of diets and their potential to compromise advice about eating a variety of foods. Third, this research points to the need to better develop methods to conduct this type of web-based research into the accuracy of nutritional information. It may be important to also think about developing these methods into practical and effective quality rating tools in order to guide health consumers through the maze of, often unreliable.
The limitations of the study are that we sampled only eight websites so that results may not be generalizable to all sites. We did not sample diet sites which we know are widely used by Canadians. We limited our non-commercial sample to one site only further limiting our ability to generalize. And, for our gold standard comparison, we relied mainly on the congruence or incongruence with Canada Food Guidelines, which may be an overly rigorous standard to comparison. Finally, inter-rater reliability was just acceptable for most sites and borderline for one site.
The strengths of this study are that we looked at sites in two languages and those that were most popular. We also systematically sampled and coded a large number of articles from sites to facilitate rigourous comparisons. We also compared coded advice with a gold standard. Finally, we have contributed to the development of methods to improve the rigour of content analyses of nutritional information conducted on websites.
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
Canadian Social Sciences and Humanities Research Council. New investigator award from the Canadian Institutes for Health Research, a scholar award from the Michael Smith Foundation for Health Research in British Columbia and the Canada Research Chair program (to A.O.).
|
Conflict of interest statement |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
None declared.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementReferences |
|---|
1. Begley A, Cardwell G. The reliability and readability of nutrition information in Australian women's magazines. Aust J Nutr Diet (1996) 53:160–6.
2. Radimer KL. Are nutrition messages in popular women's magazines consistent with dietary recommendations? Aust J Nutr Diet (1996) 53:107–13.
3. Radimer KL, Hill JM. Nutrition messages on Australian television shows. Aust J Nutr Diet (1996) 53:117–23.
4. Goldberg J. Nutrition communication in the 21st century: what are the challenges and how can we meet them? Nutrition (2000) 16:644–6.[CrossRef][ISI][Medline]
5. Readership Institute. American Newspaper Readership Survey (2001) Evanston, IL: Media Management Center, Northwestern University.
6. Stevenson K. Health information on the net. Can Soc Trends (2002) 2004:7–10.
7. Ellison J. Household Internet use survey. The Daily—Statistics Canada (2002) 2–4.
8. Crowley D. Where are we now? contours of the Internet in Canada. Can J Comm (2002) 27:469–507.
9. Dietitians of Canada. The Media and the Message: Identification of Media Habits of Consumers (2002) Dietitians of Canada. 1–48.
10. Boczkowski P. The development and use of online newspapers: what research tells us and what we might want to know. In: Handbook of New Media—Lieuvrouw LA, Livingstone S, eds. (2002) London: Sage Publications.
11. McMillan SJ. Who pays for content? Funding in Interactive Media. JCMC (1998) 4:64–73.
12. Hargittai E. Standing before the portals: non-profit websites in an age of commercial gatekeepers. J Policy Reg Strategy Telecom Inf Media (2000) 2:543–50.[CrossRef]
13. Miles J, Petrie C, et al. Slimming on the Internet. J R Soc Med (2000) 93:254–7.
14. Eysenbach G, Powell J, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA (2002) 287:2691–700.
15. Davison K. The quality of dietary information on the world wide web. J Can Diet Assoc (1996) 57:137–41.[ISI]
16. Health Canada. Using the Food Guide (1997) Ottawa, ON: Office of Nutrition Policy and Promotion, Health Canada.
17. comScore Media Metrix Canada. Key Measures Report—Health (2004) Toronto, ON: comScore Media Metrix Canada.
18. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res (2001) 16:671–92.
19. Peterson G, Aslani P, Williams KA. How do consumers search for and appraise information on medicines on the Internet? A qualitative study using focus groups. J Med Internet Res (2003) 5:e33.[Medline]
20. Lombard M, Snyder-Duch J, Bracken C. Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects. Available at: http://www.temple.edu/sct/mmc/reliability/ 2005.
21. Wikgren M. Health discussions on the Internet: a study of knowledge communication through citations. Libr Info Sci Res (2001) 23:305–17.[CrossRef]
22. Eastin MS. Credibility assessments of online health information: the effects of source expertise and knowledge of content. JCMC (2001) 6:24–9.
Received on July 5, 2006; accepted on July 9, 2007
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||