Health Education Research Advance Access originally published online on September 7, 2007
Health Education Research 2008 23(3):567-575; doi:10.1093/her/cym044
Developing a consumer evaluation tool of weight control strategy advertisements on the Internet
Division of Social Research in Medicines and Health, School of Pharmacy, University of Nottingham NG7 2RD, England, UK
* Correspondence to: K. Luevorasirikul. E-mail: paxkl{at}nottingham.ac.uk
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
To develop two evaluation tools for weight loss and weight gain advertisements on the Internet in order to help consumers to evaluate the quality of information within these advertisements. One hundred websites identified by Internet search engines for weight loss and weight gain strategies (50 websites each) were evaluated using two specific scoring instruments, developed by adapting questions from the ‘DISCERN’ tool and reviewing all related weight control guidelines and advertising regulations. The validity and reliability of the adapted tools were tested. Our evaluation tools rated the information from most websites as poor quality (70%). In the case of weight loss strategies, statements about rapid (18%) and permanent (28%) weight loss caused concern as well as lack of sensible advice about dieting and a lack of product warnings (84%). Safety concerns relating to weight gain products were the lack of warnings about side effects in products containing steroids and creatine (92%). The adapted tools exhibited acceptable validity and reliability. Quality of information within weight control advertisements on the Internet was generally poor. Problems of false claims, little advice on healthy ways to modify weight and few warnings on side effects have been highlighted in this study.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
The media serve as powerful transmitters of body image ideals which are slimness for women and muscularity for men [1]. Consequently, women perceive themselves too fat and want to lose weight, whereas men consider themselves too thin and want to gain weight or build more muscle [2]. It has been suggested that the pursuit of a healthy lifestyle, such as sensible eating and increasing exercise, is required for people who want to lose fat and/or gain muscle. However, consumers need to maintain these changes for extended periods of time. Therefore, a number of weight control products and programmes are becoming the option for those who want a fast result [3]. It has been reported that, in 2000, the US population spent almost $35 billion on weight loss products and programmes [4]. When considering weight control strategies, advertisements play an important role in informing consumers about products or programmes. Evidence has shown that most consumers select products based on information provided within advertisements [3]. However, other studies found that advertisements on weight loss products may often be imprecise and ambiguous [5].
It is evident that the Internet is becoming a key source of health and wellness information. For example, an online survey found that ‘diet, vitamin and nutritional supplements’ were the third most popular health topic sought on the Internet [6]. A study of Internet users in Germany, France, the United States of America and Japan showed that most of the Internet users perceived online health information to be of high quality [7]. It is also apparent that consumers are seeking health products, as well as information, from online suppliers. A survey of 1000 US adults found that 
20% reported purchasing over-the-counter medicines and prescription drugs on the Internet [8]. In relation to weight change strategies, a recent survey of more than two million US online consumers showed that >60% searched for information about weight loss diets and programmes on the Internet, whereas 50% used offline resources [9]. This study also found that >40% of the users reported being satisfied with the weight loss websites and almost three-quarters reported that these websites had a great impact on increasing their compliance with a weight loss programme.
In the case of weight loss products, a study by Bimal et al. [10] showed that online advertisements failed to inform consumers about side effects and the recommended dosage and contained inappropriate statements that were not in accordance with government standards. More specifically, the British Advertising Standards Authority (ASA) addressed two main problems of slimming advertisements in printed publications, which were ‘miracle claims’ and ‘exaggerated claims and guarantees’ [11]. Similarly, a report published by the US Federal Trade Commission (FTC) showed that ‘testimonials', ‘fast results’ and ‘guaranteed results’ were the most commonly made claims in weight-loss advertising [4].
Although the problems of weight gain advertisements have not been well researched, Philen's survey regarding nutritional supplement advertisements in health and bodybuilding magazines showed that most products used claims like ‘muscle growth’ or ‘enhanced strength’ (59%) but did not contain any side effect warnings (90%) [12]. Moreover, a survey by ASA found that many bodybuilding product advertisements used some claims without any scientific proof, as well as the use of ‘before and after’ pictures [13].
In 2004, the Department of Health for England and Wales launched a 3-year plan for improving quality of information: one of the main strategies is to empower people by enhancing their skills in evaluating health information [14]. To our knowledge, resources about evaluating Internet information are mostly found in the form of criteria or guidelines. We realized that using these resources to justify the quality of websites expends time, money and effort. Therefore, we developed tools which are user-friendly and could be available for consumers to use online. At present a number of evaluation tools have been developed for helping consumers to rate the quality of health information on the Internet; however, none has been created and tested for specific topics or products [15].
The main aim of this study was to develop tools for evaluating the quality of online weight loss and weight gain advertisements in order to enable consumers to gauge the reliability of promotional information about weight control products and to explore the problems of misleading advertisements on the Internet.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
Evaluation tools
In the first stage of development, we adapted questions from the DISCERN tool [16]. The DISCERN tool was originally developed to help consumers to evaluate the quality of information about treatment choices in print publications. The tool provided a summary score rating the quality from 1 (low) to 5 (high). Our aim was to create a short tool specific to weight control products and strategies; we selected questions about mode of action, benefits and risks of the product, areas of uncertainty and an overall rating of the quality of information from the original tool. We used current UK regulations and US guidelines for advertising of weight control products to develop questions about misleading information, which included seven types of weight loss claims and three types of weight gain claims [4, 11, 13]. We labelled these claims as ‘problematic’ because we were concerned that these claims may mislead consumers about the efficacy of the product. Moreover, we incorporated sections about safety of the product, scientific proof, useful information and instructions for consumer. This resulted in two evaluation tools—one for weight loss and one for weight gain products. Both tools contain three pages consisting of nine areas of concerns (Table I). Supplementary questions were added to seven of nine sections, which aimed to clarify the main aim of each section for the consumer and to enhance the consistency and simplicity of the scoring system. An example of a question about the risks of a product can be seen in Table II.
|
|
Regarding the safety of weight change products, the questions about some harmful ingredients that might be contained in the product were included in this section (Table III). Studies have shown that these ingredients could have detrimental effects on human health. For instance, Ephedra (or Ma Huang), commonly found in weight loss products, can increase blood pressure and increase risk for stroke [17]. Bodybuilding products containing creatine can cause renal damage [18].
|
The website sample
Keywords and phrases, like ‘dieting’ and ‘build muscle’ that might be used by consumers were generated by discussion among the research team to simulate searching for websites related to weight loss and weight gain products (Table IV). All the keywords were put into the most popular search engines among adolescents [YahooTM (http://www.yahoo.com and http://www.yahoo.co.uk), GoogleTM (http://www.google.com and http://www.google.co.uk) and Ask JeevesTM (http://www.ask.com and http://www.ask.co.uk)] [19]. Only the first 10 websites appearing in the first page of results for each key word/phrase were selected resulting in a total of 540 websites for weight loss products and 360 websites for weight gain products. Then websites which contained the same product, were duplicate, were non-functioning or only provided information about weight loss but listed no product or programme were excluded from the sample. The remainder was then grouped by the results from each of the six search engines. Eight to nine websites were purposively selected from each search engine in order to generate a sample of 100 websites to represent wide varieties of weight change products (50 for weight loss and 50 for weight gain products). It should be noted that ‘website’ in this context was equivalent to ‘advertisement’ in the printed media, although it is recognized that some websites might present themselves as ‘advertorials', which include information which may be purely promotional or an effort to describe evidence for use of the product or strategy.
|
Box 1. How to rate a final score from scores from each section
|
Scoring system
There were two coders in this study. The main coding was performed by K.L. and the second coding for the intercoder reliability test was performed by another researcher.
The generated sample of websites was analysed using scoring tools designed for each type of strategy (weight loss or weight gain). A product from each website was used for analysis of the quality of the website. In the case of websites containing more than one product, only one product was purposively selected that represented the range of products on that website.
The tools were developed to evaluate the overall quality of content on websites by totalling a rating score from all nine sections. Sections 3–9 comprised a set of supplementary questions and an overall rating scale. In sections with more than one question, each supplementary question was completed by answering either ‘yes', ‘no’ or ‘not applicable'. At the end of each section, the ‘yes’ or ‘no’ responses were totalled to give a score for that section. Based on the total score from each section, the final rating for overall quality of websites was ranked as ‘low', ‘moderate’ or ‘high as illustrated in Box 1. It should be noted that the approximate time used for evaluating each website was 5 min.
Validity and reliability assessment
To assess validity the tools, content validity was assessed by piloting the instruments on a small number of websites and refined through iterative discussion within the research team coupled with the reflection process within the research team before the main study. Moreover, the researcher reviewed and compared the questions about misleading claims to the standard criteria for advertising weight change products (the British advertising codes [11, 13] and the US guidelines [4] for weight change strategies) in order to ensure that all the plausible claims that might have been found in weight change advertisements were covered.
In order to test the reliability of the instruments, two reliability tests were performed by using Spearman's rho: intercoder reliability and test–retest reliability. A Spearman's rho 
0.7 is considered good reliability [20]. For intercoder reliability, 10 websites were randomly selected for a second researcher to test using the tools and then the results from both users were compared and tested. The results for intercoder reliability showed that Spearman's rho for weight loss strategies was 0.730 and for weight gain strategies was 0.722.
After 1 month, 12 websites were selected to retest the reliability of the tools. It is acknowledged that the transient nature of websites could be problematic, and some websites had undergone changes which will be commented upon later but most were in the same form. Most of the re-test sample websites were drawn at random; however, a few were purposively selected to ensure the sample included ingredients where there was particular concern. Spearman's rho was used to test the reliability by comparing the total scores from the original scores and the retest scores. When retested with a sample of 12 websites, Spearman's rho for the weight loss tool was 0.847 and for the weight gain tool was 0.936. Therefore, the results from both reliability tests were acceptable to us at this stage of development.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
Our sampled websites were grouped into three different types of websites: retail website (offers a variety of products); company website (offers their own brand product) and personal website (offers personalized weight change programme). Overall, our websites provided poor to moderate quality of information, as shown in Fig. 1. It should be noted that none of the websites merited a maximum score of five. In terms of types of products or programmes found in our websites, most weight loss strategy websites (56%) were for slimming pills and most weight gain strategy websites (38%) were for nutritional supplements containing creatine, chromium, amino acids or vitamins and minerals. Seven types of false claims were found in slimming product websites (see Fig. 2). The most frequent problematic claim was ‘rapid weight loss’ (28%), followed by ‘permanent weight loss’ (18%) and ‘block fat and carbohydrate’ (18%). Conversely, in weight gain strategies, 50% of the websites used claims like ‘increasing strength’ or ‘muscle growth’ (see Fig. 3). Moreover, 40% of them included results from studies but gave no reference sources. Furthermore, there was a lack of product warnings: only 16% of weight loss products and 8% of products containing steroids and creatine carried product warnings.
|
|
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
Our findings show that the majority of weight control products’ websites contained poor quality information, particularly inappropriate statements concerning efficacy claims for the products. As far as existing evaluation tools are concerned, our instruments were specifically designed for consumers who search for weight loss or weight gain strategies. Furthermore, each section in our tools included supplementary questions which intentionally aimed to help consumers decide whether the website had provided enough information or not. Our tools exhibited reasonable reliability and validity at this stage of development. According to the keywords used for searching websites, our samples included a wide range of products and programmes claiming to help people to lose weight or gain muscle.
Our study has several limitations. We did not have any major difficulty using these evaluation tools to analyse the content of advertisements. However, we found that some questions had areas of uncertainty and there was inconsistency when rating scores for this question. When we retested our tool, we found that some websites did change their product ingredients, as we presumed that distributors may have become aware of selling products containing prohibited substances such as ephedrine and steroids. Finally, we only chose one product to represent each website reviewed. It is possible that the product we selected was not representative of the website: selecting a different product may have resulted in a different rating score.
When comparing our findings with several related studies, similar themes emerged. In the case of false claims, ‘fast results’ was the most commonly found claim in our weight loss websites, which was similar to the finding from a FTC report [4]. In case of bodybuilding advertisements, our results were consistent with a previous study in many aspects: the most frequently mentioned ingredients; the most claims; and a lack of steroid products warning were common [10]. Moreover, our findings lead us to agree that advertisers should inform consumers more about advice on dieting, provide product warnings and provide contradictory evidence as outlined in Mason's article, the British ASA survey and Miles’ study [5, 11, 21].
Although there are legitimate guidelines regarding the advertising of weight loss and weight gain strategies, there are still a number of complaints about problematic claims made for these products and programmes [22]. Regulation of these advertisements is very difficult in practice: sites change regularly and globalization of the Internet makes any kind of meaningful sanction almost impossible [23]. The DISCERN tool was developed in order to help the public evaluate the quality of health information and our work builds upon this. By raising public awareness of these issues, and the availability of specific tools that can help them, consumers should be better able to evaluate the quality and safety of online information about weight control strategies. We were also concerned that during the first use of the tools, consumers may find that our instruments are difficult to use and time consuming. However, we believe that consumers will gain benefits from using our tools by learning and developing skills to justify whether information within advertisements is good or bad without using the instruments. Moreover, in order to enhance the accessibility of our tools, we plan to disseminate the instruments by making them widely available on the Internet and checking their inclusion in the results of simulated search engine exercises. Recent controversy over new UK guidelines from the National Institute for Health and Clinical Excellence regarding obesity, which includes a recommendation for obese adolescents to have access to stomach stapling surgery in extreme circumstances, highlights the importance of these issues and the timeliness of this study [24]. Renewed media attention may prompt more people, including more adolescents, to seek information on this topic.
We are reporting the initial development of these tools and interesting preliminary findings in simulated searches that might be of concern to health care consumers, practitioners and policymakers/regulators. The next stage of development should examine the practicality of our tools when being used by consumers who are interested in purchasing weight change products. This will enable a researcher to understand the problems, which might arise from using these tools, and to adjust the contents of questions to become more approachable and suitable for people across different age groups.
|
Conflicts of interest |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
None declared.
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
We would like to thank Dr Helen Boardman and our colleagues in the Division of Social Research in Medicines and Health at the University of Nottingham for their support with this study. The completed evaluation tools for weight loss and weight gain products are available at www.nottingham.ac.uk/csrhhc.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConflicts of interestAcknowledgementsReferences |
|---|
1. Xie B, Liu C, Chou CP, et al. Weight perception and psychological factors in Chinese adolescents. J Adolesc Health (2003) 33:202–10.[CrossRef][Web of Science][Medline]
2. Stanford JN, McCabe MP. Body image ideal among males and females: sociocultural influences and focus on different body parts. J Health Psychol (2002) 7:675–84.
3. Blanck HM, Khan LK, Serdula MK. Use of non-prescription weight loss products: results from a multistate survey. JAMA (2001) 286:930–5.
4. The Federal Trade Commission. "Reports. 2002" Weight-Loss Advertising: An Analysis of Current Trends. Available at: http://www.ftc.gov/bcp/reports/weightloss.pdf. Accessed: 21 May 2006.
5. Mason P. OTC weight control products. Pharm J (2002) 269:103–5.
6. Pew Internet and American Life Project. Online Health Search 2006. Available at: http://www.pewinternet.org/pdfs/PIP_Online_Health_2006.pdf. Accessed: 16 November 2006.
7. Harris Interactive. 4-Country Survey Finds Most Cyberchondriacs Believe Online Health Care Information is Trustworthy, Easy to Find and Understand. Available at: http://www.harrisinteractive.com/news/newsletters/healthnews/HI_HealthCareNews2002Vol2_Iss12.pdf. Accessed: 7 August 2007.
8. Burns E, Shoppers Shift to Online. Available at: http://www.clickz.com/3564781. Accessed: 7 August 2007.
9. ComScore Networks. Press Release: Sixty Percent of Consumers Researching Weight Loss use the Internet, Seeking Information on Weight Loss Programs, Diets and Medications. Available at: http://www.comscore.com/press/release.asp?id=997. Accessed: 16 November 2006.
10. Bimal H, Ashar MD, Redonda GM, et al. A critical evaluation of internet marketing of products that contain ephedra. Mayo Clin Proc (2003) 78:944–6.
11. The Advertising Standards Authority. A Survey of Slimming Advertisements. Available at: http://www.asa.org.uk/NR/rdonlyres/D4040C0F-103F-4C09-986D-9F5CFEABEE8B/0/ASA_Slimming_Report_Mar_1995.pdf. Accessed: 21 May 2006.
12. Philen RM, Ortiz DI, Auerbach SB, et al. Survey of advertising for nutritional supplements in health and bodybuilding magazines. JAMA (1992) 268:1008–11.
13. The Advertising Standards Authority. "Search: Bodybuilding" General: ASA Flexes its Muscles. Available at: http://www.asa.org.uk/asa/news/news/2004/ASA+flexes+its+muscles.htm. Accessed: 21 May 2006.
14. Department of Health. Better Information, Better Choices, Better Health: Putting Information at the Centre of Health. Available at: http://www.dh.gov.uk/assetRoot/04/09/85/99/04098599.pdf. Accessed: 21 May 2006.
15. Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet. BMJ (2002) 324:598–602.
16. Shepperd S, Charnock D, The DISCERN Instrument. Oxford: University of Oxford. Available at: http://www.discern.org.uk/discern.pdf. Accessed: 21 May 2006.
17. Dwyer JT, Allison DB. Dietary supplement in weight reduction. J Am Diet Assoc (2005) 105:S80–6.[CrossRef][Web of Science][Medline]
18. Pritchard N, Kalra P. Renal dysfunction accompanying oral creatine supplements. Lancet (1998) 351:1252–3.[CrossRef][Web of Science][Medline]
19. Gray NJ, Klein JD, Cantrill JA, et al. "How Do You Spell ‘Gonorrhea’?" Adolescents’ Health Literacy and the Internet. Abstr Acad Health Serv Res Health Policy Meet (2002) 19:21.
20. Litwin MS. How to Measure Survey Reliability and Validity: Volume 7 of the Survey Kit (1995) Thousand oaks, CA: Sage publications. 8–13.
21. Miles J, Petrie C, Steel M. Slimming on the internet. J R Soc Med (2000) 93:254–7.
22. The Federal Trade Commission. "Do the Right Thing" (Apologies to Spike Lee). Available at: http://www.ftc.gov/speeches/muris/030211rightthing.htm. Accessed: 21 May 2006.
23. Theodosiou L, Green J. Emerging challenges in using health information from the Internet. Adv Psychiatr Treat (2003) 9:387–96.
24. National Institute for Healthcare and Clinical Excellence. Obesity: The Prevention, Identification, Assessment and Management of Overweight and Obesity in Adults and Children (2006) NICE. Available at: http://www.nice.org.uk/guidance/CG43. Accessed: 14 December 2006.
Received on July 20, 2006; accepted on June 11, 2007
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. M. Whelan, T. M Jurgens, S. K Bowles, and H. Doyle Efficacy of Natural Health Products in Treating Osteoporosis: What is the Quality of Internet Patient Advice? Ann. Pharmacother., May 1, 2009; 43(5): 899 - 907. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||