Health Education Research Advance Access originally published online on May 20, 2004
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Health Education Research, Vol. 19, No. 4, 440-446,
August 2004
© 2004 Oxford University Press
Learning to DISCERN online: applying an appraisal tool to health websites in a workshop setting
1 The Kendra Consortium, Creighton House, Sandwich CT13 9HN and 2 Health Sciences, University of Oxford, Centre for Professional Development, Department of Continuing Education, Oxford OX1 1PT, UK
3 Correspondence to: D. Charnock; E-mail: deborah.charnock{at}lineone.net
 |
Abstract |
|---|
 Top Abstract BackgroundMethodResultsDiscussionReferences |
|---|
This study examined the application of DISCERN—validated criteria for judging the quality of printed information on treatment—to online health information in a workshop setting. A survey was conducted amongst 57 participants attending DISCERN Online workshops. Participants were health information users—health care and information providers, consumers (patients/carers), and consumer representatives. Workshops involved using DISCERN to appraise a health website. Participants completed questionnaires before and after the workshop, and at 2 months follow-up. Responses revealed that participants accessed online health information for professional (85.7%) and personal (75%) reasons. Less than half (41%) had applied some form of quality criteria to online information prior to attending the workshop. Despite varying levels of expertise, participants found DISCERN and the supporting materials accessible. The majority (96.2%) agreed DISCERN would help users discriminate between high- and low-quality online treatment information, and would be applicable to a wide variety of such information. At follow-up, most (89.6%) reported that their attitude to consumer health information of all types had changed—mostly becoming more critical or systematic. It is possible that general schemes such as DISCERN will provide users with simple and flexible skills for dealing with the wide range of treatment information available.
This study examined the application of DISCERN—validated criteria for judging the quality of printed information on treatment—to online health information in a workshop setting. A survey was conducted amongst 57 participants attending DISCERN Online workshops. Participants were health information users—health care and information providers, consumers (patients/carers), and consumer representatives. Workshops involved using DISCERN to appraise a health website. Participants completed questionnaires before and after the workshop, and at 2 months follow-up. Responses revealed that participants accessed online health information for professional (85.7%) and personal (75%) reasons. Less than half (41%) had applied some form of quality criteria to online information prior to attending the workshop. Despite varying levels of expertise, participants found DISCERN and the supporting materials accessible. The majority (96.2%) agreed DISCERN would help users discriminate between high- and low-quality online treatment information, and would be applicable to a wide variety of such information. At follow-up, most (89.6%) reported that their attitude to consumer health information of all types had changed—mostly becoming more critical or systematic. It is possible that general schemes such as DISCERN will provide users with simple and flexible skills for dealing with the wide range of treatment information available.
|
Background |
|---|
|
TopAbstractBackgroundMethodResultsDiscussionReferences |
|---|
The advent of the Internet has increased public access to health information through online resources such as websites and Internet discussion groups (Cline and Haynes, 2001
; Eysenbach and Kohler, 2002; Ferguson, 2002). Concerns about quality have led to a proliferation of tools and guidelines for producing and evaluating online information (Kim et al., 1999; Gagliardi and Jadad, 2002). Central to this work is the belief that the Internet represents a new information medium requiring new quality standards and raising unique evaluation issues (Lindberg and Humphreys, 1998; Tweddle et al., 1998). Established tools and guidelines for evaluating printed health information exist [e.g. (Oxman et al., 1993; Coulter et al., 1998; Entwistle et al., 1998)], but it is often assumed that these are not relevant to electronic information due to the unique way it is delivered and handled. Such assumptions are premature (Shepperd and Charnock, 2002). Little is known about how online health information is actually used and there has been no attempt to examine how validated criteria developed for use with print materials fare when applied to online information.
DISCERN (Charnock et al., 1999) is a standardized set of criteria for judging the quality of health information written for the public on treatment choices. The tool has good levels of inter-rater agreement and validity, and consumers were involved at every stage of development (Charnock et al., 1999). DISCERN provides health information users (professionals and consumers) with guidelines for appraising information and outlines standards for information producers. Reliability and credibility are assessed from aspects of the information content such as explicit sources, dates, balance and lack of bias, risks and benefits of treatment, and no treatment options. DISCERN excludes questions about presentation or delivery as these issues are covered by well-established guidelines [(Flesch, 1948; Wright, 1998) and the Royal National Institute for the Blind See it Right guidelines: http//www.rnib.org.uk/access/welcome.htm], and are largely irrelevant from a user perspective as users have heterogeneous preferences and can simply choose not to use inaccessible information (Hardy, 1999; Delamothe, 2000). Detailed hints are a key component of the DISCERN tool, as they enable inexperienced users to ‘unpack’ the criteria and to apply them consistently. Additional support is available in the handbook (Charnock, 1998) and through workshops which have been evaluated in a joint project with the Critical Appraisal Skills Programme (Clisby and Charnock, 2000). A five-star rating scheme has also been recently developed (Shepperd et al., 2002).
DISCERN was developed using printed information. However, as the tool is concerned with the content of written treatment information, it should be applicable to online information. The DISCERN Online website (www.discern.org.uk) was launched in May 1999. The aim was to provide online access to the DISCERN instrument and handbook, and to explore the application of the tool to online information through a website survey. A review of the website at 6 months post-launch revealed a high level of visitor activity: 113 875 total hits on the website and 6930 user sessions. Anecdotal feedback indicated a highly positive response to DISCERN Online and confirmed its use as an online quality tool in a variety of international settings. However, only 30 online evaluation forms providing data of variable quality were returned during this period. Our experience demonstrates the difficulties of conducting such research online.
We therefore extended the DISCERN workshop format to online treatment information. This enabled us to conduct a survey of the selection of online health information amongst health information users (professionals and consumers), to explore users' experiences of applying DISCERN to online treatment information and to assess the impact of a DISCERN Online workshop on the broader context of health information use.
|
Method |
|---|
|
TopAbstractBackgroundMethodResultsDiscussionReferences |
|---|
Venues
Five half-day workshops were held at venues with facilities for group Internet training in London (three workshops) and Oxford (two workshops). Each venue could accommodate 10–20 participants. Workshops were free to anyone interested in health information and DISCERN, and the only pre-requisite was some minimal experience of computers and the Internet.
Recruitment
The launch of the DISCERN website revealed that publicizing a health survey online can generate high levels of inappropriate queries (e.g. requests for advice on specific health problems). We therefore opted to recruit participants through a mixture of online and print methods. Workshops were publicized online through the DISCERN website, the Centre for Health Information Quality website (http://www.hfht.org/chiq/) and health professional mailing lists. Printed publicity included poster displays close to workshop venues, features and advertisements in popular and professional periodicals, and flyers sent to various national and local consumer health groups, educational and consumer health organizations in England, all Community Health Councils in England and Wales, DISCERN database contacts, and voluntary and community groups in the Oxford area.
Selecting a target website
We chose breast cancer as the target topic, as the condition is well known and treatment information of varying quality is widely available. We reviewed UK-based websites providing breast cancer treatment information specifically for the public. Material was rated for its suitability for training in terms of length, ease of access and interesting content issues relating to DISCERN. A health website developed by a UK-based breast cancer charity was selected as it appeared to be good quality in terms of its clear design and reputable producer, yet the content of the treatment information had limitations.
Workshop format, procedure and materials
Each workshop lasted 3 hours, was facilitated by D. C., and consisted of an overview, a practical session and a group discussion. During the practical session, participants used DISCERN to appraise treatment information available on the target website.
Participants were sent the URLs of the DISCERN website and target website prior to the workshop. At the workshop, each participant worked independently on an individual PC linked to the World Wide Web to assess the quality of the target website using DISCERN. Two additional websites providing good quality breast cancer treatment information (selected according to DISCERN ratings published by NHS Direct Online: www.nhsdirect.nhs.uk) were available for informal comparison after the practical session. Participants then discussed their DISCERN ratings within a small group. The workshop concluded with a facilitated group discussion.
Data collection
Prior to registering for a workshop, participants were informed they would be required to complete three questionnaires: one prior to the workshop, one at the conclusion of the workshop and one 2 months after the workshop. The questionnaires asked participants for their demographic details, views and experiences of the Internet (including online health information), and views of DISCERN as a tool for rating the quality of online and other health information on treatment choices. Data was primarily in the form of yes/no or pre-coded response categories. Up to two reminders were issued to non-responders.
|
Results |
|---|
|
TopAbstractBackgroundMethodResultsDiscussionReferences |
|---|
Response Rates
Sixty-nine people registered for a DISCERN Online workshop and 57 attended. The response rates for the three questionnaires were 56 (98.2%), 53 (93%) and 48 (84.2%).
Background Information
Participants
Results reported in this section are based on responses from 56 participants who completed the pre-workshop questionnaire. The mean age of these participants was 42 years (SD = 12). Forty-four (78.6%) were female and 48 (85.7%) described their ethnicity as white. Fifty-three (94.6%) had completed tertiary education and 49 (87.5%) were in employment. Forty-eight participants (85.7%) were regular newspaper readers: the majority (36 of 56; 64.3%) were exclusively broadsheet readers and five (8.9%) read tabloids only. [Type of newspaper read was included as an indicator of information-seeking behaviour (Jones et al., 1999).]
Thirty-eight (67.9%) participants were paid health care and information providers, 16 (28.6%) were paid or voluntary representatives of patient groups and two participants were patient/carers with no affiliations. Sixteen (28.6%) participants had used DISCERN prior to the workshop. Most had heard about DISCERN workshops through a professional organization or colleague (32; 57.1%).
All participants had experience with the Internet. The majority (38; 67.9%) had used the Internet every day in the 6 months preceding the workshop. Most accessed the Internet from the workplace (36; 64.3%) or from home (13; 23.3%). Preferences for handling online information were equally distributed: 21 (37.5%) participants preferred reading a printout, 17 (30.4%) preferred reading onscreen and 18 (32.1%) had no preference. Many reported that the choice was determined by the length and complexity of the information.
Seeking online health information
Throughout the following sections, participants could nominate or describe more than one category when providing additional details of their views and experiences.
Fifty-two participants (92.9%) had previously used the Internet to seek online health information. Details of their experiences are summarized in Table I.
|
Forty-eight (85.7%) participants had used online consumer health information in the 6 months preceding the workshop (i.e. information written specifically for patients, carers and members of the public) and 23 participants (41.1%) had employed some form of quality control to select information. Details are summarized in Table II. (It is interesting to note that six participants had used online information provided by individual patient/carer sources.)
|
Use of other forms of consumer health information
Forty-six participants (82.1%) had used other (non-online) sources of consumer health information in the 6 months preceding the workshop. Sources used are presented in Table III.
|
Evaluation of DISCERN Online and the workshop
Results reported in this section are based on responses from 53 participants who completed the questionnaire at the end of the workshop. The majority agreed that DISCERN would help users to discriminate between high and low quality online health information (51; 96.2%) and would be applicable to a wide variety of online consumer health information on treatments (50; 94.3%). Most (44; 83%) agreed DISCERN would be a useful tool for health information users and providers, and all but one (52; 98.1%) intended to use DISCERN in future.
At each workshop, discussions revealed many participants had been impressed that despite varying levels of expertise, they had independently achieved similar DISCERN scores—including the overall quality rating of the target website (which was low). This consensus contrasted with participants' varied opinions about the provenance and presentation of the website, and they were keen to explore the way DISCERN enabled them to judge information reliability and credibility without reference to such criteria. A few expressed concern that the time required to complete the appraisal would make it inaccessible to individual consumers (although it is interesting that these comments came from health professionals and consumer representatives rather than consumers themselves).
Forty-one (77.4%) participants felt the workshop had changed their attitude to consumer health information in general. The development of more critical attitudes is also demonstrated in participants' judgements about the general quality of health information available to the public on the Internet: fewer participants rated information as ‘good’ or ‘fair’ after the workshop (n = 45; Table IV).
|
Follow-up
The results in this section are based on responses from 48 participants who completed the questionnaire at 2 months follow-up. Forty-five (93.8%) agreed DISCERN was suitable for use with online health information. The majority (43; 89.6%) again reported that participation in the DISCERN Online workshops had changed their attitudes to consumer health information in general, most commonly by making them more critical (25; 52.1%) and systematic (8; 16.7%) when selecting or producing information.
Twenty-four (50%) had used DISCERN since the workshop. Applications are outlined in Table V. As can be seen, uses were diverse and not confined to online information. Several of these participants reported using DISCERN for more than one purpose and with more than one type of information.
|
|
Discussion |
|---|
|
TopAbstractBackgroundMethodResultsDiscussionReferences |
|---|
DISCERN provides an acceptable way of appraising the quality of online consumer health information. Despite varying levels of expertise, the majority of participants attending a DISCERN Online workshop found the tool and supporting materials accessible. Interestingly, the focus on content to assess information quality often proved a new concept, yet is essential to promote health literacy (Tones, 2002
). The educational potential of the workshops is illustrated by participants' reports of gaining a fresh perspective from the workshop, which equipped them with a more critical and systematic approach to online health information use and provision. Similar trends were found amongst participants of workshops where DISCERN was used to appraise print materials (Clisby and Charnock, 2000). Additional background information provided by participants revealed that use of online health information is diverse. The majority used it for both professional and personal purposes, and regularly accessed online consumer health information. Health information found on the Internet was frequently used as an aide to healthcare decision making and sometimes supplemented or replaced consultation with a health professional or other available information.
Many participants approached online information as a written medium and often used it in print form. Use of online consumer health information also occurred alongside use of health information in a wide range of other media. Reported applications of DISCERN after the workshop were equally diverse. Taken as a whole, these findings demonstrate the dynamic context in which health information is used. It is likely that general schemes for appraising the content of information, such as DISCERN, will provide users with simple and flexible core skills for dealing with the range of information available. Further research is needed, but it is possible that the demand for such schemes will increase as information provision becomes more complex and consumers achieve greater autonomy (Hardy, 1999; Eysenbach and Kohler, 2002; Ferguson, 2002; Rose et al., 2002; Shepperd and Charnock, 2002).
Generalizing from these data must be done with caution as the sample was a small select group. The majority were well educated and likely to be active information seekers. Many were also responsible for providing health information to others (yet less than half the participants had used some form of quality control prior to the workshops). Research on broader groups of online health information users and the best ways of reaching a representative sample is ongoing [e.g. (Eysenbach and Kohler, 2002; Rose et al., 2002)]. Experience following the launch of the DISCERN website revealed that recruiting participants through online user groups can generate a high level of inappropriate correspondence. Future research involving diverse recruitment methods and target materials will add to our knowledge of online health information users and identify the best ways of providing them with access to good quality materials.
|
References |
|---|
|
TopAbstractBackgroundMethodResultsDiscussionReferences |
|---|
Charnock, D. (1998) The DISCERN Handbook: Quality Criteria for Consumer Health Information. Radcliffe Medical Press, Abingdon.
Charnock, D., Shepperd, S., Needham, G. and Gann, R. (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health, 53, 105–111.[Abstract]
Cline, R.J.W. and Haynes, K.M. (2001) Consumer health information seeking on the Internet: the state of the art. Health Education Research, 16, 671–692.
Clisby, N. and Charnock, D. (2000) DISCERN/CASP Workshops 2000. Final Project Report. Critical Appraisal Skills Programme, Institute of Health Sciences, Oxford.
Coulter, A., Entwistle, V.A. and Gilbert, D. (1998) Informing Patients: An Assessment of the Quality of Patient Information Materials. King's Fund, London.
Delamothe, T. (2000) Quality of websites: kitemarking the west wind. British Medical Journal, 321, 843–844.
Entwistle, V.A., Sowden, A.J. and Watt, I.S. (1998). Evaluating interventions to promote patient involvement in decision-making: by what criteria should effectiveness be judged? Journal of Health Service Research and Policy, 3, 100–107.
Eysenbach, G. and Kohler, C. (2002) How do consumers search for and appraise health information on the world wide? Qualitative study using focus groups, usability tests and in-depth interviews. British Medical Journal, 324, 573–577.
Ferguson, T. (2002) From patients to end users. Quality of online patient networks needs more attention than quality of online health information. British Medical Journal, 324, 555–556.
Flesch, R. (1948) A new readability yardstick. Journal of Applied Psychology, 32, 221–233.[CrossRef]
Gagliardi, A. and Jadad, R.A. (2002) Examination of instruments used to rate quality of health information on the Internet: chronicle of a voyage with an unclear destination. British Medical Journal, 324, 569–573.
Hardy, M. (1999) Doctor in the house: the Internet as a source of lay health knowledge and the challenge to expertise. Sociology of Health and Illness, 21, 820–835.[CrossRef]
Kim, P., Eng, T.R., Deering, M.J. and Maxfield, A. (1999) Published criteria for evaluating health related web sites: review. British Medical Journal, 318, 647–649.
Jones, R., Pearson, J., McGregor, S., Gilmour, W.H., Atkinson, J.M. and Barrett, A. (1999) Cross sectional survey of patients' satisfaction with information about cancer. British Medical Journal, 319, 1247–1248.
Lindberg, D.A. and Humphreys, B.L. (1998) Medicine and health on the Internet: the good, the bad and the ugly. Journal of the American Medical Association, 280, 1303–1304.
Oxman, A.D., Guyatt, G.H., Cook, D.J. Jaeschke, R., Heddle, N. and Keller, J. (1993) An index of scientific quality for health reports in the lay press. Journal of Clinical Epidemiology, 46, 987–1001.[CrossRef][ISI][Medline]
Rose, P., Jenkins, L., Fuller, A. and Shepperd, S. (2002) Doctors' and patients' use of the Internet for healthcare: a study from one general practice. Health Information Libraries Review, 19, 233–235.[CrossRef]
Shepperd, S. and Charnock, D. (2002) Against Internet exceptionalism. British Medical Journal, 324, 556–557.
Shepperd, S., Charnock, D. and Cook, A. (2002) A 5-star system for rating the quality of information based on DISCERN. Health Information and Libraries Journal, 19, 201–205.[Medline]
Tones, K. (2002) Health literacy: new wine in old bottles? Health Education Research, 17, 287–290.
Tweddle, S., Avis, P., Wright J. and Waller, T. (1998) Towards criteria for evaluating websites. British Journal of Educational Technology, 29, 267–270.[CrossRef]
Wright, P. (1998) Writing and information design of healthcare materials. In Candlin, C. and Hyland, K. (eds), Writing: Texts, Processes and Practices. Addison-Wesley Longman, London, pp. 85–98.
Received on April 1, 2003; accepted on June 27, 2003
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  G Akram, A H Thomson, A C Boyter, and M J S Morton Characterisation and evaluation of UK websites on attention deficit hyperactivity disorder Arch. Dis. Child., August 1, 2008; 93(8): 695 - 700. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. P. Lintonen, A. I. Konu, and D. Seedhouse Information technology in health promotion Health Educ. Res., June 1, 2008; 23(3): 560 - 566. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 L. A. Marshall and D. Williams Health information: does quality count for the consumer?: How consumers evaluate the quality of health information materials across a variety of media Journal of Librarianship and Information Science, September 1, 2006; 38(3): 141 - 156. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||