Health Education Research, Vol. 17, No. 3, 351-364,
June 2002
© 2002 Oxford University Press
The challenge of evaluating complex interventions: a framework for evaluating media advocacy
Centre for Social Marketing, University of Strathclyde, Glasgow G4 0RQ, UK
 |
Abstract |
|---|
 Top Abstract IntroductionChallenges to the evaluation...A framework for evaluating...Using the frameworkConclusionReferences |
|---|
New health promotion and public health approaches such as media advocacy pose particular evaluation challenges. Evaluation is important to provide feedback to media advocacy practitioners on how to enhance their efforts, and to funders and researchers seeking to assess media advocacy's effectiveness as a health promotion strategy. The media advocacy evaluation literature contains some examples of promising evaluation approaches but is still evolving. A comprehensive framework for the evaluation of media advocacy is presented. Building on existing approaches to evaluation in media advocacy and on current thinking regarding evaluation in health promotion, it proposes a series of indicators and research methods for evaluating media advocacy at the levels of formative, process and outcome evaluation. The framework can be used to encourage strategic reflection on the media advocacy process, to guide evaluation of specific interventions, and to demonstrate to funders the importance and complexity of evaluation in this promising field.
|
Introduction |
|---|
|
TopAbstractIntroductionChallenges to the evaluation...A framework for evaluating...Using the frameworkConclusionReferences |
|---|
New approaches to health promotion and public health, particularly when these approaches involve complex actions and seek to produce complex changes, often over a long timescale, pose particular evaluation challenges. Media advocacy is one such approach.
The media advocacy evaluation literature is still evolving. Although small numbers of quasi-experimental studies [e.g. (Stewart and Casswell, 1993
; Rogers et al., 1995; Schooler et al., 1996; Treno et al., 1996; Holder and Treno, 1997; Voas et al., 1997)] and case studies (Adams and Jennings, 1993; Russell et al., 1995; Woodruff, 1996; De Jong, 1996; Blaine et al., 1997; Heiser and Begay, 1997) have been published, these as yet represent a limited evidence base. Further, they leave unexplored several questions which commentators have identified as crucial to an understanding of how media advocacy works and how it might contribute to public health. For example, Chapman has argued that insufficient research attention has been paid to how media advocacy actions succeed or fail in different political contexts (Chapman, 1992b) and more recently has called for more attention to be paid to the role of `media discourse' in contributing to international successes in tobacco control (Chapman, 1999). Similarly, Wallack et al. have stressed the importance of evaluation to provide feedback to media advocacy practitioners on how to enhance their efforts, and to funders and researchers seeking to assess media advocacy's effectiveness as a health promotion strategy (Wallack et al., 1999). Recent reviews for the Health Development Agency (Hastings et al., 1998; Stead et al., 2000) have called for further research into media advocacy's potential both in a UK health promotion context and as a strategy for combating health inequalities.
|
Challenges to the evaluation of media advocacy |
|---|
|
TopAbstractIntroductionChallenges to the evaluation...A framework for evaluating...Using the frameworkConclusionReferences |
|---|
What is being evaluated?
To answer the question how to evaluate media advocacy, it is first of all necessary to have a clear view on what precisely is being evaluated. This means defining what kind of intervention process is implied by the term media advocacy and how such a process might be assumed to bring about desired changes. There is no single or simple answer to this question. Most commentators agree that the label `media advocacy' refers to an approach to using the media in public health which involves generating news media coverage of public health issues in order to advocate particular policy solutions [e.g. (Jernigan and Wright, 1993
, 1996; Wallack et al., 1993, 1999; Chapman and Lupton, 1994)]. This media activity is often undertaken as part of a broader advocacy approach which might also involve networking, community development-type activities and lobbying. Some commentators define media advocacy largely as a bottom-up grassroots approach, involving community organizing and capacity building [e.g. (Wallack et al., 1993; Advocacy Institute, 1998)], although other literature suggests that media advocacy can be either a top-down or a bottom-up approach, depending on the issue and the context [e.g. (Chapman and Lupton, 1994)]. In practice, many media advocacy initiatives have been implemented by `power elites': public health authorities, professional associations such as the Australian and the American Medical Associations, large charities, and by governments themselves [e.g. (Chapman and Lupton, 1994)]. A diversity of different types of intervention have been defined as media advocacy, ranging from unpaid publicity components in large, multi-site, multi-faceted community-based health promotion interventions [e.g. (Schooler et al., 1996)], through to regional public referendum campaigns [e.g. (Donovan, 1997; Heiser and Begay, 1997)] to small-scale local activism against tobacco and alcohol promotions, and mobile phone masts [e.g. (Jernigan and Wright, 1993, 1996; Chapman and Wutzke, 1997)]. These different models of media advocacy pose challenges for defining appropriate evaluation objectives and approaches.
One way to approach the question is to locate media advocacy within the broader categorization of health promotion interventions. Several different conceptual frameworks exist which could be used to define media advocacy within health promotion. Clark and McLeroy suggest that health promotion interventions can be conceived of as operating at three levels: on individual behaviour; on the social and physical environment of the community; and on the policies regulating both (Clark and McLeroy, 1998). Community-level interventions seek to have an impact on social norms and on conditions within the community that may be deleterious to health, to develop community capacity to recognize and resolve health problems, and to increase community participation in programmes and actions related to health (Clark and McLeroy, 1998). Policy-level interventions are of two broad types, those concerned with organizational policy and those concerned with public policy. An example of an organizational policy intervention would be the implementation of a ban on smoking in a particular building or institution, while public policy comprises new laws or administrative regulations which change the physical or social environment and affect large numbers of citizens, e.g. tobacco excise taxation or the elimination of pollutants. Media advocacy could be defined as a health promotion intervention approach operating at both the policy level and, to a slightly lesser extent, the community level.
Another way of conceptualizing media advocacy is as an example of community-based intervention. Community-based interventions can be defined as those which aim to encourage empowerment of the community, and are based on a practice model which stresses partnership and dialogue between professionals and members of the community. The principles which underpin community-based health promotion interventions include adopting a socio-environmental approach, recognizing power differentials between groups in terms of their health outcomes and ability to determine their own health, a concern with addressing inequalities, the participation of local people in programme planning and implementation, and the involvement of `advocacy and the provision of a public voice for the health of the local community' [(Baum, 1998), p. 70]. Media advocacy which contains the elements of social justice and community organizing included by some north American commentators clearly fits this definition.
Nutbeam suggests that three broad categories of `health promotion actions' can be identified: education, facilitation and advocacy [(Nutbeam, 1998), p. 30] (see Figure 1). `Education', in this model, consists of the creation of opportunities for individuals and communities to improve their `health literacy', and thereby their capacity to improve and protect their health. `Facilitation' refers to action taken in partnership with individuals or social groups to `mobilise social and material resources for health' [(Nutbeam, 1998), p. 31], while `advocacy' describes `action taken on behalf of individuals and/or communities to overcome structural barriers to health' [(Nutbeam, 1998), p. 31]. These three broad types of action can lead to three broad types of `health promotion outcome': health literacy (e.g. increased knowledge and motivation), social influence and action (e.g. improved social support, community competency and empowerment), and healthy public policy and organizational practice. Media advocacy can be seen in the light of Nutbeam's model to comprise both `advocacy' and `facilitation' actions, and to lead to health promotion outcomes mostly at the level of policy and organizational practice, but also at the level of social influence and action.
|
These conceptualizations of health promotion interventions are helpful in that they illustrate the range of impacts sought by different interventions and, correspondingly, help to clarify what kinds of evaluation are most appropriate. According to the Nutbeam model, for example, evaluation approaches are needed which can assess media advocacy's contribution to changes in policy and organizational practice, and its influence on community capacity and empowerment.
What type of evaluation is most appropriate?
It is being increasingly argued that the traditional biomedical approach to evaluation, with the randomized controlled trial as its gold standard, has limited relevance for the analysis of complex health promotion interventions, particularly at the policy and community level [e.g. (Davies and MacDonald, 1998; Learmonth and Mackie, 2000; Tones, 2000)]. McKinlay (McKinlay, 1993) suggests that traditional experimental designs intended for measuring change in individual subjects are `suboptimal or even inappropriate' when the mechanisms and settings under investigation are organizations, communities and social policy. The quasi-experimental design has been attempted in a very small number of media advocacy evaluation studies. For example, the Community Trials Project intervention study attempted to assess the contribution of media advocacy to a reduction in alcohol-related car crashes by measuring multi-level outcomes and the relationship between them, in experimental and comparison communities. A causal chain was hypothesized: media advocacy training plus increased resources for police enforcement to detect and deter drink-driving could lead to, respectively, increased media coverage of drink-driving plus an increase in actual enforcement activity. These together would lead to an increase in the public's perception of risk of arrest for drink-driving, which could lead to less drinking and driving, which could produce fewer alcohol-involved crashes (Voas et al., 1997). Media coverage was measured using media analysis methods (Treno et al., 1996), while level of actual enforcement activity was measured by monitoring use of `sobriety' checkpoints and number of breath tests conducted (Voas, 1997). The combined impact of this media coverage and enforcement activity on public perceptions of the risk of arrest for drunk-driving was measured by telephone surveys of the general population in both experimental and comparison communities over the 3 years of the intervention. The telephone surveys also examined reported drink-driving behaviour (Voas et al., 1997). Here there was significant evidence that increased perception of risk of arrest was associated with less reported drink-driving. The final element in the chain—the relationship between all the above and actual alcohol-involved crash rates—was measured by analysing data on roadside breathalyser tests and accident data.
The Community Trials Project study is of a scale that few media advocacy initiatives could replicate. Furthermore, even if resources were available, the experimental design would be inappropriate for those media advocacy initiatives which are unpredictable, which take place over a long, and perhaps unforeseen, timescale or which involve changes in tactics and actions in response to environmental factors (such as statements by government or the tobacco industry) [e.g. (Samuels and Glantz, 1991; Heiser and Begay, 1997)]. Chapman notes that media advocacy activities `are not easily described in terms of programmes' [(Chapman and Lupton, 1994), p. 15], and do not lend themselves to precise statement as independent variables whose effects can be measured and which can be easily replicated. The randomized controlled trails design also leaves many important questions unanswered or uninvestigated: it fails to capture the detail and complexity of intervention inputs and tactics; it reveals little about the nature of the media coverage and how audiences interpreted it; it reveals little, if anything, about the processes by which media advocacy efforts exerted an influence on opinion formers and policy makers; and it neglects events and factors outwith the scope of the experiment. Fawcett et al. argue that `the usual research goal of establishing links between project activities...and particular outcomes' is particularly challenging when interventions use an array of actions, aimed at different targets, over varying periods of time, and whose effects are often delayed [(Fawcett et al., 1997), p. 826]. Furthermore, less easily measurable effects, which may in themselves be deemed worthwhile (such as increases in community participation in the policy process or gradual shifts in community attitudes towards a health problem), are unlikely to be captured. Allison and Rootman note that the very processes used and valued in health promotion programmes, such as community participation, tend to make randomized designs impossible (Allison and Rootman, 1996).
Interventions seeking to change policy pose additional evaluation challenges. Clark and McLeroy note that `Ideally, assessment would demonstrate that the intervention strategy produced the policy change, the policy change produced the desired behaviour and the behaviour contributed to the health outcome. Making these connections clear cut may be impossible' (Clark and McLeroy, 1998). Because the policy formation process is subject to multiple influences, and attempts to influence it are often, of necessity, opportunistic and flexible, it is difficult to `keep the intervention stable', as is required in the conventional controlled design. The nature of the policy formation process means that very often changes may not be detected for many years [e.g. (Whitehead, 1998)].
A final evaluation challenge concerns media advocacy's potential role as a health promotion strategy for addressing health inequalities. Mackenbach and Gunning-Schepers argue that evaluation of interventions to reduce health inequalities should not only assess the impact on disadvantaged groups, but should also establish the extent to which the intervention has reduced differences between disadvantaged and less disadvantaged groups (Mackenbach and Gunning-Schepers, 1997). Within the context of media advocacy, this might mean paying particular attention to whether policy changes and other actions produce differential benefits for disadvantaged segments of the community.
It would seem, then, that the most appropriate evaluation approaches for the investigation of media advocacy are those designed to analyse complex health promotion interventions. These typically concern themselves not only with intervention outcomes but also with inputs and processes, and use a mix of quantitative and qualitative methods [e.g. (Campbell et al., 2000; Tones, 2000)]. It would also seem appropriate to incorporate evaluation approaches developed for the other various fields on which media advocacy draws: communication and media studies, public relations, political campaigning, advocacy, social marketing, etc. Some of these are discussed below. Action research, in which the researchers work closely with the intervention team and use the research to change the intervention as it progresses, may have a role, as may self-evaluation, in which the interventionists themselves evaluate their actions. Ovretveit suggests that these kind of approaches are particularly useful when: the objectives of the intervention may be unclear; the boundaries of the intervention are `fuzzy'; the intervention continually changes in nature; there is uncertainty about the effects of the intervention; and there is uncertainty about what causes these effects [(Ovretveit, 1998), p. 184].
One evaluation approach which may offer a useful solution is the case study approach. This approach is valuable when broad and complex questions have to be addressed in real life and in complex circumstances [e.g. (Keen and Packwood, 1995)], and has been the most commonly used method in media advocacy evaluations to date.
|
A framework for evaluating media advocacy interventions |
|---|
|
TopAbstractIntroductionChallenges to the evaluation...A framework for evaluating...Using the frameworkConclusionReferences |
|---|
Table I
presents a framework for evaluating media advocacy interventions. The framework covers three types or phases of evaluation: formative, process and outcome. Within each phase, research objectives and research methods are listed. The final column in the framework lists potential research questions and, for the outcome evaluation section, also lists appropriate indicators by which progress towards intervention objectives can be measured.
|
Types or phases of evaluation
Formative research—also called developmental or exploratory research—can be used to guide overall strategy, e.g. by exploring how policy makers and the public perceive a particular issue, by providing insights into how credible or otherwise the media advocacy organization and its opponents are perceived to be by the public, and by gauging likely levels of support or opposition to a particular policy measure. Similarly, formative research using focus groups and pre-testing can help to design specific messages.
Process evaluation is also important. While impact and outcome evaluation can tell us whether or not an intervention succeeded, process evaluation can help to explain why it did or did not do so, by examining the factors which influence its implementation and the influence of the wider environment on the intervention [e.g. (Patton, 1997)]. Process evaluation is particularly appropriate for asking and answering the sorts of questions of media advocacy interventions which some commentators feel remain unanswered in the current literature. Among the questions which Chapman feels should be addressed in future media advocacy research are:
- What was the real and perceived power of opposition groups vis-à-vis advocacy groups?
- What tactics and strategies were used by advocacy groups to reduce or cancel out the power of opposition groups?
- How effective were different framing strategies?
- Was the media advocacy effort hampered by latent or covert censorship in the news media?
- What generally is the relationship between public opinion and the public health actions of politicians?
- What factors influence policy makers and politicians to act on a public health issue?
- What does it mean to say that some individuals are `good' at media advocacy, and what can be learned from them? (Chapman, 1992b).
As media advocacy is increasingly being applied beyond North America and Australia, there is also an obvious need to explore how different national, political, economic and cultural contexts impact on the media advocacy process.
Outcome evaluation in the framework is concerned with five kinds of outcome, relating to the media agenda, the public opinion agenda, the policy agenda, community capacity and advocacy capacity. `Media outcomes' refer to the impact of the media advocacy initiative on the local media: what coverage was generated, how this coverage was framed, and how it was received by target group audiences. `Public opinion outcomes' refer to the impact of the coverage and of accompanying media advocacy activities, such as community organizing, on public perceptions of the public health issue in question.
Nutbeam's model of health promotion actions (Nutbeam, 1998) described above proposes that evaluation of outcomes be tailored to the type of health promotion action under investigation (see Figure 1
). Nutbeam argues that the evaluation of health promotion interventions has often been directed at the higher levels of health outcomes and intermediate health outcomes rather than, as may be more appropriate, at the third and most immediate level of health promotion outcomes. As suggested earlier in the paper, the most appropriate outcomes in media advocacy interventions according to this model are those concerned with `social influence and action' and `healthy public policy and organizational practice'. These are addressed in our evaluation framework model under three headings: policy outcomes, community capacity outcomes and advocacy capacity outcomes.
`Policy outcomes' refer to impacts on policy makers and the policy process. If media advocacy seeks to influence conditions within the community that are deleterious to health and to increase the community's capacity to participate in actions to improve its health, then it is also appropriate to measure changes at this level. `Community capacity outcomes' refer to changes in community conditions and in community participation in health promotion actions. Finally, `advocacy capacity outcomes' refer to changes in the skills and status of the media advocacy practitioner/organization, which, while they might have no direct bearing on the health promotion or public health goal in question, increase the capacity of the practitioner/organization in the longer term.
Research objectives
The research objectives suggested within the framework correspond to the types/phases of evaluation. For example, at the formative phase, objectives are concerned with clarifying goals, identifying and understanding target groups, and developing appropriate strategies. At the process stage, objectives are concerned with documenting what media advocacy activities were implemented, and the extent to which these were as planned and why. At the outcome stage, objectives are concerned with measuring impacts at the five levels described above.
Research methods
The framework proposes a mix of potential research methods. These combine quantitative and qualitative approaches, and draw not only on health promotion research but also on mass-media research. Research methods derived from the latter include tracking monitors [e.g. (MacKintosh and MacAskill, 1993)] and media analysis methods for measuring volume, content, prominence, tone and `framing' of news coverage resulting from a media advocacy intervention [e.g. (Association of Media Evaluation Companies, 1997; Kitzinger and Reilly, 1997, Westwood and Westwood, 1999)]. Inevitably these studies provide feedback only on the first part of the media advocacy process, and tell little about how public and opinion former agendas were influenced by the coverage or how it might have contributed to the policy formation process. For this, approaches used in political campaigning, such as public opinion polling, can be useful [e.g. (Donovan, 1997; Heiser and Begay, 1997)]. Public opinion polls have often been used in agenda setting research to analyse the relationship between prominent news issues and the issues deemed important by the general public [e.g. (McCombs and Shaw, 1972/1991)].
More detailed insights into the influence of news media on how the public perceive issues can be provided by qualitative methods such as focus groups [e.g. (Kitzinger, 1994)]. Research by Glasgow Media Group into the influence of media depictions of mental illness on public beliefs used a range of qualitative techniques (Philo, 1996) including asking research respondents to construct their own newspaper stories and soap opera scripts, and individual in-depth interviews about personal experiences, viewing/reading habits, and other aspects of beliefs and attitudes. Such research provides detailed insights into how media messages tap into public anxieties, and how personal experiences are used both to confirm and challenge media messages. They can also provide complex and rich insights into how the general public perceive policy issues, and to whom they attribute responsibility for action.
Individual in-depth interviews with key individuals such as intervention partners and stakeholders, journalists, local opinion formers, and policy makers can provide detailed insights into all stages of a media advocacy initiative. For example, interviews with intervention partners can provide reflection on ongoing progress and satisfaction with the activities being used. Interviews with journalists can help to uncover reasons why journalists respond to some news stories and not others. Interviews with policy makers can provide insights into the many influences on the policy formation process and the role that media coverage may play. However, many media advocacy commentators note that policy makers are loathe to acknowledge that they are influenced by any media advocacy initiative; it may be useful therefore to complement interviews with secondary analysis, such as monitoring of policy-maker statements and documents, as these can serve as indicators of subtle impacts at a policy level (see next section).
Finally, monitoring, audits and observation of activities and community conditions are included in the framework.
Research questions and indicators
This section of the framework suggests potential indicators by which progress towards media advocacy objectives can be measured.
Media indicators
The impact of media advocacy activities on the news media can be assessed using a variety of indicators. At the most basic level, increases in the amount of news coverage on a particular topic serve as indicators that the subject has been placed on the agenda of media gatekeepers. Various indicators of media content have been developed, e.g. Westwood and Westwood measured Australian press reporting of public health issues according to five criteria of: prominence, which includes frequency, day of week, page, location, headline size and photographs; content; stakeholders, which refers to the groups or individuals most affected by the reported issues; orientation of reporting, which includes the author, the credibility of the sources quoted, and the type of report; and tone (Westwood and Westwood, 1999). Similar measures have been used in some media advocacy evaluation studies [e.g. (Stewart and Casswell, 1993; Schooler et al., 1996; Treno et al., 1996; Holder and Treno, 1997)]. Within commercial public relations, `media evaluation' tests for the `positive and negative significance', or favourability and unfavourability, of coverage by reference to incidence, location, weight, content, size or duration, position, prominence, and audience reach and `opportunities to see' (Association of Media Evaluation Companies, 1997). These sort of measures alone, while they are useful for quantifying impact and testing hypotheses, provide little insight into the subjective aspects of coverage and how this contributes to media discourse on health (Chapman, 1999). For this, it is helpful to apply the kind of indicators used in content or framing analysis, such as dominant argument, metaphors, symbols and vocabulary [e.g. (Chapman, 1992a; Lupton, 1994; Kennedy and Bero, 1999; Lima and Siegel, 1999)].
Measuring what appears in the news media does not tell us anything about how the audience reacted to it, if indeed they saw the coverage at all. Lindenmann makes a useful distinction between media `outputs'—the number of stories, the quantity and quality of the coverage—and media `outcomes', which refer to audience awareness and comprehension, changes in opinions and attitudes, and any resulting behaviours or actions (Lindenmann, 1997). A variety of indicators exist for assessing the impact of media messages on audiences, such as spontaneous and prompted awareness, recall, comprehension, perceived message, emotional response, credibility, tone, identification, and involvement [e.g. (MacKintosh and MacAskill, 1993)]. Qualitative indicators of audience impact might include subtle changes in how members of the public discuss and perceive specific health problems, and their interpretation of who or what is responsible.
Public opinion indicators
Indicators of changes in public opinion include measures of what policy issues are most salient to the public and levels of public support for particular policies. A common indicator used in agenda setting research is a measure of the public's top issues of concern and how these correlate with previous media coverage [e.g. (McCombs and Shaw 1972/1991; Funkhouser, 1973/1991; Gonzenbach, 1992)]. These kind of indicators are useful not only post hoc for measuring the impact of a media advocacy intervention, but also during an intervention, to provide ongoing guidance on how to shape communication strategies in accordance with public mood [e.g. (Donovan, 1997; Heiser and Begay, 1997)]. Impacts on public opinion can also be assessed by examining particular actions which indicate an increase in public concern. These might include letters to newspaper editors (Mayer, 1991), calls to helplines, citizen complaints about local businesses (Rogers et al., 1995), petition signing, etc.
Policy indicators
As noted above, the policy formation process is complex and subject to multiple influences, and it would be difficult to develop an evaluation which could measure the contribution of a specific media advocacy intervention to a specific policy change. Acknowledging the challenges of evaluating policy change, health promotion evaluators have suggested that `the challenge may be to identify indicators that most would accept as reliable and valid signs that change is occurring in the desired direction' [(Clark and McLeroy, 1998), p. 29].
Nutbeam suggests that possible indicators of policy-level changes include policy statements, legislation, regulations, organizational procedures, and funding and resource allocation (Nutbeam, 1998). Mayer, discussing ways of evaluating the agenda-setting process, suggests that indicators of the policy agenda can also include presidential speeches and the wording and passage of legislative bills (Mayer, 1991). A similar point is made in Whitehead's discussion of the process by which publicity about emerging research into health inequalities helped to put the subject on the policy agenda in Europe (Whitehead, 1998). The paper suggests that markers of progress were sometimes as slight as a change in official vocabulary, e.g. while the Thatcher government avoided making any reference to `inequalities in health', the Major government was prepared to use the phrase `social variations', a subtle marker of a shift towards acknowledging the existence of a problem. Other indicators of an effect at national level identified by Whitehead include: mention of inequalities in health in the 1995 report of the Chief Medical Officer; new research by official bodies such as the Office for National Statistics specifically into health inequalities; a change in the criteria applied by research funding agencies to make health inequalities research a legitimate area for funded study; creation of a government task force on Inequalities in Health; and establishment of action to tackle health inequalities as one of the key aims of the national health strategy in 1998.
Community indicators
Appropriate indicators of change in the conditions which shape a community's health might include a reduction in alcohol promotions in a neighbourhood [e.g. (Jernigan and Wright, 1993)] or changes in enforcement of legislation on tobacco sales to minors [e.g. (Blaine et al., 1997)]. Indicators of increases in the community's health promotion capacity might include the formation of health and advocacy groups, specific actions by such groups, an increase in their advocacy skills or enhanced participation by such groups in the policy process [e.g. (Baum, 1998; Nutbeam, 1998)].
Advocacy indicators
Finally, the framework suggests indicators of a change in the media advocacy organization's status or skills. For example, the organization may, as a result of running an intervention, become more proficient in dealing with journalists or giving television interviews, which might increase its effectiveness in the future. It might raise its profile among journalists as a credible and authoritative source of comment and information on public health issues. Another indicator that the media advocacy organization's status has been enhanced might be recognition from the policy community that it has the potential to make an important contribution to the policy process (e.g. Stead and Hastings, unpublished).
|
Using the framework |
|---|
|
TopAbstractIntroductionChallenges to the evaluation...A framework for evaluating...Using the frameworkConclusionReferences |
|---|
The framework underlines the importance of research, both in guiding individual media advocacy interventions and in helping the discipline as a whole move beyond the realm of plausible belief. It also demonstrates the complexity of the research task, and how it differs from conventional approaches. Media advocacy interventions engage with people, on their terms, reflecting their priorities, and in this way harness multiple drivers—the editor's desire for a good story, the local community group's need for better health, the politician's need for votes—to advance public health. Research should illuminate these subjective interactions. The task is quite different from that faced in clinical interventions such as drug trials, where the research challenge is to design out any such subjectivity.
The research tasks also vary because different philosophies are at work. Clinical interventions seek to do things to an essentially passive patient group, whereas media advocacy seeks to construct joint agendas and do things with active collaborators. The former focuses on changes in a very limited number of variables over a finite time; the latter is concerned with continuous progress towards a much longer-term goal. Furthermore, continuous feedback can and should enable the media advocate to adjust methods, and even objectives, in the greater cause of progress towards long-term aims. Such goalpost-moving would be anathema in a clinical trial. Research in media advocacy needs to encompass this flexibility and strategic vision.
Use of the framework should reflect these points. First, it provides a way of thinking about our activities. Objectives, target groups and intervention methods all need to be specified if the full role of research is even going to be properly considered.
Second, the framework should be used to demonstrate to the public health community as a whole—and funders, in particular—that the research challenge differs fundamentally from that faced in clinical interventions. A broad range of qualitative and quantitative procedures are needed to capture subjective processes as effectively as objectives outcomes, and the latter will always be staging posts rather than final destinations.
Third, where resources are limited and the full panoply of research tools cannot be deployed, choice should focus on the defining characteristics of media advocacy: its harnessing of the subjective in order to make progress towards wide ranging, long-term aims. Even so the choice is invidious: all research stages are important. However, in essence research becomes slightly less important as the intervention progresses. Formative research is essential; it provides the initial compass bearing, without which media advocates are flying blind and as likely to do harm as good. Process evaluation is also very important, checking coordinates and progress towards the destination, as well as alerting the media advocate to unforeseen obstacles and opportunities. Outcome evaluation is also extremely valuable, but can be truncated if resources are wanting. Good formative and process evaluation should provide some indicators of success, and the focus can legitimately be put on future progress towards long-term aims rather than proving, post hoc, that shorter-term objectives were met.
|
Conclusion |
|---|
|
TopAbstractIntroductionChallenges to the evaluation...A framework for evaluating...Using the frameworkConclusionReferences |
|---|
Media advocacy is a promising development that needs systematic research if it is to move from plausibility to proven effectiveness. This paper has presented a framework for this research.
It reflects the complexity of the challenge: the need to illuminate rather than eliminate subjective processes; to provide continuous feedback that permits adjustments; for flexibility throughout the intervention and for maintaining a strategic vision.
The framework will help media advocates both plan their activities and convince funders that a serious and innovative research resource is needed. As global interest in media advocacy increases, the need for such a research resource becomes more apparent, particularly if media advocacy is to become a relevant and useful approach in developing countries as well as in north America and Europe. It also provides a way of thinking about research priorities when the inevitably inadequate budget compels invidious choices to be made between research options. In essence media advocates should go with those techniques that best match the nature of media advocacy.
|
References |
|---|
|
TopAbstractIntroductionChallenges to the evaluation...A framework for evaluating...Using the frameworkConclusionReferences |
|---|
Adams, R. J. and Jennings, K. M. (1993) Media advocacy—a case study of Sokolof, Philip cholesterol awareness campaigns. Journal of Consumer Affairs, 27, 145–165.
Advocacy Institute (1998) Blowing away the Smoke: A Series of Advanced Media Advocacy Advisories for Tobacco Control Advocates. Revised series September 1998. The Advocacy Institute, Washington DC.
Allison, K. R. and Rootman, I. (1996) Scientific rigor and community participation in health promotion research: are they compatible? Health Promotion international, 11, 333–340.
Association of Media Evaluation Companies (1997) Guide to Media Evaluation. AMEC, London.
Baum, F. (1998) Measuring effectiveness in community-based health promotion. In Davies, J. K. and MacDonald, G. (eds), Quality, Evidence and Effectiveness in Health Promotion. Routledge, London, pp. 68–89.
Blaine, T. M., Forster, J. L., Hennrikus, D., O'Neil, S., Wolfson, M. and Pham, H. (1997) Creating tobacco control policy at the local level: implementation of a direct action organizing approach. Health Education Behaviour, 24, 640–651.
Campbell, M., Fitzpatrick, R., Haines, A., Kinmonth, A. L., Sandercock, P., Spiegelhalter, D. and Tyrer, P. (2000) Framework for design and evaluation of complex interventions to improve health. British Medical Journal, 321, 694–696.
Chapman, S. (l992a) Anatomy of a campaign: the attempt to defeat the New South Wales (Australia) Tobacco Advertising Prohibition Bill 1991. Tobacco Control, 1, 50–56.
Chapman, S. (1992b) Upgrading the academic respectability of advocacy studies in tobacco policy implementation. Tobacco Control, 2, 81–83.
Chapman, S. (1999) The news on tobacco control: time to bring the background into the foreground. Tobacco Control, 8, 237–239.
Chapman, S. and Lupton, D. (eds) (1994) The Fight for Public Health: Principles and Practice of Media Advocacy. BMJ Publishing, London.
Chapman, S. and Wutzke, S. (1997) Not in our back yard: media coverage of community opposition to mobile phone towers—an application of Sandman's outrage model of risk perception. Australian and New Zealand Journal of Public Health, 21, 614–20.[Web of Science][Medline]
Clark, N. M. and McLeroy, K. R. (1998) Reviewing the evidence for health promotion in the United States. In Davies, J. K. and MacDonald, G. (eds), Quality, Evidence and Effectiveness in Health Promotion. Routledge, London, pp. 21–46.
Davies, J. K. and MacDonald, G. (eds) (1998) Quality, Evidence and Effectiveness in Health Promotion. Routledge, London.
De Jong, W. (1996) MADD Massachusetts versus Senator Burke: a media advocacy case study. Health Education Quarterly, 23, 318–329.[Web of Science][Medline]
Donovan, P. (1997) The Colorado Parental Rights Amendment: how and why it failed. Family Planning Perspectives, 29, 187–190.[Web of Science][Medline]
Fawcett, S. B., Lewis, R. K., Paine-Andrews, A., Francisco, V. T., Richter, K. P., Williams, E. L. and Copple, B. (1997) Evaluating community coalitions for prevention of substance abuse: the case of Project Freedom. Health Education and Behavior, 24, 812–828.
Funkhouser, G. R. (1973/1991) The issue of the sixties: an exploratory study in the dynamics of public opinion. In Protess, D. L. and McCombs, M. E. (eds), Agenda setting: Readings on Media, Public Opinion, and Policymaking. Lawrence Erlbaum, Hillsdale, NJ, pp. 35–41.
Gonzenbach, W. J. (1992) A time-series analysis of the drug issue, 1985–1990—the press, the president and public-opinion. International Journal of Public Opinion Research, 4, 126–147.
Hastings, G. B., Stead, M., Whitehead, M., Lowry, R., MacFadyen, L., McVey, D., Owen, L. and Tones, K. (1998) Using the media to tackle the health divide: future directions. Social Marketing Quarterly, 4, 42–67.
Heiser, P. F. and Begay, M. E. (1997) The campaign to raise the tobacco tax in Massachusetts. American Journal of Public Health, 87, 968–973.
Holder, H. D. and Treno, A. J. (1997) Media advocacy in community prevention: news as a means to advance policy change. Addiction, 92 (Suppl. 2), S189–S199.
Jernigan, D. H. and Wright, P. A. (eds) (1993) Making News, Changing Policy: Case Studies of Media Advocacy on Alcohol and Tobacco Issue. Center for Substance Abuse Prevention, Bethesda, MD.
Jernigan, D. H. and Wright, P. A. (1996) Media advocacy: lessons from community experiences. Journal of Public Health Policy, 17, 306–330.[Web of Science][Medline]
Keen, J. and Packwood, T. (1995) Case study evaluation. British Medical Journal, 311, 444–446.
Kennedy, G. E. and Bero, L. A. (1999) Print media coverage of research on passive smoking. Tobacco Control, 8, 254–260.
Kitzinger, J. (1994) Challenging sexual violence against girls: a social awareness approach. Child Abuse Review, 3, 246–258.
Kitzinger, J. and Reilly, J. (1997) The rise and fall of risk reporting: media coverage of human genetics research, `false memory syndrome' and `mad cow disease'. European Journal of Communication, 12, 319–350.
Learmonth, A. and Mackie, P. (2000) Evaluating effectiveness in health promotion: a case of re-inventing the millstone? Health Education Journal, 59, 267–280.
Lima, J. C. and Siegel, M. (1999) The tobacco settlement: an analysis of newspaper coverage of a national policy debate. Tobacco Control, 8, 247–253.
Lindenmann, W. K. (1997) Setting minimum standards for measuring public relations effectiveness. Public Relations Review, 23, 4 391–408.
Lupton, D. (1994) Analysing news coverage. In Chapman, S. and Lupton, D. (eds), The Fight for Public Health: Principles and Practice of Media Advocacy. BMJ Publishing, London, pp. 23–57.
Mackenbach, J. P. and Gunning-Schepers, L. J. (1997) How should interventions to reduce inequalities in health be evaluated? Journal of Epidemiology and Community Health, 51, 359–364.
MacKintosh, A. M. and MacAskill, S. G. (1993) The use of tracking studies in monitoring health education campaigns. In Proceedings and Abstracts of the 3rd European Conference of the IUHE, `Health Education and Mass Media—How to Communicate Effectively'. Dutch Centre for Health Promotion and Health Education, Utrecht.
Mayer, R. N. (1991) Gone yesterday, here today—consumer issues in the agenda-setting process. Journal of Social Issues, 47, 21–39.
McCombs, M. and Shaw, D. (1972/1991) The agenda-setting function of mass media. In Protess, D. L. and McCombs, M. E. (eds), Agenda Setting: Readings on Media, Public Opinion, and Policymaking. Lawrence Erlbaum, Hillsdale NJ, pp. 17–26.
McKinlay, J. B. (1993) The promotion of health through planned sociopolitical change—challenges for research and policy. Social Science and Medicine, 36, 109–117.
Nutbeam, D. (1998) Evaluating health promotion—progress, problems and solutions. Health Promotion International, 13, 27–44.
Ovretveit, J. (1998) Evaluating Health Interventions. Open University Press, Buckingham.
Patton, M. Q. (1997) Utilization-focused Evaluation. The New Century Text, 3rd edn. Sage, Thousand Oaks, CA.
Philo, G. (1996) The media and public belief. In Philo, G. and Glasgow Media Group (eds), Media and Mental Distress. Addison Wesley Longman, Harlow, pp. 82–104.
Rogers, T., Feighery, E. C., Tencati, E. M., Butler, J. L. and Weiner, L. (1995) Community mobilization to reduce point-of-purchase advertising of tobacco products. Health Education Quarterly, 22, 427–442.[Web of Science][Medline]
Russell, A., Voas, R. B., DeJong, W. and Chaloupaka, M. (1995) MADD rates the states—a media advocacy event to advance the agenda against alcohol-impaired driving. Public Health Reports, 110, 240–245.[Web of Science][Medline]
Samuels, B. and Glantz, S. A. (1991) The politics of local tobacco control. Journal of the American Medical Association, 266, 2110–2117.
Schooler, C., Sundar, S. S. and Flora, J. (1996) Effects of the Stanford Five-City Project media advocacy program. Health Education Quarterly, 23, 346–364.[Web of Science][Medline]
Stead, M., Henderson, L., Kitzinger, J. and Hastings, G. (2000) An Investigation into the Potential of Media Advocacy as a Health Promotion Strategy. Report prepared for the Health Development Agency, University of Strathclyde/Brunel University.
Stewart, L. and Casswell, S. (1993) Media advocacy for alcohol policy support: results from the New Zealand Community Action Project. Health Promotion International, 8, 167–175.
Tones, K. (2000) Evaluating health promotion: a tale of three errors. Patient Education Counselling, 39, 227–236.[Web of Science][Medline]
Treno, A. J., Breed, L., Holder, H. D., Roeper, P., Thomas, B. A. and Gruenewald, P. J. (1996) Evaluation of media advocacy efforts within a community trial to reduce alcohol-involved injury—preliminary newspaper results. Evaluation Review, 20, 404–423.
Voas, R. B. (1997) Drinking and driving prevention in the community: program planning and implementation. Addiction, 92 (Suppl. 2), S201–S2l9.
Voas, R. B., Holder, H. D. and Gruenewald, P. J. (1997) The effect of drinking and driving interventions on alcohol-involved traffic crashes within a comprehensive community trial. Addiction, 92 (Suppl. 2), S221–S236.
Wallack, L., Dorfman, L., Jernigan, D. and Themba, M. (1993) Media Advocacy and Public Health: Power for Prevention. Sage, Thousand Oaks, CA.
Wallack, L., Woodruff, K., Dorfman, L. and Diaz, I. (1999) News for a Change: An Advocate's Guide to Working with the Media. Sage, Thousand Oaks, CA.
Westwood, B. and Westwood, G. (1999) Assessment of newspaper reporting of public health and the medical model: a methodological case study. Health Promotion International, 14, 53–64.
Whitehead, M. (1998) Diffusion of ideas on social inequalities in health: a European perspective. The Milbank Quarterly, 76, 469–492.[Web of Science][Medline]
Woodruff, K. (1996) Alcohol advertising and violence against women: a media advocacy case study. Health Education Quarterly, 23, 330–345.[Web of Science][Medline]
Received on February 15, 2001; accepted on July 31, 2001
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. L. Granner, P. A. Sharpe, E. L. Burroughs, R. Fields, and J. Hallenbeck Newspaper content analysis in evaluation of a community-based participatory project to increase physical activity Health Educ. Res., September 13, 2009; (2009) cyp049v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. Stead, L. McDermott, and G. Hastings Towards evidence-based marketing: the case of childhood obesity Marketing Theory, December 1, 2007; 7(4): 379 - 406. [Abstract] [PDF]
|
|
|
|
 |
|
 J. Niederdeppe, M. C Farrelly, and D. Wenter Media advocacy, tobacco control policy change and teen smoking in Florida Tob. Control, February 1, 2007; 16(1): 47 - 52. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. Wammes, B. Breedveld, S. Kremers, and J. Brug The 'balance intervention' for promoting caloric compensatory behaviours in response to overeating: a formative evaluation Health Educ. Res., August 1, 2006; 21(4): 527 - 537. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. E. Martinson and D. B. Hindman Building a health promotion agenda in local newspapers Health Educ. Res., February 1, 2005; 20(1): 51 - 60. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. J. Kelly, R. W. Edwards, M. L. G. Comello, B. A. Plested, P. J. Thurman, and M. D. Slater The Community Readiness Model: A Complementary Approach to Social Marketing Marketing Theory, December 1, 2003; 3(4): 411 - 426. [Abstract] [PDF]
|
|
|
|
|
|
R Durrant, M Wakefield, K McLeod, K Clegg-Smith, and S Chapman Tobacco in the news: an analysis of newspaper coverage of tobacco issues in Australia, 2001 Tob. Control, September 1, 2003; 12(90002): ii75 - 81. [Abstract] [Full Text]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||