Health Education Research, Vol. 15, No. 6, 743-756,
December 2000
© 2000 Oxford University Press
Utilizing research in practice and generating evidence from practice
County Durham Health Authority, Appleton House, Lanchester Road, Durham DH1 5XZ, UK
 |
Abstract |
|---|
 Top Abstract Introduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
This paper gives an overview of evidence-based practice in health promotion, with reference mainly to the National Health Service (NHS) context within the UK, but with wider international relevance. It starts by looking at the tensions raised at the interface of the two activities of research and health promotion. It goes on to explore two aspects of evidence-based practice: incorporating research evidence into health promotion activity and developing robustly evaluated practice in such a way as to feed the developing research agenda. Each of these two aspects is explored using a specific example, from within the UK. Finally, the paper goes on to make eight recommendations that taken together would help create an iterative process contributing to the development of health promotion theory and practice.
|
Introduction: the interface between research and health promotion |
|---|
|
TopAbstractIntroduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
This article aims to draw together some contemporary themes in relation to two key processes: the transfer of research findings into health promotion practice and the capacity for evaluation done by practitioners in the field, to generate evidence or lead to wider research. Each of these will be explored briefly in theory and through an illustrative example.
In order to address this main subject it is necessary to clarify the nature of both research and health promotion. The primarily divergent purposes of these two activities may set up a tension if the difference is not clearly recognized and addressed. This tension can be both practical and ethical, leading to role conflict if the health promotion researchers are also acting as change agents or facilitators.
One definition of research, developed with the task in mind of integrating research into nursing practice, is [(MacVicar, 1999, p. 299]:
...the word research is being used to mean either an activity to discover, develop and test knowledge (i.e. doing research) or as a source of scientific knowledge accessed through research reports for use in practice (i.e. reading and critiquing).
The core purpose of primary research is to discover new knowledge.
For health promotion the WHO has endorsed the following definition [(WHO, 1986
), p. 1]:
...health promotion is the process of enabling people to increase control over, and to improve, their health.
Wimbush and Watson [(Wimbush and Watson, 2000), p. 306] expand on this:
In seeking to tackle the root causes of ill health and reduce inequalities in health outcomes, health promotion interventions adopt a `whole systems approach' where cross-sectoral partnerships and high levels of community and user involvement are essential characteristics. What should count as evidence of effectiveness when it comes to such complex health interventions is a highly contested issue.
From the researchers' viewpoint, is the need to generate new knowledge so strong that the requirements this imposes has a fundamental effect on the nature of ongoing health promotion practice? If so, the research being conducted is on an artificially modified activity. From the health promotion practitioners' perspective, how far does the wish to evaluate work suggest that in order to conduct good practice, they need to conform to the requirements of research?
Formulating the question in this way leads us to address the concept of evaluation. One recent study which included practitioners understanding of evaluation and research found that while the distinction was not always clear, a number of aspects emerged (South and Tilford, 2000):
- Research: larger scale, broader research questions, focus on developing new knowledge and exploring possibilities for interventions, more structured, more likely to be generalizable, greater depth, more academically rigorous, output in academic papers, occasional, resource intensive, needs outside support;
- Evaluation: smaller scale, narrower questions, focus on process and outcomes of interventions, less structured, less likely to be generalizable, less depth, less academically rigorous, output in shorter reports and briefings in practice contexts, everyday part of practice, less resources needed, less need for outside support (except where external evaluation required)'.
A formal definition of evaluation that may be of assistance in clarifying the distinction from research is offered by Bowling (Bowling, 1997):
Evaluation is the use of the scientific method, and the rigorous and systematic collection of research data to assess the effectiveness of organizations, services and programmes in achieving pre-defined objectives...
Formative evaluation involves the collection of data while the organization or programme is active, with the aim of developing or improving it. Summative evaluation involves collecting the data about the active (or terminated) organization or programme with the aim of deciding whether it should be continued or repeated...
The evaluation of health services is usually based on the collection of data about the structure, inputs, process, outputs and outcomes of the service...
Evaluation therefore is less concerned with new knowledge and more with assessing what has resulted from the implementation of specific programmes.
The fourth and final sections of this paper explore further the question of evaluation and the contribution it can make to the research agenda.
Stepping back to the perspective of the researcher, it may be useful to review from an academic stance what research methods are available that are in keeping with the goals of health promotion. Participative inquiry, or `research with people' [(Reason, 1994, p. 325)], is one developing area of research which meets the criteria of both health promotion practice and research. Reason explores three methods within the field of participative inquiry. In co-operative inquiry [(Reason, 1994), p. 326)]:
...all those involved in the research are both co-researchers, whose thinking and decision-making contribute to generating ideas, designing and managing the project, and drawing conclusions from the experience, and also co-subjects, participating in the subject being researched.
Participatory action research gives explicit recognition to the political aspects of knowledge production [(Reason, 1994, p. 328)].
One aim is to produce knowledge and action directly useful to a group of people—through research, adult education and socio-political action. The second aim is to empower people at a second and deeper level through the process of constructing and using their own knowledge.
Action science and action inquiry are a third aspect of this style of research. In this case, the emphasis is on the development of effective action that may [(Reason, 1994, p. 330)]:
...transform organizations and communities into collaborative, self-reflective communities of inquiry.
With emphasis on individuals, groups and organizations, these three approaches (and there are more in the area of participative inquiry) are developing methods which involve and empower the subjects, users and researchers, as part of the research process.
Policy-related research, with its focus on the `real-world' evaluation of policies and services, is another academic source area of relevance to health promotion practice. Again, it may be at the interface with evaluation that much useful work can be done. One developing area is utilization-focused evaluation, which begins with the premise that evaluations should be judged by their utility and actual use. The focus is on intended use by intended users, i.e. how real people in the real world apply evaluation findings and experience the evaluation process (Patton, 1997). This gives rise to ethical considerations in terms of which users have the loudest voice (e.g. the end users or those commissioning the evaluation), but the method gives scope for the consideration and weighting of these issues. Like the participative inquiry approaches, the involvement of users is central to the process. These approaches may offer a way forward in terms of health promotion research, which is compatible with the goals of health promotion practice.
However, in England, the lead to promote the health of the nation lies with the National Health Service (Department of Health, 1999c). Here the dominant paradigm has `the gold standard' in a hierarchy of evidence is generated by the randomized controlled trial (Cochrane, 1972; US Preventive Services Task Force, 1989). This hierarchy has informed the series of 13 systematic reviews commissioned by the Health Education Authority (http://www.hda-online.org.uk). The way in which the hierarchy has been applied shows considerable diversity in the inclusion criteria used in these systematic reviews (Oliver et al., 1999; Peersman et al., 1999). Nevertheless there is a substantial evidence base emerging for health promotion activity, within the constraints of the methodology.
The tensions that the primacy of the randomized control trial and clinical models of effectiveness establish for health promotion in view of its collaborative participative principles and goal of empowerment have been explored elsewhere (Learmonth and Watson, 1999). Methodological problems with the application of randomized control trials to health promotion have also been identified including: random allocation is difficult to achieve in practice and artificial; ethical issues in withholding educational `treatments'; it is virtually impossible with community-based interventions to avoid contamination from the experimental to the comparison area; and it is not possible to meet the requirements for `blind' control and experimental groups if you are working participatively (Tones, 1999). Whilst particularly acute for health promotion practitioners, the need to recognize diverse forms of evidence applies in many other aspects of health care. Muir Gray (Muir Gray, 1997) carried out some initial mapping work to identify the different forms of research method able to generate evidence most appropriate to assess safety, acceptability, appropriateness and quality in addition to effectiveness. This trend has been endorsed at a policy level in England with the publication of Saving Lives: Our Healthier Nation [(Department of Health, 1999c), para. 11.35]:
...in the past it [the RCT] has been the gold standard for research but it is no longer applicable to all the kinds of research questions that need to be asked.
This shift in guidance may help open the doorway to the development of research methods that are in keeping with the aims and principles of health promotion. Part of this process will be the cross-fertilization with other disciplines. For example, in the field of education there has been a parallel debate emerging around grappling with the difficulty of generating meaningful indicators for use by and with schools. Organizational behaviour is notoriously subject to change, in order to modify indicators, particularly in the context of a culture where performance and league tables are the method for judging success. Once particular measures have been selected to act as indicators of the status of the school overall, they become management targets. An indicator may be selected based on sound research that linked a particular feature (e.g. truancy rates) with a wide range of other features of the school (e.g. overall ability to address bullying and poor performance). This relationship changes as soon as it becomes the focus of attention in measuring the success of the school (Visscher et al., 1999). This phenomenon is especially pertinent to those like health promotion practitioners working in health-related areas where the move to evidence-based practice is often aligned with performance management and cost/efficiency savings.
To conclude this first scene-setting section then, research methods are developing which are compatible with the principles and practice of health promotion, particularly participative inquiry and utilization-focused evaluation. There is an opportunity now to develop the application of these methods to generate new knowledge to draw on in a way that is appropriate to work with communities, organizations and at a policy level. In the meantime, conclusions for some aspects of health promotion practice are emerging from a series of systematic reviews, despite the constraints of the methodology. Focusing for the moment on research-driven activity and the contribution this may make to health promotion practice, the next section looks at the infrastructure required in order to ensure conclusions from research activity are transferred into practice.
|
Utilizing research in practice |
|---|
|
TopAbstractIntroduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
This question has been thoroughly investigated in relation to developing evidence-based health care. Practitioners need the ability to fulfil three steps: finding the evidence, appraising the evidence and implementing the evidence (Thomson, 1998
). In discussing the transfer of these three elements to health promotion activity, this part of the paper talks about Health Promotion Specialists. This group in the UK form an important part of the infrastructure for health promotion. Their professional organization describes their role as (Society of Health Education and Promotion Specialists, 1999):
Advisors, consultants, researchers, trainers, project leaders, coordinators, policy development officers, enablers, mediators and advocates. Through these roles they help many thousands of workers such as doctors, nurses, teachers, police officers, local authority representatives, company directors and community representatives to carry out health promotion work within their own setting. Health Promotion Specialists are the key catalysts and facilitators for the vast majority of health promotion work carried out within the UK.
They are therefore a pertinent group to focus on in considering the transfer from research into practice.
A survey of 20 Health Promotion Specialist departments in 1997 found all of them considered that the drive towards evidence-based practice was impacting on their work (Learmonth and Watson, 1999). This confirms results from surveys in the Northern and Yorkshire Region (Delaney et al., 1997; Tilford and South, 1999). In the Learmonth and Watson paper some Health Promotion Managers articulated the way in which their marginalized position within the NHS as a whole, impacted on their credibility in addressing this issue, both by highlighting the focus on their ability to respond and in the way in which the response was judged. An adaptation of the definition of evidence-based health care developed by Hicks (Hicks, 1997) was accepted as useful by most participants [(Learmonth and Watson, 1999), p. 322]:
Evidence-based health promotion takes place when decisions that affect our interventions are taken with due weight according to all valid, relevant information.
All were tackling the first step identified by Thomson, finding evidence, to inform decision making. They were using a wide variety of sources, including databases, Internet sites and national agencies such as the Health Education Authority and Institute for the Study of Drug Dependence, as well as academic publications and systematic reviews. There has of course been considerable duplication of effort involved in this process since it has been repeated in local departments up and down the country.
In addressing the second step, appraising the evidence, all participants were confident in assessing the relevance of research to their local situation, using criteria of feasibility in terms of local needs, resources, culture and stakeholder views. However, many were less sure how to judge the question of validity or deciding how accurate the research findings are. This is not surprising since the debate referred to in the first section is part of a diversity of views in the academic community. For example, Silverman (Silverman, 1993), with reference to the validation of qualitative data, discusses the value of triangulation and respondent validation, before going on to reject both of these in favour of rigorous analytic induction as his preferred method, in which exceptions are eliminated by revising the hypothesis until all data fit.
Saving Lives: Our Healthier Nation identifies as part of the Health Development Agency's new role: `maintaining an up-to-date map of the evidence base for public health and health improvement' [(Department of Health, 1999c), para. 11.6]. The early documentation suggests that the developmental areas of research referred to in the first part of this paper will form an important part of the map (Gillies, 2000). As a national (and international) resource this will begin to address the question of duplication of effort in finding the evidence and consistency in judging validity when appraising it.
The third step after finding and appraising evidence is its implementation. Despite the technical difficulties surrounding steps one and two, the third is in many ways the hardest. It requires a commitment of senior staff to change management, and often an ability to influence at a policy and resource allocation level. At this juncture there is a crucial interface between the aspiration for a rational search for evidence-based practice, and the realpolitik of decision making based on power and influence. Clark and McLeroy, reviewing the evidence for effectiveness of health promotion in the US, find [(Clark and McLeroy, 1998), p. 38]:
Despite its limitations, the social-behavioural approach has contributed substantially to our knowledge of effective settings, strategies, theories and principles, and evaluation methods for health promotion.
This is despite the restriction in implementing this evidence arising from the fact that prevention efforts consume approximately 1% of health-related dollars. They go on to conclude that [(Clark and McLeroy, 1998), p. 39]:
...social ecology and empowerment approaches may expand our ideas about prevention to include more emphasis on needed system changes and greater recognition of the effects of social and economic disparities on health.
The illustrative example below goes on to look at one situation where the potential to implement evidence-based practice existed at a national level, with the potential to produce the system changes referred to by Clark and McLeroy.
|
An illustrative example: implementing evidence at policy level |
|---|
|
TopAbstractIntroduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
The aim of this section is to explore the degree to which one specific piece of piece of highly relevant research appears to have influenced policy nationally in terms of the strategy for health in England. The material for discussion includes Saving Lives: Our Healthier Nation (Department of Health, 1999c
) together with the parallel government guidance related to performance indicators, partnership working and Health Improvement Programmes.
The Health of the Nation—A Policy Assessed (Department of Health, 1998a) is selected as the research base for this exploration of implementing evidence in practice, for three reasons. Firstly, the research is unusual in being related to health promotion at the highest policy level, where arguably the opportunity for significant impact is far greater than in interventions affecting much smaller parts of the system. That there is a lack of research in the policy implementation area was indicated by Oldenburg et al. (Oldenburg et al., 1999). Secondly, it was a major piece of work based on two studies commissioned by the Department of Health carried out by three universities, involving 250 semistructured interviews across 16 Health Authorities. Thirdly, it was published at a time when, as Professor John Swales says in the foreword:
These studies were conducted at the same time as the Government initiated development work on a new health strategy for England, Our Healthier Nation. They will form an important strand in the thinking for that new strategy.
The research is indeed referred to in Saving Lives: Our Healthier Nation [(Department of Health, 1999c), para. 1.48] and the discussion that follows should be seen in the light of this acknowledgement of the research.
This discussion is based on an analysis of the 17 key findings from The Health of the Nation—A Policy Assessed and the 25 recommendations arising from them. These were assessed initially by the author in relation to the degree to which they had apparently impacted on national policy guidance and its local implementation. These initial findings were discussed as a workshop `Policy and Practice: Factors Affecting Implementation' at the Society of Health Education and Promotion Specialists Conference `Narrowing the Health Gap' in April 1999. This elicited views from a group representing a range of Health and Local Authorities across the UK. As a result, five areas of particular importance in policy terms at a national level were identified: investment, alliance working, process and outcomes, shared ownership, and skill development. Discussion of these five areas forms the basis of the rest of this section.
In relation to investment, The Health of the Nation—A Policy Assessed found:
[Our Healthier Nation] did not cause a major readjustment in investment priorities by Health Authorities. (para. 2.5)
This finding may have influenced the inclusion in Saving Lives: Our Healthier Nation of funding as one of the six key principles and programmes which would enable the Government to meet the defined targets (para. 1.7), in a way which was not explicit in the preceding Green Paper. Recognition that this major policy direction requires a shift in funding priorities is welcome. Having said this, the funding referred to is attached to the following specific programmes: smoking cessation, healthy living centres, health action zones, NHS Direct and public health development. The White Paper also refers to the much greater sum (£21 billion compared to £800 million) allocated to the NHS as a result of the Comprehensive Spending Review. The responsibility for driving a move from secondary care to primary care and prevention at a local level is devolved to the Regional NHSE Offices, which approve the Health Improvement Programmes developed by each Health Authority, and to Primary Care Groups, increasingly responsible for allocating the majority of NHS funds in relation to services. If we look to the guidance underpinning this strategic approach, given by the Department of Health to inform decision making in this area, there is no attempt to establish a target for expected marginal shifts in mainstream investment towards identifiable preventive activity [HSC 1998/167 and HSC 1999/243 (Department of Health, 1998c, 1999b)]. Whether without a target, existing priority setting is strong enough to drive a shift in investment overall, remains to be seen.
The second important area, alliance working, will be discussed here along with the third element, measuring process as well as outcome. The material to be discussed includes Saving Lives: Our Healthier Nation, High Level Performance Indicators and Health Improvement Programmes. The Health of the Nation—A Policy Assessed recommends `use existing evidence on alliance building and outcomes' (para. 3.0).
Saving Lives: Our Healthier Nation sets targets related to mortality: reducing deaths from cancer in people under 75 by at least a fifth, reducing deaths from coronary heart disease and strokes in people under 75 by at least two-fifths, reducing deaths from accidents by at least a fifth, and reducing deaths from suicide by at least a fifth. There is an intention to develop standards and local targets. The White Paper does refer briefly to common purpose, resolving conflict, agreeing a shared approach, clear vision, adapting and developing common training programmes, as part of good practice. However, these are neither put in such a way as to attract priority attention in the way the targets do nor as specific as the evidence would allow.
That this evidence exists is widely known in the field of Health Promotion Specialists in the UK and related practitioners. Internationally, the Jakarta Declaration states [(WHO, 1996), p. 2]:
...there is now clear evidence that: comprehensive approaches to health development are the most effective...settings offer practical opportunities for the implementation of comprehensive strategies...participation is essential to sustain efforts...and health learning fosters participation.
An international systematic review of health alliances found that they do work to tackle the broader determinants of health and well-being in a sustainable manner as well as individual behaviour change. Key factors in their success are: level of community involvement in setting the agenda, desirable structures to facilitate planning and policies to promote health in rural areas (Gillies, 1997). Within the UK, the Wessex Institute of Public Health work on effective partnerships (Funnell et al., 1995) was based on both literature searches and field trials, and identified process indicators related to: commitment, joint working, resources, community involvement, accountability, communication and joint working.
Given the recommendation from The Health of the Nation—A Policy Assessed and the availability of evidence, it would seem reasonable to expect that process indicators would be referred to in key strategic documents such as Saving Lives: Our Healthier Nation and in the guidance related to performance management, which directs priority setting at Chief Executive level.
Turning to performance management, one aspect is the formulation of High Level Performance Indicators (NHSE, 1999). The indicators are grouped under six headings: health improvement (e.g. deaths under 75 from cancer), fair access (e.g. percent screened for breast cancer), effective delivery (e.g. percent of target population vaccinated), efficiency (e.g. day case rate), patient/carer experience (e.g. delayed discharge) and health outcomes (e.g. conceptions among girls aged 13–15). As this list suggests, despite the promising headings, the High Level Performance indicators reflect solely on clinical and service uptake outcomes. They are only one aspect of the Performance Management Framework, which will be complemented by local agreements shaped by the National Priorities Guidance [HSC 1998/159 and HSC 1999/242 (Department of Health, 1998b, 1999b)]. However, it is likely that if partnership working is not included in performance management at the highest level, it will be taken less seriously than those areas that are included.
As indicated in the discussion on investment, Health Improvement Programmes are a local planning tool where some of the recommendations from The Health of the Nation—A Policy Assessed might be met in terms of coordinated approaches to partnership working and measuring process. The Society of Health Education and Promotion Officers produced a set of indicators for Health Improvement Programmes, based on a social model of health (French and Learmonth, 1998). These included resource allocation in relation to addressing determinants of health, evidence of work to build social capital, effective public involvement and integration with other strategies such as Agenda 21. An informal scan of 12 documents in the Northern and Yorkshire Region suggested that the targets set by Saving Lives: Our Healthier Nation (i.e. reduce heart disease, cancer, suicide and accidents) had been consistently addressed. However, this small sample indicated a wide variety in the quality and depth of the partnerships informing the Health Improvement Programmes and that the sort of indicators proposed are the exception rather than the rule. In terms of evidence-based practice to continue to attempt to performance manage Health Improvement Programmes using nationally set targets for heart disease, cancer, suicide, etc., seems at best naïve. It is certainly not following the recommendation from The Health of the Nation—A Policy Assessed: `hold each group responsible for its contribution—both process and outcomes'. This in turn links with the fourth aspect to be explored, shared ownership, where The Health of the Nation—A Policy Assessed recommended: `shared ownership, horizontally and vertically...with a statutory framework and accountability, and wider ownership outside the NHS' (para. 3.0).
At a strategic level Saving Lives: Our Healthier Nation does declare a new duty of partnership which places on Local Authorities a duty to promote the economic, social and environmental well-being of their areas (HSC 1998/159). In the meantime at a local level organizations, both Health and Local Authority, are still responding primarily to guidance from their relevant departments. The programme of guidance includes Health Improvement Programmes, Best Value, Community Safety, Agenda 21 and a whole range of legislation related to the wider determinants of health. The National Priorities and Planning Guidance 1999–2002 (1998/159) sets out shared lead areas between Health and Social Services as being cutting health inequalities, mental health and promoting independence. Social Services has a lead on inter-agency working. Health leads on primary care, coronary heart disease and cancer. The much wider issues of shared ownership across all relevant agencies are left to local initiatives, with the exception of the special freedoms given to Health Action Zones. Some of the early findings from Health Action Zones are explored briefly in the final section of this paper. However for areas which are not Health Action Zones, the development of shared ownership remains a rather idiosyncratic process.
The final area to explore is skill development. The Health of the Nation—A Policy Assessed found a need for `a development strategy to equip managers and practitioners with the requisite skills and competencies' (para. 3.0).
While Saving Lives: Our Healthier Nation sets out a clear agenda for multidisciplinary public health in terms of nursing and related roles, the crucial role of managers in understanding and giving priority to its messages is not included. The lack of planning in relation to middle managers generally has been recognized nationally as a serious deficit (Baker, 1999). The devolution of training needs assessment to Training and Education Consortia without a clear national mandate to include all levels of skill development means there is still no coherent strategy within the NHS, never mind across all the agencies whose role is crucial to the implementation of Saving Lives: Our Healthier Nation.
In conclusion, in none of the five key areas examined (investment, alliance working, measuring process as well as outcomes, shared ownership and skill development) was research evidence consistently transferred into policy guidance. Even though research evidence is available, and the opportunity and intention to incorporate findings into practice is there, the process of implementation is at best partial. This is not a new finding [(Harrison, 1999), p. 128]:.
Weiss [(Weiss, 1991
In the UK the coordinating and facilitating role of Health Promotion Specialists has tended to focus attention on the professional group and its ability to deliver evidence-based practice. The previous section of this paper examined the three components of the process necessary for practitioners to be able to transfer evidence from elsewhere into practice. This exploration of the third element, implementation, at a national level illustrates that it is not always possible to put recommendations into practice. The level of responsibility held by Health Promotion Specialists for the degree to which evidence is implemented should therefore be tempered by recognition of the other political, social, economic and cultural factors likely to be involved. This concludes the discussion of the ways research-driven activity may contribute to health promotion activity.
The next two sections focus on health promotion-driven activity, which is contributing to the creation of an evidence base.
|
Generating evidence from practice |
|---|
|
TopAbstractIntroduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
The WHO [(WHO, 1998
), p. 6] has recommended that policymakers should:
Ensure that a mixture of process and outcome information is used to evaluate all health promotion initiatives.
and
A minimum of 10% of the total financial resources for health promotion initiatives should be allocated to health promotion.
On the face of it these recommendations may seem to advance the cause of effective and useful evaluation of health promotion. However, there are precedents for not attempting to monitor outcomes from small initiatives, but simply to accept good process measures. This is the rule adopted by Healthway, the prestigious Western Australian Foundation which uses tobacco taxes to sponsor arts and sports activities that advance health promotion. All projects funded by Healthway are subject to evaluation. However, projects which cost less than around £70 000 are only expected to record process indicators rather than attempt impact and outcome measures (Open University, 1997). The difficulty of tracing outcomes in a population from small local interventions means that the resource invested in trying to achieve this is simply disproportionate to the amount spent or the improvements in process which could be gained.
This leads us to consider the second recommendation from WHO. For England, using figures from 1997–98, the identified budget for Research and Development (£425 million) is around 1% of expenditure on the NHS. It seems disproportionate to demand 10% for activity related to health promotion. This is particularly so when only an estimated 1% of the health budget in the US and the UK is allocated to planned health promotion interventions (Limb, 1996; Clark and McLeroy, 1998). Given the difficulty in diverting real new resources into this area, this requirement could actually reduce the amount of health promotion activity going on, weakening the work rather than strengthening it.
Finally, the recommendations do not clarify appropriate roles for practitioners and researchers. This often results in Health Promotion Specialists, and other practitioners, trying to devise and implement robust evaluation, according to the criteria of scientific proof, in relation to their own practice while delivering it. This is like asking a GP to prove that clinical procedure x has better outcomes than procedure y. While this may occasionally be feasible with careful research design within a practice, it is more likely that a GP is simply asked to participate in a multicentre trial. As a general rule, if GPs are able to show auditable procedures, indicate the source material for their decision making and adhere to clinical protocols where they exist, then they would have fulfilled the expectations placed on them in terms of evidence-based practice at an individual level. Surely standards developed for Health Promotion Specialists should similarly be appropriate for the nature of their practice.
This role confusion leads to another dis-service to the field, because it is then assumed that practitioner-led evaluation is poor research. In fact, the parallel in the above example would be for evaluation to be like clinical audit for a GP. Audit has been defined as [(Last, 1995), p. 11]:
An examination or review that establishes the extent to which a condition, process or performance conforms to predetermined standards or criteria.
Effectiveness questions should be considered as part of the choice of procedure and audit may pick up unusual variations in outcome. However, audit is not expected on its own to demonstrate effectiveness, generate new knowledge or prove one procedure is better than another.
This does not mean to say that practitioner-led evaluation is not important. Carefully carried out, it may indicate useful effects, encourage good practice and prompt questions which generate larger-scale research. As Baum (Baum, 1998) has pointed out, these aspects are valuable in their own right, without attempting to provide a cost-effectiveness analysis or scientifically valid comparison between two methods.
If it involves other stakeholders, it may also help to involve them in the process of decision making and thinking through realistic aims for health promotion activity, which in itself is a valuable process. Baum's (Baum, 1998) example is developed in relation to a community based health promotion initiative. Initial outcomes expected by the stakeholders changed during the course of the work. For example, the community went from expecting to stop pollution, to gain skills in lobbying, prevent new polluting industries from setting up and giving a community voice to local environmental work. The Community Health Centre Manager went from expecting to reduce asthma admissions to seeing the Community Health Centre as responsive to local needs and an active member of the environmental forum. Local industry went from no expectations at all, to being able to communicate with the community group and developing a means to seem responsive to community concerns.
The next illustrative example examines one local piece of evaluation in practice.
|
An illustrative example: evaluation at local level |
|---|
|
TopAbstractIntroduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
The aim of this section is to assess the benefits that resulted from local evaluation of practice in terms of identifying useful effects, encouraging good practice, prompting questions that generate larger-scale research and influencing other stakeholders.
There are a number of valuable local frameworks for evaluation (Coyne and Jackson 1997; Duck, 1997; South and West Devon Health Authority, 1998). These are especially helpful in deciding when and what to evaluate. This point is conversely clearly expounded by Wright (Wright, 1999) when describing when not to evaluate. Some reasons she suggests are that time, skills or funding are not available; the work has been researched elsewhere; the results are likely to be ignored; managerial support is lacking.
This section concerns the framework developed and applied by the North Durham Health Promotion Service from 1994 to 1998. It was based on that developed by the Society of Health Education and Promotion Specialists (Society of Health Education and Promotion Specialists, 1992), enriched by adding indicators related to partnership working when these were produced by the Wessex Institute of Public Health (Funnell et al., 1995). Every Specialist in the department carried out a written-up evaluation of one piece of work each year, identified according to a number of criteria, e.g. innovative, resource intensive, a priority area, etc. This resulted in 55 evaluation reports over the period 1994–1997.
An analysis of this set of evaluations was carried out with the help of an independent assessor from Public Health, using a data extraction protocol which was piloted and then applied to a sample of 20 of the reports (Mackie and Learmonth, 2000). The aim was to ascertain the quality and value of the evaluation process, in developing good practice and assessing impact, in relation to the specific objectives set for each piece of work.
Demonstrating useful effects is the first benefit suggested for local evaluations. Many of the evaluations reported on by Mackie and Learmonth (Mackie and Learmonth, 2000) picked up good participation by the target audience, and changes in the practices of health promoters, organizational change, including policies, procedures and provision of services. This is especially encouraging given the wider impacts likely to arise from change at an organizational level, rather than simply the individual one. Some reports also identified outcomes, most commonly changes in service uptake, but also including changed knowledge or attitudes, health status, behaviour and very occasionally (as would be expected in a service with no capacity for longitudinal tracking) morbidity.
This leads on to the second benefit suggested in relation to local evaluation—encouraging good practice. The analysis found that one of the most important criteria in choosing areas for evaluation should be whether it was an important question for the Specialist concerned. This generates a higher level of interest and enthusiasm in following through the implications of the evaluation into practice. Conversely, the model developed could be applied routinely, almost mechanistically, if this ownership was missing. The recommendation from this was to integrate reflective practice into the evaluation process and to value the perceptions of the Specialist (Schon, 1991).
Thirdly, has the local evaluation process generated questions for the wider research agenda? While this information has not been systematically collected, there is no doubt that on a number of occasions the answer is yes. Two examples are: that an evaluation on the value of a service for young gay men led to an initial proposal for a systematic literature review to identify an appropriate and relevant tool to assess self-esteem among this group; and an evaluation study on the benefits of the healthy school award fed into the recent national evaluation and standard setting process. While the answer in this local example is partially yes, there is little doubt that this process of capturing research questions from the `front line' could be strengthened through closer links with academic bodies specializing in health promotion research.
Finally, did the work done influence other stakeholders and develop their understanding of the processes involved? The format chosen for the evaluation and the criteria for selecting pieces of work to evaluate were both formed by the major stakeholder—the Health Authority. The results of the evaluation work undertaken were shared with a wider range of stakeholders using a variety of formats: a single seminar, themed seminars and written reports. In addition in some years, both clinical audit and a Research Fellow from the University of Durham were involved in discussing the evaluations. However, none of these efforts was fully successful in creating ownership of the results of the evaluation, e.g. attendance by Primary Care representatives at the seminars was low.
To conclude this example then, the evaluation work was at least partly successful in relation to the four benefits identified from evaluation as part of local practice.
|
Conclusions: completing the circle |
|---|
|
TopAbstractIntroduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
This paper began by outlining some of the issues at the interface between health promotion and research, concluding that the research approaches taken by participative inquiry methods and utilization-focused evaluation are most in keeping with the goals of health promotion practice. It then examined the practical requirements necessary for the key process of transferring research findings into practice: being able to find, appraise and implement findings. The third of these, implementation, was explored in relation to a piece of research in relation to national policy. None of the five areas discussed showed consistent transfer of evidence-based recommendations into practice.
The paper then went on to look at the complementary process of generating evidence from practice, through identifying the benefits of local evaluation. These were then applied to a local example, which demonstrated useful effects and encouraged good practice. The example was less successful in consistently generating ideas for further research or in involving stakeholders.
The first of these, the interface between health promotion and research, requires the extension of networks involving practitioners and researchers, to facilitate the transfer of ideas for research from practitioner-based evaluation, to generate academically rigorous research. This endorses recommendations from research in this area (Tilford and South, 1999).
In terms of involving stakeholders, there are recent developments in the field that specifically address this aspect of evaluation. Wimbush and Watson (Wimbush and Watson, 2000) have developed a framework for the Scottish Health Education Board, which recognizes the different values placed on evaluation by different stakeholders: policy makers, programme managers, practitioners, community groups, users and professional evaluators (including academic researchers). The framework also identifies both evaluation and research questions which may be asked at each stage of the process from planning, through to early start up, establishment and fully operational stages of the implementation process, and finally dissemination.
This is one specific example where the related activities of evaluation and research have been drawn together, in a framework designed for health promotion practitioners. Another example, also with strong links on stakeholder involvement, is offered as part of a Training Manual for Community Development (http://www.scdc.org.uk). This work builds on the substantial wealth of experience developing from Canada in this area (Labonte, 1998).
Another developing strand in the UK, this time at the level of policy implementation, where the problems of evaluating complex community interventions are being addressed relates to the Health Action Zones. The initial report from the Personal Social Research Unit (Personal Social Research Unit, 1999) for the National Evaluation of the 26 Health Action Zones recognizes the difficulty in determining causality in a context where disparate elements have an impact on outcomes. A method is proposed based on realistic evaluation, which identifies context–mechanism–outcome configurations. The aim is to increase knowledge about what works for whom in what circumstances, through cumulative comparisons within and between different elements of the programme. In addition, a theory of change specifies how activities will lead to intermediate and long-term outcomes, and the contextual conditions that may help them. This presents the challenge of eliciting a theory of change among the diverse groups of individuals involved in planning and delivering initiatives. It also requires an analytic stance different from the intuitive stance of most practitioners. However [(Jacobs, 1999), pp. 12–13]:
The approach helps to surface information about the organizational, management and political processes at work within partnerships and the ways in which these influence how stakeholders deliver programmes.
The funding for the evaluation of the Health Action Zones is currently around 0.25% of the investment being made in them. Yet the sort of methodology being developed here relates directly to the concerns and realities of evaluation of health promotion.
So, to close the circle between the activities of research and health promotion, what is required? Four key recommendations emerge in relation to the transfer of evidence into practice. The first is to apply more widely appropriate research methods compatible with the goals of health promotion, including participative inquiry and utilization-focused evaluation. The second is to develop more inclusive criteria to consistently assess the validity of a range of research approaches, complementing the hierarchy of evidence used for systematic reviews to date. The third is to streamline the collection and dissemination of this material in England through the Health Development Agency new role. Finally, we need to ensure the rigorous implementation of evidence at a policy level.
A further four key recommendations emerge in relation to the flow from practice to research. The first is to build on the approaches taken by the Health Action Zone evaluation team in addressing complex interactive whole system interventions. The second is to develop coordinated use of frameworks such as the one developed by Wimbush and Watson (Wimbush and Watson, 2000) to facilitate the transfer of experience within the field. The third is to actively involve stakeholders in evaluating specific health promotion activity. Finally, there is a need to develop effective research networks involving practitioners and researchers to facilitate the transfer of ideas from the field into research.
These eight steps taken together would help create an iterative process, in which the flow of evidence from research to practice, and of ideas from practice to research, would strengthen and enrich the theory and practice of promoting health.
|
Notes |
|---|
The paper is based on a presentation to the Health Promotion Research Conference: Challenges and Opportunities, Leeds Metropolitan University, September 1999.
|
References |
|---|
|
TopAbstractIntroduction: the interface...Utilizing research in practiceAn illustrative example:...Generating evidence from...An illustrative example:...Conclusions: completing the...References |
|---|
Baker, A. (1999) It's tough in the middle. Health Management, April, 24–25.
Baum, F. (1998) Community-based health promotion. In Davies, J. K. and Macdonald, G. (eds), Quality, Evidence and Effectiveness in Health Promotion: Striving for Certainties. Routledge, London, pp. 68–89.
Bowling, A. (1997) Research Methods in Health Investigating Health and Health Services. Open University Press, Buckingham.
Clark, N. M. and McLeroy, K. R. (1998) Reviewing the evidence for health promotion in the United States. In Davies J. K. and Macdonald G. (eds), Quality, Evidence and Effectiveness in Health Promotion. Routledge, London, pp. 21–39.
Cochrane, A. (1972) Effectiveness and Efficiency: Random Reflections on Health Services. Nuffield Provincial Hospital Trust, London.
Coyne, A. M. and Jackson, D. 1997 Evidence Based Practice—A Framework for Operational Guidance. Northwest Health Promotion Specialists and Managers Group.
Delaney, F., South, J. and Tilford, S. (2000) Northern and Yorkshire Region: Research and Development Needs in Health Promotion. Centre for Health Promotion Research Faculty of Health and Environment, Leeds Metropolitan University, Leeds.
Department of Health (1998a) The Health of the Nation—A Policy Assessed. HMSO, London.
Department of Health (1998b) HSC 1998/159 Modernising Health and Social Services: National Priorities Guidance 1999/00–2002/03. HMSO, London.
Department of Health (1998c) HSC 1998/167 Health Improvement Programmes Planning for Better Health and Better Health Care. HMSO, London.
Department of Health (1999a) HSC 1999/242 Modernising Health and Social Services National Priorities Guidance 2000/01–2002/03. HMSO, London.
Department of Health (1999b) HSC 1999/243 Health Authority Revenue Cash Limits 2000/01. HMSO, London.
Department of Health (1999c) Saving Lives: Our Healthier Nation. HMSO, London.
Duck, J. (1997) A Framework for Evaluation. Redbridge and Waltham Forest Health Authority.
French, J. and Learmonth, A. (1998) Health Improvement Programmes—The Role of Health Promotion Services and the Application of a Social Model of Health to Guide the Process. Society of Health Education and Promotion Specialists, Glasgow.
Funnell, R., Oldfield, K. and Speller, V. (1995) Towards Healthier Alliances: A Tool for Planning, Evaluating and Developing Healthy Alliances. Wessex Institute of Public Health Medicine, Health Education Authority, London.
Gillies, P. (1997) The effectiveness of alliances or partnerships for health promotion. Paper presented to the Fourth International Conference on Health Promotion, Jakarta.
Gillies, P. (2000) Evidence Base 2000. Health Education Authority, London.
Harrison, D. (1999) Social System Intervention. In Perkins, E. R., Simnett I. and Wright, L. (eds), Evidence-based Health Promotion. Wiley, Chichester, pp. 125–137.
Health Education Authority Series of Systematic Reviews. Website: http://www.had-online.org.uk.
Hicks, N. (1997) Evidence based health care. Bandolier, The Churchill, Oxford, 4(5), 8.
Jacobs, B. (1999) Partnerships in Pittsburgh: the evaluation of complex local initiatives. In Osborne, S. P. (ed.), Managing Public Private Partnerships for Public Services: An International Perspective. Routledge, London.
Labonte, R. (1998) A Community Development Approach to Health Promotion: A Background Paper on Practice Tensions, Strategic Models and Accountability Requirements for Health Authority Work on the Broad Determinants of Health. Prepared for the Health Education Board of Scotland Research Unit on Health and Behavioural Change, University of Edinburgh.
Last, J. M. (1995) A Dictionary of Epidemiology. Oxford University Press, Oxford.
Learmonth, A. and Watson, N. (1999) Constructing evidence based health promotion: perspectives from the field. Critical Public Health, 19, 317–333.
Limb, L. (1996) Health of the Nation under scrutiny. Health Service Journal, 14, 8.
Mackie, P. and Learmonth, A. (2000) Evaluating effectiveness in health promotion: a case of re-inventing the millstone? Health Education Journal, 59(3), in press.
MacVicar, M. (1999) Integrating research into nursing practice. In Perkins, E. R., Simnett, I. and Wright, L. (eds), Evidence-based Health Promotion. Wiley, Chichester, pp. 298–308.
Muir Gray, J. A. (1997) Evidence-based Health Care: How to Make Health Policy and Management Decisions. Churchill-Livingstone, New York.
NHSE (1999) Quality and Performance in the NHS: High Level Performance Indicators. Quarry House, Leeds.
Oldenburg B. F., Sallis, J. F., French, M. L. and Owen, N. (1999) Health promotion research and the diffusion and institutionalization of interventions Health Education Research, 14, 121–130.
Oliver, S., Peersman, G. and Oakley, A. (1999) Discrepancies in findings from effectiveness reviews: the case of health promotion for older people in accident and injury prevention Health Education Journal, 58, 66–77.
Open University (1997) K301 Promoting Health Skills, Perspective and Practice. Audio Cassette 4: Evaluating Health Promotion.
Patton, M. Q. (1997) Utilisation-Focused Evaluation: The New Century Text. Sage, London.
Peersman, G. Harden, A., Oliver, S. and Oakley, A. (1999) Discrepancies in findings from effectiveness reviews: the case of health promotion interventions to change cholesterol levels, Health Education Journal, 58, 192–202.
Personal Social Services Research Unit (1999) Health Action Zones: Learning to Make a Difference. Website: www.ukc.ac.uk/pssru/.
Reason, P. (1994) Three approaches to participative inquiry. In Denzin, N. K. and Lincoln, Y. S. (eds), Handbook of Qualitative Research. Sage, Thousand Oaks, CA.
Schon, D. (1991) The Reflective Practitioner: How professionals Think in Action Arena. Ashgate, Aldershot.
Scottish Community Development Centre (1999) Evaluation Indicators—An Holistic Perspective. Training Manual for Community Development. Website: www.scdc.org.uk.
Silverman, D. (1993) Interpreting Qualitative Data Methods for Analysing Talk, Text and Interaction. Sage, London.
Society of Health Education and Promotion Specialists (1992) Framework for Developing Quality in Health Education and Health Promotion. SHEPS GMS, Belfast.
Society of Health Education and Promotion Specialists (1999) A Quality Framework. Available from S. Maclennan, 64 Terregles Avenue, Glasgow G41 4LX, UK.
South, J. and Tilford, S. (2000) Perceptions of research and evaluation in health promotion practice and influences on activity. Health Education Research, 15, 729–741.
South and West Devon Health Authority and the University of Bristol Health Promotion Research Programme (1998) Developing a Framework for Effectiveness in Health Promotion. South and West Devon Health Authority and the University of Bristol Health Promotion Research Programme.
Thomson, R. (1998) Sources of evidence and information. Presented at the Clinical Effectiveness Workshop, Durham.
Tones, K. (2000) Evaluating health promotion: a tale of three errors. Patient Education and Counselling, 39, 227–236.
US Preventive Services Task Force (1989) Clinical Guide to Preventive Services: Report of the US Preventive Services Task Force. Williams & Wilkins, Baltimore, MD.
Visscher, A., Karstens, S. and Bosker, R. (1999) Evidence on the intended and unintended effects of publishing school performance indicators. Paper presented to Evidence Base Policies and Indicator Systems, Second International Inter-disciplinary Conference, University of Durham.
Weiss, C. H. (1991) Policy research: data, ideas or arguments? In Wagner, P., Weiss, C. H., Wittrock, B. and Wollman, H. (eds), Social Sciences and Modern States. Cambridge University Press, Cambridge, pp. 307–332.
Wimbush, E. and Watson, J. (2000) An evaluation framework for health promotion: theory, quality and effectiveness Evaluation, 6, 301–321.[Abstract]
WHO (1986) Ottawa Charter for Health Promotion. WHO, Geneva.
WHO (1998) Health Promotion Evaluation: Recommendations to Policymakers. WHO, Geneva.
WHO (1996) The Jakarta Declaration on Health Promotion into the 21st Century. WHO, Geneva.
Wright, L. (1999) The proof of the pudding. In Perkins, E. R., Simnett, I. and Wright, L. (eds), Evidence Based Health Promotion. Wiley, Chichester, pp. 393–403.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  R. Cameron, S. Manske, K. S. Brown, M. A. Jolin, D. Murnaghan, and C. Lovato Integrating Public Health Policy, Practice, Evaluation, Surveillance, and Research: The School Health Action Planning and Evaluation System Am J Public Health, April 1, 2007; 97(4): 648 - 654. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. Rosen, O. Manor, D. Engelhard, and D. Zucker In Defense of the Randomized Controlled Trial for Health Promotion Research Am J Public Health, July 1, 2006; 96(7): 1181 - 1186. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. Pankratz, D. Hallfors, and H. Cho Measuring perceptions of innovation adoption: the diffusion of a federal drug prevention policy Health Educ. Res., June 1, 2002; 17(3): 315 - 326. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||