Health Education Research, Vol. 18, No. 6, 664-677,
December 2003
© 2003 Oxford University Press
The European Smoking Prevention Framework Approach (ESFA) project: Observations by Six Commentators
 |
ESFA Mini-series: an experiment |
|---|
 Top ESFA Mini-series: an experiment Commentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
In issue 18(5) of the Journal we published the first two papers of the European Smoking Prevention Framework Approach (ESFA) mini-series. The third and final paper of this series appears above. All three papers describe an ambitious smoking prevention project involving collaboration between six European nations. As promised in that issue, we now present the reactions of a ‘Commentary Group’ to these papers. The three articles had already been subjected to the Journal’s customary review process. Accordingly, the members of the Commentary Group were asked not to merely repeat that procedure, but rather to provide more general reflections and reactions, e.g. to the subject matter or the research paradigm itself.
We are very grateful to our commentators but, above all, our thanks go to our friends and colleagues at Maastricht University for sharing their research with us and agreeing to cooperate with this ‘experimental’ mini-series.
We also welcome observations from readers—perhaps in the form of Letters to the Editor!
Keith Tones
Editor
|
Commentary 1: The ESFA Mini-series |
|---|
|
TopESFA Mini-series: an experimentCommentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
Professor Jackie Green, Centre for Health Promotion Research, Leeds Metropolitan University
The ESFA team is to be congratulated on the ambition of this initiative and the rigor with which the researchers have carried it through. Studies of this nature, located towards the positivist end of the positivist–interpretivist spectrum, will inevitably be caught up in epistemological debates—and this pre-supposes that a spectrum exists rather than discrete and mutually exclusive positions. However, for those who subscribe to a methodologically plural position—and there is considerable support for this position in relation to health promotion research (WHO, 1998
)—the selection of papers published here makes a valuable contribution to our understanding of smoking in adolescence and the issues associated with developing effective interventions, both generally and at the European level.
Of particular interest is the insight into the effects of friends’ and parents’ smoking status—issues which young people themselves may be unaware of, or reluctant to acknowledge, given adolescents’ concern with the establishment of autonomy. Some time ago, May (1993) challenged simplistic notions of peer pressure. Qualitative researchers such as Michell (1997) have demonstrated the complexity of peer influence. The fact that the ESFA study does not provide support for conventional notions of peer pressure is consistent with this emerging view and helps to establish the need for more sophisticated interpretations.
A somewhat paradoxical outcome of the study is that it provides evidence of the difficulty of obtaining appropriate control or comparison groups and random assignment in community intervention trials. This has been a major criticism of experimental approaches to evaluation [see, e.g. (Green and Tones, 1999)].
A further interesting feature of the study is the approach to the development of the intervention itself. Rather than attempting to impose a standardized protocol on all participating countries, the intervention includes core objectives derived by consensus along with additional optional objectives. The methods used to achieve the objectives also vary from country to country in line with cultural differences and pedagogic practice. While this could potentially be a source of criticism from a purely experimental perspective, it has the advantage of testing the intervention under naturalistic conditions. The evaluation therefore focuses on effectiveness, which is concerned with the achievement of objectives under normal conditions rather than efficacy, a term which, as used by Brook and Lohr (Brook and Lohr, 1985), refers to the effectiveness of a programme under ideal conditions which may be impossible to replicate or maintain over the longer term. Maintenance of programme fidelity is highly problematic and there is a tendency for teachers to modify programmes to meet curriculum needs at the school level—and to circumvent any attempts to teacher-proof materials. For example, the Drug Abuse Resistance Education Programme (DARE) was subject to considerable re-invention and modification [see (Rogers, 1993)]. Building some variability into the ESFA programme, while at the same time adhering to a common theoretical framework, is a pragmatic response to this issue. It ensures that programmes are relevant to the national context of participating countries and that the evaluation takes place under relatively naturalistic conditions.
A final point concerns context. There can often be a tendency for evaluations to focus on interventions in isolation. Consideration of context is central to the approach of realistic evaluation developed by Pawson and Tilley (Pawson and Tilley, 1997). The international scope of the ESFA study highlights the importance of context both in the development and implementation of the intervention and also in influencing outcomes.
|
Commentary 2: The ESFA Mini-series |
|---|
|
TopESFA Mini-series: an experimentCommentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
Dr Lilian Lechner, Department of Psychology, Open University of The Netherlands
The three papers describe a large and unique project on smoking prevention, including six different European countries. Few studies have been done on this scale in Europe. It is a great achievement both to get funding for such a project, and to persuade six countries to participate and cooperate! There is much to learn from this large-scale approach. However, using six different countries proves to be both strength and a potential weak point of the study.
When working with so many countries, cultural differences need to be acknowledged. These differences present a great dilemma. On the one hand, as a researcher you want all interventions to be equal to each other. However, on the other hand, from the health promotion angle, it is important to encourage all the parties involved to make modifications to the intervention in order to meet the particular needs of the target population. This inevitably leads to alterations in the intervention. Now, while attempts to achieve a perfect research design and the best possible intervention could result in having the best of both worlds, it might equally result in the worst of both. For instance, community participation is a sine qua non of true health promotion and local initiatives are ideally placed to achieve this goal. However, local initiatives typically lead to a variety of different approaches and all these differences make it difficult or even impossible to evaluate programme effects, when these are assessed using traditional evidence-based medicine methods.
It is understandable that due to cultural differences the interventions have some differences in details. Moreover, following the community-based approach it is even a very relevant and essential step that initiatives should come from the populations in which the interventions should take place. However, the differences in interventions as described in the ESFA papers are so substantial that comparability between the different countries seems hardly possible—in fact, they give the impression that six different interventions were simply assembled under one name! This makes it questionable to put all experimental interventions together as one condition.
In recent years, a good deal of discussion has taken place regarding the relevance of more traditional research methods for evaluating health promotion activities (Rootman et al., 2001) and a good deal of critical thought has been devoted to the question of what counts as evidence of effectiveness? Some of the older traditions suggest that the randomized controlled trial is the only acceptable method providing valid evidence of effectiveness. ESFA tried to use this method, in order to assess the effects of the intervention on smoking behavior and its determinants. The conclusion from the results seems to be that it is unclear whether the ESFA interventions were effective or not.
Increasing numbers of researchers and practitioners in health promotion and related fields argue that the nature of health promotion requires a variety of evaluation approaches, and that more appropriate techniques need to be developed (Rootman et al., 2001). As was concluded clearly by Stewart-Brown (Stewart-Brown, 2001), concentration on experimental methodologies, especially the randomized controlled trial, is likely to be misleading in school health promotion research. Evaluations of school health promotion interventions need to take into account the nature of the intervention. Researchers should recognize the limitations of the experimental model in interventions that require active engagement of participants. Research and development that take equal account of the contribution made by different methodologies are more likely to be successful in the long run (Stewart-Brown, 2001).
Knowing that there were so many differences between the interventions in the six countries, it would probably have been better to put more emphasis on the process evaluation side of the research. For instance, what are the elements in an intervention that have the capability to be effective in preventing smoking behavior? The results of the process evaluation are needed in order to be able to really interpret the results found in the effect evaluation. With the data as presented now, little can be said of what elements are more or less effective and what the reasons were for (lack of) effectiveness.
The ESFA papers mention goals, objectives and theoretical methods without explicitly describing them. It would have been useful to gain more insight into the way in which each country developed its own materials taking into account the consensus of goals and objectives. Seeing the many differences in the interventions between the different countries, one may wonder if more strategic rules or guiding principles should have been presented to the different countries in order to get more comparable interventions. Furthermore, an additional question seems to be whether the goals and objectives of the project were precisely related to the theory and methodology used. Measurement is particularly challenging when concepts that are used are unclear or subject to debate, as may have been the case in this project (and indeed is not uncommon in many health promotion initiatives).
The ESFA project elected not to follow the practice of piloting programmes, i.e. of first designing an intervention, then testing it on a small scale for effectiveness, and only after effectiveness had been proven, implementing it on a larger scale. Instead, the intervention was based on more general principles of potential effective smoking interventions in the literature. Several different intervention components were placed together and were directly implemented in the six countries.
The increasing complexity of health promotion interventions poses considerable methodological challenges. This complexity requires multiple strategies at multiple levels. For example, it would have been useful to not only gather data from students by means of written questionnaires, but to also include other measurement methods (particularly qualitative methods) and other target groups (parents, peers outside of school). Designing and constructing appropriate evaluations in complex projects such as ESFA is extremely difficult. A key challenge is to determine the optimal intervention package and the efficacy of each of its parts. However, this probably can only be done by first testing the effectiveness of the different intervention components on a smaller scale and by using multiple evaluation methods. By using very complex interventions and only limited evaluation methods, it is virtually impossible to ‘filter out’ from the results what are the effective or ineffective components of the interventions—and what are the key factors for success. As a result, it is difficult to provide useful guidelines on intervention aspects for future projects.
A basic step in planning a large intervention project such as ESFA would, therefore, seem to be to first test the effectiveness of several components on a small scale before implementing an intervention with so many different components on a very large scale. On the other hand, such an approach conflicts with the current emphasis on a ‘formative’ model of health promotion that stresses the importance of modifying the intervention in response to the changing needs and wants of the community.
There is an additional serious risk with the approach used by ESFA. Although the project has provided valuable insights into the design and management of large-scale interventions—particularly the problems of achieving inter-sectoral collaboration—it could also have a downside. Since no definite conclusions can be drawn as to whether ESFA was or was not effective, it could result in feelings of disappointment in both participating parties as well as those who have provided the funding! In other words, a lack of explicit results could lead to feelings of failure and consequently a determination never to repeat a project of this kind. In that way, it could impede the opportunities for future large-scale projects, even those that might have been better tested for effectiveness on a smaller scale.
|
Commentary 3: The ESFA Mini-series |
|---|
|
TopESFA Mini-series: an experimentCommentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
Alan S. Rigby, Academic Department of Cardiology, University of Hull, Kingston-upon-Hull, UK
...smoking prevention activities should be embedded in a comprehensive approach that specifically addresses youngsters, but should also aim at changing the smoking and parenting behaviour of parents and teachers, and at creating non-smoking policies where youngsters congregate. (Charlton et al., 1990)
That smoking is a major preventable form of premature mortality is not in question. It must also be disappointing to health care researchers that while smoking prevalence has declined in adults, this has not been matched in teenagers. The tobacco industry continues to be ever more subtle in its advertising. Thus, the research aims of the ESFA study were laudable though, perhaps, never actually achievable.
The ESFA team carried out what they termed a community intervention trial in which six European countries participated.
Within each country two regions took part, one designated the control the other the intervention. Randomization into control of intervention was possible in most if not all countries. Such trials have had widespread application, including smoking cessation studies (Hyland et al., 2002). The parallel design in clinical trials research is the cluster randomized controlled trial. Cluster randomized controlled trials have had an impact in general practice research where the ‘cluster’ is the general practice and the ‘cluster size’ is the number of patients per practice. In the ESFA trial, the ‘cluster’ would be the region and the ‘cluster size’ would be the number of students per region, for example. The key thing is that there are specific design issues in cluster randomized controlled trials that must be addressed. One of these is sample size estimation. Hence, variation within a cluster is assumed to be smaller than variation between a cluster (this seems obvious if we refer to the general practice example quoted above); thus, larger sample sizes are indicated (Donner et al., 1990; Klar and Donner 2001). Looking at the ESFA study (De Vries et al., 2003a–c), although power calculations were carried out, no attempt was made to take account of the cluster research design. Hence, it is likely that the sample size calculation was an underestimate of the numbers of students actually required.
Sample size estimates for cluster randomized controlled trials are inflated by calculating something called the design effect (DE) as follows: DE = 1 + (c – 1) * ICC where c = average cluster size and ICC = intraclass correlation coefficient. The ICC is often unknown and we have to resort to a guess (e.g. 0.05, 0.1). Otherwise estimates of the ICC for various situations are available in a Health Technology Assessment report (Ukoumunne et al., 1999).
Why is it so important to get the sample size correct? In their paper discussing short-term effects of the ESFA study, a logistic regression model showed no overall effect for the intervention (De Vries et al., 2003c). The question that De Vries et al. would doubtless like to answer is whether this non-significant finding is simply due to lack of power. We shall never know the answer!
|
Commentary 4: The ESFA Mini-series |
|---|
|
TopESFA Mini-series: an experimentCommentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
Professor Jim Connelly, Centre for Primary Care and Public Health, University of Reading, UK
General
The original vision and motivating purpose of this exercise seems to be as much to do with contributing to the development of a common ‘European identity’ as it is to do with testing an intervention to decrease regular smoking in adolescents. It is ironic, therefore, that what started out as a bid for coherence in preventing smoking seems to end with a call for valuing separate national cultures and contexts.
The precedents for the ESFA were not promising. Large community intervention trials aimed at smoking cessation (e.g. the COMMIT study) should have been a salutary basis for planning of this study. For instance, the lessons from COMMIT include the need to consider the effectiveness of the intervention both in terms of its theoretical appropriateness and in its delivery (COMMIT Research Group, 1995). Don Nutbeam singles out three elements as ‘basic’ to evaluating a programme (Nutbeam, 1998)—with slight modifications I suggest that thinking about these issues at the planning stage may well have proved useful for the ESFA team:
(1) Consider programme reach—will the programme reach all the target population?
(2) Programme acceptability—is the programme acceptable to the target population?
(3) Programme integrity—will the programme be implemented as planned?
The subsequent history of the ESFA studies points up the necessity of these considerations at the time of, not after, the implementation of the programme.
For each of Nutbeam’s issues the ESFA studies is found to be wanting. Coverage was only, at most, 75%, acceptability (by the collaborators) was low and the actual intervention delivered differed greatly from place to place. In their conclusions the authors at times seem to celebrate the observed diversity of approaches and findings, calling finally to a return to ‘dismantling’ strategies—so much for a common European identity. Why did the ESFA studies generally fail in its aims, what is behind the weaknesses that become all too obvious as the studies proceeded? Above all, what lessons should we learn from the ESFA events?
Empiricism versus realism
In my view the major problem inscribed in the ESFA study from its inception was its slavish attempted adherence to a randomized controlled trial design. The randomized controlled trial derives from the special and contrived circumstances of laboratory science, where all environmental influences other than the experimental intervention are strictly controlled. The experimenter is conceived as an irrelevant figure, all that is required is that he/she observes accurately and reliably what is there to observe. The amount or dose of the intervention is precisely measured and is delivered without any deviation from the experiment’s protocol. In other cases is may be titrated until an effect is observed. What is seen (observations) are simply ‘seen’ or are read off the recording instruments, are unambiguous and need little or no interpretation to become meaningful. Such is the paradigm of experiment, at least in its positivist incarnation (Connelly, 2001). What is surely now apparent is the large gap in reality between this paradigm and the social, economic, political, psychological and cultural circumstances that are likely to impact on a six country anti-smoking effort produced in a Europe at the turn of the second millennium.
A listing of the ‘protocol deviations’ from an envisioned randomized controlled trial paradigm misses the central critical point. Once again the problem is not that the ESFA was a poorly conducted randomized controlled trial, it is that the reality of the social world was not even considered in the design and conduct of this anti-smoking effort. It is hard not to conclude that the insistent power of the empiricist paradigm for whom only the closest approximation to the experimental set-up is acceptable science seems to have trumped all other perspectives. So what are the alternatives?
Realism in health promotion
It is fascinating to read the interpretations brought to their results by the authors of the ESFA. Rather than rely on aggregated and general findings, they time and again draw attention to distinctions and contrasts between countries. This is shown in terms of distinctions in national or regional policies (e.g. regarding existing school based anti-smoking education), in actual circumstances for the delivery of the interventions (such as the use of different communications media and differing numbers of sessions) and in the position of adolescents themselves (whether receiving an academic or technical curriculum). The authors are here demonstrating a ‘taken-for-granted’ appreciation of the different contexts existing in different countries and regions. They are using this tacit knowledge to help explain (and occasionally justify) both the reasons for ‘protocol deviations’ and the actual observations themselves. They are, in my re-description of their actions, engaged in a post hoc realist evaluation of the ESFA observations. This realism, however, remains constrained by its continued adherence to the empiricist paradigm and by its, unavoidable, meagerness. The latter is due to the post hoc realist reconstruction—which as things turned out is all that can be attempted.
A realist reconstruction of the ESFA
In my earlier brief description of the main elements of critical realism as applied to health promotion (Connelly, 2001), the importance of a theoretical standpoint was emphasized. In the ESFA it appears that a rather general social cognition theory was chosen without adequate consideration of its likely effectiveness in the contexts encountered. A realist approach would be to first construct groups of adolescents (grouped as a starting point by taking cultural and material differences as important). Second, each group would be consulted to explore and identify what pathways and influences might be operating to lead to trying smoking or not smoking regularly and quitting smoking. These pathways contribute the ‘generative mechanisms’ (Bhaskar, 1975, 1989; Connelly, 2001) that will be the targets for interventions. Possible interventions are also explored with the adolescent groups, these are placed in order of perceived likely effectiveness, and this information is used to elaborate the actual intervention content, form and delivery. So far this realist approach has paid attention to differing contexts, and has (most likely) identified different generative mechanisms and interventions. As important is the elaboration of expected observations. These observations are specifically tailored to the context and triggered mechanism of intervention. For example, the accounts given by adolescents who successfully quit smoking may highlight the importance of improving self-efficacy which itself was brought about by a ‘new start’ life-event (e.g. a change of teacher, exclusion of a bully, etc.). The task then is to attempt to engineer one or more ‘new starts’ to have a group (or subgroup) impact. Importantly, the ‘new start’ may well consist in improving material conditions and prospects. The observations that are important in this example derive from the ‘fit’ between the new start event and its perception as such by the adolescent, and the effect this has on self-efficacy. Realist theory would then predict that anti-smoking education would be more effective (in terms of the observation of regular smokers quitting) amongst those who had increased their level of self-efficacy. Given this alternative realist paradigm we may now suggest how the realist reconstruction of the ESFA study might be undertaken.
Realism and the ESFA
From a realist perspective randomly allocating half of the study population is wasteful of half the population. The inherent variability in the efficacy of an intervention determines the number of observations required to reach conventional statistical significance, immediately losing half of your study population is therefore very inefficient. Moreover, when one is dealing with behavioral outcomes the ‘control condition’ cannot be assumed to be inert (Connelly, 2002). Adolescents like other human beings who are informed that they are not in the intervention group will construct their own meanings—some of these might become causes for starting, maintaining or quitting smoking. A ‘matched pair’ comparison between regions or schools looking at differences (effect size) may miss rather large (but shared) positive or negative effects (Connelly, 2002).
In their book Realistic Evaluation, Pawson and Tilley (Pawson and Tilley, 1997) point out the sizeable variation in the smoking cessation rates both within matched pairs and between pairs of communities randomized in the COMMIT study. This study lumped these communities together and their ‘intention-to-treat’ analysis revealed no significant overall effect. Yet this illustrates nicely the realist point—why was there the observed variation? What contextual factors meant that the COMMIT interventions were successful in some areas but unsuccessful in others? On the other hand where communities (‘contexts’) seemed very similar were the different observed effects due to differences in intervention type or intensity? The ESFA researchers should therefore attempt to learn from their varied findings—for instance, what are the common characteristics of those schools that reported large or small quit rates? In the longitudinal analysis they could distinguish between those adolescents who quit between T1 and T2, those who initiated and those who maintained their smoking, looking at whether their best friends or parents smoking changed. It is only by setting up prior hypotheses, based on experience, consultation and, where appropriate, theory and by specifying what mechanisms might work, producing which observations (when?, where?), that we begin to use rigorous scientific designs that are appropriate.
What is considered appropriate here must engage with the reality of the contemporary socio-cultural world of adolescents in different European countries, it must acknowledge the inescapable ‘openness’ of the social world, which cannot be closed even in principle, and it must question the usefulness of very general theories applied to specific contexts. The ESFA study is indeed very instructive, but, finally, its lessons are about taking the amassed experience we all have about our social world seriously, and being scientific in our social and health research.
|
Commentary 5: The ESFA Mini-series |
|---|
|
TopESFA Mini-series: an experimentCommentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
Chris Roberts, Head of Research and Evaluation, Health Promotion Division, Welsh Assembly Government, Wales
Background comments
The evidence base for interventions that are effective in preventing uptake of smoking is inconclusive at best. However, a number of recent reviews do seem to be in agreement that a whole school approach has potential and could also be applied in other areas, such as the promotion of healthy eating and physical activity [e.g. (Anderson, 1999; NHS Centre for Reviews and Dissemination, 1999; Rees et al., 2001; Shepherd et al., 2001)]. The ESFA project was an ambitious international attempt to add to the evidence base on adolescent smoking prevention and the published series of papers are a welcome addition. My brief response to these papers outlines what I believe to be the key achievements of the study and some of the main implications for future practice. It is hoped that the issues raised below will contribute to the ongoing debate in this important area of public health policy, practice and research.
What does the ESFA study tell us?
The ESFA authors have presented an honest portrayal of their experiences, emphasizing the study weaknesses that are apparent with the benefit of hindsight. Before drawing out some of the themes to emerge from the study, it is important to acknowledge that a number of valuable findings have been identified during the course of the project. One of the study’s stated objectives was to conduct theoretical research on smoking onset and this can be seen in, for example, the focus on peer pressure at the expense of parental influence. Useful descriptive evidence has also been produced, such as confirmation of gender differences in smoking, suggesting a need for dedicated interventions. Importantly, the multilevel approach underpinning the study recognizes the need for complex solutions to complex problems, although this brings with it challenges for programme implementation and evaluation. There is also much to be said for international collaboration, both in terms of building capacity and having the ability to examine differences (and commonalties) between countries.
Whilst recognizing these achievements, it should be noted that the authors sensibly describe their work as a pilot study and inevitably a variety of lessons can be learnt to shape future work. I will touch on three areas for further debate: implementing school-based health promotion interventions, evaluation of integral prevention programmes and the importance of context.
Lessons for implementing school-based health promotion interventions
Much recent practice, including the ESFA project and the wider health-promoting school movement, has been based on an integral approach to prevention, recognizing the importance of the broader environment. Evidence is available to suggest that exposure to teachers’ smoking or comprehensive school smoking policies, for example, are associated with young people’s smoking behavior (Moore et al., 2001; Poulsen et al., 2002). In the ESFA study the goal of introducing five lessons to schools did not materialize in a number of countries and there were notable differences in teachers’ attitudes towards the programme. This should come as no great surprise given the demands on schools, but it provides a timely reminder of what we can and cannot expect of schools, and the need for adequate resources to be made available. Nutbeam’s (Nutbeam, 1993) concerns about the unrealistic expectations of both researchers and practitioners as to what schools can contribute to tackling smoking are as valid today as they were 10 years ago.
Lessons for evaluating integral prevention programmes
The difficulties associated with evaluating health promotion interventions continue to be the subject of much debate—which need not be repeated at length here. The ESFA authors acknowledge that as a community trial the study had weaknesses. For instance, there was substantial variation in the implementation of the programme between countries, the randomization process was problematic and data were only collected at the individual level. Based on the ESFA experience, it could be concluded that community-based randomized controlled trials have a limited part to play in the evaluation of complex health promotion interventions. Whilst it has been argued for some time that health promotion needs to broaden the repertoire of evaluation approaches that it draws upon (Nutbeam et al., 1990), I would argue that experimental designs still have the potential to make a significant contribution to the public health evidence base if undertaken alongside well-planned and adequately resourced process evaluations.
Given the above, the immediate and longer-term effects of the ESFA programme must be treated with some caution. However, perhaps the more important weakness in terms of learning is the study’s apparent inability to shed more light on the reasons for some of the important findings reported in the papers, with potential implications for policy and practice. For example, what might explain differences in programme implementation (e.g. use of the school smoking policy manual) or smoking behavior (e.g. counter-productive findings in the UK and Denmark). Comprehensive process evaluation using a range of appropriate methods can be time consuming and expensive, but should nevertheless form an integral part of an evaluation of this nature. It is not my intention to suggest that the study has ignored process evaluation, as some details are provided. Rather, I would suggest that a greater emphasis could have been placed on this element of the evaluation.
The importance of context
In their recent review of process evaluation of health promotion interventions, drawing in particular on current smoking prevention studies in Wales, Parry-Langdon et al. (Parry-Langdon et al., 2003) suggest that a pragmatic approach is required—a move away from epistemological to design issues. They argue that for this approach to be successful, the evaluator must choose the most appropriate methods for the job in hand and that there should be a focus on establishing what it is about an intervention that makes it work (or not) [i.e. the realist focus on context, mechanism and outcome—see, e.g. (Pawson and Tilley, 1997)]. The ESFA authors have clearly considered context when interpreting their findings, preferring to make comparisons between countries, for example, rather than focusing on aggregate findings. Indeed, the study can best be viewed as a series of national evaluations. This makes perfect sense given that the programme was operating at different levels and in six countries.
Despite setting out to evaluate a common intervention across countries, the ESFA team concludes with a call for a dismantling strategy in future, explicitly recognizing diversity. This seems entirely logical and if integrated into a study at the earliest planning and developmental stages, should provide an opportunity to test approaches that are sensitive to the social and political circumstances in which young people find themselves.
Note: The above observations represent the personal views of the author and not his employer.
|
Commentary 6: The ESFA Mini-series |
|---|
|
TopESFA Mini-series: an experimentCommentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
Professor Maurice B. Mittelmark, Research Centre for Health Promotion, University of Bergen and President of the International Union for Health Promotion and Education
There is much that is positive in these papers, which describe a European collaboration to test a community-based youth smoking prevention project. The intervention approach and the study design are state of the art, building on 25 years of programmatic research in the US and Europe. One can trace the heart of the project—the focus on social influence processes and refusal skills training—back to Richard Evans et al.’s (Evans et al., 1978) seminal work in Houston, Texas, in which I had the opportunity to participate as a graduate student. Importantly, the ESFA group has both followed, and contributed to, a degree of critical examination of those early ideas. The social influences-oriented interventions of today are clearly more sophisticated than the original, even if the core seems to remain the same.
Yet, the result is less than hoped for, certainly in ESFA’s eyes. The project did not contribute to less experimentation with smoking. The paper in this issue by de Vries et al. (de Vries et al., 2003c), on short-term effects, concludes ‘more fundamental research is needed to test which elements within smoking prevention projects can contribute to their effectiveness’. Specifically, they recommend ‘dismantling’ designs that permit component analysis combined with subgroup analyses, to pinpoint what works for whom, when and how. It is this conclusion, among the many drawn in this collection papers, that I wish to remark on. Indeed, I seek to discourage researchers from considering such designs, and to discourage funders and journal editors from prioritizing such studies in the future. This commentary concludes with suggestions for alternative priorities.
There are two problems with dismantling studies such as those recommended by the ESFA collaborators. First, after 25 years, the paradigm launched by Evans et al. (Evans et al., 1978) has expanded the envelope of knowledge about as far as can be reasonably expected. It is time to move on. Second, even if pursued, dismantling studies of complex interventions cannot be done well enough to justify the time and expense. These points are elaborated on in the following paragraphs.
At the time Richard Evans launched the Social Psychological Deterrents of Smoking in Schools project in the early 1970s, the state of the art of school-based smoking prevention in the USA was about as follows. The teacher used lectures, films and visual aids to expose children in the last years of Grammar School (about 10–12 years of age) to the devastation to one’s health that followed from taking up the smoking habit. Exhibits containing cross-sections of smoke-damaged and smoke-free lungs were particularly popular, as the children invariably reacted nauseously, most gratifying to a teacher trying to make a strong impression. Every soul present pledged never to start smoking! Yet, when the children crossed the yard from the Grammar School to the Middle School about the time they turned 13, and came under the influence of older children, their pledges seemed to go up in...smoke.
The jump from this type of tobacco prevention to the ‘social influences’ model of tobacco prevention was profound. Many hundreds of studies after Evans, by many research teams in many places, have fine-tuned the approach to the nth degree. The learning curve was steep, at first, but seems to have flattened out considerably in recent years. It should be possible, today, to accept the state of the science for this type of programme as mature and concentrate on dissemination research to ensure high quality in the state of the art of implementation. But it is important to have realistic expectations. The level and quality of background anti-tobacco ‘noise’ is many decibels higher today than for 25 years since. Youth who are immune to the crescendo of messages from every quarter about the evils of tobacco will be hard to convince by even the best-conceived school-based education programme. (I use the term school-based’, since the only out-of-school activities with any real penetration were brochures and posters, not representing a significant community-based component.) This is born out by the ESFA data. About one in 10 T1 non-smokers became weekly smokers at T2 in the control group and the same pattern was observed in the experimental group. This small but determined minority, in both study groups, resisted every intervention. Indeed, the term ‘control group’ is a misnomer since the level of exposure to interventions did not differ between the two groups for two of eight indicators and differed only marginally (although statistically significantly) for three of the eight indicators. The term ‘comparison group’ is more appropriate, since both groups enjoyed almost comparable levels of intervention, due to the background noise of intervention referred to above.
Looking at the challenge another way, consider that a smoking prevention programme, to be successful, must reduce significantly an annual smoking uptake rate of just 10% (using ESFA data as a starting point). To reduce that to 5%, half of the expected smoking uptake would have to be prevented; to reduce that to 7.8%, 25% of the expected smoking uptake would have to be prevented. It is hard to imagine any social intervention in the public school sector that could manage such a challenge. Therefore, modest results, at best, can be expected, even from the best of programmes aimed at reducing smoking uptake rates. Indeed, modest results are what we get and futile scrambling for slightly better results, by fine-tuning our best interventions, is a poor use of our energies from the perspective of the above analysis.
So, what to do? First, new thinking is needed, that expands our ideas of what the risk factors are and of how to deal with them. One example of such thinking is presented to illustrate the point. There is a small minority of youth that are determined to be tobacco consumers and who will not be dissuaded by any school-based education programme, no matter how sophisticated. Clearly, one of the main risk factors for adolescent tobacco use in this cohort is the availability of tobacco products. Responding frailly, many public health authorities have mounted education campaigns and sales restrictions to limit youth access to tobacco products.
But these do not work well. Sting operation after sting operation reveals the ease with which youth acquire tobacco, despite existing restrictions. In Norway, there is now serious consideration being given to addressing this problem by treating the sale of tobacco products the same as alcohol products. That would include drastically reducing the type and number of legal outlets, with draconian penalties for those that break the law against providing tobacco products to minors, including seizing licenses to operate.
In addition to considering a full range of policy responses to the tobacco problem, research to improve interventions of all types, including school-based interventions, is needed. However, as already asserted in an oblique way, studies intended to further tweak already over-tweaked methodologies may not be the best way to use scarce resources.
Let us acknowledge, instead, that Meryvn Susser had it right when he stated that ‘small effects nestling under the wings of large ones are difficult to elicit.’ (Having kept carefully the quote, I have failed to keep the citation, and will be grateful for readers’ help in tracing it.) As every graduate student knows, weak effects and weak designs do not mix well, yet that is precisely the formula we work with in community-based quasi-experimental studies. There are other options, but psychologists and educators working in the substance prevention research arena will have to learn new ways of thinking and master some new (to them) methodologies as well. That is the subject to which I now turn.
I have claimed that we need to consolidate 25 years of experience and begin disseminating widely the state of the science on tobacco uptake prevention. Such dissemination work must have a research component, to understand and improve dissemination processes. The research should simultaneously contribute to the development of ever better programmes. In a robust research programme with these mixed aims, quantitative methods will be accompanied ideally by a robust qualitative approach. The emphasis in such an approach would to describe processes and holistic effects. This can be done by appreciating diversity across programme sites, by tracing and attempting to understand deviations from initial plans, by studying adaptation to local conditions, needs and interests, and by engaging in an open-ended, discovery-oriented frame of inquiry that hopes to describe developmental processes. The approach I describe has good potential to provide depth and detail about strengths and weaknesses, it describes important, non-quantifiable attributes of the programme environment, it focuses on the nuances of quality, it seeks to understand the quality of experience of all who participate; staff, clients and researchers alike.
The qualitative approach sketched here seeks to build on a foundation of best practices, by seeking them out and documenting them using a triangulation of methods. It is concerned with health-related conditions as much as with health behaviors, it is intensely interested in the social and physical environments that are the context for health development. It is committed to truly participatory research methods, not just window dressing ‘advisory committees’ whose role is legitimize the researchers’ activities.
I have a strong hunch that programmes of research characterized by the features listed above will provide more value for effort than will quantitative dismantling studies. This is of course merely a belief, not a contention I can document. The substance abuse prevention research arena is populated with few people who are trained to function in the qualitative mode. Funders do not receive many research proposals of high quality that describe such research. Journal editors do not receive many well-prepared manuscripts documenting such research. (I am exaggerating the dearth, but not by much.)
But even if I am wrong in my hunch, dismantling studies are not a viable alternative, for two reasons.
First, in programmes with many components—those specified by the researchers and those that are spawned by the intervention setting—any attempt at rigorous component analysis requires tighter control over the timing and nature of intervention than any research group has the resources to mount. No single study can accomplish a tight component analysis, and in any series of studies set up to accomplish the task, secular trends and naturally occurring changes in characteristics of the test settings will mar seriously our confidence in the results, however they come out.
Second, there are important psychosocial aspects to community-based interventions that influence study outcome and that neither component analysis, nor any quantitative scientific method, can reveal. A preventive intervention is not merely the mechanical implementation of a protocol. It is a human enterprise, in which the diverse actors’ levels of motivation, care, commitment, skill (and luck!) make an immeasurable difference. A multi-component intervention that ‘works’ in one setting may be replicated with great fidelity in another setting, yet fail to reproduce the expected results. The intervention might even fail to reproduce results within the same setting at a different time.
This phenomenon is well know to sports coaches, whose players and strategies produce winning results at one point in the season, and result in failure, under seemingly almost identical conditions, at another point. The 64 000 dollar question is ‘what went right the first time, and what went wrong the second time’? Similarly, why does the theatre group sparkle one night and fall on its face the next? Why do the students in the cooking class produce widely divergent results, despite working with the same instructor, recipes, materials and equipment?
The answer is the same in every case: the human element. The subtle aspects of human performance illustrated above differ only superficially from the sublimely subtle, human aspects of every preventive intervention. We have no technique to control them with predictability and their effects cannot be accounted for by simple-minded ‘manipulation checks’. The psychosocial environment in which a community-based intervention takes place is infinitely complex, beyond documentation by any known method and perhaps only approachable by a triangulation of methods.
Taken together—the technical complexity of component analysis and the complex social context within which any component analysis takes place—these factors confound component analysis utterly. For complex, community-based interventions, component analyses have just one guaranteed outcome—the need for more component analyses!
However, the pressure to mount ever-more-of-the-same-but-slightly-better studies seems inexorable. If we cannot count the number of angels on the head of the pin, let us use progressively stronger magnifying glasses, never mind the possibility that the angels are on the point of the pin!
Why is this so? What can we do about it?
The answer to the first question has several strands, as I see it. The psychologists and educators that dominate the health behavior intervention research area, and the related groups of funders, editors and peer group, share a research philosophy that exerts pressure to demonstrate ‘effectiveness’ using methods suited to measure ‘change’. There are important professional rewards for ‘proving’ the worth of programmes and for operating within the safe confines of a hypothetico-deductive frame. The studies that get funded and published tend to be of the ‘more-of-the-same with-a-twist’ variety, everyone knows it and everyone plays along. Our love of techno-tweaking results in ever more sophisticated designs and statistical analyses, aimed at helping us detect effects invisible to the naked eye. Summarized, all the forces at work have conspired to emphasize the development of quantitative methods to better detect weak effects, rather than the development of a qualitative approach as outlined earlier.
This commentary has resorted to the exaggerations of an expository style, in the hope that exhortation may provoke further exchange on this matter. The danger is that of going too far. We continue to need studies like ESFA—we need techno-tweaking. But we need also fresh methods, approaches and perspectives if we are to have any hope to experience, again, the large jump in programme effectiveness and quality that was launched when exhibits of ‘how to say no and keep your friends’ replaced exhibits of good and bad lungs.
|
References |
|---|
|
TopESFA Mini-series: an experimentCommentary 1: The ESFA...Commentary 2: The ESFA...Commentary 3: The ESFA...Commentary 4: The ESFA...Commentary 5: The ESFA...Commentary 6: The ESFA...References |
|---|
Anderson, P. (1999) The smoking gun. In European Commission/International Union for Health Promotion and Education. The Evidence of Health Promotion Effectiveness. Shaping Public Health in a New Europe. European Commission, Brussels/Luxembourg.
Bhaskar, R. (1975) A Realist Theory of Science. Harvester, Brighton.
Bhaskar, R. (1989) The Possibility of Naturalism, 2nd edn. Harvester, Brighton.
Brook, R. and Lohr, K. (1985) Efficiency, effectiveness, variations and quality. Medicinal Care, 23, 710–722.
Charlton, A., Melia, P. and Moyer, C. (1990) A Manual on Smoking and Young People for the Industrialized World.International Union against Cancer, Geneva.
COMMIT Research Group (1995) Community intervention trial for smoking cessation (COMMIT) 1. Cohort results from a four-year community intervention. American Journal of Public Health, 85, 183–192.
Connelly, J. (2001) Critical realm and health promotion: effective practice needs an effective theory [Editorial]. Health Education Research, 16, 115–119.
Connelly, J. (2002) The Behavior Change Consortium studies: missed opportunities—individual focus with an inadequate engagement with personhood and socio-economic realities [Commentary]. Health Education Research, 17, 691–695.
De Vries, H., Mudde, A., Leijs, I., Charlton, A., Vartiainen, E., Buijs, G., Clemente, M.P., Storm, H., Navarro, A.G., Nebot, M., Prins, T. and Kremers, S. (2003a) The European Smoking Prevention Framework Approach: an example of integral prevention. Health Education Research, 18, 611–626.
De Vries, H., Engels, R., Kremers, S., Wetzels, J. and Mudde, A. (2003b) Parents’ and friends’ smoking status as predictors of smoking onset: findings from six European countries. Health Education Research, 18, 627–636.
De Vries, H., Mudde, A., Kremers, S., Wetzels, J., Uiters, E., Ariza, C., Vittoria, P.D., Fielder, A., Holm, K., Janssen, K., Lehtuvuori, R. and Candel, M. (2003c) The European Smoking Prevention Framework Approach (ESFA): short-term effects. Health Education Research, 18, 649–663.
Donner, A., Brown, K.S. and Brasher, P. (1990) A methodological review of non-therapeutic intervention trials employing cluster randomisation, 1979–1989. International Journal of Epidemiology, 19, 795–800.
Evans, R.I., Rozelle, R.M., Mittelmark, M.B., Hansen, W.B., Bane, A.L. and Havis, J.G. (1978) Deterring the onset of smoking in children: knowledge of immediate physiological effects and coping with peer pressure, media pressure, and parent modelling. Journal of Applied Social Psychology, 8, 126–135.[CrossRef]
Green, J. and Tones, K. (1999) Towards a secure evidence base for health promotion. Journal of Public Health Medicine, 21, 133–139.
Hyland, A., Garten, S., Giovino, G.A. and Cummings, K.M. (2002) Mentholated cigarettes and smoking cessation from COMMIT. Community Intervention Trial for smoking cessation, Tobacco Control, 11, 135–139.
Klar, N. and Donner, A. (2001) Current and future challenges in the design and analysis of cluster randomized trials. Statistics in Medicine, 20, 3729–3740.[CrossRef][ISI][Medline]
May, C. (1993) Resistance to peer group pressure: an inadequate basis for alcohol education. Health Education Research, 8, 1–14.[Medline]
Michell, L. (1997) Loud, sad or bad: young people’s perceptions of peer groups and smoking. Health Education Research, 12, 1–14.
Moore, L., Roberts, C. and Tudor-Smith, C. (2001) School smoking policies and smoking prevalence among adolescents: multilevel analysis of cross-sectional data from Wales. Tobacco Control, 10, 117–123.
NHS Centre for Reviews and Dissemination (1999) Effective Health Care Bulletin. Preventing the Uptake of Smoking in Young People. University of York, York.
Nutbeam, D. (1993) Exposing the myth. What schools can and cannot do to prevent tobacco use by young people. Promotion and Education, 2, 11–14.
Nutbeam, D. (1998) Evaluating health promotion—progress, problems and solutions. Health Promotion International, 13, 27–44.
Nutbeam, D., Smith, C. and Catford, J. (1990) Evaluation in health education: a review of possibilities and problems. Journal of Epidemiology and Community Health, 44, 83–89.[ISI][Medline]
Parry-Langdon, N., Bloor, M., Audrey, S. and Holliday, J. (2003) Process evaluation of health promotion interventions. Policy and Politics, 31, 207–216.
Pawson, R. and Tilley, N. (1997) Realistic Evaluation. Sage, London.
Poulsen, L.H., Osler, M., Roberts, C., Due, P., Damsgaard, M.T. and Holstein, B. (2002) Exposure to teachers smoking and adolescent smoking behaviour: analysis of cross sectional data from Denmark. Tobacco Control, 11, 246–251.
Rees, R., Harden, A., Shepherd, J., Brunton, G., Oliver, S. and Oakley, A. (2001) Young People and Physical Activity: A Systematic Review of Research on Barriers and Facilitators. EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, London.
Rogers, E.M. (1993) Diffusion and re-invention of project DARE. In Backer, T.E. and Rogers, E.M. (eds), Organizational Aspects of Health Communication Campaigns: What Works? Sage, Newbury Park, CA.
Rootman, I., Goodstadt, M., Hyndman, B., McQueen, D., Potvin, L., Springett, J. and Ziglio, E. (2001) Evaluation in Health Promotion: Principles and Perspectives. European Series 92. WHO, Copenhagen.
Shepherd, J., Harden, A., Rees, R., Brunton, G., Garcia, J., Oliver, S. and Oakley, A. (2001) Young People and healthy Eating: A Systematic Review of Research on Barriers and Facilitators. EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, London.
Stewart-Brown, S. (2001) Evaluating health promotion in schools: reflections. In Rootman, I., Goodstadt, M., Hyndman, B., McQueen, D., Potvin, L., Springett, J. and Ziglio, E. (eds), Evaluation in Health Promotion: Principles and Perspectives. European Series 92. WHO, Copenhagen.
Ukoumunne, O.C., Gulliford, M.C., Chinn, S., Sterne, J.A.C. and Burney, P.G.J. (1999) Methods for evaluating area-wide and organization-based interventions in health and health-care: a systematic review. Health Technology Assessment, 3, 1–99.[Medline]
WHO (1998) Health Promotion Evaluation: Recommendations to Policymakers. WHO, Copenhagen.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  H. de Vries, F. Dijk, J. Wetzels, A. Mudde, S. Kremers, C. Ariza, P. D. Vitoria, A. Fielder, K. Holm, K. Janssen, et al. The European Smoking prevention Framework Approach (ESFA): effects after 24 and 30 months Health Educ. Res., February 1, 2006; 21(1): 116 - 132. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||