Health Education Research, Vol. 16, No. 5, 609-622,
October 2001
© 2001 Oxford University Press
Evaluation of a primary school drug drama project: methodological issues and key findings
Department of Social Medicine, University of Bristol, Bristol BS8 2RR
1 Faculty of Health and Social Care, University of the West of England, Bristol BS16 1DD, UK
Correspondence to: J. Orme
 |
Abstract |
|---|
 Top Abstract IntroductionMethodResultsDiscussionReferences |
|---|
This paper describes the impact evaluation of a primary school drug drama project developed by a health promotion service and a theatre's education department in England. The project targeted 10–11 year olds in 41 schools with an interactive drama production and workshop day on attitudes, choices, decisions and risks of alcohol, tobacco and illegal drug use. Parents were also involved in parents' evenings and watching children's performances. The research consisted of both process evaluation, consultation with pupils, teachers, parents, actors and health promotion staff on the project itself, and impact evaluation which looked at potential changes in children's knowledge, attitudes and decision-making skills. This paper reports findings of the impact evaluation, from six of the schools participating in the project. The impact evaluation consisted of pre- and post-project testing using a `draw and write' and a problem-solving exercise. These findings suggest that the project had a significant impact on the children's knowledge of names of specific illegal drugs, and on their awareness that alcohol and cigarettes were also drugs, and secondly encouraged the children to think in less stereotypical terms about drugs and drug users. The problem-solving exercise, involving decision-making scenarios, showed small but positive trends between pre- and post-project solutions in more than half of the response categories. Methodological difficulties relating to evaluating such a project are discussed.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
The research and policy contexts
In the light of increasing levels of drug use amongst young people in England (Blackman, 1996
; Ramsay and Spiller, 1997; Roker and Coleman, 1997; Standing Conference on Drug Abuse, 1997) and evidence of a decreasing average age of first experimentation with drugs (Roberts et al., 1995; Standing Conference on Drug Abuse, 1997; Balding, 1998), the development of school-based drug education has become an area of both national and local education and public health policy focus. Government policy also includes an emphasis on the need to start drug education in primary school (Cabinet Office, 1998).
The Government's most recent White Paper, Tackling Drugs to Build a Better Britain (Cabinet Office, 1998), builds on the previous Government's 1995 White Paper, Tackling Drugs Together (Home Office, 1995), to include a continuing commitment to drug education for young people in order to enable them to resist drug misuse. The 1998 White Paper emphasizes the importance of beginning drug education for young people at an early age via life-skills approaches at primary schools, and ensuring that any drug education is appropriate for the age and circumstances of the young people involved [(Cabinet Office, 1998), p. 13].
The Department for Education and Employment (DfEE) circular, Drug Prevention and Schools (Department for Education and Employment, 1995), intended to complement the objectives relating to drug education laid out in the 1995 White Paper, provides a comprehensive set of guidelines for local education authorities and schools on the development of appropriate drug education. This circular emphasizes the Government's view of drug misuse as an educational issue, and recommends the development of effective drug education programmes and drugs policies within all primary and secondary schools. In particular, the circular focuses upon the value of drug education as part of an integrated health education programme, spanning all stages of the school career; skills-based approaches to foster young people's ability to make informed and responsible choices; and interactive methods of drug education, including role play.
Building upon the 1995 DfEE circular on drug prevention and schools, the Government recently published supplementary guidance on drug education in schools and the youth service entitled Protecting Young People (Department for Education and Employment, 1998). This guidance further emphasizes the government's view of primary schools as having a central role to play in drug education, as they are able to reach young people who are unlikely to have started experimenting with drugs and to be more receptive to messages from teachers at this stage of their school careers.
The Home Office Drugs Prevention Initiative, on the basis of its programme of work during the 1990s, identifies key messages relating to drug education with young people (Home Office, 1998), including the crucial importance of drug education at all stages of the school curriculum, the need for drug education to start in primary school and form part of a broader spiral Personal Social and Health Education programme, and the importance of emphasizing life-skills approaches. The Drugs Prevention Initiative reports an evaluation of `Project Charlie', a life-skills programme for primary school children in London, and concludes that `it is possible to influence young people's behaviour and attitudes away from drugs starting at an early, pre-experimentation, primary school age' [(Home Office, 1998), p. 4].
With regard to the use of drama, Theatre in Education has been identified by the DfEE (Department for Education and Employment, 1995), Ofsted (Office for Standards in Education, 1997) and SCODA (Standing Conference on Drug Abuse, 1998) as an innovative and popular approach to drug education, although evaluation data on Theatre in Education's effectiveness as a method of drug education is lacking in the literature.
There is therefore strong impetus from both government policy and government-funded research to address the delivery of drug education in primary schools, and to consider the use of interactive approaches such as drama in this delivery.
The Primary Drug Drama Project
The Primary Drug Drama Project was developed by Health Promotion Service Avon who commissioned the education department of a local theatre to work on the project. The evaluation of the project forms the basis of this paper. The project targeted Year 6 pupils (i.e. 10 and 11 year olds), in 41 schools, presenting an interactive drama production and workshop day in which actors worked with pupils to develop their own performances.
The project aimed to make a positive contribution to each school's drug education programme, help pupils to explore attitudes and develop relevant skills, raise pupils' awareness of the consequences of different decisions and increase parental involvement.
The main themes which the project proposed to explore were:
- Attitudes: how they influence behaviour.
- Choices: how these are influenced and exercised.
- Decisions: their different consequences and how these may affect people.
- Risks: how to recognize and assess them, and decide what action to take.
An original drama script was commissioned for the project, and actors with teaching or workshop experience were recruited both to perform in the play and to work with the children throughout the day.
The content of the play consisted of an allegorical story focusing on a young girl whose parents had split up and whose grandfather had died. She was bullied at school and her mother did not listen to her. She met a wizard who offered her a potion to make her feel better; each time she tried this potion it had different effects and she had to give the wizard first her arms and then her legs as payment. When the wizard asked for her heart, she realized that this was too great a price to pay and that she needed to live in the real world. Although her parents did not get back together, her father did come to see her and communication was re-established between the family members. As well as looking at drugs via the `wizard' and his `potion', the play raised other issues including bereavement, loneliness, divorce, lack of communication, bullying, self-blame, guilt, depression, escapism and obsession.
The main focus of the project was a whole day spent in each school by four actors. The structure of the day included the actor-performed play, followed by four workshops leading up to performances for parents of plays rehearsed by the children during the day. The actors moved around to work with different groups of children for each workshop. Although the day included mention of a wide range of legal and illegal drugs, the role-plays and performances in which the children were involved focused primarily on cigarettes and alcohol.
All coordinating teachers in the participating schools were invited to attend a pre-project briefing session held at the theatre and teachers were asked to be present throughout the drama day in their school with the children. A teacher evaluation session was also held at the theatre after the project to obtain feedback from teachers on the day itself and on the project process.
|
Method |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
In line with the project aims, the evaluation was intended to assess the effectiveness of the Primary Drug Drama Project in terms of its contribution to each school's drug education programme, its impact on pupils' attitudes, skills and awareness of the consequences of different decisions, and its effect on parental involvement.
The research consisted of both process evaluation, in the form of consultation with pupils, teachers, parents, actors and Health Promotion Service Avon staff on the project itself, and impact evaluation looking at changes in children's knowledge, attitudes and decision-making skills. This paper reports on the results of the impact evaluation with the pupils. Findings of the process evaluation are detailed in the project report (Orme and Starkey, 1998).
Evaluation tools for pupils were administered by the coordinating teachers, who were fully briefed on this process by the researchers at the pre-project briefing session. To ensure consistency, they received detailed notes on how to administer the tools, which included instructions not to allow the children to comment out loud, ask questions or share their ideas with others. Participating schools (except the first school, which received the project before the evaluation was set up) were asked to complete the evaluation exercises for pupils. To ensure comparability of data, schools were asked to administer the evaluation activities a week before and 4 weeks after the project visited their school. Of 40 schools asked, 28 (70%) did complete and return all pupil evaluation materials. However, only six schools completed the post-project questionnaires at the required 4-week interval, due to the range of pressures on schools' time. These six schools were therefore used as the sample for the present analysis.
Two pre- and post-project exercises were carried out with pupils as part of the impact evaluation. The first was a `draw and write' exercise, completed prior to the project and again 4 weeks after the visit to each school. This exercise aimed to explore children's knowledge and attitudes relating to drugs and drug use, and was based on an established evaluation instrument (Williams et al., 1989). In the exercise, children were asked a range of questions (presented in Results) and had to draw and/or write a response. In the six schools sampled, a total of 297 `draw and write' questionnaires were completed prior to the project and 253 post-project. The difference in sample size was due to absences and illness in participating classes.
The second evaluation activity was also a pre- and post-project exercise on problem solving, based on an evaluation tool adapted from the `Alternatives and Consequences' test developed by Ahlgren and Merrick (Ahlgren and Merrick, 1984) and used in a primary school drug education project (McGurk and Hurry, 1995). This evaluation instrument aimed to assess children's decision-making skills and their ability to resist peer pressure. The exercise consisted of four problem situations (described in Results) to which children had to generate as many solutions as they could and also to identify what they would do themselves in each situation. The names of children in the scenarios were different for the pre- and post- project versions to minimize any effect of recall on responses. The order in which the scenarios were presented was also changed in the post-project version.
Decision-making skills were assessed by recording the overall number of solutions generated to all four problems. These solutions were then analysed according to their quality, using the following system of categorization devised by Ahlgren and Merrick:
- First-class solutions are those in which the children take responsibility for their own actions and intend to use verbal or written communication to resolve the issues. Such solutions can be identified for scoring purposes by the mention of direct interaction between the child and the other.
- Second-class solutions are those where children resolve the interpersonal or problem issues through indirect means, such as telling an adult or, in the case of peer pressure, running away.
- Third-class solutions are those where children do not accept responsibility for their actions, e.g. in the case of peer pressure items, the solution either involves ambivalence or going along with peers. This category also includes solutions of a very hostile or aggressive tone, such as `beat them up'. Such solutions are more likely to create a problem than resolve issues.
In total, 291 problem-solving questionnaires were completed pre-project and 285 post-project.
The `draw and write' exercises for pupils were categorized and thematically analysed in accordance with Wetton's framework (N. Wetton, unpublished). A z-test was then applied to look at differences in proportions between the pre- and post-project results. The problem-solving exercises also used a classification system similar to that used by McGurk and Hurry (McGurk and Hurry, 1995). Chi-square analysis was then carried out in order to test for significant differences between pre- and post-project results.
|
Results |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
Impact evaluation: `draw and write' exercise
The questions asked in the `draw and write' exercise were as follows:
- Hannah was walking home when she found a bag with drugs inside it. Draw what you think was in the bag.
- Who do you think lost the bag? Draw the person. What kind of person is this?
- What do you think the person was going to do with the drugs in the bag before he/she lost it?
- What did Hannah do with the bag when she found it?
- What would you do if you found the bag?
- Can a drug be good for you? Can it help you? If it can, when?
- Can a drug be bad for you? Or hurt you? If it can, when?
The results from the pre- and post-project exercises are presented below.
Contents of the bag
The most common responses given by children to the question of what was in the bag of drugs Hannah found are presented in Table I.
|
The percentage of children who were able to name at least one specific drug increased significantly from 53% (n = 157) in the pre-project exercise to 71% (n = 180) in the post-project exercise.
In terms of legal drugs, significant increases occurred in the proportion of children who named as drugs cigarettes (from 20 to 27%), alcohol (from 3 to 10%) and solvents (from 2 to 8%).
With regard to illegal drugs, children in both the pre- and post-project exercises named a range of drugs. Illegal drugs which children named significantly more often following the project included ecstasy (from 10 to 18%) and speed (from 5 to 17%).
Children also strongly identified needles or syringes with drugs: 35% of children cited syringes, needles or injections in the pre-project exercise and 40% in the post-project exercise. A small number of children also named a range of drug `paraphernalia', including a belt, spoon, mirror, straw and a bong.
Reflecting the increase in the number of children citing specific drugs, there was a highly significant decrease in the proportion of children who mentioned only non-specific `drugs', from 27% pre-project to 14% post-project and a significant decrease in those who cited `pills', `capsules' or `tablets' (from 37 to 30%).
These findings suggest that even prior to the project there was a fairly high level of awareness amongst children of what drugs might be called: children were able to name a whole range of legal and illegal drugs. A significant increase in levels of awareness of the names of specific drugs, both legal and illegal, appeared to have occurred after the project. This was particularly noticeable with cigarettes, alcohol, speed and ecstasy. The children's understanding of what the named drugs actually were was not explored in the evaluation as this was not a primary aim of the project.
The owner of the drugs bag
When asked who might have dropped the bag of drugs, the children responded as indicated in Table II.
|
Again, the children suggested a whole range of people who might have dropped the bag of drugs in both pre- and post-test exercises. The most significant difference, although only amongst a relatively small group of children, was the increase in the number of children saying that `anyone' or `a normal person' could have dropped the bag (from four to 20 children, 1–8%). This finding suggests that some children may have begun to think in a less stereotypical way about drug users following the project.
As well as saying who dropped the bag, the children were asked to think about what kind of person that might be. The range of characteristics identified by the children are presented in Table III.
|
It is interesting that in both the pre- and post-test exercises, children were far more likely to draw males than females as the person who lost the bag. In the pre-project exercise, children generally cited negative characteristics such as `bad', `horrible' and `nasty'. There was a significant decrease in the number of these responses post-project (from 9 to 2%), as in the number of children who identified characteristics such as `silly' or `stupid' (from 4 to 0.5%). In the post-project exercise, some of the children's responses were more sympathetic, although this difference was not significant, and a small number of children gave characteristics such as `unhappy', `stressed', `lonely', `shy' and `someone with problems'. This may reflect the increase in the number of children post-project who felt that the person who dropped the bag could have been a `normal' person.
Using the drugs
The children were asked what the person was intending to do with the drugs in the bag. Their responses are presented in Table IV.
|
They came up with a very wide range of uses to which the drugs might have been put in both the pre- and post-project exercises, with the most popular category being to sell or deal the drugs. Only small numbers of children referred to drugs as medicines, to make someone better. There were no significant differences in children's responses to this question between the pre- and post-project exercises.
Things to do with the bag of drugs
When asked what they thought Hannah did with the bag of drugs, the children responded as in Table V.
|
Over half the children in both the pre- and post-project exercises said they thought Hannah took the bag of drugs to the police. Smaller numbers (7% pre- and 10% post-project) stated that Hannah gave the bag to her mum or dad. Small numbers of children pre-project felt that Hannah would have told an adult (parents, police or other adults) but not picked the bag up and this number increased significantly in the post-project exercise (from 2 to 7%).
Small and comparable numbers of children in both the pre- and post-project exercises stated that they thought Hannah actually tried the drugs.
The children were also asked what they themselves would do with the bag of drugs if they found it and responded as in Table VI.
|
Again, nearly half of the children stated in both the pre- and post-project exercises (44 and 46%, respectively) that they would take the bag to the police station. Although this demonstrates the clear connection between drugs and breaking the law, it also raises the need to clarify with children health and safety issues about handling drugs and paraphernalia, particularly bearing in mind the large number of children who included syringes in the contents of the bag. There was a significant increase in the number of children who stated that they would phone or tell the police (from 5 to 9%).
There was a small but non-significant increase between the pre- and post-project exercises in the number of children who stated that they would not touch the bag if they found it.
`Good drugs'
Children were asked whether a drug could be good for you and in what way. Their responses are categorized in Table VII.
|
There was a significant increase between the pre- and post-test exercise in the number of children who were aware that a drug could be good for you (from 78 to 87%). Similar numbers of these children (74% pre- and 77% post-exercise) qualified this as being when you were ill or when obtained from a doctor. Small numbers of children both pre- and post-test gave a range of other circumstances in which drugs could be good for you, including when in labour, when asthmatic, to relieve pain and to calm nerves. There was also a significant decrease in the number of children who stated that drugs would kill you (from 6 to 2%).
`Bad drugs'
The children's responses to whether a drug could be bad for you or could hurt you are presented in Table VIII.
|
The great majority of children both pre- and post-project were aware that drugs could be bad for you. Other comments made by children related to awareness of the impact of drugs on physical and mental health, and on people's families. Their comments also demonstrated an awareness of appropriate ways of taking drugs such as medicine, e.g. by not taking them too often or for too long. Children's responses to this question were comparable in the pre- and post-project exercises.
Summary of `draw and write' findings
These findings suggest that the project had a significant impact on the children's knowledge of names of specific illegal drugs, and on their awareness that alcohol and cigarettes were also drugs.
In terms of attitudes towards drug users, there was some indication that following the project some children were thinking in less stereotypical terms about who might be a drug user or in possession of drugs, with significant decreases in the number of children post-project who attributed negative characteristics to the owner of the bag of drugs. The vast majority of children post-project still gave adult drug dealers as the likely owner of the bag.
In terms of appropriate responses to finding a bag of drugs, children both pre- and post-project recognized the importance of involving adults and the police, with a significant increase in the number of children who stated that they would tell the police if they found the bag. Nonetheless, exploration with primary age children of health and safety issues about handling drugs and syringes would be valuable in the light of the number of children who said they would pick up the bag.
There was a significant increase in the number of children who were aware that a drug could be good for you following the project. In terms of the negative effects of drug use, children both pre- and post-test demonstrated an impressive awareness of the range of negative implications that drug use might have for individuals and their families.
Problem solving
The four scenarios to which the children had to generate solutions in the problem-solving exercises were as follows:
Scenario 1
James is at Tom's house. James takes one of Tom's tapes without asking, when Tom is out of the room. A few days later, James hears Tom telling someone that he has lost his favourite tape. It is the tape that James took.
- What things could James do in this situation? Write down as many different solutions as you can think of. They don't need to be what you would do.
- Imagine if you were James, what would you do in this situation?
Scenario 2
Lilian is at her best friend Amy's house. Amy's older brother and his friends ask the girls to have a glass of cider. Lilian doesn't really want to drink alcohol, but she wants to impress the boys.
- What can Lilian do? Write down as many different solutions as you can think of. Remember, they don't need to be what you would do.
- Imagine if you were Lilian, what would you do in this situation?
Scenario 3
Rob wants to get on with his work in class. But Rob's friend Tony has been annoying him and putting him off his work.
- What can Rob do to make Tony stop putting him off his work? Write as many answers as you can.
- Imagine if you were Rob, what would you do in this situation?
Scenario 4
Mel's friends meet after school down the park and they are all smoking cigarettes. They offer one to Mel. Mel knows that smoking isn't really good for you but she sees that her friends are enjoying it.
- What can Mel do? Write down as many different solutions as you can think of. Remember, they don't need to be what you would do.
- Imagine if you were Mel, what would you do in this situation?
Tables IX–XII –contain the pre- and post-project results for each scenario, categorized by the class of solution generated.
|
|
When post-project solutions are compared with pre-project responses using the
2 test, more second-class solutions are identified in all of the scenarios and fewer third-class solutions in two of the scenarios (see Tables IX–XII
–). With regard to the finding that more children generated second-class solutions after the experience of the project compared to before, it may be that for children of this age group these are the most appropriate solutions. To anticipate first-class solutions may be too high an expectation of this age group. The positive but lesser responsibility associated with second-class solutions may be more appropriate to the children's age.
The scenario involving drinking alcohol demonstrated a significant difference in some categories of solutions. This involved increased numbers of children responding with second-class solutions post-project, and fewer children responding with first- and third-class solutions post-project. This finding may be due to the fact that peer pressure to drink alcohol was more explicitly addressed during the drama day compared to some of the issues raised by the other scenarios.
The solutions generated by the children need to be considered in the context of the scenario to which they were responding. For example, hostile action taken by one child toward another featured more as a response to the `irritation in the classroom' scenario than to any other scenario; resolving the problem by indirect methods such as telling the teacher was another popular response to the same scenario.
The children were also asked to indicate what they would do themselves in each situation. A fifth of their responses were the same as those generated for the main character in each scenario, which may indicate that the children giving these responses were identifying with the protagonist.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
The project evaluation found that the Drug Drama Project did meet its stated aims of making a positive contribution to participating schools' drug education programmes and of increasing parental involvement. In addition, the impact evaluation carried out showed some encouraging trends in terms of developing children's knowledge of and attitudes towards drugs and drug users, and their decision-making skills. The pre- and post- project `draw and write' exercise results suggest that the project encouraged some children to think in less stereotypical terms about drugs and drug users. The problem-solving exercise also showed small but positive trends between pre- and post-project solutions in more than half of the response categories, with more second-class solutions given for all of the post-project scenarios. It is beyond the scope of this paper to present both the results of the process and impact evaluation. In discussing the results, it is appropriate to draw on the results of the process evaluation undertaken with the project team, teachers, pupils, parents and actors for clarification. The following factors may have influenced the results of the impact evaluation.
Programme issues
Sampling
A major issue relating to the findings of the study is the structure of the pupil sample, in that the evaluators were not able to stratify or control the sample. In addition, the sample represented only 15% of the schools involved in the project. It is possible that this sample reflected the schools where teachers were more committed to the evaluation aspect of the project or at least were better organized in relation to the follow-up work.
Teacher involvement
The teachers were asked to be present with the children throughout the drama day in their school, yet not all teachers stayed in the workshops that the actors ran with the children. Increased teacher involvement was identified as an area for development by both actors and health promotion specialists, in order to provide the actors with more information on the children and facilitate follow-up work back in school. These factors may have influenced the impact of the initiative on the children both in terms of the actors being able to gear the work to the needs of the children, and the teachers having sufficient knowledge of issues covered during the day to develop these in follow-up work and discussion.
Evaluation tools
The overall aims of the drama day did not change over the course of the project but the content of the workshops was flexible and evolved throughout the project. As the evaluation was designed to correspond to the aims of the project rather than the actual content of the workshops, this raises the question of whether the impact evaluation tools were measuring the same variables with children who took part in the project at different stages of its evolution.
Actors' abilities
The need for the actors to receive some input on working in educational settings, particularly on child protection and confidentiality issues, and also more input on drugs to increase their confidence, was identified by the project team and some teachers as an area for improvement. While the provision of such training clearly has resourcing and staffing implications, preparing actors appropriately for this innovative intensive work in the educational setting is an important consideration in the planning of such a project and a factor that is likely to influence the impact of the workshops on the children.
Methodological issues
The `draw and write' technique
The `draw and write' technique, first developed for use in relation to young people and drug issues by Williams et al. (Williams et al., 1989), was used as a pre- and post-project evaluation tool because of its widespread acceptance by practitioners and educators as a way of obtaining information from children about drugs [e.g. (Taylor, 1996; Office for Standards in Education, 1997)]. O'Connor et al. [(O'Connor et al., 1997), p. 23], claim that:
The `draw and write' technique...provides a means by which it is possible to gauge the level of primary school children's knowledge and perceptions without `priming' them, and limiting and/or distorting their responses through inappropriate questioning and inadequate literacy skills.
However, Backett-Milburn and McKie mount a strong challenge to the prevailing assumption within health education and health promotion research that the `draw and write' technique does achieve this goal. They argue that the information elicited is crucially affected by the social context in which the method is administered and completed, and the representations and `pictures' to which they are exposed in their habitual environment. They conclude that `it is likely that children will reproduce images of the dominant discourses of health and health education associated with the culture of which they are members' [(Backett-Milburn and McKie, 1999), p. 390].
Further, Backett-Milburn and McKie argue that the information provided by children via `draw and write' will depend on the way in which they perceive the research task and its meaning to them. They suggest that if the task is introduced in a classroom environment, by a teacher whose questions are usually posed in order to obtain a `correct' answer rather than to elicit perceptions, children may provide the information that they feel their teacher would want to see and therefore limit their drawings and writings to what they perceive as publicly acceptable representations.
Drawing on some of the process evaluation data, it was clear in the evaluation reported here that some of the teachers involved in the project viewed the evaluation exercises as `work' for the children rather than as research tools for an external evaluation. This may have affected the way in which those teachers introduced the tasks and consequently how the children completed them, encouraging some children to try and provide what they felt might be a `correct' rather than an honest response.
Problem solving
Again, the problem-solving exercise was used due to its established use in evaluations of other health education projects focused on life-skills development, most recently and notably `Project Charlie' (McGurk and Hurry, 1995; Hurry and Lloyd, 1997). However, the concern of Backett-Milburn and McKie that adult interpretations may be imposed upon children's meanings is also valid here (Backett-Milburn and McKie, 1999).
The first-class form of solution identified by `Project Charlie' evaluators (as outlined in Methods) could be seen as more of an `adult' response to a difficult situation than children of primary school age are likely to want to engage in. Therefore, the finding in this evaluation that more children generated second-class solutions after the experience of the project compared to before, may reflect a lack of acknowledgement by adults of what children might understand and see as appropriate actions in the face of difficulty.
Pre- and post-project testing: measuring outcomes?
In order to be able to assess the aims of the project, i.e. its impact on pupils' attitudes, skills and awareness of the consequences of different decisions, pre- and post-project evaluation exercises were necessary. However, this design presented a number of issues, partly due to the nature of the tasks themselves and partly due to the way in which the tasks were administered. Although the post-project problem-solving scenarios were altered so that they would not be identical to the pre-project exercise, four teachers still commented on the difficulties caused by giving children what they viewed as identical `work'. Quotations from two of the teachers illustrate the issue:
My only quibble is in the follow-up work being the same as the introductory work. This method of follow-up seems to me to be very crude and uninteresting for the children (essentially being asked to repeat two pieces of work)—so we haven't done it.
... the post-project questionnaires being identical to the pre-project ones was very uninspiring for the children, who all wanted to know why they had to do it again. Would it be possible to rephrase questions so children weren't aware they were doing the same?'
There is clearly a tension here between research aims of achieving methodological rigour and consistent data collection, and the teachers' aims of giving the children `interesting work' to do. Whether this tension would have been resolved had the researchers administered the questionnaires themselves is not clear. Although the teachers were carefully briefed and asked to explain the purpose of the evaluation exercises, it is not clear that they all fully understood why they were being asked to administer the same tasks again. This lack of understanding may have had an impact on the priority given to the post-project tasks both by the teacher and the children, and consequently on the quality of the data received.
Backett-Milburn and McKie suggest that children's views on and knowledge of health-related issues could be more effectively obtained by adopting more interactive research approaches, such as face-to-face discussion with the children (Backett-Milburn and McKie, 1999). While this view is shared by the authors of this paper, the expense of such approaches would require that the sample of children in the evaluation exercise be smaller. While more interactive approaches may provide more `meaningful' data, albeit from a smaller number of children, it again creates a tension with other stakeholders—in this case the current government and its emphasis on identifying, and setting performance indicators to measure, `what works' in drug education in the sense of what reduces drug use.
Reviews of drug education literature suggest that the available evaluation literature is unable to prove what is `effective' due to its emphasis on process rather than outcome measures [e.g. (Health Education Authority, 1997)]. There appears to be a push amongst policy makers, therefore, to become `more scientific' in evaluating drug education initiatives, in terms of applying quantitative outcome measures relating to drug-taking behaviour. However, at the same time, researchers and educators are querying whether such research approaches, based upon a positivist paradigm and using methods designed for the natural sciences, are appropriate for evaluation of health promotion and health education initiatives based on principles of participation, empowerment and capacity building [e.g. (WHO European Working Group on Health Promotion Evaluation, 1998)].
Difficulties relating to measuring the outcomes of drug education have been recognized by government policy, which notes that [Department for Education and Employment, 1998), p. 18]:
Establishing the effectiveness of drug education is a complex task. Almost all evaluations of programmes have been inconclusive in terms of perceived results in reducing or preventing drug use. Where programmes have shown positive results, the benefits have been in improved knowledge, decision-making skills, and improved self-esteem. Personal and social skills, however, have not been shown in themselves to relate directly to the prevention of drug use.
The findings of the Home Office `Project Charlie' evaluation (Hurry and Lloyd, 1997), which showed limited differences in drug use with very small numbers of children, have been used to suggest that life-skills programmes delivered at primary school age can delay the onset of first drug use, which in turn is likely to reduce the prevalence of problematic drug use. While this assertion may turn out to be true, more comprehensive and longitudinal outcome evaluation of such programmes is now required before primary school life-skills programmes can be heralded as the long-awaited proof of drug education `effectiveness', as has been suggested in recent government documentation [e.g. (Cabinet Office, 1998; Department for Education and Employment, 1998; Home Office, 1998)].
The present study also highlights how the context within which drug education initiatives are received and developed within schools varies enormously between each institution. This context includes the children themselves and their circumstances, the teachers' characteristics, preparatory and follow-up work undertaken, including the integration of an initiative into the school's health education curriculum, and pressures both within and external to the school, such as demands on teaching time and preparation for Standardized Attainment Tests. All of these factors are likely to affect the impact of any drug education initiative within a school, making comparability of data difficult.
These complexities relating both to the nature of drug education, its desired outcomes and the means by which these might be evaluated all suggest the crucial importance of ensuring that evaluations of local initiatives encompass both outcome measures, which clearly relate to the aims of the intervention itself, and process measures which are sensitive to the contextual factors involved. Such evaluations clearly require adequate resourcing to ensure that a range of research methods, both qualitative and quantitative, can be used to measure outcomes in a way which allows the children's voices to be heard in a meaningful way. As Backett-Milburn and McKie comment, `health education researchers need to create the potential for children to have their own ideas and explanations heard and understood' [(Backett-Milburn and McKie, 1999), p. 397].
|
|
|
Acknowledgments |
|---|
Mollie Gilchrist and John Cobby of UWE advised on the statistical analysis for this paper. Thanks go to all of the children and teachers who participated in the project evaluation, as well as to Health Promotion Service Avon and the Bristol Old Vic Education Department for their support and guidance throughout the project. Grateful thanks are also expressed to the funders of the drama project and its evaluation. Any views expressed in this paper, however, are those of the authors and do not necessarily reflect the views of the funding organizations. We would also like to thank the referees for their comments on drafts of this paper.
|
References |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
Ahlgren, A. and Merrick, S. (1984) Evaluation of the effects of the Project Charlie Curriculum in Public Schools of Richfield and Edina, Minnesota. Report for Storefront/Youth Action, Richfield, MN and the University of Minnesota, Alcohol and Other Drug Abuse Programming, MN.
Backett-Milburn, K. and McKie, L. (1999) A critical appraisal of the `draw and write' technique. Health Education Research, 14, 387–398.
Balding, J. (1998) Young People and Illegal Drugs in 1998. Schools Health Education Unit, University of Exeter, Exeter.
Blackman, S. (1996) Has drug culture become an inevitable part of youth culture? A critical assessment of drug education. Educational Review, 48, 131–142.
Cabinet Office (1998) Tackling Drugs to Build a Better Britain: The Government's 10-Year Strategy for Tackling Drug Misuse. Stationery Office, London.
Department for Education and Employment (1995) Drug Prevention and Schools. DfEE, London.
Department for Education and Employment (1998) Protecting Young People: Good Practice in Drug Education in Schools and the Youth Service. DfEE, London.
Health Education Authority (1997) Health Promotion in Young People for the Prevention of Substance Misuse. Health Education Authority, London.
Home Office (1995) Tackling Drugs Together: A Strategy for England 1995–1998. HMSO, London.
Home Office (1998) Guidance on Good Practice: A Supplement to the DPI's Overview Guidance to Drug Action Teams on Developing Local Drugs Prevention Strategies. HMSO, London.
Hurry, J. and Lloyd, C. (1997) A Follow-up Evaluation of Project Charlie: A Life Skills Drug Education Programme for Primary Schools. Drugs Prevention Initiative Paper 16. HMSO, London.
McGurk, H. and Hurry, J. (1995) Project Charlie: An Evaluation of a Life Skills Drug Education Programme for Primary Schools. Drug Prevention Initiative Paper 1. HMSO, London.
O'Connor, L., Best, D., Best, R. and Rowley, J. (1997) Young People, Drugs and Drug Education: Missed Opportunities. Roehampton Institute, London.
Office for Standards in Education (1997) Drug Education in Schools. Stationery Office, London.
Orme, J. and Starkey, F. (1998) Evaluation of HPSA/Bristol Old Vic Primary Drug Drama Project 1997/98. University of the West of England, Bristol.
Ramsay, M. and Spiller, J. (1997) Drug Misuse Declared in 1996: Latest Results from the British Crime Survey. Home Office, London.
Roberts, C., Moore, L., Blakey, V., Playle, R. and Tudor-Smith, C. (1995) Drug use among 15–16 year olds in Wales, 1990–1994. Drugs: Education, Prevention and Policy, 2, 305–316.
Roker, D. and Coleman, J. (1997) Education and advice about illegal drugs: what do young people want? Drugs: Education, Prevention and Policy, 4, 53–64.
Standing Conference on Drug Abuse (1997) Drug-related Early Intervention: Developing Services for Young People and Families. SCODA, London.
Standing Conference on Drug Abuse (1998) The Right Choice: Guidance on Selecting Drug Education Materials for Schools. SCODA, London.
Taylor, T. (1996) Drugs Education: A Practical Guide for Primary School Teachers. Groups in Learning, Bristol.
Williams, T., Wetton, N. and Moon, A. (1989) A Way In: Five key areas of Health Education. Health Education Authority, London.
WHO European Working Group on Health Promotion Evaluation (1998) Health Promotion Evaluation: Recommendations to Policymakers. WHO Europe, Copenhagen.
Received on June 16, 2000; accepted on March 11, 2001
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  D. Neumark-Sztainer, J. Haines, R. Robinson-O'Brien, P. J. Hannan, M. Robins, B. Morris, and C. A. Petrich 'Ready. Set. ACTION!' A theater-based obesity prevention program for children: a feasibility study Health Educ. Res., July 11, 2008; (2008) cyn036v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 N. Daykin, J. Orme, D. Evans, D. Salmon with, M. McEachran, and S. Brain The Impact of Participation in Performing Arts on Adolescent Health and Behaviour: A Systematic Review of the Literature J Health Psychol, March 1, 2008; 13(2): 251 - 264. [Abstract] [PDF]
|
|
|
|
 |
|
 D. Peerbhoy and C. Bourke Icebreaker: The evaluation Health Education Journal, September 1, 2007; 66(3): 262 - 276. [Abstract] [PDF]
|
|
|
|
|
|
J. M McWhirter, A. J Young, and N. Wetton In a class of its own: Introducing a new tool for understanding adolescents' perceptions of the world of drugs Health Education Journal, January 1, 2004; 63(4): 307 - 323. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||