Health Education Research Advance Access originally published online on May 20, 2004
Health Education Research 2004 19(5):570-575; doi:10.1093/her/cyg073
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Health Education Research Vol.19 no.5, © Oxford University Press 2004; All rights reserved
Developing complex interventions for rigorous evaluation—a case study from rural Zimbabwe
1 Centre for Sexual Health and HIV Research, University College London, London WC1E 6AU, UK, 2 London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, UK and Departments of 3 Community Medicine and 4 Psychology, University of Zimbabwe, Harare, Zimbabwe
5 Correspondence to: R. Power; E-mail: rpower{at}gum.ucl.ac.uk
 |
Abstract |
|---|
 Top Abstract IntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
Much attention has been placed on the need to develop and evaluate complex interventions targeting public health issues, such as reproductive health. However, and as has been the case in the recent past, even well-designed trials will be flawed unless meticulous attention is paid to ensuring the most appropriate intervention is designed and developed. This requires a well-resourced and carefully planned feasibility study, incorporating both formative and process evaluation, with particular attention being paid to the context of the proposed intervention. In this paper, we describe the way in which a feasibility study helped redesign and shape a complex intervention targeting adolescent sexual health in rural Zimbabwe. By using a mixture of in-depth interviews, focus groups and participant observation with pupils, parents, teachers and education officers, we were able to show that the intervention as originally conceived was unlikely to be deliverable. Process evaluation findings from the feasibility study led to substantial changes to both the content and delivery of the proposed intervention, which is now subject to testing for effectiveness in a large community randomized trial.
Much attention has been placed on the need to develop and evaluate complex interventions targeting public health issues, such as reproductive health. However, and as has been the case in the recent past, even well-designed trials will be flawed unless meticulous attention is paid to ensuring the most appropriate intervention is designed and developed. This requires a well-resourced and carefully planned feasibility study, incorporating both formative and process evaluation, with particular attention being paid to the context of the proposed intervention. In this paper, we describe the way in which a feasibility study helped redesign and shape a complex intervention targeting adolescent sexual health in rural Zimbabwe. By using a mixture of in-depth interviews, focus groups and participant observation with pupils, parents, teachers and education officers, we were able to show that the intervention as originally conceived was unlikely to be deliverable. Process evaluation findings from the feasibility study led to substantial changes to both the content and delivery of the proposed intervention, which is now subject to testing for effectiveness in a large community randomized trial.
|
Introduction |
|---|
|
TopAbstractIntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
There is increasing recognition of the need for rigorous evaluation of complex interventions to determine effectiveness, especially when social behavioural as well as biological outcomes are targeted (Stephenson and Imrie, 1998
; Campbell et al., 2000). However, trials of complex interventions are challenging for two reasons. First, due to the problems associated with developing, identifying and reproducing the intervention itself. For example, few of the many interventions aiming to improve reproductive health of young people have been theoretically based (Fisher and Fisher, 1992; Oakley et al., 1995; UNAIDS, 1997; Wight et al., 1998; Harden et al., 2001). Second, because of the logistic difficulties associated with trial design and implementation. Even well-designed trials will be flawed unless meticulous attention is paid to ensuring the most appropriate intervention is designed and developed. This requires a well-resourced and carefully planned feasibility study, incorporating both formative and process evaluation, with particular attention to the context of the proposed intervention. Even if an intervention is conceptually based, its effectiveness is likely to be context specific. In one recent study, a ‘diffusion of innovation’ model was adopted in a peer-led HIV intervention in gyms patronized by gay men (Elford et al., 2001). However, it proved ineffective largely because the setting for the intervention was inappropriate, in that the etiquette and social norms in the gyms militated against face-to-face health promotion. Similarly, a randomized controlled trial of sex education set in Scottish schools, although based on a piloted psycho-social model, was unable to demonstrate substantial differences between those receiving the intervention and the controls (Wight et al., 2002). In retrospect, the authors noted that the low priority and non-examinable status of sex education, timetabling constraints and teacher mobility impacted on the intervention's delivery. In Masaka, Uganda, a school-based randomized controlled trial involving 2000 students from 31 schools showed no effect. It was subsequently established that the intervention had been incompletely implemented due to shortage of curricula time, teacher reticence and lack of skills in the participatory methods required (Kinsman et al., 2001).
As we describe below, a well-designed feasibility study can forestall some of the pitfalls in intervention delivery that have beset other trials of complex interventions. This is especially the case regarding ‘fidelity assessment’ which examines the extent to which an intervention is implemented as intended (Rossi and Freeman, 1985).
The primary aim of our study was to determine the feasibility and acceptability of conducting a community randomized trial of an adolescent reproductive health intervention in rural Zimbabwe (Cowan et al., 2002). We will use our experience to reflect on some pertinent issues in the design and implementation of complex interventions.
|
How process evaluation at the feasibility study stage helped shape our intervention in Zimbabwe |
|---|
|
TopAbstractIntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
Following lengthy discussion with the Ministry of Education, Sport and Culture (MoESC) it was agreed to use Zimbabwe's life skills curriculum as the basis for the feasibility study (O'Donoghue, 1996
). This would dovetail with the MoESC initiated provision of weekly Guidance and Counselling classes (G & C) to pupils throughout secondary school (Chief Education Officer, 1993) and for life skills teaching (including those relating to sexual relationships and reproductive health) to take place within this class. Yet it soon became clear there were difficulties associated with the delivery of this intervention. Consequently, we decided to include elements of process evaluation during the feasibility study to guard against recommending an undeliverable intervention for the main trial. In line with sound process evaluation techniques, we aimed to address the context in which the intervention was being delivered as well as the delivery and content of the intervention itself (Rychetnik et al., 2002).
The feasibility study took place in four rural secondary schools in Masvingo Province, Zimbabwe. Teachers were trained to deliver reproductive health materials to Form 1 and Form 2 pupils (age 12–17) in weekly lessons for two terms. All teachers completed a recording sheet at the end of each lesson indicating the strengths and weaknesses of the materials, and how the session had worked. Qualitative data were collected from pupils, parents, teachers, headmasters, regional education officials, community stakeholders and health care providers by means of focus discussion groups, in-depth interviews and 4 weeks of participant observation in the schools. Additional focus groups were held to explore issues raised in the research. The qualitative data were tape-recorded, transcribed and translated verbatim from the Shona into English, and then analysed thematically to create broad ‘categories’ and ‘subcategories’ (Glaser and Strauss, 1966).
To illustrate the importance of this phase of research, and its implication for the development and delivery of the intervention to be subsequently tested in the randomized controlled trial, we will highlight some of the key ‘categories’ and describe how they would come to influence our planning.
|
The classroom may not be an appropriate ‘context’ for delivering the intervention |
|---|
|
TopAbstractIntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
A key issue of contention to arise from the feasibility study was that of the classroom as the setting to deliver such an intervention to Zimbabwean youth. In traditional Zimbabwean society, sex education of teenagers is undertaken, in private, by designated relatives (maternal uncle for boys, paternal aunt for girls) (Gelfand, 1965
). Urban migration, and societal and familial shifts have reduced such kinship influences, leaving a vacuum in informal education (Sherman and Bassett, 1999). In community discussions, parents reported feeling unable to talk to their children about sex. Discussions with teachers, headmasters and parents revealed that adults think it is inappropriate to talk directly and publicly (as in a classroom) about sexual intercourse. In addition, parents, as well as some teachers and headmasters, were wary about discussing sex, unless in promoting abstinence.
...I was uncomfortable discussing topics like reproductive organs because according to our Shona culture it's taboo. [Teacher: Focus Group Discussion (FGD)]Concern was expressed by adults that informing young people about safer sex would inevitably lead to experimentation. Participant observation of class lessons noted some teachers discouraging pupils from sexual activity, telling them condoms are ineffective.
These kids do not know that what they learn should end in the classroom. They want to go and put what they learn into practice, causing great suffering. [Parent: FGD]Doubt was also cast as to the appropriateness of teachers delivering the intervention, due to prevailing power relations. One major obstacle to sex education programmes is the difficulty in creating an open and participatory ambience for G & C classes. Generally, teaching takes place in a less permissive environment, where the use of corporal punishment by teachers is both accepted and widely practised:
...if our teacher sees us quiet and looking down, he will beat us. [Student: FGD]Furthermore, both teachers and students indicated that there was sexual abuse of children by some schoolteachers in return for favours, such as good exam grades:One day a teacher beat up a student resulting in him bleeding profusely. This was for peeping through the window whilst another teacher was having a lecture. [Field notes, participant observation]
The problem is that the teachers are male teachers with us girls. I will be afraid. Some of the problems are difficult ones to the extent that male teachers propose love to you, saying that he would help you in your problem. If you turn them down, they will hate you and they will beat you all the time. [Student: FGD]This presents a potential barrier of trust for any program that is delivered by teachers.Other teachers will ask you to come for extra tuition on a Saturday and there automatically you will be driven into sexual intercourse. [Student: FGD]
Compounding this problem, pupils commonly reported sexual abuse from outside school. The reality for many young people is that Zimbabwe's difficult economic situation increases pressure on adolescents to exchange sex for money, goods and other services, such as food, school fees or free accommodation close to the school:
If your parents are deceased and you stay with another relative who will be ill-treating you all the time, to the extent that they sometimes have sex with you or they oppress you or do not give you enough food, what should you do? [Student: FGD]On the other hand, teachers reported difficulty knowing what to do when abuse was reported to them by pupils, having received little or no training to deal with such sensitive issues:The main reason why young people are having sexual intercourse with old people is that they want money... [Student: FGD]
For example, there was a girl who came to me. She was being abused by other boys in the classroom. I did not know what to say to her. I just told her that you should have reported to me earlier. But I just felt I could have said more. [Teacher: FGD]All in all, there was plenty of evidence to doubt whether a teacher-led, classroom-based intervention would be deliverable. We did not need to run a randomized controlled trial to come to this conclusion.We find it difficult to counsel pupils with problems. [Teacher: FGD]
|
The school infrastructure did not provide a robust ‘conduit’ to ‘deliver’ the intervention materials |
|---|
|
TopAbstractIntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
Despite MoESC support for G & C, we found that in practice G & C classes had low status in schools and were not taught regularly, echoing the experience in Scotland (Wight et al., 2002
) and Uganda (Kinsman et al., 2001). Furthermore, the topic was not examinable, one consequence being that its delivery was not monitored by the authorities:The problem is that most people do not take G & C seriously because it is not tested...there is no assessment from the Ministry. [Teacher: FGD]The low status attributed to G & C lessons led to a number of practical implications for the delivery of sexual and reproductive health teaching in schools. We found that G & C classes were seldom included on the master timetable. This meant classroom space was not allocated, with teachers having to teach the subject either outdoors or out of school time.
The subject is not taken seriously...we teach, but the subject does not even appear on the master timetable. We use the student's free time. [Teacher: FGD]It was evident from record report forms and discussions with pupils and teachers that many lessons were cancelled. Teachers reported lack of support from other teaching colleagues and stigmatization for taking on the extra teaching load without remuneration.There are no classroom facilities, so the class is taught outside. When it rained last week, class was cancelled. Teaching happens at different times for each form class. [Teacher: FGD]
G & C is not considered part of the teaching load. Like you can't say ‘I have 31 loads’ and if other teachers ask you which makes them 31, and you mention G & C, they will laugh at you. [Teacher: FGD]By observing lessons on subjects other than G & C we noted that most teaching in schools was didactic. Within G & C classes teachers were encouraged to use participatory teaching methods including drama, role-play and group discussion—all important for creating an enabling environment for positive behaviour change (Kirby et al., 1994
). However, the majority of teachers had little experience of these methods prior to the specific training we provided. Observation of teaching sessions illustrated some difficulty teachers had in creating an interactive learning environment, leading to a lack of openness between students and teachers:We do not go to ask questions from teachers because some teachers are rough to school children. This causes children to fear to go and ask. [Student: FGD]
|
Existing materials may not provide relevant ‘content’ for the intervention |
|---|
|
TopAbstractIntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
The feasibility study was used to examine and pilot the content of various materials used in the school. Pupils and teachers indicated that some exercises were too modern, urban-based and unrelated to life in rural areas:
I wanted to raise the issue of ‘dating’. Here in the rural areas they don't know anything about dating. I think this happens in towns. [Teacher: FGD]Conversely, some important topics for those living in rural areas were not addressed:
The other thing would be to use examples of what is happening here in the rural areas like kugara nhaka [wife or husband inheritance in the event of death], chiramu [brothers-in-law playing with the wife's young sister]. These are familiar to the students and they would understand better if these examples were used. [Teacher: FGD]In Zimbabwe, the MoESC requires that all secondary education be conducted in English. However, comprehension of English is often poor among children in rural areas:
The language used at times can be a problem. Speaking in English is wasting time because it is as good as speaking to yourself...so if we could have the material in Shona, that would be easier for the students to understand. [Teacher: FGD]Sexual and reproductive health teaching lends itself to using indigenous languages (few rural children will need to use their sexual refusal skills in English). However, teachers, parents, and pupils all agreed that, in order to overcome the sensitive nature of the subject matter English should be used when presenting reproductive health biology and that indigenous languages should be employed when discussing reproductive health issues with an emotional content. We observed that lessons taught in indigenous language were much more interactive than those in English.I got the impression that student participation is very high when students are told to use any language at the beginning of the session. [Field notes, participant observation]
|
Discussion |
|---|
|
TopAbstractIntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
It has been some 7 years since Oakley and colleagues called for more methodological rigour in social behavioural research (Oakley et al., 1995
), yet, as described above, we have already witnessed a number of examples where poor intervention design has resulted in flawed studies. Our study in Zimbabwe illustrates the importance of a thorough feasibility study. Process evaluation findings from the feasibility study have led to substantial changes to both the content and the delivery of the proposed intervention, which we are now funded to test for effectiveness in a large community randomized trial. In relation to the content, we will use a programme, relevant to rural Zimbabwe, integrating relevant and evaluated resource materials. Notably, all the materials relating to development of negotiation skills or on emotional issues will be in indigenous language. It is clear that teachers found it difficult to deliver the programme for a variety of logistic reasons (low morale, lack of support and competing priorities at school) and contextual reasons (difficulty teaching about sensitive issues, switching from their traditional teacher role, and lack of trust between pupils and teachers). The programme will now be delivered by carefully selected and trained school-leaver volunteers, who will live full-time in the study communities. In addition, it has become apparent that an intervention is unlikely to be effective without wider societal support. The school-based intervention will therefore be complemented by a community-based programme aimed at parents, designed both to raise their awareness of issues relating to adolescence and at improving parent–child communication. Commensurate with this approach, the programme has been extended to include both school pupils and out-of-school youth. This is relevant, as those who drop out of school early appear to be particularly vulnerable to HIV.
Had we not explored, in detail, the context, content and, crucially, the delivery of the proposed intervention, prior to starting the effectiveness trial, we may have spent considerable time and resources trialing a suboptimal or undelivered intervention, as, for example, was the case in Masaka, Uganda (Kinsman et al., 2001). This could have serious implications. First, 20 years into the HIV epidemic we have little evidence on the effectiveness of HIV prevention for adolescents. As few of these expensive and difficult trials are ever funded, an inconclusive trial taking 5 years to run would further delay finding out what works and what does not. Second, and potentially more importantly, a trial that showed, for example, that ‘sex education does not work’ could have major implications for policy and funding priorities, even if this was because the intervention itself was inappropriate or not properly implemented.
|
References |
|---|
|
TopAbstractIntroductionHow process evaluation at...The classroom may not...The school infrastructure did...Existing materials may not...DiscussionReferences |
|---|
Campbell, M., Fitzpatrick, R., Haines, A., Kinmonth, A.L., Sandercock, P., Speigelhalter, D. and Tyrer, P. (2000) Framework for design and evaluation of complex interventions to improve health. British Medical Journal, 321, 694–696.
Chief Education Officer (1993) Education Circular: AIDS Action Programme for Schools. Circular Minute 16:D/109/3.
Cowan, F.M., Langhaug, L.F., Mashungupa, G., Nyamurera, T., Hargrove, J., Jaffar, S., Peeking, R.W., Brown, D.W., Power, R., Johnson, A.M., Stephenson, J.M., Bassett, M.T., Hayes, R.J.; Regai Dzive Shiri Project (2002) School-based HIV prevention in Zimbabwe: feasibility and acceptability of evaluation trials using biological outcomes. AIDS, 16, 1673–1678.[CrossRef][ISI][Medline]
Elford, J., Boding, G. and Sherr, L. (2001) Peer education has no significant impact on HIV risk behaviours among gay men in London. AIDS, 15, 535–580.[CrossRef][ISI][Medline]
Fisher, J.D. and Fisher, W.A. (1992) Changing AIDS-risk behaviour. Psychological Bulletin, 111, 455–474.[CrossRef][ISI][Medline]
Gelfand, M. (1965) African Background: The Traditional Culture of the Shona. Juta, Cape Town.
Glaser, D. and Strauss, A. (1966) The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine, Chicago, IL.
Harden, A., Oakley, A. and Olivers, S. (2001) Peer delivered health promotion for young people: a systematic review of different study designs. Health Education Journal, 60, 339–353.
Kinsman, J., Nakiyingi, J., Kamali, A., Carptenter, L., Quigley, M., Pool, R. and Whitworth, J. (2001) Evaluation of a comprehensive school-based AIDS education programme in rural Masaka, Uganda. Health Education Research, 16, 85–100.
Kirby, D., Short, L., Collins, J., Rugg, D., Kolbe, L., Howard, M., Miller, B., Sonenstein, F. and Zabin, L.S. (1994) School-based programmes to decrease sexual risk behaviours: a review of effectiveness. Public Health Report 109, 339–360.[ISI][Medline]
Oakley, A., Fullerton, D., Holland, S., Franc-Dawson, M., Kelley, P. and McGrellis, S. (1995) Sexual health education interventions for young people: a methodological review. British Medical Journal, 310, 158–162.
O'Donoghue, J. (1996) Zimbabwe's AIDS action programme for schools: a case study. Promotion and Education, 3, 7–12.
Rossi, P. and Freeman, H. (1985) Evaluation: A Systematic Approach. Sage, London.
Rychetnik, L., Frommer, M., Hawe, P. and Shiell, A. (2002) Criteria for evaluating evidence on public health interventions. Journal of Epidemiology and Community Health, 56, 119–127.
Sherman J.B. and Bassett, M.T. (1999) Adolescents and AIDS prevention: a school-based approach in Zimbabwe. Applied Psychology, 48, 109–124.[Medline]
Stephenson, J. and Imrie, J. (1998) Why do we need randomised controlled trials to assess behavioural interventions? British Medical Journal, 316, 611–613.
UNAIDS (1997) Impact of HIV and Sexual Health Education on the Sexual Behaviour of Young People: A Review Update. WHO/GPA, Geneva.
Wight, D., Abraham, C. and Scott, S. (1998) Towards a psycho-social theoretical framework for sexual health promotion. Health Education Research, 13, 317–330.
Wight, D., Raab, G., Henderson, M., Abraham, C., Buston, K., Hart, G. and Scott, S. (2002) Limits of teacher delivered sex education: interim behavioural outcomes from a randomised trial. British Medical Journal, 324, 1430–1437.
Received on May 27, 2003; accepted on October 9, 2003
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  L. Brereton, C. Carroll, and S. Barnston Interventions for adult family carers of people who have had a stroke: a systematic review Clinical Rehabilitation, October 1, 2007; 21(10): 867 - 884. [Abstract] [PDF]
|
|
|
|
 |
|
 N. C Campbell, E. Murray, J. Darbyshire, J. Emery, A. Farmer, F. Griffiths, B. Guthrie, H. Lester, P. Wilson, and A. L. Kinmonth Designing and evaluating complex interventions to improve health care BMJ, March 3, 2007; 334(7591): 455 - 459. [Full Text] [PDF]
|
|
|
|
|
|
A. Oakley, V. Strange, C. Bonell, E. Allen, J. Stephenson, and RIPPLE Study Team Process evaluation in randomised controlled trials of complex interventions. BMJ, February 18, 2006; 332(7538): 413 - 416. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||