Health Education Research, Vol. 16, No. 2, 215-226,
April 2001
© 2001 Oxford University Press
Principles in practice: reflections on a `postpositivist' approach to evaluation research
Research Unit in Health and Behavioural Change, University of Edinburgh Medical School, Teviot Place, Edinburgh EH8 9AG, UK
 |
Abstract |
|---|
 Top Abstract IntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
User participation is currently seen as an ethically appropriate way to proceed when researching disadvantaged groups and it is encouraged by funding agencies. However, the literature rarely discusses the methodological and practical implications for researchers attempting to incorporate user participation into evaluation studies which are informed from an epistemologically opposed (positivist) research paradigm. The paper explores this issue by drawing on the evaluation of a community-based smoking intervention to describe and reflect upon the recruitment, training and employment of local residents as survey interviewers. While the evaluation methodology adopts a quasi- experimental approach, the appointment of local residents as survey interviewers reflects an alternative (interpretive) research tradition. The combined strategy constitutes a postpositivist methodology in that it combines a data collection strategy more akin to interpretive social science while retaining a positivistic epistemological framework. The paper describes some logistics of this approach and problems encountered during the course of survey. While many of the problems described may be routinely associated (although seldom aired) with survey work, particularly in disadvantaged areas, the paper suggests they are also a function of the postpositivist research strategy which we adopted. The failure to involve interviewers in the conception and development of the evaluation meant that they lacked identification with our endeavour and this had practical implications for the survey interviewing. Although the survey was successfully executed and the employment of local residents was a valuable and worthwhile experience, the authors recognize that this narrow conception of user involvement meant that many of the potential benefits (both to the research and the participants) associated with participatory approaches were forfeited.
|
Introduction |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
The paper draws upon an ongoing evaluation study, based at the Research Unit in Health and Behavioural Change at the University of Edinburgh, in order to examine the use of `postpositivist' methods in the evaluation of a community health intervention on smoking (`Breathing Space'). To this end the paper describes and reflects upon the recruitment, training and employment of community residents as survey interviewers. Although user participation is seen as an ethically appropriate way to proceed when researching disadvantaged groups and is encouraged by funding agencies, the literature offers little practical advice for researchers attempting to incorporate such qualitative procedures into an epistemologically distinct (positivistic) research framework.
|
Background |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
The evaluation study is intended to assess the potential of the intervention to alter the culturally normative status of smoking in a low-income area in Edinburgh. In common with many large-scale community interventions, `Breathing Space' targets the entire community on the understanding that the context in which the individual risk resides is a key contributor to the cause of individual risk and highlights the importance of creating cultural receptivity (as evidenced by shifts in attitudes, values, norms, etc.) towards health-related risk behaviour (Lefebvre et al., 1987
; Altman et al., 1995). The design of the evaluation involves a pre- and post-intervention survey conducted in the experimental (intervention) area and in three control areas. In addition, a detailed process recording chronicles the development and implementation of the intervention in the experimental area as well as mapping smoking-related developments in smoking in the control areas. Moreover, the evaluation includes a form of `user participation' in an endeavour to reflect the community-based ethos of the intervention. This is achieved through consultation with community groups during the survey design and questionnaire development stage of the research, and via the commitment to the training and employment of local residents as survey interviewers.
While the evaluation methodology adopts a quasi-experimental approach, the appointment of local residents as survey interviewers reflects an alternative (interpretive) research tradition. The combined strategy constitutes a postpositivist methodology in that it engages with some of the heuristic insights associated with interpretive social science and also retains a positivistic epistemological framework (Denzin and Lincoln, 1998). Such a combined or pluralist approach is not uncommon in evaluation research (Pawson and Tilley, 1997) where an attempt is made to make the best of both worlds by simultaneously incorporating the reliability afforded by quantitative measurement, and celebrating concepts such as user empowerment and participation.
Although utilizing a form of user participation in our evaluation, we recognize that user participation is a complex phenomenon which does not necessarily lend itself to every research circumstance. In this paper we specifically focus upon the recruitment, training and employment of interviewers for the baseline survey in order to examine the feasibility of the endeavour.
Participatory research is currently regarded as an appropriate and desirable way in which to study disadvantaged lives (Rappert, 1997). Its ascendancy has been facilitated by the growing acceptance of qualitative (and particularly feminist) methodologies which challenge traditional dichotomies between both researcher and researched and professional and lay knowledges. Moreover, participatory research embraces the principle of user participation which is fundamental to a community-based approach to health and social welfare.
Community-based initiatives are underpinned by the concept of `enabling environments'. These are contexts in which community members are encouraged to participate in the planning, implementation and management of health interventions as well as the dissemination of health-related information (Gillies, 1998; Campbell et al., 1999).
The concept of user involvement, as a crucial component of community-based approaches to health, has gained popularity as a result of the general shift in thinking about those who provide and those who receive health and social welfare services (Beresford and Croft, 1993). By allowing the community to have more say in the shaping of policies influencing health, it is claimed that community participation represents a break with earlier traditions of public health associated with top-down social engineering (Petersen and Lupton, 1996).
Participatory approaches have been described as an optimum way of building upon and increasing social capital accruing to communities (Gillies, 1998). Social capital (Putman, 1993a,b, 1995) has been developed as a useful construct for understanding the relationship between communities and health because of the emphasis which it places upon community coordination, cooperation and reciprocity (Lomas, 1998). Social capital encompasses both formal and informal community networks, adopting an intermediary space between micro (individual) and macro (social) levels where much community-based work takes place (Gillies, 1998). Moreover, the construct highlights the importance of alliances or partnerships among different stakeholders. These alliances assist in promoting health across sectors, across professional and lay boundaries, and between public, private and non-governmental agencies, and arguably form the cornerstone of successful community-based development work (Campbell et al., 1999).
Despite the advantages of community-based approaches, several problems have been highlighted in the literature. First, it is suggested that the `ideal' of participation, encompassed by the approach, often amounts to little more than tokenism. At the extreme, it has been argued, users may be coopted to participate in the very power structures which they set out to oppose (Petersen and Lupton, 1996). Second, some community participation programmes are criticized because they have been established around priorities identified in a top-down (rather than a bottom-up) way, undermining the aim to increase users' self-efficacy through grassroots ownership (Asthana and Oostvogels, 1996). Third, existing social networks and resources within which community-based approaches must work may be limiting, and the extent to which these conditions can be created is contestable. Fourth, assumptions about particular communities, which inform the design and execution of programmes, may not always be accurate. For example, neither neighbourhood proximity nor the proliferation of particular health-related risk behaviours necessarily implies levels of social interaction or solidarity conducive to successful community-based work (Campbell et al., 1999).
On a practical level there are costs to users associated with the approach. It has been argued that a `discourse of active citizenship' has served to obscure the interpersonal demands and responsibilities required of those called on to conform to the participatory ideal (Petersen and Lupton, 1996). These may include unanticipated levels of commitment and input, a high demand upon personal resources, and frustration at lack of impact. Moreover, when user participation, as a strategy, goes wrong it may simply reinforce personal feelings of anger and powerlessness (Barnes, 1994; Raynes, 1998).
|
Participatory research |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
Moves towards an increased empowerment of service users in the health and welfare sector have been paralleled by concern about the inability of positivist research approaches to mount an effective challenge upon the social processes of discrimination and marginalization. This has led to demands for a `stakeholder approach' in which individuals who are themselves discriminated against, marginalized or disadvantaged, undertake a research role (Hills, 1998
). Moreover, it is argued that greater involvement leads to better outcomes. Hence, community members who have the opportunity to reflect critically on the research process and participate as decision makers will, it is argued, feel empowered as `creators' and `choosers' rather than simply people who apply techniques and follow rules (Meulenberg-Buskens, 1996).
Despite its current ascendancy, participatory research constitutes a fundamental challenge to the positivistic research paradigm within which most systematic evaluation work resides. This is because participatory research contests the very premises (neutrality, objectivity and value freedom) on which traditional social science methodology is based. The challenge offered by interpretive methodologies is that they effectively interrupt the distance between the researcher and researched, and related to this, the dichotomy of subject and object. In turning the traditional research model on its head, the participatory approach advocates that, rather than fixing particular users' needs, these should be actively constructed or shaped, at an early stage in the research, in a reflexive way between users and researchers (Woolgar, 1997).
In addition to questioning traditional reliance upon quantitative measurement, the participatory approach challenges accepted patterns of control and ownership of knowledge (Maguire, 1996; Tandon, 1996). In an expert dominated culture, such as our own, there exists an unstated assumption that scientific knowledge is superior (Petersen and Lupton, 1996). The predilection to attribute `knowledge problems' to the scientific ignorance or naivety of participants or to problems with lay knowledge or lay rationality, or both, may create barriers to the recognition of lay expertise.
Despite raising fundamental questions about the appropriateness of research methodologies and the epistemological and political status of knowledge (Woolgar, 1997), there has been little analysis of the effects of user participation on the research process. Furthermore, the current preference for user involvement is presented as wholly compatible with the newly emerging (positivist) market model of academic research (Rappert, 1997). Indeed this movement towards the commercially viable product, in which user needs are prioritized, is reflected in current Research Council policy [(HMSO, 1993), p. 270].
|
Involving local interviewers |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
The study was designed to evaluate the effectiveness of `Breathing Space' through the measurement of difference between the experimental area and combined control areas in the amount of change between initial and follow-up surveys in respect of several cultural indicators. These include knowledge and belief about smoking, its causes and consequences (especially for health), perceived norms about smoking (local, national), personal attitudes to smoking, perceived community support for quitting, and community toleration for smoking (in public and private spaces).
The quasi-experimental approach which we used required that the techniques and types of data to be collected were standardized. At the same time we determined to employ local interviewers and there were several reasons for this. First, the involvement of users in the data collection would, it was felt, reflect the community-based ethos of the intervention. Second, the involvement of users during data collection was intended to assist the aim of the intervention by raising awareness of `Breathing Space' among the interviewers and, through them, other members of the community. Third, it was anticipated that the intention to identify, train and employ local people would provide a powerful motivation for communities to participate in the research. This would apply particularly in the case of the three control areas whose potential gains from participation in the study were less apparent than those of the experimental area. Fourth, it was thought advantageous to involve participants who would be, to some extent, knowledgeable about their communities and connected to local networks. Fifth, by avoiding travelling expenses we thought that the employment of local interviewers would be a cost-effective strategy. Finally, given that is a practice currently favoured by many funding bodies, it was felt the inclusion of users in the research process might enhance the likelihood of a successful outcome to the grant application.
It is generally agreed that participation should not be restricted to situations where local people work with a researcher simply for the latter's convenience (de Koning and Martin, 1996). However, because total participation (especially in situations of extreme disadvantage) has been described as an unrealistic expectation (Madan, 1987), programmes of developing participation starting with at least one aspect of users' or communities' lives are more often advocated (Payne, 1995; Forbes and Sashidharan, 1997). Although we did not expect that our strategy of employing local interviewers would provide them with the advantages associated with participatory research, we did anticipate that the employment would benefit some community residents who were disadvantaged in the job market
|
The baseline survey |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
The baseline survey aimed to achieve 1000 interviews in the experimental area and in each of the three control/reference areas. In order to achieve the sample, households in each area were randomly selected from the Postcode Address File (PAF), which is a listing of all active address points maintained by the Post Office. Working on an expected minimum response rate of 70%, 1400 addresses were randomly drawn from the PAF in each area. At each of the selected households, one respondent (aged 12 years or older) was to be randomly selected and interviewed.
We aimed to recruit and train 40 local interviewers who would be employed for the duration of the survey, which was estimated to be 10 weeks. Each interviewer was to be allocated to one of the four area teams, comprising 10 interviewers, headed by an area supervisor. Recognizing there are both advantages and disadvantages to employing local interviewers, we determined to ensure that the interviewers would not be allocated addresses in the actual locality where they lived. The intention here was capitalize upon the interviewers' local knowledge while not jeopardizing the response rate by raising respondent concern about disclosure of information to individuals whom they might know.
Interviewers were recruited in several ways. These included advertisements in the local press, via our contacts with community groups, job centres, a local jobs fair, by word of mouth, and advertisements at Edinburgh University and further education colleges located within the survey catchment. Advertisements to attract students to the research were aimed at home-based students living in the areas identified for the research. In addition, a professional organization was approached with a view to engaging trained interviewer supervisory support. However, the organization declined to be involved because their members refused to work in `disadvantaged areas' such as the ones we had identified. The reasons given were fear for personal safety and anticipated difficulties in recruiting respondents to the survey. The research team then sought to identify potential supervisors among applicants residing in each of the research areas.
The initial selection and training sessions, which were held prior to the onset of the survey, took place over 2 days (a full day and 2 half days). Forty-seven applicants attended the first (selection and training) day. The (42) successful applicants who were not selected out (by either the research team or themselves) participated in a further 2 half-day training sessions. During the initial training and selection days, potential group supervisors were identified to head teams of interviewers allocated to each area and at the final half-day session interviewers were allocated to these teams.
At the training sessions participants were provided with background information about both the intervention and the evaluation. Following this introduction, participants were instructed on the role of interviewer and tutored on selection procedures, use of the questionnaires and interview techniques. This training focused upon the necessity of reading fluently and speaking clearly as well as techniques to improve the rapport and the interview process by, for example, creating a relaxed atmosphere, seating arrangements, discouraging third parties, using eye contact, not leading respondents and avoiding offering the interviewer's opinion.
Following familiarization with the survey instruments participants carried out supervised role play. Participants were also trained to use the extensive collection of survey documentation developed to facilitate fieldwork and enable the systematic recording of progress. In addition, a range of issues associated with survey interviewing, such as confidentiality and personal safety, was covered at these sessions.
The participants levels of skill varied considerably. A few had previous interviewing experience (mainly market research) and some had utilized communication skills in other lines of work. The interviewing ability of the majority of participants improved remarkably over the course of the training and those few who did not improve were not selected to take part in the survey.
At each recruitment session the issue of the participants' own smoking was raised by the research team. Participants were told that interviewers would be recruited irrespective or whether or not they smoked, but that it would not be permissible for them to smoke while interviewing, under any circumstance.
The research team attempted to offset difficulties which they anticipated the interviewers might face. A level of training was provided to the limits of the available resources. Participants attending training sessions were reimbursed for their travel expenses, provided with lunch and paid a training fee. Interviewers were offered an attractive rate of pay and were paid by the hour rather than the number of interviews completed. This strategy sought to avoid interviewers rushing through interviews (or even falsifying questionnaire responses) in order to achieve maximum payment.
|
The outcome |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
Several weeks into the survey it became apparent that the interview completion rate was much slower than anticipated. On average 35 completed interview schedules were being returned each week, instead of the expected 100. The discrepancy between desired and achieved outcome was due to a combination of factors, which we recognize to be associated, at least to some extent, with carrying out research in areas of disadvantage. These included the large number of ineligible addresses, a high non-contact rate and unanticipated levels of interviewer turnover.
The ineligible addresses
Despite postponing drawing the sample until the latest PAF update, our source of addresses was not as up to date as anticipated. Although we expected a shortfall between addresses supplied and those eligible, the actual discrepancy was far higher than anticipated. Of the 6522 sampled addresses, 975 (14.9%) were ineligible. Mostly the ineligible addresses were either demolished or unoccupied, but a few transpired to be non-residential or untraceable. The percentage of ineligible addresses differed considerably between areas, ranging from 7 to 22%, suggesting that overall the PAF may be less up to date in deprived areas, and its accuracy may be a function of area-related characteristics such as demolition activity, migration rates and levels of population decline. The high level of ineligible addresses, while undoubtedly a technical problem for the survey, did provide an insight into housing conditions which characterized the types of disadvantaged areas in which we were working.
The non-contact rate
Of the remaining (5547) eligible addresses, 1021 (18.4%) could not be contacted by the end of the survey. The interviewers, despondent about the high non-contact rate, returned voicing suspicions about giro-drops (addresses maintained solely for the purposes of benefits receipt), shift workers employed late into the evening and residents (particularly of high-rise buildings) loathe to open their doors to strangers. The high level of ineligible addresses resulted in an increase in the ratio of time spent attempting to secure interviews as opposed to actual interviewing. Consequently, interviewers, who had originally been instructed to make up to four visits to secure interviews, were visiting some addresses (to no avail) as many as 10 times. Unsurprisingly, many interviewers became dispirited and requested additional addresses while only achieving outcomes on a very small proportion of those issued to them.
Upon successfully recruiting respondents to the study, interviewers experienced few difficulties in administering the interview schedule which mainly comprised structured questions on health-related beliefs and behaviours. Personal income was the only area for which problems were anticipated but these did not materialize.
Interviewer turnover
The strategy of employing local residents as interviewers contributed to a high interviewer turnover rate. Although it was originally anticipated that, once trained, interviewers would remain employed for the duration of the survey, this was not the case. Of the 42 participants from the initial selection and training day who were recruited to the survey, 34 actually started work and only 10 remained until the survey was completed.
There were several reasons for the high rate of interviewer turnover. First, because the survey could only offer temporary employment some interviewers were tempted away by more permanent work. Second, the interviewing mainly attracted individuals who appeared to have other more pressing priorities and who were only prepared to work part-time on the survey. There was a tendency for the commitment of those offering minimal input to dwindle to the point where their continued employment was counter-productive. Third, the majority of local residents employed as interviewers were receiving state benefits. Aspects of the benefits system, particularly relating to the penalization of claimants working over a specific number of hours, did not match the needs of these temporary, part-time employees. Fourth, the inflexible University payroll system, which required employees to work a month in hand, was similarly ill equipped to meet the immediate needs of these employees. It was unreasonable to expect interviewers ineligible for state benefits to work a minimum of 1 month before payment.
Fifth, many of our interviewers were unprepared for dealing with the exigencies of the work. The survey team had, for example, underestimated the extent to which the survey would have to be `sold' on the doorstep to potential respondents. In some cases the training provided was insufficient for the task in hand. Moreover, working in the survey areas was experienced as emotionally difficult by several of the interviewers who commented on the conditions of disadvantage which indexed some of the respondents' lives. Interviewers, on occasion, were faced with ethical dilemmas resulting, for example, from learning of children left alone while parents worked, or when facing respondents clearly under the influence of drugs or alcohol. One interviewer resigned after a respondent claimed to know the identity of the perpetrator in a police murder investigation.
Although personal safety issues were covered at the training sessions and interviewers were equipped with personal alarms, some respondents (and their significant others) expressed fears for personal safety and resigned because of these anxieties. Some female interviewers did not start work because of their partners' wishes and one student dropped out when her parents threatened to withdraw her rent allowance if she continued to work. Several other interviewers claimed, on occasion, to feel unsafe or uneasy and this increased their reluctance to conduct interviews during the evening, and doubtless exacerbated the high non-contact rate.
Addressing the problems
In order to achieve an acceptable sample size the data collection period was extended by 8 weeks. As a result of the high turnover of interviewers, the recruitment and training program had to be extended throughout this period.
The original intention to employ only local residents was abandoned simply because the supply of applicants dried up. The interviewer recruitment strategy was widened to encourage students from the local universities and other higher education colleges to apply. This was a timely move because the second round of recruitment coincided with the end of the spring term when many students became available for temporary work.
The period of training was reduced to 1 day and the balance of training was changed so that supervisors assumed a more active role in training, following input from the research team. Moreover, while participants still received travelling expenses and a free lunch, payment for training was restricted to those who successfully completed 10 interviews. Still further into the survey period, training payment was made dependent upon continued employment until the end of the survey.
The employment conditions in respect of those recruited following the onset of the survey were altered so that these interviewers were required to work a minimum of 20 h per week. Interviewers were also expected to be more flexible about where they worked. In order to raise the contact and response rates in areas with a poor return of completed interviews, interviewers were moved between the four survey areas.
To allay concerns about personal safety, interviewers had originally been instructed to cease interviewing (except in cases where a prior arrangement had been agreed) before 8 p.m. Due to longer daylight hours and in order to raise the level of successful contacts, interviewers were encouraged to interview up until 10 p.m., and to make additional visits on evenings and weekends.
Support to supervisors was provided by means of regular weekly meetings with the research team throughout the survey period. These meetings served as a forum where problems and issues relating to the survey could be shared and addressed. Supervisors received weekly updates on the interview completion rate in their own and the three other areas, and were alerted to the performance of individuals in their teams.
To combat the high non-contact rate further addresses were obtained from the PAF and distributed to interviewers working in areas where the problem was most acute. In addition, demolition lists were obtained from the city Council and addresses at demolition sites were recalled.
Despite the revised strategy a high drop out rate among the interviewers continued to be a problem, as a result of which four additional selection and training sessions were carried out. A breakdown of interviewer selection and drop-out is given in Table I.
|
The ratio of local residents to non-local interviewers recruited to the study shifted over the course of the survey. At the initial recruitment sessions there were 29 local residents, one of whom was a student. Of the remaining 13 successful candidates, who lived outside the survey areas, seven were students. At the subsequent recruitment sessions there were 18 local residents, one of whom was a student. Of the remaining 27 successful candidates, who lived outside the survey areas, 14 were students. Overall, there were 47 local residents, two of students. Of the remaining 40 successful candidates, who lived outside the survey areas, 21 were students. The most notable difference between student and resident interviewers lay in their apparent priorities. To reiterate, the life circumstances of many of the local interviewers were such that they were unable to fully commit themselves to the survey. In contrast, the student interviewers were able to make a fuller time commitment to work and tended to put in longer hours over the period of their employment. In many cases the students needed funds for a planned holiday and many left the survey upon realizing this objective. It is our impression, however, that the students were less committed than the local interviewers to the goals of the project.
Despite extending the duration of the survey, the target of 1000 completed interviews in each area proved impossible to meet. By the time the survey closed, 2678 interviews were completed (67% of the target of 4000). In order to increase the statistical power of the sample in the experimental area, resources were focused there in the final stages of the survey. As a result of this strategy, 88% of the target sample in the experimental area was reached compared to 60% in the control areas.
Resource implications
Time wastage constituted a major problem for the survey team. The Research Fellow and the four area supervisors spent increasing amounts of time and energy attempting to retrieve addresses from interviewers who had resigned, and redistributing them among the remaining team members and new recruits. Further time was dedicated to chasing interviewers who were sitting on addresses which they were not following up. When chased up, interviewers tended to promise to do better, yet ultimately many of them resigned after doing very little or nothing at all. Some interviewers appeared to vanish completely. They failed to respond to telephone messages, were not in when supervisors visited and were lax about returning survey documentation.
Recruitment drives and training days subsequent to the onset of the survey had enormous implications for survey administration, necessitating repeat advertising as well as the organization and execution of four additional selection and training sessions. They also increased the burden on supervisors who were responsible for some basic training and induction of new interviewers.
The strategy of reissuing non-contact addresses to more experienced interviewers towards the end of the survey period, which we anticipated would improve the response rate, was curtailed due to time constraints. While unable accurately to estimate the total impact of reissuing addresses we believe that good interviewers might have turned a substantial proportion of the non-responses into completed outcomes. This is based upon an improved response rate on the limited number of reissued addresses which were given to the more experienced interviewers. These interviewers were identified by a sustained involvement with the survey and good personal response rate. Because many of the interviewers did show an appreciable improvement in their response rate with prolonged experience, it is suspected that early refusals may in some cases been converted into successful interviews.
The high turnover of interviewers necessitated careful scrutiny of the completed questionnaire surveys, to check for mistakes made in the recording of responses. It is doubtful whether this high degree of scrutiny would have been necessary professional interviewers been engaged.
The survey also incurred increased financial costs as a result of unanticipated contingencies such as the application for additional PAF addresses as well as costs relating to interviewing. The non-contact rate and amount of time spent revisiting addresses significantly increased the cost per interview. Moreover, to avoid losing good interviewers, the survey team resorted, on some occasions, to paying child-care expenses. The practice of moving interviewers between areas meant that we were obliged to pay travel expenses to work. Although each completed interview cost over 30% more than anticipated, the survey did not run over budget since fewer interviews were achieved than had been intended.
|
Discussion and recommendations |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
Our experiences suggest that researchers intending to integrate aspects of participatory research into an overall positivist research framework should be aware of the methodological and practical implications for the research endeavour. It is to the practical considerations which we initially turn.
Our baseline survey provided training and temporary employment to a group of individuals who could be described as disadvantaged in the job market without involving them at an earlier stage in the research process. Under these conditions, unlike conditions of genuine participatory research, any real sense of ownership of the project is absent and it may be difficult to get interviewers to identify with the goals of the evaluation and to assess their own contribution to the study as worthwhile. To this end, interviewer supervisors who are themselves residents of the research areas have an important role to play in motivating their team members and in developing a sense of area team membership. Supervisors can also contribute to interviewer training, assisting with administrative tasks relating to the interviews, and fulfilling an intermediary role between the interviewers and the research team.
When locating research in disadvantaged areas it is easy to overestimate the numbers of potential interviewers who could be recruited to the research. Certainly we were surprised, given our broad ranging recruitment drive, by the relatively small number who attended the initial training sessions. It is important to remember that the work will not suit everybody. Potential interviewers differ in the level of personal and material resources which they can offer. In our experience, strategies which channel time and energy into chasing up errant interviewers and attempt to motivate those who contribute little to the survey are unproductive. A more efficient strategy might be to encourage competitiveness (with support) among interviewers through rewarding good performance.
In an attempt to satisfy the sometimes disparate needs of the interviewers, the conditions of employment may run counter to the overall aims of the research. It is well to bear these in mind. The minimum number of hours expected of any interviewer should be determined by the overall needs of the survey rather than by individual preferences. Employing interviewers for a few hours a week is not a productive strategy. The proficiency of these interviewers does not appear to improve as quickly as those who have higher input into the survey and they are more likely than interviewers working longer hours to lose interest and drop out.
Given that interviewer recruitment may not be particularly easy, preparations should be made to maintain recruitment and training at a high level throughout the survey period to ensure there are no shortages of workers in the field. Recruitment strategies should be broad to maximize the number of potential interviewers at recruitment and training days. Attempts should also be made to attract a range of potential applicants, particularly those (such as students) who are likely to be highly motivated to secure and complete casual work. Any payment for training should be made subject to contractual obligations.
To ensure the survey runs smoothly, all necessary preparatory work should be completed in advance of the start of the survey. Information regarding demolition and empty housing stock should be obtained so that redundant addresses can be removed from the file. Addresses should be allocated into batches ready for issue so that interviewing is not delayed by depleted supplies. Rather than calculating the number of addresses to be allocated according to individual need (based on agreed hours), each interviewer should receive the same number of addresses. Recognizing that some interviewers will resign and other may fail to exhaust their supply of addresses, batches of addresses issued to each interviewer should be fairly small.
The research team should be sympathetic to the financial circumstances of the interviewers. Because eligibility for state benefits varies according to individual circumstances, employees should be directed to seek advice from the relevant agencies. Interviewers should be supplied with the appropriate income tax (including exemption) forms. Methods of payment, wherever possible, should be tailored to address the needs of casual temporary employees.
In the interests of the survey outcome, the conditions of employment, once determined, should be non-negotiable and contractually agreed. Failure to comply with the conditions should lead to termination of the employment contract. It must be fully appreciated that, for interviewers who drop out, the experience will not be particularly rewarding. Moreover, the anticipated benefits (of capitalizing upon local knowledge and user networks) to both the evaluation and the intervention may be counter-productive where interviewers have an unsatisfactory experience. The onus is therefore on the research team to provide a realistic picture of what the job entails and applicants should be encouraged to consider carefully whether or not the job is suitable for them prior to signing the contract.
Despite the problems which we have described here, employing local interviewers was a very rewarding experience. The quality of the completed interviews was generally high. Random telephone checks were undertaken to eliminate fraud and ensure the interviews were being conducted in a satisfactory manner. Feedback from respondents contacted was uniformly positive and no instances of fraud were discovered. Moreover the interviewers achieved a fairly reasonable response rate (just under 60%) among those addresses which they successfully contacted. Overall, women appeared to be the more successful interviewers while the young male interviewers performed least well. Supervisors were encouraged to assist interviewers in overcoming their difficulties in recruiting respondents to the survey and it must be noted that great improvements in interviewing success were apparent following their input. Many of the interviewers, and particularly the four supervisors, claimed to have enjoyed the experience and expressed an interest in participating in the post-intervention survey. Although we do not currently have the resources to follow up interviewers, in order to ascertain whether the experience helped them to gain work, we have received several requests for references on their behalf since the survey closed.
Doubtless many of the practical problems which we encountered have been experienced, although seldom aired, by other researchers working in disadvantaged areas. However, we can be equally certain that some of these problems arose as a consequence of our decision to incorporate a participatory element into our wider (positivist) research design. The failure to involve interviewers in the conception and development of the evaluation meant that they lacked identification with our endeavour, and this had practical implications for the survey interviewing. Had the research been truly participatory, the data collection would have benefited from the local knowledge, relationships and experiences which local interviewers could bring to the work. Instead the survey required the interviewers to develop a technical relationship with the work in hand. Although the survey was successfully executed and we estimate the employment of local residents to have been a valuable and worthwhile experience, we make no claims that the experience was particularly enabling or empowering for the interviewers themselves.
In outlining some of the difficulties which we encountered and providing some guidelines for those undertaking similar research, we have addressed some implications of using a postpositivist approach in evaluation research. Our experiences clearly indicate that the strategy of involving users, within a research framework which is epistemologically opposed to that of participatory research, should be adopted only in the full understanding of the implications for both the users themselves and the research outcomes. Researchers are ill advised to take this course of action merely because it appears ethically appealing or in an attempt to court funding bodies.
|
Acknowledgments |
|---|
The research upon which the paper draws is funded by the Department of Health. We gratefully acknowledge the respective contributions of the survey interviewers and team supervisors who participated in the baseline study. We are also indebted to the two anonymous referees who provided detailed and extremely useful feed back on an earlier version of the paper. The Research Unit in Health and Behavioural Change is jointly funded by the Scottish Executive Health Department and the Health Education Board for Scotland. All opinions expressed in the article are those of the authors and not necessarily those of the funders.
|
References |
|---|
|
TopAbstractIntroductionBackgroundParticipatory researchInvolving local interviewersThe baseline surveyThe outcomeDiscussion and recommendationsReferences |
|---|
Asthana, S. and Oostvogels, R. (1996) Community participation in HIV prevention problems and prospects for community based strategies among female sex workers in Madras. Social Science and Medicine, 45, 133–148.
Altman, D. G. (1995) Strategies for community health interventions: promises, paradoxes, pitfalls. Psychosomatic Medicine, 57, 226–233.
Barnes, M. and Wistow, G. (1994) Achieving a strategy for user involvement in community care. Health and Social Care, 2, 347–356.
Beresford, P. and Croft, S. (1993) Citizen Involvement: A Practical Guide for Change. Macmillan, Basingstoke.
Campbell, C., Wood, R. and Kelly, M. (1999) Social Capital and Health. Health Education Authority, London.
Denzin, N. K. and Lincoln, Y. S. (1998) Introduction: entering the field of qualitative research. In Denzin, N. K. and Lincoln, Y. S. (eds), The Landscape of Qualitative Research; Theories and Issues. Sage, London, pp. 1–34.
Forbes, J. and Sashidharan, S. P. (1997) User involvement in services—incorporation or challenge? British Journal of Social Work, 27, 481–498.
Gillies, P. (1998) The effectiveness of alliances and partnerships for health promotion. Health Promotion International, 13, 1–21.
Hills, D. (1998) Engaging new social movements. Human Relations, 51, 1457–1476.[Abstract]
HMSO (1993) Realising Our Potential. HMSO, London.
de Koning, K. and Martin, M. (1996) Participatory research in health: setting the context. In de Koning, K. and Martin, M. (eds) Participatory Research in Health. Zed Books, Johannesburg, pp. 1–14.
Lefebvre, R. C., Lasater, T. M. Carleton, R. A. and Petereson, G. (1987) Theory and delivery of health programming in the community: the Pawtucket Heart Health Program. Preventive Medicine, 16, 80–95.[Web of Science][Medline]
Lomas, J. (1998) Social capital and health: implications for public health and epistemology. Social Science and Medicine, 47, 1181–1188.
Madan, T. (1987) Community involvement in health policy: socio-structural and dynamic aspects of health beliefs. Social Science and Medicine, 25, 615–620.
Maguire, P. (1996) Proposing a more feminist participatory research: knowing and being embraced openly. In de Koning, K. and Martin, M. (eds), Participatory Research in Health. Zed Books, Johannesburg, pp. 27–39.
Meulenberg-Buskens, I. (1996) Critical awareness in participatory research: an approach towards teaching and learning. In de Koning, K. and Martin, M. (eds), Participatory Research in Health. Zed Books, Johannesburg, pp. 40–49.
Pawson, R. and Tilley, N. (1997) Realistic Evaluation. Sage, London.
Payne, M. (1995) Focused participation in social work and social action. Social Welfare, 4, 197–203.
Petersen, A. and Lupton, D. (1996) The New Public Health: Health and Self in the Age of Risk. Sage, London.
Putman, R. (1993a) Making Democracy Work: Civic Traditions in Modern Italy. Princeton University Press, Princeton, NJ.
Putman, R. (1993b) The prosperous community: social capital and public life. American Prospect, 13, 35–42.
Putman, R. (1995) Bowling alone: America's declining social capital. Journal of Democracy, 6, 65–79.[Web of Science]
Rappert, B. (1997) Users and social science research: policy, problems and possibilities. Sociological Research On-line, http://www.socresonline.org.uk/socresonline/2/3/10.html.
Raynes, N. V. (1998) Involving residents in quality specification. Ageing and Society, 18, 65–78.
Tandon, R. (1996) The historical roots and contemporary tendencies in participatory research: implications for health care. In de Koning, K. and Martin, M. (eds), Participatory Research in Health. Zed Books, Johannesburg, pp. 19–26.
Woolgar, S. (1997) Accountability and Identity in the Age of UABs. CRICT Discussion Paper. University of West London, London.
Received on February 2, 2000;
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. Titterton and H. Smart Can participatory research be a route to empowerment? A case study of a disadvantaged Scottish community Community Dev. J., January 1, 2008; 43(1): 52 - 64. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||