Health Education Research Advance Access originally published online on July 16, 2007
Health Education Research 2008 23(2):272-286; doi:10.1093/her/cym031
Process evaluation of a multi-institutional community-based program for diabetes prevention among First Nations
1 Department of International Health, Bloomberg School of Public Health, Johns Hopkins University, 615 North Wolfe Street, Baltimore, MD 21205, USA
2 Centre For Studies in Family Medicine, University of Western Ontario, 245-100 Collip Circle, London, Ontario, Canada N6G 4X8
3 Cancer Etiology Program, Cancer Research Center of Hawaii, University of Hawaii, 1236 Lauhala Street, Honolulu, HI 96813, USA
* Correspondence to: A. M. Rosecrans. E-mail: arosecra{at}jhsph.edu
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
Epidemic rates of diabetes among Native North Americans demand novel solutions. Zhiiwaapenewin Akino'maagewin: Teaching to Prevent Diabetes was a community-based diabetes prevention program based in schools, food stores and health offices in seven First Nations in northwestern Ontario, Canada. Program interventions in these three institutions included implementation of Grades 3 and 4 healthy lifestyles curricula; stocking and labeling of healthier foods and healthy recipes cooking demonstrations and taste tests; and mass media efforts and community events held by health agencies. Qualitative and quantitative process data collected through surveys, logs and interviews assessed fidelity, dose, reach and context of the intervention to evaluate implementation and explain impact findings. School curricula implementation had moderate fidelity with 63% delivered as planned. Store activities had moderate fidelity: availability of all promoted foods was 70%, and appropriate shelf labels were posted 60% of the time. Cooking demonstrations were performed with 71% fidelity and high dose. A total of 156 posters were placed in community locations; radio, cable TV and newsletters were utilized. Interviews revealed that the program was culturally acceptable and relevant, and suggestions for improvement were made. These findings will be used to plan an expanded trial in several Native North American communities.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
Background
American Indians and First Nations populations in the United States and Canada (henceforth collectively referred to as Native North Americans) have been disproportionately affected by the epidemic of Type 2 diabetes mellitus, with some of the highest rates in the world [1–8]. The Pima Indians of Arizona were reported to have an adult prevalence of 50% [8], the highest ever documented, and Sandy Lake First Nation in Canada had an age-standardized prevalence of 26.1% [9]. The epidemic in North American populations continues to increase and shows no signs of slowing [4, 6, 10].
Diabetes is a complex disease caused by a combination of genetic and environmental risk factors [4], including fatty food preparation resulting in high caloric and fat intake and increasing rates of obesity coupled with low levels of physical activity [11, 12]. Native diets in northwestern Ontario have been shown to be high in protein and fat and low in dietary fiber [11], increasing diabetes risk [13, 14]. Modification of risk factors such as sedentariness, obesity and diet quality could drastically reduce the risk of developing diabetes [11, 13, 15–20].
Several diabetes and obesity prevention programs for Native North American communities have aimed to improve overall health through diet and/or physical activity [21–26]. The relative success of these programs can be linked to their emphasis on formative research, community participation, integration of local knowledge and learning styles, monitoring and evaluation plans and flexibility in program design [21–23, 25, 27]. However, most of these programs have focused on only one institution, usually the school, which does not allow for exposure through multiple channels [28] and targets only a select sample of the community, namely, schoolchildren. Effective community-based health promotion programs need to mobilize communities, integrate program activities into multiple community settings and employ individual, environmental and social changes [29]. Implementation and evaluation of the Zhiiwaapenewin Akino'maagewin: Teaching to Prevent Diabetes (ZATPD) program attempted to fill this gap.
Project description and study design
The ZATPD program was adapted from two community-based programs, the Sandy Lake School Diabetes Prevention Program [24] and the Apache Healthy Stores program [30], with activities added in partnership with local health and social services (HSS) agencies. Extensive formative work was conducted in the ZATPD communities [31], which included dietary assessment to select target foods (S. Sharma, in preparation).
ZATPD used an integrated approach and aimed to reach the majority of community members by working in multiple institutions simultaneously. Activities were divided into five phases, each with targeted foods and behaviors, and integrated across institutions. Each phase was lasted 6–10 weeks, averaging 8 weeks per phase. The schools implemented yearlong curricula for Grades 3 and 4, which used story-telling and participatory activities to teach and reinforce healthier eating habits and promote physical activity, as well as educate students about diabetes risk. For the store component, food store owners were asked to stock healthier food choices, utilize shelf labels identifying promoted foods and display educational posters. Additionally, cooking demonstrations/taste tests (CD/TTs) were held which demonstrated healthier cooking methods and distributed samples of healthier foods. Finally, the HSS component included community events and workshops intended for integration with ongoing HSS activities. The program utilized local communication methods such as radio, cable television, newsletters and bulletin boards for educational materials.
Seven First Nations communities in four sites in northwestern Ontario, Canada, participated in the feasibility trial. Researchers gave presentations to several community leaders, and interested communities volunteered to participate. Three of the seven were part of the formative research. Baseline household interviews were conducted in all communities, and Round 1 of the intervention began in September 2005 in four communities (in two study sites referred to as Site 1 and Site 2). Process evaluation took place in these communities. Subsequently follow-up interviews were conducted, and then Round 2 communities received the intervention beginning in September 2006. Results from the pre–post evaluation are reported elsewhere (L. Ho, in preparation) [32].
Site 1 is a remote fly-in community with winter road access 2 months of the year and a population of 
1100 on reserve. The reserve has two medium-sized stores, two convenience stores, a school, a nursing station, local radio and local cable television. Almost all food is flown in, and supplies of fresh foods are limited. Site 2 encompasses three semi-remote communities with year-round road access to a nearby town, and on-reserve populations range between 120 and 300 people. Communities are 30–60 min from town, and have access to convenience stores, medium-sized stores and a large supermarket. Children go to public schools off reserve, and community members have access to health centers on reserve and in town. In baseline surveys of Round 1 and Round 2 communities, 27% of adults interviewed (n = 133) reported that they were diabetic, and 80% were obese or overweight (body mass index 
25; L. Ho, in preparation) [32]. The actual prevalence of diabetes is likely to be higher due to undiagnosed cases. Respondents reported that supermarkets or grocery stores were the most common place they got food, while on the remote reserves 54% supplemented with hunting and fishing and 28% ordered or bought food off reserve. Only 5% of respondents from the semi-remote communities reported hunting or fishing as a main source of food. Also, 23% of households had at least one child in Grades 3 or 4 and 73% had at least one person under the age of 18.
Each community was intended to have one program assistant and one process evaluator, but this was not possible due to limited applicants and two resignations. Site 1 had one person for both positions, while Site 2 had two program assistants and one process evaluator for three communities. All program staff received pre- and mid-program training on nutrition, diabetes and program activities. Investigators and program staff presented the program and curriculum to teachers and interested school staff at the beginning of the program. The field supervisor and project coordinator provided on-the-job support and site visits.
Standards for intervention delivery were set for each component. Teachers were asked to deliver a lesson approximately every 2 weeks, for a total of 17 lessons in Grade 3 and 16 lessons in Grade 4, with four family packs sent home over the year aimed at increasing parental involvement and knowledge. Store owners were asked to stock promoted foods during each phase and to continue stocking these foods throughout the intervention. Program assistants were required to conduct at least three CD/TTs per phase per community and three per phase at the large supermarket; to place new posters and flyers for each phase at all feasible community locations including band offices, schools, stores, HSS offices, community halls, nursing stations and in school and community newsletters and to organize and conduct at least one community event per phase. Additionally, ZATPD assistants were encouraged to work with HSS staff to plan workshops and other events.
Process evaluation
Researchers have demonstrated the need for comprehensive process evaluation in health promotion interventions to improve program quality [33–36]. Process evaluation can help researchers avoid a Type III error [37], the measurement of something that does not exist, by giving a clear picture of how well program activities are implemented and the context in which implementation occurs [38]. With proper documentation, negative outcomes due to ineffective programs can be distinguished from negative outcomes due to insufficient program delivery. Additionally, variability in delivery among intervention sites can elucidate and explain relative successes and failures [39].
Process evaluation can be both formative and summative [40]. Formative use includes ‘fine-tuning’ an intervention during implementation [40]. Data used for summative purposes evaluate the extent to which an intervention was implemented as planned and reached the intended audience. Both uses are helpful in the design, evaluation and streamlining of community interventions. Process evaluation of ZATPD served both roles.
The ZATPD process evaluation focused on five constructs: ‘fidelity', the quality of program delivery and extent to which it is delivered as planned; ‘dose delivered', the number of units delivered by interventionists; ‘dose received', the extent to which the target audience actively engages in and receives intervention activities; ‘reach', the amount of target audience that participates in the intervention and ‘context', the larger sociopolitical and environmental factors that may influence the intervention [41].
There has been little process evaluation of multi-institutional interventions, and because ZATPD was a feasibility trial, process data were particularly relevant. This paper explores the following main research questions:
- (i) Was the ZATPD program implemented with high fidelity, reach and dose?
- (ii) Is ZATPD a feasible, acceptable and sustainable model for this context?
- (iii) How can ZATPD be improved for expansion to additional Native North American communities?
- (ii) Is ZATPD a feasible, acceptable and sustainable model for this context?
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
Process evaluation instruments
Qualitative and quantitative process evaluation instruments were developed for the school, store and HSS components (Table I). Development of evaluation instruments and methods were based on previous work and lessons learned [30, 42, 43]. School lesson implementation was assessed through teacher interviews, and teachers kept logs for family pack returns. Stocking of foods and poster placement at stores was assessed by a checklist form which was completed several times per phase for each store. Participation in CD/TTs was assessed by counting participants and food samples given and assessing participant reaction in addition to a self-administered evaluation form given to participants. Mass media was evaluated by recording posters, flyers, radio and cable TV spots and newsletter publications each phase. Semi-structured interviews were done with teachers, school staff, students’ families, store owners, HSS employees and ZATPD employees assessing acceptability, feasibility and sustainability. Exposure instruments were used in other analyses to fully assess reach of all components.
|
Data collection, management and analysis
Process evaluators, research staff and investigators collected all process data. All forms were regularly sent by fax or hand delivered to the field supervisor, who checked and entered data into a Microsoft Access database. STATA version 8.2 [44] statistical software was used for quantitative analysis. Process indictors for which standards were set, such as fidelity (i.e. % of minimum foods stocked) or dose received (i.e. % of family pack cards completed and returned), were assigned to categories of implementation as follows: low (0–49%), moderate (50–74%) or high (75–100%). These were post hoc categories agreed upon by the authors to ease interpretation of results. For indictors where no specific criteria were set, such as reach and dose delivered at CD/TTs, no categorization was assigned. Most interviews were tape-recorded and transcribed, though in some cases notes were taken during interviews instead. Field guides were semi-structured with questions and topics for probing. Transcripts were divided into sections based on topic and read several times by the first author until themes emerged and representative quotes were selected.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
School
Fidelity, reach and dose
An average of nine out of 17 Grade 3 lessons and eight out of 16 Grade 4 lessons were taught completely or partially, with an average of 16 students in attendance at each lesson (Table II). Each lesson had multiple components, and overall moderate fidelity was achieved with 63% of the planned lesson elements delivered. Out of 139 family packs distributed (dose delivered), 40% were completed and returned (dose received). Return rate for Grade 4 was particularly low at only 19%. Teachers reported enjoying 82% of the lessons and judged that students enjoyed 75% of those lessons taught.
|
Feasibility, acceptability and sustainability
Two principals, all seven teachers and 16 parents participated in a total of 34 semi-structured interviews. Teachers expressed overall satisfaction with the acceptability and operability of lessons, but cited difficulty with time constraints. As one teacher explained, ‘Our curriculum, especially in grade 3, is jam-packed with math and language because we are shooting for that EQAO test [a standardized test for Ontario] in May. It is somewhat difficult to get anything other than math and language in'. The program was perceived as relevant and interesting for the students. One teacher stated:
It's brought some information forward that the children hadn't been aware of before. And if nothing else, it's making them more aware that they need to be more physically active. Now they're talking about grandparents and other family members who are diagnosed already with diabetes, and some who have complications from it. They're putting it together that ‘whoa, I can do something about this’.Some teachers observed a trend for healthier lunches and more physical activity among students, while others noted persistent unhealthy habits despite classroom discussions.
Many parents reported not seeing the curriculum material, but those who had found it relevant and culturally acceptable. They enjoyed reading the stories and cooking recipes with their kids. Some parents noticed changes in their children's eating habits and activity levels. One mother said that her daughter who used to like McDonald's now says, ‘Mom, McDonald's, its junk food, I know it is not good for me'. Parents on the remote reserve expressed difficulty obtaining the requested foods due to limited availability and high prices.
Suggestions for improvement
Suggestions for improvement from teachers included shortening lessons, providing more hands-on activities and exposure to promoted foods, more discussion of the general Canadian population and including audio pronunciation guides for Ojibway words. Teachers and principals welcomed the idea of taste tests in the schools. Parents requested more information and contact from teachers about the program, perhaps involving phone calls. Teachers also requested that phasing in of the curriculum take place in a way that Grade 4 students do not miss the Grade 3 curriculum. Recognizing the importance of starting young, principals and teachers suggested expanding the curriculum to start earlier and last throughout the school years.
Store
Fidelity, reach and dose
Three small/convenience stores, four medium-sized stores and one large supermarket participated in the program. A total of 93 store evaluation visits were made over five phases (plus eight for post-intervention evaluation), an average of two visits per store per phase. Stocking of foods was evaluated for all promoted foods and minimum standard foods, a less stringent set of foods considered essential for each phase. Availability of promoted foods was assessed before, during and after the phase in which the food was promoted since stores were encouraged to continue to stock healthy foods.
The program achieved moderate fidelity in posting of shelf labels and posters and stocking of foods. The overall availability of all promoted foods over five phases was 64% (Table III). Overall availability of minimum standard foods (assessed separately for foods in each phase) was 60% before, 70% during and 71% after each phase. For available foods, shelf labels were placed appropriately 60% of the time. Overall fidelity of the poster goal was 78%.
|
Though originally planned as part of the store component, CD/TTs were conducted most often at community locations such as band offices, community halls and HSS offices due to limited space and minimal traffic in smaller stores. The goal of three at the large supermarket and three per community per phase had a fidelity of 71% overall with a total of 53 demos (Table IV). There were 572 participants who actively participated in the demonstrations (average 11/demo), 620 food samples distributed (average 12/demo) and 412 flyers and recipes distributed (average 8/demo). On a scale of 1–5, the average participant reaction to the food (as judged by the process evaluators) was 4.08 (SD 0.58) and participant interest was 3.83 (SD 0.75) (Table V). From 429 self-administered surveys, participants gave the taste of the food an average of 4.58 out of five (SD 0.70), and rate that they would buy or make the food as 4.22 (SD 0.90). The ratings and positive reactions suggest high participant engagement and interest (dose received).
|
|
Feasibility, acceptability and sustainability
A total of 13 semi-structured interviews were conducted with 10 managers, owners or employees at all eight stores and with ZATPD staff. All store owners stated that the program was easy to have in their stores, and most thought the shelf labels were useful and educational for customers. Some noted increases in sales of promoted items, but exact records were not consulted. The manager of the supermarket commented positively on the program, but stated the difficulty of conveying messages to the public:
I think the intent of it is good [...] But I think just the shelf tickets and the awareness campaigns, I think are just not seen [...] You know when every box of cereal has got some marketing campaign on it of ‘high in this’ and ‘healthy this’, you know for you to put up a shelf talker that says ‘high in fiber’—well, every cereal box already has that, if it honestly is high in fiber. And you know what, the message has become so diluted that if it has any fiber in it at all they'll call it a healthy alternative [...] They're just bombarded with so much health information and marketing that they don't know who to trust anymore.Other comments echoed this sentiment, and owners of smaller stores stated that they would only stock what people buy, which meant potentially discontinuing some healthy items that did not sell. Overall, store owners appreciated the list of healthy foods and were willing to stock new foods, especially if they were requested by customers, but the remote reserve had difficulty obtaining items from suppliers and keeping them in stock. At the end of the program, most owners agreed to continue stocking healthier foods, and some stated they would post shelf labels and posters if given the materials.
The one-on-one communication with people through CD/TTs and community events like store tours was appreciated and perceived as more effective than simple shelf labels and posters. The supermarket manager stated:
I would think the most effective thing you're doing is when you bring them in here on a tour and personally speak to them all, and say ‘hey, Sunny Delight is not orange juice.’ You know, ‘this has fiber, and you need this, and this is why’. The personal touch, you know, they'll listen to that. That's the only way I think it's going to be truly effective. Because right now I think your sign campaign is wonderful and I'm glad we do it and I want to keep it going, but are you getting bang for the buck? I doubt it.Community health representatives (CHRs) felt that the CD/TTs were a successful way to educate the community, as one CHR explains:
They're very visual people, and this gives them something to see. You know the actual seeing the food being cooked, or having it right there so they can taste it and then see it. Whereas a lot of the pamphlets we have, it's a lot of reading [...] But I find that the food sampling gives them the opportunity to feel comfortable, a little bit more relaxed, and they seem to think of questions to ask as they're eating.
Suggestions for improvement
Store owners had very few suggestions for improvement aside from laminating shelf labels and finding sturdy ways to affix them. ZATPD staff suggested working to lower prices of healthier foods or having sales on ZATPD promoted foods. One employee emphasized this and pointed to the necessity of higher structural interventions:
You know I've heard this so many times, all the good stuff is jacked right up. And all the junk that they buy, it's no choice you know because it's cheaper, it's cost efficient for them. All the good stuff should be cheaper [...] Not only that, but I think this should come from the factories themselves you know. They should involve the government, the factory makers, you know decide if you want everybody to be diabetic in 100 years [...] That's my main concern, you guys need to step up a few more steps to push for this prevention that we're doing.Parents also raised the concern of high prices, so the program should work with store owners and suppliers on this issue.
Health and social services/community events
Fidelity, reach and dose
Kickoff feasts were held in each community at the beginning of Phase 1, three of which included a community walk. Kickoff events had good attendance, with a total of 122 people, 58 in Site 1 and an average of 21 in Site 2 communities. During Phase 1, a pedometer walking challenge was held. Community members were enrolled in a 4-week challenge and given pedometers to monitor their steps for prizes. A total of 87 people enrolled; 47 in Site 1 and an average of 13 people in Site 2 communities. Four people out of 87 finished the 4-week challenge. Comments reported increased motivation, but many participants reported lost, forgotten or unreliable pedometers. Additional activities during this phase included two workshops held with HSS programs and attendance of program staff at a health screening.
Community events planned for subsequent phases had lower participation. One community held walking clubs in the community hall up to twice a week with a total of 31 participants. During Phases 2 and 3, announcements were made for Healthy Cooking Contests and Turn Off the TV Day, but no one signed up to participate. Rather than have the scheduled Family Fun Night in Phase 3, program staff had information and food sampling tables at local health fairs in two communities in Site 2 (participant data were counted with CD/TTs).
A total of 156 posters (average 39/community) were placed at band offices, nursing stations, schools, HSS offices, community halls and stores throughout the program (Table VI). A total of 565 flyers, cartoons and recipes were distributed at CD/TTs or posted on walls. Flyers, cartoons and program updates were printed in community and school newsletters 30 times. In Site 1, lesson supplements from the school curriculum were aired for two consecutive days on the cable station, eight flyers or announcements were run for 1 month each and five announcements or stories were read on the local radio. In Site 2, 12 informational radio spots were purchased each for Phases 3 and 4 from a radio station in town that claimed broadcast coverage to 35 000 people.
|
Feasibility, acceptability and sustainability
Eight interviews were conducted with five CHRs and HSS employees in all four communities, as well as ZATPD program staff, assessing the program as a whole. Interviewees perceived the program as well-received, culturally acceptable and relevant in their communities. CHRs wanted to integrate the program more completely into their current diabetes activities, which often focused on diabetes care rather than prevention. One CHR commented on this:
I think what we're trying to do is incorporate everything so that it just smoothly rolls, and I think this would be a really important component. I really like it and in fact, our healthier lifestyle challenge is a spin-off of this [...] I love that I don't have to put something together; it's right there for me.All communities expressed a willingness to continue program activities if supplied with materials, but there was no plan made for such continuation.
ZATPD staff enjoyed the opportunity to help their communities, but felt that they were not sufficiently prepared to provide health advice. The ZATPD training in diabetes and nutrition was appreciated, but was ultimately inadequate. There was dissatisfaction with the short time frame for implementation since community members were gaining awareness of the program only toward the end of the feasibility trial.
Suggestions for improvement
The most repeated suggestion from all sources was to do more: more CD/TTs, more recipes, more community events, more family oriented activities, more workshops and more communication with the community about program activities. Despite advertising, community members were unaware of program goals or activities. Specifically to get more involvement in events, more incentives would be needed. There was a call for greater coordination between health workers and program staff and more training on nutrition and diabetes. Members of the reserves also noted that the Ojibway program name was hard to understand.
Suggestions to improve the walking challenge included shortening the challenge to less than a month, using time as a counter instead of distance, and involving nurses who could screen for blood pressure and blood sugar during the challenge. As with the stores, structural problems were identified, namely, that dogs and dusty roads kept people from walking. One CHR stated the need for community action:
Walking around the gym I guess for us would be a band-aid solution. I want to do two things: get people out walking and get rid of the dogs. So the only way to get rid of the dogs is to get something going like a walking group, and this way the chief and council can do action.Despite these perceived barriers, people seemed aware that action could be taken to improve opportunities for healthy lifestyles.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
The ZATPD program achieved overall moderate levels of fidelity, dose and reach. The school curriculum was implemented with moderate fidelity and dose. In stores, maintenance of shelf labels and posters and stocking of healthier foods achieved moderate fidelity. CD/TTs and community events were both implemented with moderate fidelity, while CD/TTs had high dose received and community events had low participation. The program was considered culturally acceptable and relevant by all groups, and sustainability would be possible with modifications and planning.
Strong implementation of all components was difficult for many reasons and may have reduced program impact. At the school level, teachers expressed difficulty fitting lessons into schedules packed with other requirements and testing, which has been a problem in other school-based programs [26, 45]. There was a range of completion levels and satisfaction among teachers, pointing to the impact of individual priorities and teaching abilities. Because the teachers were not employees of the ZATPD program, there was limited opportunity for quality control. A suggestion was made to have health experts visit schools to deliver the curriculum instead of teachers, which could further integrate the school program with other ZATPD components.
Implementation of the store program takes commitment on the part of store owners. The food stores differed in eagerness and willingness to participate. In Site 1 where access to healthier foods is more limited, store owners readily agreed to participate and order the requested foods, but ordering systems were irregular and many foods were either not stocked at all or only for a short time. Also, store owners may have limited understanding of nutrition and therefore limited ability to make healthy purchases. At convenience stores, owners stock only what people will buy, and according to store owners, this does not include some healthy options. There are no clear trends and not enough data to conclude that the ZATPD program had a long-term effect on stocking of healthier foods. One store was under construction for several months, limiting program implementation. Similar fidelity was seen with Apache Healthy Stores [30], though the latter phases of ZATPD showed problems in implementation.
Restructuring of the CD/TTs for community locations was necessary, but ultimately limited the ability to impact buying patterns at the point of purchase. The program assistant in Site 1 expressed discomfort in conducting demonstrations alone, and required foods were not always available for demonstration. Insufficient advertising for most CD/TTs meant that community members were unaware of demonstrations, limiting attendance.
Mass media components varied considerably through phases. Only Site 1 has access to local radio and cable TV, which was minimally utilized. Posters achieved high reach and fidelity in posting, but printing in community and school newsletters was sporadic. Educational displays were only used during CD/TTs because there were no available spots for permanent display.
Another challenge was raising enthusiasm for participation in community events. Many people signed up for the pedometer challenge, but high attrition rates suggest limited impact on people's physical activity. Attendance at workshops was low and no one signed up for other advertised events. When activities were integrated into existing events such as health fairs attendance was much higher. Another community-based diabetes prevention program cited limited community ‘activation’ when working with a Native North American population and suggested that the limited time frame may have been insufficient to penetrate the community, which is a likely explanation for this program as well [46]. Addition of a ‘stages of change’ evaluation prior to program implementation could provide further information to guide program development.
Strengths and limitations
We developed a multi-component process evaluation plan to document and improve a multi-institutional community-based diabetes prevention program for First Nations. To our knowledge, this is the first detailed process evaluation of such an intervention program. There are both strengths and limitations to the data collected.
Collection of quantitative and qualitative data strengthens the evaluation plan. While careful documentation of program activities is important to evaluate reach, dose and fidelity, it is equally relevant to explore explanations of successes and failures, potential for sustainability and overall acceptability of the program through semi-structured interviews. Data were collected at the institutional, community and individual level to provide a comprehensive evaluation of the program. Additionally, the process data were used to iteratively improve program components that showed weak implementation.
There are several limitations to the data. First, the teacher interviews were retrospective and not based on direct observation. Second, availability of foods at stores of different sizes and locations varied considerably, and data from the large supermarket skews percentages of food availability. Third, the process data were recorded by several program assistants and process evaluators and were sometimes inconsistent between evaluators. Fourth, the level to which program activities were timed successfully across institutions was not assessed.
Implications for practice: lessons learned
Fidelity, dose and reach can be improved in future implementations of ZATPD if adjustments are made. Lessons learned are drawn from the data, experiences of the field coordinator and literature on other community-based programs [30, 47].
Programs should be modified based on community size, location and setup
Remote reserves are quite different than semi-remote reserves. For example, stores in the northern reserve had little access to fresh, affordable foods, while semi-remote reserves had access to a fully stocked supermarket. Remote reserves have on-site schools and nursing stations, while semi-remote reserves have access to local public schools and health centers. The issues on these reserves are fundamentally different, as are cultures and daily activities. Variations of the program should be created to address the differing needs. For example, increasing the availability of healthy foods should not be a program goal when working in large supermarkets, and lifestyle differences between rural and urban communities should be considered.
Programs require ongoing supervision
Working in remote and semi-remote communities made supervision of program implementation difficult. Program assistants were given freedom to plan activities appropriately for their individual communities, which often resulted in poor planning and insufficient delivery of activities. Being motivated and self-started is difficult under any condition, and a local supervisor who can monitor progress is crucial.
Comprehensive nutrition, diabetes and program training must be provided
Program assistants had to fill many roles: community motivator, event planner, nutrition and diabetes expert, chef, physical fitness coordinator, workshop moderator and public speaker. Limited skills and knowledge to perform these roles inhibited their ability to carry out the ZATPD program. Training of individuals should be extensive and focus on diabetes, nutrition and physical activity, while also providing skills training in public speaking, conducting workshops, internet and computer use and event organizing. Care should be taken to build confidence, capacity and ownership in individuals, as well as fostering relationships among program assistants, HSS staff and other community and health resources.
Programs should be implemented by full-time employees or professional health workers
Hiring local staff proved difficult because most people wanted full-time employment. Several of those who were hired had other jobs, making it difficult for them to prioritize ZATPD activities. Should ZATPD choose to hire program assistants in the future, two paths may prove more productive: (i) provide full-time jobs based in health offices with local supervisors who are invested in the program or (ii) employ local diabetes or nutrition workers to implement the program as part of ongoing activities and initiatives.
Alternatives to hiring program assistants include (i) forming community coalitions of individuals and organizations with the skills and interest to sustain the program or (ii) organizing and training volunteer groups to carry out ZATPD activities. Both methods increase motivation through group support and feedback, avoid issues of limited funds for employee payment, promote community planning and ownership, increase community capacity to address health issues and leave sustainable models for action [48].
Further institutionalization of the program will promote effectiveness and sustainability
Institutionalization of ZATPD program activities was difficult to achieve in the feasibility trial because it was a pilot of the activities with a short duration. Ultimately, ZATPD was still an outside program implemented with external funding, and program activities operated peripherally from other HSS activities. Integration and sustainability are particularly relevant for ZATPD because diabetes prevention requires long-term lifestyle changes and, thus, continued program activities. Daniel and Green [33] emphasize the importance of institutions in changing the social norms of communities for prevention programs. Many authors have suggested that the key to sustainability of prevention programs is capacity building at the community, organizational and individual level [35, 49–53]. Capacity building involves increasing knowledge, skills and access to resources so that community members can serve as their own health promotion experts and problem solvers [49]. Training should be provided for the school staff, HSS staff, store owners and other community members in nutrition and diabetes prevention. Building capacity will ensure quality program implementation [50], as well as contribute to sustainable community development, enabling community members to take ownership over this program and diabetes prevention as a whole.
Conclusion
ZATPD was the first multi-institutional diabetes prevention program for Native North Americans. Detailed process evaluation using qualitative and quantitative methods enabled iterative improvements in program strategies, provided vital qualitative feedback from community members and elucidated necessary changes for expansion to future communities.
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
The American Diabetes Association (Clinical Research Award: Award no. 7-04-CR-15) and the Canadian Institutes of Health Research.
|
Conflict of interest statement |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
None declared.
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
The authors would like to thank the participating First Nations, schools and stores.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflict of interest statementAcknowledgementsReferences |
|---|
1. Acton KJ, Burrows NR, Moore K, et al. Trends in diabetes prevalence among American Indian and Alaska native children, adolescents, and young adults. Am J Public Health (2002) 92:1485–90.
2. First Nations and Inuit Regional Health Survey National Steering Committee. Available at http://www.naho.ca/firstnations/english/pdf/key_docs_1.pdf.
3. Fox C, Harris SB, Whalen-Brough E. Diabetes among Native Canadians in northwestern Ontario: 10 years later. Chronic Dis Can (1994) 15:92–6.
4. Young TK, Reading J, Elias B, et al. Type 2 diabetes mellitus in Canada's first nations: status of an epidemic in progress. Can Med Assoc J (2000) 163:561–6.
5. Knowler WC, Bennett PH, Hamman RF, et al. Diabetes incidence and prevalence in Pima Indians: a 19-fold greater incidence than in Rochester, Minnesota. Am J Epidemiol (1978) 108:497–505.
6. CDC. Diabetes prevalence among American Indians and Alaska Natives and the overall population—United States, 1994–2002. Morb Mortal Wkly Rep (2003) 52:702–4.
7. Naylor JL, Schraer CD, Mayer AM, et al. Diabetes among Alaska Natives: a review. Int J Circumpolar Health (2003) 62:363–87.[Medline]
8. Bennett PH, Burch TA, Miller M. Diabetes mellitus in American (Pima) Indians. Lancet (1971) 2:125–8.[Web of Science][Medline]
9. Harris SB, Gittelsohn J, Hanley A, et al. The prevalence of NIDDM and associated risk factors in native Canadians. Diabetes Care (1997) 20:185–7.[Abstract]
10. Burrows NR, Geiss LS, Engelgau MM, et al. Prevalence of diabetes among Native Americans and Alaska Natives, 1990–1997: an increasing burden. Diabetes Care (2000) 23:1786–90.
11. Gittelsohn J, Wolever TM, Harris SB, et al. Specific patterns of food consumption and preparation are associated with diabetes and obesity in a Native Canadian community. J Nutr (1998) 128:541–7.
12. Lee ET, Welty TK, Cowan LD, et al. Incidence of diabetes in American Indians of three geographic areas: the strong heart study. Diabetes Care (2002) 25:49–54.
13. Wolever TM, Hamad S, Gittelsohn J, et al. Low dietary fiber and high protein intakes associated with newly diagnosed diabetes in a remote aboriginal community. Am J Clin Nutr (1997) 66:1470–4.
14. Hanley AJ, Harris SB, Gittelsohn J, et al. Overweight among children and adolescents in a Native Canadian community: prevalence and associated factors. Am J Clin Nutr (2000) 71:693–700.
15. Bazzano LA, Serdula M, Liu S. Prevention of type 2 diabetes by diet and lifestyle modification. J Am Coll Nutr (2005) 24:310–9.
16. Knowler WC, Barrett-Connor E, Fowler SE, et al. Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin. N Engl J Med (2002) 346:393–403.
17. Bazzano LA, Serdula MK, Liu S. Dietary intake of fruits and vegetables and risk of cardiovascular disease. Curr Atheroscler Rep (2003) 5:492–9.[Medline]
18. Manson JE, Nathan DM, Krolewski AS, et al. A prospective study of exercise and incidence of diabetes among US male physicians. J Am Med Assoc (1992) 268:63–7.
19. Manson JE, Rimm EB, Stampfer MJ, et al. Physical activity and incidence of non-insulin-dependent diabetes mellitus in women. Lancet (1991) 338:774–8.[CrossRef][Web of Science][Medline]
20. Venn BJ, Mann JI. Cereal grains, legumes and diabetes. Eur J Clin Nutr (2004) 58:1443–61.[CrossRef][Web of Science][Medline]
21. Gittelsohn J, Davis SM, Steckler A, et al. Pathways: lessons learned and future directions for school-based interventions among American Indians. Prev Med (2003) 37:S107–112.[CrossRef][Web of Science][Medline]
22. Macaulay AC, Harris SB, Lévesque L, et al. Primary prevention of type 2 diabetes: experiences of 2 aboriginal communities in Canada. Can J Diabetes (2003) 27:464–75.
23. Macaulay AC, Paradis G, Potvin L, et al. The Kahnawake Schools Diabetes Prevention Project: intervention, evaluation, and baseline results of a diabetes primary prevention program with a native community in Canada. Prev Med (1997) 26:779–90.[CrossRef][Web of Science][Medline]
24. Saksvig BI, Gittelsohn J, Harris SB, et al. A pilot school-based healthy eating and physical activity intervention improves diet, food knowledge, and self-efficacy for native Canadian children. J Nutr (2005) 135:2392–8.
25. Vastine A, Gittelsohn J, Ethelbah B, et al. Formative research and stakeholder participation in intervention development. Am J Health Behav (2005) 29:57–69.[Web of Science][Medline]
26. Paradis G, Levesque L, Macaulay AC, et al. Impact of a diabetes prevention program on body size, physical activity, and diet among Kanien'keha:ka (Mohawk) children 6 to 11 years old: 8-year results from the Kahnawake Schools Diabetes Prevention Project. Pediatrics (2005) 115:333–9.
27. Gittelsohn J, Harris SB, Burris KL, et al. Use of ethnographic methods for applied research on diabetes among the Ojibway-Cree in northern Ontario. Health Educ Q (1996) 23:365–82.[Web of Science][Medline]
28. Harris SB, Zinman B. Primary prevention of type 2 diabetes in high-risk populations. Diabetes Care (2000) 23:879–81.[Web of Science][Medline]
29. Merzel C, D'Afflitti J. Reconsidering community-based health promotion: promise, performance, and potential. Am J Public Health (2003) 93:557–74.
30. Curran S, Gittelsohn J, Anliker J, et al. Process evaluation of a store-based environmental obesity intervention on two American Indian Reservations. Health Educ Res. (2005) 20:719–29.
31. Ho LS, Gittelsohn J, Harris SB, et al. Development of an integrated diabetes prevention program with First Nations in Canada. Health Promot Int (2006) 21:88–97.
32. Ho LS. Diabetes prevention in northwestern Ontario first nations: a multi-institutional program to improve diet and increase physical activity (2007) Baltimore, MD: Department of International Health, Johns Hopkins Bloomberg School of Public Health.
33. Daniel M, Green L. Community-based prevention and chronic disease self-management programmes: problems, praises and pitfalls. Dis Manage Health Outcomes (1999) 6:185–92.[CrossRef]
34. Johnson CC, Osganian SK, Budman SB, et al. CATCH: family process evaluation in a multicenter trial. Health Educ Q (1994) ((Suppl 2)):S91–106.
35. Armstrong DL. A community diabetes education and gardening project to improve diabetes care in a Northwest American Indian tribe. Diabetes Educ (2000) 26:113–20.
36. Fortmann SP, Flora JA, Winkleby MA, et al. Community intervention trials: reflections on the Stanford Five-City Project Experience. Am J Epidemiol (1995) 142:576–86.
37. Dobson D, Cook T. Avoiding type III error in program evaluation: results from a field experiment. Eval Program Plann (1980) 3:269–76.[CrossRef]
38. Harachi TW, Abbott RD, Catalano RF, et al. Opening the black box: using process evaluation measures to assess implementation and theory building. Am J Community Psychol (1999) 27:711–31.[CrossRef][Web of Science][Medline]
39. Baranowski T, Stables G. Process evaluations of the 5-a-day projects. Health Educ Behav (2000) 27:157–66.
40. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract (2005) 6:134–47.
41. Linnan L, Steckler A. Process evaluation for public health interventions and research: an overview. In: Process Evaluation for Public Health Interventions and Research—Linnan L, Steckler A, eds. (2002) San Francisco, CA: Jossey-Bass. 1–23.
42. Steckler A, Ethelbah B, Martin CJ, et al. Lessons learned from the pathways process evaluation. In: Process Evaluation for Public Health Interventions and Research—Linnan L, Steckler A, eds. (2002) San Francisco, CA: Jossey-Bass. 268–85.
43. Steckler A, Ethelbah B, Martin CJ, et al. Pathways process evaluation results: a school-based prevention trial to promote healthful diet and physical activity in American Indian third, fourth, and fifth grade students. Prev Med (2003) 37:S80–90.[CrossRef][Web of Science][Medline]
44. StataCorp. Stata Statistical Software: Release 8 [computer program]. Version 8.2 (2003) College Station, TX: StataCorp, LP.
45. Gittelsohn J, Merkle S, Story M, et al. School climate and implementation of the pathways study. Prev Med (2003) 37:S97–106.[CrossRef][Web of Science][Medline]
46. Daniel M, Green LW, Marion SA, et al. Effectiveness of community-directed diabetes prevention and control in a rural Aboriginal population in British Columbia, Canada. Soc Sci Med (1999) 48:815–32.[CrossRef][Web of Science][Medline]
47. Bang AT, Bang RA, Reddy HM. Home-based neonatal care: summary and applications of the field trial in rural Gadchiroli, India (1993 to 2003). J Perinatol (2005) 25((Suppl. 1)):S108–22.[CrossRef][Medline]
48. Kegler MC, Steckler A, Malek SH, et al. A multiple case study of implementation in 10 local Project ASSIST coalitions in North Carolina. Health Educ Res. (1998) 13:225–38.
49. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. (1998) 13:87–108.
50. Chinman M, Hannah G, Wandersman A, et al. Developing a community science research agenda for building community capacity for effective preventive interventions. Am J Community Psychol (2005) 35:143–57.[CrossRef][Web of Science][Medline]
51. Johnson K, Hays C, Center H, et al. Building capacity and sustainable prevention innovations: a sustainability planning model. Eval Program Plann (2004) 27:135–49.[CrossRef][Web of Science]
52. Potvin L, Cargo M, McComber AM, et al. Implementing participatory intervention and research in communities: lessons from the Kahnawake Schools Diabetes Prevention Project in Canada. Soc Sci Med (2003) 56:1295–305.[CrossRef][Web of Science][Medline]
53. Taylor-Ide D, Taylor CE. Just and Lasting Change: When Communities Own Their Futures (2002) Baltimore, MD: Johns Hopkins University Press.
Received on November 20, 2006; accepted on May 10, 2007
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  L. S. Ho, J. Gittelsohn, R. Rimal, M. S. Treuth, S. Sharma, A. Rosecrans, and S. B. Harris An Integrated Multi-Institutional Diabetes Prevention Program Improves Knowledge and Healthy Food Acquisition in Northwestern Ontario First Nations Health Educ Behav, August 1, 2008; 35(4): 561 - 573. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||