Health Education Research, Vol. 14, No. 3, 371-386,
June 1999
© 1999 Oxford University Press
Measuring participatory strategies: instrument development for worksite populations
1 The Miriam Hospital, Division of Behavioral Medicine, 164 Summit Avenue, Providence, RI 02906,
2 Brown University, Providence, RI 02906,
3 The University of Rhode Island, North Kingston, RI 02852,
4 Fred Hutchinson Cancer Research Center, Seattle, WA 98109,
5 University of Texas School of Public Health, Houston, TX 77030,
6 Pennsylvania State University, University Park, PA 16802,
7 Dana Farber Cancer Institute, Boston, MA 02115 and
8 National Cancer Institute, Bethesda, MD 20814, USA
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
A participatory strategies approach which involves employees in the planning and delivery of worksite health promotion programs was utilized in the 55 experimental worksites included in the national, NCI-funded Working Well Trial. According to study protocol, Employee Advisory Boards (EABs) were organized in each experimental worksite. This paper describes two substudies designed to develop and measure participatory strategies associated with the EABs in the Working Well Trial. Study 1 determined characteristics of the EABs, developed subscales and assessed the internal consistency of the scales. Study 2 used a confirmatory factor analysis to examine the structure of the developed questionnaire. The four subscales include: Autonomy/Independence, Management Involvement, Institutionalization/Commitment and Others Involvement. Results from Study 1 indicate that the four subscales of the 24-item instrument demonstrated strong internal consistency and three were sensitive enough to register differences by Study Center at the baseline. Study 2 results found that the EAB subscales again demonstrated good internal consistency, structural stability and acceptable sensitivity. An initial validity analysis was performed and yielded results which supported some but not all of the hypothesized associations. Implications for further refinement and application of this new instrument in worksite settings are explored.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
Behavior change interventions applied to large defined populations frequently utilize a community organization approach in planning and delivery (McAlister et al., 1982
; Hunkeler et al., 1990; Fortmann et al., 1990; Thompson et al., 1991). This approach, recognized as a method for achieving change in the social environment (Green and Raeburn, 1990; Thompson and Kinne, 1990; Thompson and Pertschuk, 1992), is based on the principle of participation, i.e. large-scale behavioral change is much more likely to occur when the people affected by a problem are involved in defining the problem, planning and instituting steps to resolve the problem, and establishing structures to ensure that the desired change is maintained (Green, 1986; Kuriji et al., 1988). Pragmatically, this requires members of the target population to participate in program development, and to have a sense of responsibility for and control over programs promoting change (Kelly, 1979; Thompson et al., 1991; Thompson and Pertschuk, 1992). Participatory strategies have also been used in worksite health promotion programming and research (Sorensen et al., 1990, 1992; Glasgow et al., 1994; Glasgow et al., 1995). Employee and citizen involvement have been operationalized by establishing systems such as employee advisory boards (EABs), task forces, coalitions and community boards to assist with decision-making, prioritizing and the delivery of intervention activities (Florin and Wandersman, 1990; Sorensen et al., 1990; Thompson et al., 1991; Glasgow et al., 1995). Although the time and resources required to establish these structures are extensive (Thompson et al., 1993), few studies have measured the impact of participatory strategies on intervention outcomes (Lewis et al., 1996; Fawcett et al., 1997).
While participatory approaches are very appealing on a theoretical level and have been applied in many settings (McAlister et al., 1982; Fortmann et al., 1990; Hunkeler et al., 1990; Sorensen et al., 1990, 1992; Thompson et al., 1991; Glasgow et al., 1994, 1995), little research has been conducted to delineate specific components of participatory strategies which appear to be most salient. Anecdotal data suggest that advisory boards add important dimensions to research projects (Hunkeler et al., 1990; Thompson et al., 1993). However, exactly what such groups `do' remains a matter of speculation. Factors such as their involvement in planning, degree of ownership, and perceptions of support and commitment are likely to be important in determining the success of participatory strategies.
Another gap in the literature on advisory boards involves a study of the characteristics of advisory boards that may be related to outcomes. Descriptions of board members, the representativeness of the board and the enthusiasm of the board are variables which may have a significant impact on the implementation of health programs. Other variables such as the nature of the community or worksite under study may be connected with board commitment to the project or the ability to plan and implement activities. Measures and tools to assess characteristics of boards as well as the larger entities within which they operate are necessary to build a better understanding of their relationship to intervention outcomes.
In this paper, we discuss the development and evaluation of an instrument which measures some key characteristics of advisory boards within the context of the Working Well Trial, a 5-year cooperative agreement funded by the National Cancer Institute. The Working Well Trial tested the effectiveness of worksite health promotion interventions in achieving individual and organizational changes to reduce cancer risk. A study overview (Abram et al., 1994), baseline results (Heimendinger et al., 1995) and final results (Sorenson et al., 1996) are published elsewhere. Since little is known about factors that influence employee participation in implementing health promotion programs, we will describe two substudies which examine this issue. The first study determined the characteristics of the EAB at each experimental worksite, developed subscales and assessed the internal consistency of the scales. The second study used a confirmatory factor analysis to assess the structure of the developed questionnaire. The development of a tool to measure participatory strategies represents a first step in understanding the impact of employee involvement in worksite health promotion programs.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
Background on subjects and settings
The Working Well Trial was a randomized, prospective field experiment with 114 worksites and more than 28 000 workers, and included companies and workers with broad geographic and industrial diversity (Abrams et al., 1994
). Four project Study Centers, a coordinating center and the National Cancer Institute collaborated on common elements of design, data collection and analysis, and intervention standards. The four Study Centers and the number of worksites involved at each were University of Florida (UF) with 24 worksites, M. D. Anderson Cancer Center (MDACC) with 40 worksites, Dana Farber Cancer Institute (DFCI) with 24 worksites and Brown University/Miriam Hospital (BROWN) with 26 worksites. A key component of the Working Well intervention was the establishment of partnerships between each worksite and its corresponding Study Center. Partnerships ensured that employees in the worksites were able to participate in decisions about intervention planning and implementation. Key players in this partnership were the Intervention Specialist (a paid staff person from the Study Center), the Worksite Contact (an EAB member and main liaison to the Study Center) and EAB members (employee volunteers from the company). One Intervention Specialist was assigned to each worksite. The Worksite Contact was usually a member of Human Resources, Benefits or the Medical Department. The Intervention Specialist was responsible for delivering the intervention with assistance from the Worksite Contact and EAB. Each EAB was designed to be representative of the various constituents within the worksite (e.g. demographic, ethnic, union, shift, etc.). Formulation of the EABs was a task of the Intervention Specialist and the Worksite Contact, who worked together with sample job descriptions and a list of criteria (e.g. that the group be representative of the overall workforce, that management and line workers be included, smokers were represented, etc.). Additional members were recruited as needed or desired by the EABs. The EAB was charged with tailoring project activities to the needs of the worksite, providing opportunities for project activities to be integrated with other organizational activities, developing strategies to incorporate employee ideas into project activities, developing a process for worker involvement in developing and supporting healthier company policies, and helping to mobilize employee support/interest in project activities.
According to study protocol, EABs were organized in all experimental worksites by October, 1991. The Working Well Intervention Tracking System (WWITS) monitored the extent to which the intervention was delivered in all sites, and maintained descriptive information about the size, turnover and composition of the EABs (Abrams et al., 1994).
Since Working Well utilized EABs in all of the experimental worksites, we had a unique opportunity to survey EAB members participating in similar intervention activities across multiple worksites.
|
Study 1: Development of the EAB (EAB) questionnaire |
|---|
|
TopAbstractIntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
Instrument development and pilot test
Item generation
Four a priori constructs of interest were identified as being important to the measurement of conditions that would reflect participatory strategies and facilitate the success of health behavior change programs at worksites: Autonomy/Independence: degree to which EAB members function independently from Study Center Staff; Enthusiasm: general enthusiasm for the project from EAB members and perception of employee and management enthusiasm; Participation/Involvement: degree to which key individuals of the group (top managers, middle managers, EAB members and employees) are actively involved in the project; and, Institutionalization/Commitment: degree to which the site has shown support and commitment to keeping this project going beyond the funding period. These constructs were identified based on theoretical frameworks of community organization approach, other studies and the combined practical experience from investigators in the national trial (Rothman, 1970
; Carlaw et al., 1984; Farquhar et al., 1984; Lasater et al., 1984; COMMIT Study, 1988). In the spring of 1992, after several reviews, the items and constructs underwent a pilot test. At least one worksite from each of the Study Centers was included in the pilot test of the EAB questionnaire. In total, 26 worksite contacts and EAB members completed the pilot survey, and their input was invaluable in revising the survey. Mean response time to complete the questionnaire was 7.9 min (range = 3–20 min). Based on results of the pilot, a cover page with instructions and definitions of key terms was added to the final questionnaire, along with editorial changes, and `don't know' response categories for several items. Although the employees surveyed in the pilot were re-tested and included in the Study 1 analysis, the potential overlap could not have exceeded 4% of the total number of respondents thereby minimizing any test–re-test effects.
A total of 33 items were included in the final item set selected to measure the four originally hypothesized constructs of interest: Autonomy/Independence (11 items), Enthusiasm (seven items), Participation/Involvement (five items) and Institutionalization/Commitment (10 items).
Subjects
The population (N = 657) for the administration of the first EAB questionnaire was comprised of all current and former EAB members as well as the worksite contact from all 55 experimental worksites in the Working Well Trial (two experimental worksites withdrew from the study and were not included in this analysis). The purpose of the questionnaire was to assess the functioning of the EAB at two time points during the project, during the early stages of their operation and later. Because it requires some time for an EAB to become oriented to the study protocol and implementation procedures, we chose 12 and 24 months timeframes for questionnaire administration.
Procedure
The initial EAB questionnaire was administered in the fall of 1992, approximately 12 months after the intervention began. The 4-page survey included a cover page with instructions and definitions of terms used in the questionnaire. The survey was voluntary and while responses were kept confidential they were not anonymous. Individuals who did not respond to the initial request were given or mailed a second survey. No additional attempts were made to assess non-respondents. Intervention Coordinators were asked to distribute the EAB survey at a regularly scheduled EAB meeting and collect them at the same meeting. One Study Center (M. D. Anderson) elected to mail the survey to each potential respondent, but this did not appear to negatively impact response rates.
Results—Study 1
Response rate
Overall response rate to the Time 1 EAB questionnaire was 63% (N = 414) and ranged from 58 to 70% across Study Centers.
Initial dimensional analysis
The 33 items of primary interest from the EAB questionnaire were first examined for their individual item level characteristics, such as, mean, standard deviation, skew, kurtosis and endorsement level. One item, which asked about the enthusiasm of the union to the health promotion program, was removed from further analysis due to very low union representation among the studied companies.
Two key questions related to the internal validity of the EAB questionnaire were specifically addressed using principal components analysis (PCA), an exploratory statistical procedure often used with initial instrument development. The first question that PCA allowed us to examine was whether or not the instrument measured the four proposed constructs of interest as separate dimensions (components). The second question on which PCA provided information concerned the allocation of the individual items to the proposed underlying dimensions of the instrument. The allocation of items was based on the size of the component loadings for each item on the different components.
PCA with a varimax rotation (Kaiser, 1958) was thus conducted on the remaining 32 items. A pairwise correlation matrix with a maximum sample size of 408 was input to the PCA. The MAP procedure (Velicer, 1976), two parallel analysis approximations (Allen and Hubbard, 1984; Lautenschlager, 1989) and the Scree Test (Cattell, 1966) were employed to determine the most parsimonious solution to the PCA. A four component solution, employing 24 of the 32 items, was judged to be the best solution based on the convergence of these several methods. The eight items that were deleted from the final solution were items with low loadings (less than 0.40 for all components) or were complex items (items with loadings above 0.40 on two or more components). A second PCA with varimax rotation was conducted on the reduced 24-item pairwise correlation matrix. On the basis of the PCA we revised two of our initially proposed subscales: Enthusiasm and Participation/Involvement. Items were reallocated to dimensions measuring constructs that we called Management Involvement and Others Involvement. Table I presents the items on the revised subscales, the individual item response formats and the resulting varimax-rotated solution. (The original item set available by request from the first author.)
|
Internal consistency analysis
The resulting four subscales of the EAB questionnaire were examined separately for internal consistency using the
(Cronbach, 1951) statistic. Scale 1 (Autonomy/Independence) had an of 0.92; scale 2 (Management Involvement and Support) had an of 0.89; scale 3 (Others Involvement) had an of 0.72; and scale 4 (Institutionalization/Commitment) had an of 0.73. Based on the number of items in the scales these values were judged to be excellent for scales 1 and 2, and good for scales 3 and 4. The correlations among the scales was low to moderate and ranged from –0.24 to 0.18. The inter-scale correlations appear in Table II.
|
Study center comparisons
The sensitivity of the four identified subscales to describe differences among the Study Center level grouping of worksites that would relate to the acceptance of and support for health promotion programs at the four study sites was examined. Four separate ANOVAs were conducted using the four study sites as grouping variables and the four EAB subscales as dependent measures. All analyses were examined for statistical significance using a Bonferroni adjusted
rate of P < 0.0125. The PROC mixed procedure of SAS version 6.09 was used to examine Study Center differences and controlled for positive within-center correlations. Differences among study sites were found on three EAB subscales: Autonomy/Independence, Institutionalization and Management Involvement. The specific results of these analyses are presented in Table III. Note that smaller values indicate greater Autonomy/Independence at a worksite and greater Institutionalization Support, while a larger value indicates greater Management Involvement.
|
|
Study 2: Confirmation of the structure of the EAB questionnaire |
|---|
|
TopAbstractIntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
Method
Subjects
The EAB questionnaire was administered at Time 2, 12 months after the initial assessment, to a sample of 560 Worksite Contacts and EAB members who had been active between Year 1 and 2 of the intervention at the same 55 experimental worksites. Due to EAB members rotating off the board, being laid off or permanently separated, some attrition between baseline and final survey administration occurred.
Response rate
Three hundred twenty-two employees responded to the EAB questionnaire at Time 2 for an overall response rate of 56%, which ranged from 49 to 72% across Study Centers. Employee respondents were not matched with unique identifiers, therefore, it was not possible to determine the percentage of the initial respondents who also completed the Time 2 questionnaire.
Procedure
A stringent test of the EAB questionnaire structure was conducted using confirmatory factor analytic methods. Confirmatory factor analysis (CFA) allowed for further examination of the theoretical instrument structure (four subscales) by precise specification of the measured items to their proposed dimensions (factors). Also, the significance of subscale (factor) correlations was examined within the CFA. A final advantage of the CFA procedure allowed for the comparison of nested alternative models of instrument structure that might best explain the underlying measured data structure. The structure of the refined 24-item EAB questionnaire was examined using the LISREL 7 (Joreskog and Sorbom, 1989) statistical package to perform a CFA. First, the items within each of the four scales were combined to form item parcels (Hull et al., 1995; West et al., 1995). Composite item parcels are more reliable indicators of the measured constructs than single items and also facilitate convergence of the solution within the statistical package. Each of the item parcels were arbitrarily formed by summing two adjacent items within a given scale, except for scale 4 (Institutionalization/Commitment). Mathematical constraints related to attaining a convergent solution necessitated a minimum of three item parcels (or items) per factor. Thus, in scale 1 the eight items were combined to form four item parcels, the six items within each of scales 2 and 3 were combined to form two sets of three item parcels, and the four items of scale 4 were combined to form one item parcel and two single items. The resulting 11 item parcels and two single items were correlated to form a 13x13 pairwise matrix of inter-item and inter-item parcel correlations with a maximum sample size of 309 that was examined within the context of the CFA. Five competing models were examined that might represent alternative explanations of the data structure of the EAB.
Model 1 was a Null Model, which assumes that all the measured variables are unrelated. The Null Model represents a baseline or comparison model for the four other competing models and is used in the calculation of several fit indices. Typically the Null Model is not considered a plausible explanation of the data structure. Model 2 represented a Single Factor Model which posits that there is only one general factor or construct that will account for the correlations among the measured variables. Model 3 represented an Uncorrelated Four Factor Model, which posits that the four previously identified EAB scales are orthogonal or independent of one another. Model 4 represented a Correlated Four Factor Model, which posits that the four previously identified EAB scales represent separate but overlapping dimensions. Model 5 represented a Hierarchical Model, which posits a single higher- or second-order factor that can account for each of the four first-order factors (i.e. the four previously identified EAB scales) and would suggest that there is a single overall dimension (e.g. readiness to employ participatory strategies in support of health promotion) that can explain the correlations among the four separate EAB subscales.
Five common indices of model fit were calculated for Models 2, 3, 4 and 5 to help determine which model provided the best explanation for the data structure. These were: (1) the Likelihood Ratio 
2 statistic, (2) the Goodness of Fit Index (Joreskog and Sorbom, 1989), (3) the Root Mean Square Residual (Joreskog and Sorbom, 1989), (4) the Tucker–Lewis Index (Tucker and Lewis, 1973) and (5) the Delta2 Index (Bollen, 1989b). For Model 1 (Null Model) only three fit indices could be calculated since one of the indices, the Likelihood Ratio 2 statistic for the Null Model, is used to calculate both the Tucker–Lewis Index and the Delta2 Fit Index for the other models. As was mentioned above, the Null Model is primarily examined for comparison with other more plausible models. It is customary to compare several fit indices across models as there is no consensus (Bollen, 1989b; Bollen and Long, 1993) on a best fit index or even specific values for each index that guarantee `good' or `bad' fit. It has been generally accepted that values above 0.80 represent a good fit to the data and values above 0.90 represent an excellent fit to the data for the Goodness of Fit Index, the Tucker–Lewis Index and the Delta2 Index, while a low value (close to 0.05) (Byrne, 1989) represents an excellent fit for the Root Mean Square Residual when used with standardized variables. In addition, the models were compared and tested using the 2 difference test (Bollen, 1989b) in accordance with suggested methods for examining nested alternative models (James et al., 1982; Anderson and Gerbing, 1988; Bollen, 1989b; Hull et al., 1995).
Results—Study 2
Confirmatory dimensional analysis
Table IV presents the results of the CFA for each of the alternative models. An examination of all the fit indices indicated that both the Four Correlated Factors Model and the Hierarchical Model provided excellent fits to the data. The results of the 2 difference test indicated significant improvement in fit for the Four Correlated Factors Model over: (1) the One Factor Model, 
2 (Green and Raeburn, 1990) = 765.40, P < 0.001; (2) the Four Uncorrelated Factors Model, 2 (Green and Raeburn, 1990) = 65.12, P < 0.001; and (3) the Hierarchical Model, 2 (McAlister et al., 1982) = 15.57, P < 0.001. The Four Correlated Factors Model was judged slightly superior to the Hierarchical Model and examination of the parameter values in Fig. 1 reveals that while five of the six possible correlations among the subscales were significant at P < 0.01, the correlations are generally moderate and do not reflect the presence of a strong second-order factor as an explanation for the correlation among the scales.
|
|
Internal consistency analysis
The four subscales of the EAB questionnaire were again examined separately for inter-item consistency using the
(Cronbach, 1951) statistic. Scale 1 (Autonomy/Independence) had an of 0.89; scale 2 (Management Involvement and Support) had an of 0.90; scale 3 (Others Involvement) had an of 0.77; and scale 4 (Institutionalization/Commitment) had an of 0.65. Based on the number of items in the scales these were again judged to be excellent for scales 1 and 2, good for scale 3, and adequate for scale 4.
Study center comparisons
The sensitivity of the four identified subscales to describe differences among the Study Center level grouping of worksites that would relate to the acceptance of and support for health promotion programs at the four study sites was examined again. Four separate ANOVAs were conducted using the four study sites as grouping variables and the four EAB subscales as dependent measures. As before, analyses were examined for statistical significance using a Bonferroni adjusted rate of P < 0.0125. The PROC mixed procedure of SAS version 6.09 was again used to examine Study Center differences. Differences among study sites were found on three of the EAB subscales: Autonomy/Independence, Institutionalization and Management Involvement. The specific results of these analyses are presented in Table V. Note that small values indicate greater Autonomy/Independence at a worksite and greater Institutionalization Support, while a larger value indicates greater Management Involvement and Others Involvement. Because the samples were not exactly the same at each assessment of the EAB questionnaire, a repeated measures ANOVA was not possible. However, if one simply compares the means of the scales for each Study Center across the two time points, one can assume that all four Study Centers have become more autonomous and independent at providing worksite health promotion activities, while there has been relatively little change in Management Involvement or Institutionalization; and, while values are not presented in Table III
, the mean values for Others Involvement were also essentially unchanged across time points. Differences among Study Centers on each measure have changed little and the relative ordering of differences are similar at the second time point.
|
Initial validity analyses
Three types of validity are important in the development of new instruments: content validity, criterion validity and construct validity. Initial content validity was established through the expert review process and pre-testing described previously. Criterion-related validity requires that the scales of interest have an empirical association with some criterion or `gold standard', while construct validity ensures that the instrument or subscales measure what they were intended to measure (DeVellis, 1991
). Construct validity demands an ongoing assessment of subscale performance across different times, places and populations. In the case of the EAB questionnaire, our initial examination of the subscales included both criterion and construct validity analyses by using selected items from three additional sources of data collected during the same time period within the Working Well Trial. These validating items came from the final Personnel Director or CEO interviews, and from the process tracking data collected as part of the WWITS (Abrams, 1994). Because the four EAB subscales represent a new instrument and `gold standard' items for `participatory strategies' are not readily available or agreed upon, we choose a set of items that were thought to relate most directly to the latent constructs of interest. Correlational analyses between these validating items and the scales of interest are summarized in Table VI.
|
We initially posited which of the specific validating items should be most related to each specific EAB subscale and these relationships are bolded in Table VI
. However, because the subscales were moderately correlated with each other and the item content of each of the validating items often suggested possible relationships with several of the other subscales, we also explored the full set of other possible relationships between the validating items and the four subscales. These other correlational analyses are represented by non-bolded typeface in Table VI. All of the correlational analyses were conducted at the worksite unit of analysis and the overall sample sizes ranged from 24 to 55 worksites depending on the validating variable in the analysis. These relatively low sample sizes also translated to generally low statistical power, with the result that a bivariate correlation often had to account for enough variance to be considered a medium effect (Cohen, 1977) to reach statistical significance (P < 0.05), and in cases of the lowest sample size (n = 24), even a medium effect size was not strong enough to guarantee statistical significance. Consequently, in Table VI we have also noted those relationships that are significant at the P < 0.10 level to help control for possible Type II statistical errors.
The results of these initial validity analyses found that Autonomy/Independence was related to company paid hours expended (P < 0.01), CEO beliefs regarding the benefits of health promotion (P < 0.10) and average EAB participation (P < 0.10). Management Involvement was related to company paid hours expended (P < 0.01), proportion of company sponsored activities (P < 0.01) and average EAB participation (P < 0.05). Others Involvement was related to number of EAB activities (P < 0.05) and definite plans to offer health programs next year (P < 0.10). Institutionalization was related to definite plans to offer health programs next year (P < 0.01). Further details of these analyses, including actual correlation coefficients, are given in Table V.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
Community-based health promotion efforts have long embraced the notion of empowering local constituents to become engaged in and to participate in change efforts. Investigators have also recognized the need for methods and instruments to measure the impact of participatory approaches on communities. Some have designed methods to measure community competence (Cottrell, 1976
; Knight et al., 1991) and others have evaluated the impact of community health coalitions (Francisco et al., 1993; Lewis et al., 1996; Fawcett et al., 1997). The instrument described here extends this measurement research to the worksite setting and focuses on participation, one important variable which impacts the health behavior of community and worksite populations. The Working Well Trial adopted a participatory strategies approach—organizing EABs in all 55 worksites assigned to the experimental condition. The measurement instrument developed by investigators from the national trial assessed four key domains hypothesized to be integral to the participatory strategies approach: Autonomy/Independence, Management Involvement, Others Involvement and Institutionalization/Commitment. The four subscales of the 24 item instrument demonstrated strong internal consistency and were sensitive enough to register differences by Study Center on the baseline assessment. The subscales again demonstrated good internal consistency, structural stability and acceptable sensitivity when re-administered 12 months later.
Initial validity analyses conducted on these subscales gave mixed results. We identified a series of dependent measures a priori and investigated initial validity of these items against the four EAB subscales. We found evidence of concurrent validity on two of the four subscales as hypothesized (Others Involvement and Institutionalization) and found items which proved helpful in establishing validity for the other two subscales (Autonomy/Independence and Management Involvement) during post hoc analyses. The latter findings must be interpreted with caution, but do make intuitive sense given previous knowledge of the subject area. Given several item-specific limitations which may have precluded our finding additional significant relationships between the hypothesized variables (e.g. low power), we are satisfied that the first set of analyses with these subscales warrant further testing and refinement in subsequent research efforts.
The instrument also yielded results consistent with expected outcomes differentiated by Study Center. For example, the M. D. Anderson Study Center worksites reported more Autonomy/Independence and less Others Involvement. This difference could be partially explained by the geography and staffing of the M. D. Anderson Study Center. Specifically, the M. D. Anderson Study Center included 20 intervention worksites spread over several southern states. Because of the distances involved and the large number of worksites, Intervention Specialists from the Study Center provided intervention support primarily by telephone and mail. Out of necessity, the Worksite Contact and other EAB members were required to do most of the tasks referred to in the Autonomy/Independence and Others Involvement subscales. Such anecdotal evidence confirmed by subscale differences further strengthens our confidence in these results and the potential usefulness of the discriminant validity of the subscales.
However, we recognize that our initial validity analyses did not provide evidence which fully supports our original hypotheses about these scales. One plausible explanation for the low correlations of some of these validity analyses may relate to the specific items selected as our `gold standard' measures. Although we chose the best set of items available to us, this instrument was developed after the Working Well baseline instruments for employee surveys and key informants interviews (e.g. Personnel Director and CEO) were already in the field. Ideally, we would have triangulated similar items across various data collection instruments at the design phase but that was not possible in this case. A second possibility for low validity coefficients may be related to different methods of administration which may have lowered the reliability of our `gold standard' items. For example, employee surveys (paper and pencil), interviews (in-person or by phone) and records maintained by intervention staff were all sources of our validity items. Differences in these methods may have attenuated the correlation coefficients.
Patterns which emerged during the correlation analysis that were inconsistent with our hypothesized relationships may yield important clues for further refinement of the subscales. For example, perceptions of the Personal Director (r = 0.078 [NS]) about responsibility for health promotion tasks were lower than the were CEO beliefs about the benefits of offering these programs (r = 0.241, P < 0.10) or the amount of time paid by the company for employees to engage in these programs (r = –0.348, P < 0.001). Further refinements of the Autonomy/Independence subscale may need to take into account different types or levels of autonomy. Perceptions of autonomy are likely to differ based on different respondents. Subscale improvements based on a review of the patterns from the initial validity analyses are currently being weighed as well as consideration of appropriate validity criteria to include in future studies.
The worksite health promotion literature is quite limited with regard to the availability of measurement tools which address participatory approaches. However, since the practice of involving employees with health promotion activities is increasing in worksite settings, we need to know what characteristics make for an effective and successful group. What level of autonomy or direction do they need to be successful? Do they need more support or less? Should we encourage using outside resources or not? This instrument may be useful in helping to quantify these issues and may help build a better understanding of the factors involved.
Practitioners may find the EAB questionnaire to be a useful feedback and planning tool when working with EABs or other wellness committees. Periodic assessments of EAB members' perceptions, opinions and concerns can empower (and re-invigorate) individual board members, and may also help to identify training or development needs. A majority of EAB members in the present study gave written feedback about their appreciation for the project and their pride in being included on the EAB. Intervention staff who worked with the EABs found the survey results very helpful in planning and shaping the structure, function and future direction of the committees. Feedback regarding how to best provide leadership for EABs can be a powerful learning tool for administrators and those who chair or direct these committees. Glasgow and colleagues felt that their EABs needed greater direction from the Study Center research team earlier in the intervention period (Glasgow et al., 1995). Periodic administration of the EAB questionnaire may help EAB members and the leaders directing these groups clarify roles, perceptions and how to move forward successfully.
Several important limitations are acknowledged regarding these findings. First, no individual matching of initial to final survey respondents was possible. Future applications will match individual responses at each administration and strengthen our ability to track our findings over time. On the other hand, there is some indication that EAB membership changed over time; thus, it may not be feasible to get repeated measures on individual EAB members.
A second limitation is that the results may be influenced by the timing of the administration of the instrument. Specifically, the lack of change we saw in dimensions other than Autonomy/Independence may be due to the fact that surveys were administered at the 12 and 24 month time frame within a 2.5 year intervention cycle. Since we did not administer a survey immediately after formation of the EAB and then allowed 12 months to pass, we may have missed the opportunity to measure the early, important changes in the dimensions of management involvement and others involvement during the first 12 months of the intervention (Stryker et al., 1997). We recommend periodic administrations of the questionnaire in future research efforts, including an initial measurement after the EAB has met several times, and is oriented to their roles and responsibilities.
Third, the subscales are based on the impressions of EAB members about the level of autonomy, management involvement, institutionalization and others involvement at their particular worksite rather than specific observed or other behavioral measures of participation levels. However, this tool was meant to be used as an adjunct to more behaviorally based data already being collected as part of process tracking in the larger Working Well Trial. For example, actual employee participation records, EAB member attendance in health promotion activities, number of hours spent by company volunteers (independent of the EAB) were also being collected and have served to begin to initially validate the subscales. As the subscales are further refined and construct validity is built over time, we expect to determine the relative importance of behavioral and observational measures, and incorporate other validity criteria which will enhance future the concurrent and predictive validity analyses.
Since this was part of the largest federally funded, randomized-controlled worksite health promotion trial, there is strength in the quantity and diversity of the worksite participants. However, our findings may not generalize to all worksites in all parts of the country. Certainly worksites that agreed to participate in this research study are likely to be different from those who did not (or would not) agree. Additional testing with worksites from a wider variety of sizes and industry classifications will strengthen the external validity of the instrument.
Further refinements of the instrument will include revising item response categories (e.g. Yes/No response replaced with 1–5 Likert on the four-item Institutionalization scale); adding new items and (potentially) developing an overall index of `participatory strategies'. Understanding participatory strategies and how levels of autonomy commitment and involvement contribute to differential outcomes are needed. The next step facing our research team is to test whether (and to what extent) these subscales relate to key outcome variables for the Working Well Trial. The EAB questionnaire will allow investigators to measure differences across worksites and Study Centers. For now, there is clearly much to be learned about the impact of participatory strategies for worksite health promotion. The development of this tool brings us one step closer to understanding how EAB members regard their participation in worksite health promotion and how to make use of this information to enhance intervention effectiveness.
|
Note |
|---|
Instrument available upon request from corresponding author.
|
Working Well study investigators and committee members |
|---|
|
TopAbstractIntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
National Cancer Institute (NCI)
Division of Cancer Prevention and Control. Program Director: Jerianne Heimendinger. Statisticians: Charles Brown, Don Corle. Fellows: Nancy Cotugna (1990), Susan Sullivan (1990–1991).
Study Centers
Brown University School of Medicine/The Miriam Hospital
Principal Investigator: David B. Abrams. Co-Principal Investigator: Lois Biener. Co-Investigators: Karen M. Emmons, Laura Linnan. Statisticians: Joe Fava, Wayne Velicer. Project Director: Laura Linnan. Intervention Specialists: Edward Galuska, Mary Lynne Hixon, Sheila Jacobs, Suzanne Moriarity. Data Specialists: David Vampola, Kate Lapane (1989–1992). Research Associate: Janine Costa. Administrative Assistant: Mark Morgenstern.
Dana-Farber Cancer Institute/University of Massachusetts Medical School
Principal Investigator: Glorian Sorensen. Co-Principal Investigators: Jay Himmelstein, Judith Ockene. Co-Investigator: Katherine Hammond, James Hebert, Ruth Palumbo, Anne Stoddard. Project Director: Mary Kay Hunt. Evaluation Coordinator: Jean Hsieh. Intervention Coordinators: Lynda Grahm-Meho, Elizabeth Harden, Jane Ellen Thompson. Nutrition Coordinator: Judith Phillips. Industrial Hygienist: Richard Youngstrom. Process Tracking Coordinator: Steve Potter. Administrative Assistant: Elizabeth Farr.
University of Florida at Gainesville
Principal Investigator: Jill Varnes. Co-Principal Investigator: Claudia Probart. Co-Investigator, Medical Consultant: David Schapira. Statistician: David Miller. Intervention Coordinators: Jeanine Lahey, Steve Livesay, Valerie Studnick (1990–1992). Interventionist/Tracking Coordinator: Susan Guckenberger. Office Manager, Data Manager: Kathy Galloway.
University of Texas M. D. Anderson Cancer Center
Principal Investigator: Ellen R. Gritz, W. Bryant Boutwell (1991–1993), Michael Ericksen (1989–1991). Co-Principal Investigator: Carlo DiClemente. Co-Investigators: Robert Chamberlain, Guy Parcel, Karen Basen-Engquist, John Foryet, Larry Whitehead, Alan Herd, Jane Mayfield. Project Coordinator: Scott R. Cummings. Intervention Specialists: Scott R. Cummings, Ingrid Nielsen, James Eldridge, Bill Mann. Consultants: Karen Glanz, Robert Butera.
Coordinating Center
Fred Hutchinson Cancer Research Center
Principal Investigator: James Grizzle. Co-Investigators: Ziding Feng, Susan Kinne, Alan Kristal, Beti Thompson, Nicole Urban (1990–1992). Fellow: Ruth Patterson. Coordinating Center Manager: Sonya Olsen, Gwen Glaefke (1989–1992). Statistical Research Associates: Cynthia Lostoski, Steve Thomson, Dale McLerren. Database Manager: Allan Williams. Programmers: Stephanie Stafford, Peter Dueber, Lynette Brown (1990–1991). Cost Analysts: Shelly Hager, Addy Tseng, Carole Shaw (1990–1993). Support Staff: Catherine Cohen, Ellen Powell (1991–1992), Estella Bennett (1990–1992), Jenny Devaney (1990–1991).
Working groups and subcommittees
Steering committee
Ed Lichtenstein [Oregon Research Institute, Chair], David B. Abrams, Ellen R. Gritz, James Grizzle, Jerianne Heimendinger, Glorian Sorensen, Jill Varnes, W. Bryant Boutwell (1991–1993), Michael Ericksen (1989–1991), Ellen Gritz (1993–1994).
Data managers/tracking coordinators
Allan Williams [Chair], Peter Dueber, James Eldridge, Kathy Galloway, Susan Guckenberger, Jean Hsieh, Cynthia Lostoski, Steve Potter, Stephanie Stafford, David Vampola, Kate Lapane (1989–1992).
Evaluation Working Group
James Grizzle [Chair], Don Corle, James Eldridge, David Miller, Anne Stoddard, Wayne Velicer.
Nutrition Working Group
Karen Glanz [Chair], James Hebert, Jerianne Heimendinger, Mary Kay Hunt, Alan Kristal, Laura Linnan, Ruth Patterson, Claudia Probart. Additional Nutrition Intervention Subcommittee Participants: Mary Lynne Hixson, Judith Phillips, Patricia Pillow.
Organizational variables working group
Lois Biener [Chair], Susan Kinne [Co-Chair], Karen Basen-Engquist, Jerianne Heimendinger, Glorian Sorensen, Jill Varnes.
Process evaluation working group
Glorian Sorensen [Chair], Karen Basen-Engquist, Shelly Hager, Jean Hsieh, Mary Kay Hunt, Laura Linnan, Cynthia Lostoski, David Miller, Stephanie Stafford, Addy Tseng, Nicole Urban (1990–1992).
Participatory strategies working group
Claudia Probart [Chair], Karen Basen-Engquist, Karen Emmons, Jerianne Heimendinger, Mary Kay Hunt, Laura Linnan, Ingrid Nielsen, Beti Thompson.
Publications and presentations subcommittee
Jill Varnes [Co-Chair], Karen Emmons [Co-Chair 1991–1992, 1994], Ziding Feng, Ellen Gritz, Jerianne Heimendinger, Anne Stoddard, David B. Abrams [Chair 1989–1991], W. Bryant Boutwell [Co-Chair 1992–1993].
Smoking working group
Karen Emmons [Chair], Carlo DiClemente [Chair 1989–1991], Jerianne Heimendinger, Judith Ockene, Beti Thompson, David B. Abrams (1989–1992).
|
Acknowledgments |
|---|
This work was supported by a Cooperative Agreement from the National Cancer Institute, grants U01 CA51687, U01 CA51671, U01 CA51686, U01 CA51688 and P01 CA50087. The authors thank Drs Glorian Sorensen, David Abrams and Ziding Feng for their helpful review and comments on early drafts; Barbara Doll for her typing expertise; Intervention Specialists at each of the four Study Centers; and members of the EABs who made this effort possible.
|
References |
|---|
|
TopAbstractIntroductionMethodsStudy 1: Development of...Study 2: Confirmation of...DiscussionWorking Well study investigators...References |
|---|
Abrams, D. B., Boutwell, W. B., Grizzle, J., Heimendinger, J., Sorensen, G. and Varnes, J. (1994) Cancer control at the workplace: the working well trial. Preventive Medicine, 23, 15–27.[Web of Science][Medline]
Allen, S. J. and Hubbard, R. (1984) Regression equations for the latent roots of random data correlation matrices with unities in the diagonal. Multivariate Behavioral Research, 21, 393–398.
Anderson, J. C. and Gerbing, D. W. (1988) Structural equation modeling in practice: a review and recommended two-step approach. Psychological Bulletin, 103, 411–423.[Web of Science]
Bollen, K. A. (1989a) A new incremental fit index for general structural equation models. Sociological Methods of Research, 17, 303–316.
Bollen, K. A. (1989b) Structural Equations with Latent Variables. John Wiley, New York.
Bollen, K. A. and Long, J. S. (eds) (1993). Testing Structural Equation Methods. Sage, Newbury Park, CA.
Byrne, B. M. (1989) A Primer of LISREL: Basic Applications and Programming for Confirmatory Factor Analytic Models. Springer-Verlag, New York.
Carlaw, R., Mittlemark, M., Bracht, N. and Luepker, R. (1984). Organization for a community cardiovascular health program: experiences from the Minnesota Heart Health Program. Health Education Quarterly, 11, 243–252.[Web of Science][Medline]
Cattell, R. B. (1966) The scree test for the number of factors. Multivariate Behavioral Research, 1, 245–276.
Cohen, J. (1977) Statistical Power Analysis for the Behavioral Sciences, revised edn. Academic Press, New York.
COMMIT Study (1988) A Handbook for Community Change. National Institutes of Health, National Cancer Institute, Division of Cancer Prevention and Control, Bethesda, MD.
Cottrell, L. S. (1976) The competent community. In Kaplan, B. H., Wilson, R. N. and Leighton, A. H. (eds), In Further Explorations in Social Psychiatry. Basic Books, New York, pp. 195–209.
Cronbach, L. J. (1951) Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.[Web of Science]
DeVellis, R. F. (1991) Scale Development: Theory and Applications. Sage, Newbury Park, CA.
Farquhar, J. W., Fortmann, S. P., Maccoby, N., Wood, P. D., Haskell, W. L., Taylor, C. B., Flora, J. A., Solomon, A. S., Rogers, T., Adler, E., Breitrose, P. and Weiner, L. (1984) The Stanford Five City Project: an overview. In Matarazzo, J. D., Weiss, S. M., Herd, J. A., Miller, N. E. and Weiss, S. M. (eds), Behavioral Health: A Handbook of Health Enhancement and Disease Prevention. John Wiley, New York, pp. 1154–1165.
Fawcett, S. B., Lewis, R. K., Paine-Andrews, A., Francisco, V. T., Richter, K. P., Williams, E. L. and Copple, B. (1997) Evaluating community coalitions for prevention of substance abuse: the case of project freedom. Health Education and Behavior, 24, 812–828.
Florin, P. and Wandersman, A. (1990) An introduction to citizen participation, voluntary organizations, and community development: insights for empowerment through research. American Journal of Community Psychology, 18, 41–54.[Web of Science]
Fortmann, S. P., Winkleby, M. A., Flora, J. A., Haskell, W. L. and Taylor, C. B. (1990) Effect of long-term community health education on blood pressure and hypertension control: the Stanford five-city project. American Journal of Epidemiology, 132, 629–646.
Francisco, V. T., Paine, A. L. and Faucett, S. B. (1993) A methodology for monitoring and evaluating community health coalitions. Health Education Research, 8, 403–416.
Glasgow, R. E., Terborg, J. R., Hollis, J. F., Severson, H. H., Fisher, K. J., Boles, S. M., Pettigrew, E. L., Foster, L. S., Strycker, L. A. and Bishchoff, S. (1994) Modifying dietary and tobacco use patterns in the worksite: the take heart program. Health Education Quarterly, 21, 69–82.[Web of Science][Medline]
Glasgow, R. E., Terborg, J. R., Hollis, J. F., Severson, H. H. and Boles, S. M. (1995) Take heart: results from the initial phase of a worksite-wellness program. American Journal of Public Health, 85, 209–216.
Green, L. W. and Raeburn, J. (1990) Contemporary developments in health promotion: definitions and challenges. In Bracht, N. (ed.), Health Promotion at the Community Level. Sage, Newbury Park, CA, pp. 278–303.
Green, L. W. (1986) The theory of participation: a qualitative analysis of its expression in national and international health policies. Advances in Health Education Promotion, 1A, 211–236.
Heimendinger, J., Feng, Z., Emmons, K., Stoddard, A., Kinne, S., Biener, L., Sorensen, G., Abrams, D., Varnes, J. and Boutwell, B., for the Working Well Research Group (1995) The Working Well Trial: baseline dietary and smoking behaviors of employees and related worksite characteristics. Preventive Medicine, 24, 180–193.[Web of Science][Medline]
Hull, J. G., Tedlie, J. C. and Lehn, D. A. (1995) Modeling the relation of personality variables to symptom complaints: the unique role of negative affectivity. In Hoyle, R. H. (ed.), Structural Equation Modeling: Concepts, Issues, and Applications. Sage, Thousand Oaks, CA, pp. 217–235.
Hunkeler, E. F., Davis, E. M., McNeil, B., Powel, J. W. and Polen, M. R. (1990) Richmond quits smoking: a minority community fights for health. In Bracht, N. (ed.), Health Promotion at the Community Level. Sage, Newbury Park, CA, pp. 278–303.
James, L. R., Mulaik, S. A. and Brett, J. M. (1982) Causal Analysis: Assumptions, Models, and Data. Sage, Beverly Hills, CA.
Joreskog, K. C. and Sorbom, D. (1989) LISREL 7: A Guide to the Program and Applications, 2nd edn. SPSS, Chicago, IL.
Kaiser, H. F. (1958) The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23,187–200.[Web of Science]
Kelly, J. G. (1979) Tain't what you do, it's the way you do it. American Journal of Community Psychology, 7, 239–261.[Web of Science][Medline]
Knight, E. A., Johnson, H. H and Holbert, D. (1991) Analysis of the competent community: support for the community organization role of the health educator. International Quarterly of Community Health Education, 11K, 145–154.
Kuriji, K., Ostbye, T. and Bhatti, T. (1988) Initiating community self-help: a model for public health workers. Canadian Journal of Public Health, 79, 208–209.[Web of Science][Medline]
Lasater, T., Abrams, D. and Artz, L., Beaudin, P., Cabrera, L., Elder, J., Ferreira, A., Knisley, P., Peterson, G., Rodrigues, A., Rosenberg, P., Snow, R. and Carleton, R. (1984) Lay volunteer delivery of a community-based cardiovascular risk factor change program: the Pawtucket experiment. In Matarazzo, J. H., Weiss, S. M., Herd, J. A., Miller, N. E. and Weiss, S. M. (eds), Behavioral Health: A Handbook of Health Enhancement and Disease Prevention. John Wiley, New York, pp. 1166–1170.
Lautenschlager, G. L. (1989) A comparison of alternatives to conducting Monte Carlo analyses for determining parallel analysis criteria. Multivariate Behavioral Research, 24, 365–395.
Lewis, R. K., Paine-Andrews, A. L., Fawcett, S. B., Francisco, V. T., Richter, K. P., Copple, B. and Copple, J. (1996) Evaluating the illegal sales of alcohol and tobacco products to minors. Journal of Community Health, 21, 429–436.[Web of Science][Medline]
McAlister, A., Puska, P. Salonen, J. T., Toumilehta, J. and Koskela, K. (1982) Theory and action for health promotion: illustrations from the North Karelia Project. American Journal of Public Health, 72, 43–50.
Rothman, J. (1970) Three models of community organizational practice. In Cox F. M. (ed.), Strategies of Community Organization. Peacock, Itasca, IL.
Sorensen, G., Glasgow, R. and Corbett, K. (1990) Involving worksites and other organizations in health promotion. In Bracht, N. (ed.), Organizing for Community Health Promotion: A Guide. Sage, New York.
Sorensen, G., Hsieh, J., Hunt, M. K., Morris, D. H., Harris, D. R. and Fitzgerald, G. (1992) Employee advisory boards as a vehicle for organizing worksite health promotion programs. American Journal of Health Promotion, 6, 443–450, 464.[Medline]
Sorensen, G., Thompson, B., Glanz, K., Feng, Z., Kinne, S., Di Clemente, C. and Emmons, K. (1996) Worksite based cancer prevention: Primary results from the Working Well Trial. American Journal of Public Health, 86, 939–947.
Stryker, S. A., Foster, L. S., Pettigrew, L., Donnelley-Peny, J., Jordan, S. and Glasgow, R. E. (1997) Steering committee enhancements on health promotion program delivery. American Journal of Health Promotion, 11, 437–440.[Web of Science][Medline]
Thompson, B. and Kinne, S. (1990) Social change theory: applications to community health. In Bracht, N. (ed.), Health Promotion at the Community Level. Sage, Newbury Park, CA, pp. 45–65.
Thompson, B. and Pertschuk M. (1992) Organizing communities for risk-factor change. In Ockene, I. S. and Ockene, J. K. (eds), Prevention of Coronary Heart Disease. Little-Brown, Boston, MA, pp. 493–515.
Thompson, B., Wallack, L., Lichtenstein, E. and Pechacek, T. for the COMMIT Research Group. (1991) Principles of community organization and partnership for smoking cessation in the Community Intervention Trial for Smoking Cessation (COMMIT). International Quarterly of Community Health Education, 11, 187–203.
Thompson, B., Corbett, K., Bracht, N. and Pechacek, T. (1993) Community mobilization for smoking cessation: lessons learned from COMMIT. Health Promotion International, 8, 69–83.
Tucker, L. R. and Lewis, C. (1973) A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10.
Velicer, W. F. (1976) Determining the number of components from the matrix of partial correlations. Psychometrika, 41, 321–337.[Web of Science]
West, S. G., Finch, J. F. and Curran, P. J. (1995) Structural equation models with nonnormal variables: problems and remedies. In Hoyle R. H. (ed.), Structural Equation Modeling: Concepts, Issues, and Applications. Sage, Thousand Oaks, CA.
Received on July 22, 1996; accepted on May 22, 1998
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. Lassen, M. Bruselius-Jensen, H. M. Sommer, A. V. Thorsen, and E. Trolle Factors influencing participation rates and employees' attitudes toward promoting healthy eating at blue-collar worksites Health Educ. Res., October 1, 2007; 22(5): 727 - 736. [Full Text] [PDF]
|
|
|
|
 |
|
 K. M. Pollack, G. S. Sorock, M. D. Slade, L. Cantley, K. Sircar, O. Taiwo, and M. R. Cullen Association between Body Mass Index and Acute Traumatic Workplace Injury in Hourly Manufacturing Employees Am. J. Epidemiol., July 15, 2007; 166(2): 204 - 211. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||