Health Education Research, Vol. 16, No. 6, 735-745,
December 2001
© 2001 Oxford University Press
Can health professionals learn qualitative evaluation methods on the World Wide Web? A case example
Department of Health Behavior and Health Education,
1 Department of Maternal and Child Health, and
2 The North Carolina Institute for Public Health, School of Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599,
3 Department of Public Health, Southern Connecticut State University, New Haven, CT 06515 and
4 National Centers for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention, Atlanta, GA 30333, USA
 |
Abstract |
|---|
 Top Abstract IntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
The Enhancing Data Utilization Skills through Information Technology (EDUSIT) project trained Maternal and Child Health professionals to collect, analyze and interpret data via a year-long web-based course. The overall goal of the project was to strengthen the technology and analytic skills of the public health workforce. This article describes and analyzes a web-based module for training public health professionals to use qualitative research and evaluation methods that was one of six offered within the EDUSIT project. The qualitative module consisted of six units: overview of qualitative methods, planning qualitative studies, conducting field observations, qualitative interviewing, analyzing qualitative data and presenting qualitative findings. Evaluation results found no statistically significant changes in specific knowledge or beliefs about qualitative methods. However, the change in participants' self-efficacy was statistically significant. Participants' self-reports also showed significant changes in perceived skill levels in `collecting qualitative data through an interview' and `analyzing and interpreting qualitative data'. Most participants rated each lesson within the qualitative methods module as valuable, and most found the teaching methods used satisfactory, emphasizing the value of both the didactic teaching and the practical exercises and team project. The most common difficulty reported was finding the time to complete the module requirements while also working full-time. Implications of these findings for web-based teaching of public health professionals are discussed.
|
Introduction |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
Computer technology has been used in qualitative research for a variety of purposes. Perhaps the best-known and most frequently used computer technology in qualitative research is data analysis software such as NUD*IST, ATLAS and the ETHNOGRAPH (Higgins, 1998
). An emerging use of computer technology in qualitative research is voice recognition software for transcribing data (Anderson, 1998).
Computer technology in the form of the Internet has also been used in the qualitative data collection process. For instance, E-mail has been used to conduct focus groups for data collection purposes (Murray, 1997). The Internet's capacity for video conferencing is also beginning to be used for qualitative data collection, for communication among researchers, and for consulting and teaching purposes (Fetterman, 1997). The Internet has been used to collect qualitative data from hard to reach populations. For example, Fleitas used the Internet to gather data from children with chronic illnesses and other medical problems (Fleitas, 1998). E-mail has been used by qualitative researchers located in different countries to assist in the data analysis process and in developing theory from their analysis (Bunting et al., 1998).
Local computer networks and software have been used for teaching qualitative methods and for collecting qualitative data (Krothe and Pappas, 1998). Electronic conferencing technology has also been used in teaching qualitative data analysis and report writing (Davie, 1996). Publishing qualitative study reports is another emerging use of the World Wide Web (WWW). Increasingly, the WWW also is being used to teach qualitative methods. This article describes and analyzes a web-based module for training public health professionals to use qualitative research and evaluation methods.
|
Background: the Enhancing Data Utilization Skills through Information Technology (EDUSIT) course |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
The EDUSIT project trained Maternal and Child Health professionals from 13 southeastern states to collect, analyze and interpret data via a year-long web-based course. The course was offered by the Department of Maternal and Child Health, School of Public Health, University of North Carolina at Chapel Hill (UNC) in collaboration with the School's Center for Distance Learning. The overall goal of the project was to strengthen the technology and analytic skills of the public health workforce. The course was composed of the following six modules:
- Essential skills, which included basic epidemiological and statistical concepts
- Qualitative data collection and analysis
- Measurement of social inequalities in health
- Development of quantitative data collection instruments and planning data analysis
- Economic analysis
- Use of Geographic Information Systems and Epi-Info software
Six hours of graduate credit were offered for completing all six modules. Each module was presented for a period of 6 weeks. The modules were self-paced, and participants could access material and complete assignments at their convenience within the specified period of 6 weeks. Before beginning the EDUSIT course, states were provided with the technology specifications needed to fully participate in the course, e.g. they needed RealPlayer installed on their computers in order to access the audio and video portions of the course. A short pre-course module that covered technical skills necessary for participating in the course was developed and implemented for all participants. In addition to didactic material, participants worked as a team with other participants from their state to apply the skills learned in the course to a final course project involving a health problem of importance to their state.
|
The qualitative methods module |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
The qualitative methods module was based on a graduate level, three-credit hour, semester-long course taught by one of the authors (A. S.). Because the qualitative methods module was only one of six in the EDUSIT course and because the participants were busy professionals, it was decided to attempt to create an online course that while covering the basic concepts and skills of qualitative methods would be relatively brief. The online module, therefore, was very skills oriented. With the exception of the `Overview module', each module had some type of skill that was learned and practiced by the participants. The overall goal of the module was for participants to feel comfortable in using qualitative methods, and to be able to plan and carry out a simple qualitative project. It was hoped that this skills-focused approach would be the most useful to the public health professionals who were the EDUSIT participants. The regular three-credit hour course uses the same framework as the online module and is also skills oriented, but each teaching unit is in much more depth. In the regular course, there is a textbook and more journal articles. There is also a more extensive required qualitative project. The development of the online module was done by two course instructors (A. S. and J. B.) and an instructional designer (A. T.) who had prior experience developing web-based courses.
To accomplish these aims, the qualitative methods online module consisted of six units, each unit was taught for 1 week, for a total of 6 weeks for the complete module (January–February 1999). See Table I for a list and explanation of the topics covered in the module.
|
Each of the six units within the qualitative module included:
- An on-line `tutorial', which was the written equivalent of a brief lecture
- A journal article with accompanying guided reading questions
- An experiential field assignment leading to completion of a small-scale qualitative project
For each unit, participants read the tutorial and the journal article, and then posted their responses to the questions to the course discussion forum. Using the web-course software capability, the module instructors then gave feedback to participants concerning their answers to the guided reading questions. In addition, participants from the same state health department worked in teams to develop and carry out a small-scale qualitative study (qualitative field project). For each unit within the qualitative module, there was an experiential field assignment related to their field project. For instance, in the unit on interviewing, each participant was required to conduct at least one interview related to their team's field project. One state team examined the prevalence of Traumatic Brain Injury in their state; they conducted a small study of helmet use by children while riding bicycles. Members of this state team observed children in several locations riding bicycles, then interviewed children and their parents concerning their beliefs and attitudes about the use of bicycle helmets. Then they analyzed the data and wrote a brief report. Each team then posted a report to their assigned presentation space in the course website. Students were not required to obtain institutional review board (IRB) approval for these small-scale field projects. However, students in future online courses that require a field project will be required to obtain IRB approval. The UNC School of Public Health IRB approval form is now available online and distance students can apply for IRB approval online.
In the field observation unit, participants watched a brief video clip and then wrote field notes about it. In the interviewing unit, participants listened to a brief audio clip of an interview that had been conducted by one of the instructors and then wrote a critique of the interview that was posted to the assigned presentation space in the course website. Both the video and audio clips were accessed via the course website.
The instructors for the qualitative module included a faculty member and a graduate student teaching assistant. The instructors checked participants presentation spaces approximately three times per week and provided feedback on the participants' written assignments that had been posted. In addition, twice over the course of 6 weeks, the instructors held `chat rooms' with the participants. Participants were able to ask questions and receive answers immediately. In turn, the instructors were able to ask participants questions about how the module was going for them. Instructors were available for weekly `office hours' so participants could call with questions. Instructors were available through E-mail throughout the module. On average, the qualitative module took about 19 h to complete, spread out over the 6 weeks. Approximately half this time was spent online. Table I
presents a summary of the module's requirements.
|
Evaluation of the qualitative module |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
The evaluation of the online qualitative module was based on a model for changing behavior of health professionals involved in continuing education derived by Umble et al. (Umble et al., 2000
) (Figure 1). The model postulates that training affects behavior by influencing factual and procedural knowledge (Shuell, 1986), beliefs related to the benefits and costs of a behavior (Janz and Becker, 1984), self-efficacy (Bandura, 1997), and skill levels (Kraiger et al., 1993). Participation in distance learning and application of skills learned (on-the-job practice) are further shaped by workplace factors such as peer and supervisor support, time, and the availability of proper equipment (Davis et al., 1995; Ottoson, 2000).
|
|
Methods |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
The qualitative methods module was evaluated using an online evaluation instrument, a pre-test and post-test, and a 6-month follow-up survey.
The online module evaluation instrument
An on-line questionnaire administered to participants immediately after the module included questions about the value of each lesson, participants' gains and level of satisfaction, time spent on the module, and suggestions for future modules.
The pre-test and post-test
An identical pre-test and post-test given immediately before and after the module measured changes in participants' knowledge, beliefs and self-efficacy with regard to the use of qualitative methods. Each of these variables was measured using a composite score from a set of questions designed to measure that variable.
Each participant's `knowledge' score was computed by totaling the participant's correct responses to five multiple choice or true/false questions, for a maximum score of 5. For example, one question asked, `Which of the following statements about qualitative interviewing is true?' (a list of statements was given).
Each participant's `belief' score was computed by totaling how strongly participants agreed with four statements about qualitative methods, using a four-point scale, from `strongly agree' to `strongly disagree', for a maximum score of 16. For example, one statement read, `Using qualitative methods is useful in discovering the point of view of program participants'.
Self-efficacy refers to `people's belief in their capability to organize and execute the course of action required to deal with prospective situations' [(Maibach and Murphy, 1995), p. 38]. Each participant's self-efficacy score was computed by totaling the participant's responses to four questions. Each question used a four-point scale ranging from `not at all confident' to `completely confident', for a maximum score of 16. For example, one question asked, `Your state team has recently conducted 20 qualitative interviews with women and their children regarding a new service in your state. How confident are you in your ability to use the code and retrieve method of analysis to analyze these interviews, if you had to do that right now?'.
The 6-month follow-up survey
A follow-up survey in which the participants evaluated the EDUSIT course was administered 6 months after the entire course ended. The instrument measured participants' self-assessed skill levels before and after the course on a range of representative skills taught in the courses, including qualitative methods skills of `collecting qualitative data through an interview' and `analyzing and interpreting qualitative data'. The instrument also asked participants to report the number of times they had used those skills on the job in the 6 months before and after the course, using a scale of `0 times, 1 time, 2 or more times'. This method, known as the `retrospective pre-test method', has been shown in many studies to be more valid than the traditional pre-test/post-test method due to the `response shift bias' which occurs when participants tend to be more realistic about their level of skills and knowledge after attending a training program than they were before they attended (Howard, 1980, 1981). All data were collected online.
Data analysis
Paired samples t-tests performed using Excel (Microsoft) were used to examine whether the module produced changes in knowledge, belief, and self-efficacy. The Wilcoxon signed-ranks test (Pett, 1997) was used to examine whether the course had produced changes in the frequency with which particular skills were performed in the work setting (SAS version 8.0; SAS Institute, Cary, NC). The evaluation questionnaire also contained several open-ended questions. Participants were asked to describe in their own words the strengths, weaknesses, benefits and suggestions for improvement of the online qualitative module. The brief written responses were reviewed and coded. The coded data were then organized into main themes following the method described by Patton (Patton, 1990). These themes and direct quotations illustrating them are included in Results.
|
Results |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
Participants
Forty participants originally enrolled in the EDUSIT course, but only 27 participated in the qualitative methods module. Some participants dropped out of the EDUSIT course due to changes in their jobs or inability to complete the activities on time. Of the 27 who took the qualitative methods module, 19 (70%) completed all the activities in the module. Eighty-two percent of participants were 40 years or older. They were well educated (67% had Master's degrees or higher) and, on average, their last degree had been awarded more than 10 years prior to the project. Most participants (90%) worked at the state rather than the local level. As a cohort, the group reported strong beliefs about the value of using data and technology before the program, although confidence in their abilities to use data and technology was low.
Satisfaction with and effectiveness of format and teaching methods
Most participants completing the online evaluation rated each lesson within the qualitative methods module as `valuable' (Table II).
|
In their comments, most participants also found the teaching methods used to be satisfactory, emphasizing the value of both the didactic teaching and the practical exercises and team project. The following are typical comments made by participants about the qualitative module:
I learned there are other ways of analyzing besides quantitative. I [gained because I] was able to practice a method of obtaining qualitative data and comparing what we did in our group with some of the qualitative data that was provided [in the course materials].
Another emphasized the value of the team project:
The process of performing a team qualitative research project was a powerful hands-on learning experience. The theory part of the Module was built upon through each step of the team project.
Another remarked:
In spite of my own erratic participation, I think all of us have benefited from working on a distance-learning model. We are much more comfortable with the technology and the skills learned have been helpful for improving individual skills and developing a context within which to study other MCH issues.
The most common difficulty reported was finding the time to complete the module requirements while also working full-time. The qualitative methods module was the second module in the EDUSIT course series. Having received comments about the quantity of work required in the first module, the requirements for the qualitative methods module were somewhat reduced so that this module was `more workable', as one participant put it. Still, participants had difficulty completing the module requirement while also working full-time. As one participant put it:
The field observation and interviews were time consuming, and we had some difficulty getting all of the permissions necessary to complete this activity, in terms of timing and other things going on in the state.
Another participant suggested breaking up the EDUSIT course into smaller pieces and tailoring content to specific responsibilities of employees:
...I frankly did not have the time to devote to the module that was required. This was due to my own time allotment since I am MCH director and not caused by others above me. I find I have had too many competing demands during this module time period. The length of the modules is also somewhat daunting. If I have a half hour to devote, I still can't get much done. Perhaps in the future the level of person engaged in distance learning should be considered. Breaking modules into more discreet components might also be helpful.
Effects on knowledge, beliefs, self-efficacy, skills and practice
There were no statistically significant changes in specific knowledge or beliefs about qualitative methods over the course of the module (Table III). The change in participants' self-efficacy was statistically significant.
|
As described above, the qualitative methods module was one of six modules in the EDUSIT course. It is useful, therefore, to compare the impact of this module to the other EDUSIT modules. First we tested whether the modules differed statistically in their level of impact on knowledge, beliefs and self-efficacy using a generalized repeated measures analysis of variance involving methods for generalized estimating equations (GEE) (1). Computations were done using SAS version, PROC GENMOD (SAS Institute, Cary, NC). Difference scores (post–pre) were the outcome and module was treated as a class variable. The pre score was also included as a covariable and subject ID was included in the repeated statement with exchangeable covariance structure. For each of the three outcomes, effect of module was non-significant (Type III Score statistic). Since the modules were not statistically different in their effects, responses were pooled across modules and the effect of intervention via paired t-test (2) on the difference scores for each outcome was computed separately. (Note that in the paired t-test analysis, all response scores were treated as independent. No steps were taken to account for multiple responses from the same individual.) When data for all modules were pooled and compared using the paired t-test, the knowledge (P < 0.0001), belief (P < 0.01) and self-efficacy (P < 0.0001) measurements were all significantly higher after the modules. Thus, while the qualitative module showed a significant increase only in self-efficacy, its effects were not significantly different from the other modules and when all modules' data were combined, all three variables showed significant changes due to the EDUSIT course.
Participants' self-reports showed significant changes in perceived skill levels in `collecting qualitative data through an interview' and `analyzing and interpreting qualitative data' from before the EDUSIT course (retrospective self-assessment at the 6-month follow-up) to after the EDUSIT course (current self-assessment at the 6-month follow-up) (Table IV).
|
Participants' self-reports about the frequency of use 6 months before and 6 months after the course showed an increase in the frequency of `collecting qualitative data through an interview' and `analyzing and interpreting qualitative data' as practice behaviors (Table V
). Participants were asked to include in their self-reported practices their activities for completing the major course project, so it is possible that the reported increases were due to completion of module activities.
|
Participants' comments in the module evaluation complement and support these results. Many responded that they had gained a greater fundamental understanding of the nature of qualitative methods, the distinction between qualitative and quantitative methods, and ways they could use qualitative and quantitative methods in their work. They also said they were more skilled and comfortable with using these methods, and that they were using these methods in practice and planned to continue doing so. The following quotations are typical:
First, [I gained] an understanding of how qualitative methods differ from quantitative methods. Second, [I gained] some ideas on how to apply qualitative methods to work programs for assessment and improvement.In this module, I learned important techniques for collecting and analyzing quantitative and qualitative data, as well as the distinct difference between the two data sources. I also learned how to develop meaningful questions for questionnaires as a means of collecting relevant data. Also, this module afforded the opportunity for me to learn more about and improve my interviewing skills.
I feel more comfortable conducting interviews with clients, and knowing what questions to ask to get at the information that I want.
I plan to conduct or assist in conducting focus group(s) to better target the efforts of our SIDS Coalition. I did improve my interviewing skills somewhat and feel much more comfortable in analyzing that data.
The module was much more interesting and practical than I originally thought. The qualitative methods taught will be particularly useful in working on program planning and monitoring for children with special health care needs (CSHCN). Much of the success of the work with CSHCN is based upon family needs. Success is more easily measured through family attitudes and perceptions rather than quantitative evaluation.
In addition to indicating intention to use the methods, the last quotation above also indicates affective outcomes, specifically, an increase in appreciation of the practical value of qualitative methods under specific circumstances of interest to the participant and an increase in interest in carrying them out. In a similar way, another participant remarked, `When I began [this module], I did not expect to come away with such an appreciation of qualitative research methods'.
In summary, completing the module appeared to increase participants' overall understanding of the nature of qualitative methods, their distinction from quantitative methods and when to use them. Furthermore, when participants now use qualitative methods, they are more comfortable and skilled in doing so, in their own view. Finally, they appear to be more likely to apply qualitative methods, as the frequency of self-reported use increased from before to after the course and as indicated by comments.
|
Discussion |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
The main limitation of this study was the small number of participants in the qualitative module. Twenty-seven people began the module but only 19 completed it. This is a small number of people from which to draw meaningful conclusions. Another limitation was the fact that the whole EDUSIT program was a first-time effort. It was the first experience of online learning for the participants, for the instructors and for the technical support personnel. For all those involved there was a steep `learning curve'. In addition, the measurements, especially those concerning knowledge and beliefs, could have been improved. An important issue in online teaching is the impact on faculty workload. Faculty and university administrators are anxious to know how teaching an online course compares to teaching a regular `residential' course in terms of demand on faculty time. In this study, unfortunately, we did not keep data on the amount of time that faculty put into developing and teaching the qualitative module. This will be an important issue for future research.
Despite these limitations, the findings are useful in furthering our understanding of online teaching/learning directed at the practicing public health workforce. For instance, the quantitative measurements found increases in self-efficacy, self-assessed skill level and self-reported frequency of use of qualitative methods. While the testing found non-significant change in knowledge and beliefs, several participants' comments indicated that their factual and procedural knowledge of, and their appreciation for, qualitative methods had increased. Several participants described ways they were using or were planning to use the methods taught in more `skilled' and `comfortable' ways. It is likely that the knowledge and belief questions were not adequate measures of the knowledge and beliefs taught in the course, so this finding may indicate measurement error rather than a lack of a program effect.
These results may indicate that even a relatively short exposure to qualitative methodology can make health professionals more willing to explore the use of these methods. Further improvements will require opportunities to practice the skills on-the-job as well as support from their supervisors and agency environments (Ottoson, 1995; Umble and Cervero, 1996; Green et al., 1988).
Over the course of the year-long web-based project, we learned a great deal about the public health workforce. First, it is very fluid. Professional staff change jobs, are given new assignments and some leave the workforce altogether. Second, although supervisors may give their support and in some cases attempt to protect time for staff to complete training activities, the immediacy of day-to-day tasks often takes precedence over training. Thus, the combination of tempered supervisory support coupled with the difficulty of devoting several hours to completing an assignment may have undermined the participants' ability to complete the qualitative module. Of the 27 participants who took the pre-test for the qualitative methods module, ultimately only 19 completed it. One participant said that additional responsibilities because of staff attrition and demands of work left her no time to complete the assignments. Seven others, who did not complete all the module requirements, stayed on as team members, however, and may have stayed abreast of the new qualitative skills their peers were learning.
Using computer technology and particularly web-based instruction to teach qualitative methods appears to be a promising approach for reaching health professionals. The advantages of web-based instruction is that it is relatively easy to access and can be accessed anytime from work or home. All that is required is a desktop computer, some software and a connection to the Internet. With just this minimum amount of technology, health professionals can have access to high-quality, graduate-level training. The problems that we found in this first attempt to reach health professionals with web-based instruction, however, were not technical but organizational. The main barrier was the time necessary to complete requirements for the module in relation to the participant's work-related time demands. To say that participants were busy professional people would be an understatement. While more research is needed on how to foster public health workers' completion of distance-learning continuing education programs, we believe that having a worker's supervisor sign a memorandum of understanding (MOA) would be helpful in reducing the non-completion problem. Through such an MOA, the supervisor would indicate institutional commitment for the worker to take the time necessary to complete the program's requirements.
At the end of the EDUSIT course, participants suggested breaking the qualitative module into smaller, self-contained units, such as a unit just on conducting focus groups. As previously mentioned, the qualitative module used in the EDUSIT project was based on an existing, three semester-hour, graduate-level course. The online qualitative module was equivalent to approximately a one-half to a one semester-hour course. While creating even smaller units is possible, it may not be appropriate. The possibility of brief, self-contained modules raises the dilemma captured in two aphorisms: `a half a loaf is better than none' and `a little bit of knowledge can be dangerous'. In learning about a body of knowledge and skills, such as qualitative methods, there is probably minimum content that should be included so that the learner does not do more harm than good, or at least can use the methods and techniques appropriately. For any body of knowledge, it is difficult to determine where the line is between a minimum amount of content and an insufficient amount.
In conclusion, we are convinced that teaching high-quality courses to working health professionals using modern computer and web-based technologies is possible. The challenge we see is designing courses in such a way that they can fit into professionals' busy schedules, while at the same time providing enough content so that the learner is able to use the new knowledge and skills appropriately. Whenever feasible, it will also be important for distance learning programs to take steps to improve supervisory support and resources in the work organizations, including released work time and equipment, to foster completion of distance learning activities and application of learned skills on the job.
|
Acknowledgments |
|---|
Enhancing Data Utilization Skills through Information Technology (EDUSIT) was supported by grant # 6 U93 MC 00150-03 S1 RO from the Maternal and Child Health Bureau, Health Resources and Services Administration, US Department of Health and Human Services. The authors thank Gary Koch and Molly Young of the UNC Biometrics Consulting Laboratory for their assistance with data analysis; and Sarah Giersch and Sarah Paliulis for their help with data management.
|
References |
|---|
|
TopAbstractIntroductionBackground: the Enhancing Data...The qualitative methods moduleEvaluation of the qualitative...MethodsResultsDiscussionReferences |
|---|
Anderson, J. F. (1998) Transcribing with voice recognition software: a new tool for qualitative researchers. Qualitative Health Research, 8, 718–723.
Bandura, A. (1997) Self-efficacy: The Exercise of Control. Freeman, New York.
Bunting, S. M., Russell, C. K. and Gregory, D. M. (1998) Use of electronic mail (email) for concept synthesis: an international collaborative project. Qualitative Health Research, 8, 128–135.
Davie, L. E. (1996) Learning qualitative research: electronic learning circles. Qualitative Health Research, 6, 453–457.
Davis, D., Thomson, M. A., Oxman, A. D. and Haynes, R. B. (1995) Changing physician performance: a systematic review of the effect of continuing medical education strategies. Journal of the American Medical Association, 274, 700–705.
Fetterman, D. M. (1997) Videoconferencing over the Internet. Qualitative Health Research, 7, 154–163.
Fleitas, J. (1998) Spinning tales from the World Wide Web: qualitative research in an electronic environment. Qualitative Health Research, 8, 283–292.
Ford, J. K. and Weissbein, D. A. (1997) Transfer of training: an updated review and analysis. Performance Improvement Quarterly, 10(2), 22–41.
Green, L. W., Eriksen, M. P. and Schor, E. L. (1988) Preventive practices by physicians: Behavioral determinants and potential interventions. American Journal of Preventive Medicine, 4, 101–107.[Medline]
Higgins, J. W. (1998) Social Marketing and MARTIN: tools for organizing, analyzing, and interpreting qualitative data, Qualitative Health Research, 8, 867–876.
Howard, G. S. (1980) Response shift bias. A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4, 93–106.
Howard, G. S. (1981) On validity. Evaluation Review, 5, 567–576.
Janz, N. K., and Becker, M. H. (1984) The health belief model: decade later. Health Education Quarterly, 11, 1–47.[Web of Science][Medline]
Kraiger, K., Ford, J. K. and Salas, E. (1993) Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.[Web of Science]
Krothe, J. S. and Pappas, V. C., (1998) Using technology for collaborative research and writing. Qualitative Health Research, 8, 563–569.
Maibach, E. and Murphy, D. A. (1995) Self-efficacy in health promotion research and practice conceptualization and measurement. Health Education Research, 10, 37–50.
Murray, P. J., (1997) Using virtual focus groups in qualitative research. Qualitative Health Research, 7, 542–549.
Ottoson, J. M. (1995) Use of a conceptual framework to explore multiple influences on the application of learning following a continuing education program. Canadian Journal of Study of Adult Education, 9, 1–18.
Ottoson, J. M. (2000) Evaluation of continuing professional education: toward a theory of our own. In Mott, V. M. and Daley, B. J. (eds), Charting a Course for Continuing Professional Education. New Directions for Adult and Continuing Education. Jossey-Bass, San Francisco, CA, pp. 43–55.
Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. Sage, Thousand Oaks, CA.
Pett, M. A. (1997) Nonparametric Statistics for Health Care Research. Sage Thousand Oaks, CA.
Shuell, T. J. (1986) Cognitive conceptions of learning. Review of Educational Research, 56, 411–436.
Umble, K. E., and Cervero, R. M. (1996) Impact studies in continuing education for health professionals: a critique of the research syntheses. Evaluation and the Health Professions, 19, 148–174.
Umble, K. E., Cervero, R. M., Yang, B. and Atkinson, W. (2000) Effects of traditional classroom and distance continuing education: a theory-driven evaluation of a vaccine-preventable diseases course. American Journal of Public Health, 90, 1218–1224.
Received on November 7, 2000; accepted on August 15, 2001
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  K. E. Umble and L. M. Dooley Planning Human Resource Development and Continuing Professional Education Programs That Use Educational Technologies: Voices That Must be Heard Advances in Developing Human Resources, February 1, 2004; 6(1): 86 - 100. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||