Health Education Research Advance Access originally published online on January 21, 2008
Health Education Research 2008 23(3):427-439; doi:10.1093/her/cym094
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure
1 Department of Health Education and Health Promotion, School for Public Health and Primary Care (Caphri), Maastricht University, PO Box 616, 6200 MD Maastricht, The Netherlands
2 Department of Public Health, Erasmus University Medical Centre, Rotterdam, The Netherlands
3 EMGO Institute, VU University Medical Centre, Amsterdam, The Netherlands
* Correspondence to: R. Crutzen. E-mail: rik.crutzen{at}gvo.unimaas.nl
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
It appears that in practice exposure to Internet-delivered behaviour change interventions, encouraging a healthy lifestyle for adolescents with regard to health risk behaviours, is quite low. There is, however, a lack of evidence-based insight into how to disseminate such interventions and how to reach optimal exposure. A more extensive exploration is therefore timely since this knowledge is crucial to improve the public health impact of such interventions. By means of a three-round Delphi study factors associated with dissemination of and exposure (first visit, stay long enough and revisit) to Internet-delivered interventions aimed at adolescents had been identified, as well as the extent to which experts agree on the importance of these factors. Results showed that there was a high rate of consensus among experts from several disciplines with regard to the importance of factors like word of mouth recommendation, the interface of the intervention and utilization of all features provided by the Internet. Experts do not agree, however, on a gold standard for successful dissemination. Overall, the results of this exploration serve as a handle for the formation of further research questions to be tested and answered in research among adolescents.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
From evidence of efficacy trials, it appears that in practice exposure to Internet-delivered behaviour change interventions is quite low [1], and in real situations exposure may be even lower [2]. Explosive growth of the Internet has caused an increase in Internet-delivered interventions aimed at promoting behaviours related to the primary prevention of chronic diseases and therefore encouraging a healthy lifestyle with regard to health risk behaviours [3–5]. Previous research indicated that factors to improve dissemination of, and exposure to, such interventions include high search-engine ranking [6], an appropriate domain name (URL) [7] and perceived credibility [8]. There is, however, a lack of evidence-based insight into how to disseminate such interventions and how to reach optimal exposure, so a more extensive exploration, which provides an overview of factors associated with dissemination and exposure of Internet-delivered interventions, is timely. There were three reasons to focus on interventions aimed at adolescents. First, while many of these health risk behaviours, such as lack of physical activity, low consumption of fruit, cigarette-smoking, unprotected sex and excessive alcohol consumption, are acquired during adolescence, they track into adulthood, thereby affecting not only current health but also health in later life [9]. Second, the current generation of adolescents grew up with the Internet [10] and is more open towards new possibilities offered by this medium [11]. They use the Internet differently compared with adults [12–14] and their main online activities are instant messaging, gaming, downloading and visiting social-networking websites [13, 15]. The expectation is therefore that adolescents are susceptible to other factors associated with dissemination of and exposure to interventions. Third, this age group poses specific challenges to health promoters. In contrast to patients looking for treatment or medical information, adolescents without any chronic disease are not likely to be internally motivated to search for Internet-delivered behaviour change interventions [1]. Researchers in this study therefore expect other factors to be associated with dissemination of and exposure to Internet-delivered interventions aimed at promoting behaviours related to the primary prevention of chronic diseases.
Dissemination involves distribution of the intervention among the target population and also bringing the intervention to the attention of the target population. Successful dissemination depends on the target population, the source and the intervention itself [16]. Exposure involves actual exposure of individuals to the intervention content, through use of the intervention, and is necessary because attention is a prerequisite to establish desired behaviour change [17]. The Reach, Efficacy, Adoption, Implementation and Maintenance (RE-AIM) model is a very useful framework to look at dissemination and exposure. According to this model, the public health impact of an intervention can only be evaluated by the assessment of five dimensions: Reach, Efficacy, Adoption, Implementation and Maintenance (hence the acronym RE-AIM) [18, 19]. So, besides the importance of evaluating efficacy of interventions [4, 5], these other four dimensions should also be addressed. The potential reach of Internet-delivered behaviour change interventions aimed at adolescents is enormous since already 95% of Dutch people between 12 and 25 years had access to the Internet by 2006 [20]. Internet penetration rates are also high in other countries in western Europe, North America and Australia [21]. The other two dimensions, adoption and maintenance, refer, respectively, to the proportion and representativeness of settings that adopt an intervention and the extent to which an intervention is sustained over time [18]. In terms of Internet-delivered interventions, there are three different aspects with regard to exposure: (i) a first visit, (ii) staying on the intervention long enough actually to use and process the information and (iii) revisiting the intervention. The latter can be important since there is a possible dose–response relationship between the number of visits and behaviour change outcomes [22–24]. With regard to staying ‘long enough’, it is hard to indicate an exact length of time since this differs per intervention. It depends, for example, on the length of the questionnaire which precedes tailored feedback. Furthermore, there is also a difference between individuals. For example, some people are faster in processing information than others. In general, staying long enough is defined as the time which an individual needs to process the information provided at a specific intervention. In line with the RE-AIM model, these three different aspects of exposure relate to implementation: the extent to which an intervention is delivered as intended [18]. Our focus in this study was on this dimension.
The aim of this study was (i) to identify factors that are associated with dissemination of and exposure (first visit, stay long enough and revisit) to Internet-delivered behaviour change interventions aimed at adolescents from the point of view of experts in different areas and (ii) to identify the extent to which experts agree on the importance of these factors.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
A three-round Delphi study was conducted through the Internet by the use of SurveyMonkey.com and within a 3-month time frame (September–December 2006). In a Delphi study, a panel of experts is consulted over several rounds. All questionnaires of the three-round Delphi study were pre-tested among health promotion researchers and employees of health promotion institutions.
The Delphi methodology was chosen for three reasons: First, because this method is useful to generate ideas where scientific knowledge is scarce. Second, the main characteristics of a Delphi study (anonymity, iteration and controlled feedback) allow participants to give and change (after having received feedback) their opinion freely [25]. This is highly relevant since different disciplines are involved in carrying out Internet-delivered behaviour change interventions, for example. health promotion research, e-marketing and communication, health promotion institutions and technical implementation. A heterogeneous sample of experts was therefore used in this Delphi panel to gain an insight into the opinion of their peers from different disciplines. Finally, a Delphi methodology can be helpful in situations where face-to-face discussions are impractical [26]. For example, experts from different parts of the world (mainly western Europe and Northern America) could be included because this study could be conducted through the Internet and there were no barriers in distance. Furthermore, experts could complete each questionnaire at their own convenience (within a 4- or 5-week period), which partly circumvents difficulties from non-matching schedules.
First round
Procedures and participants
For the first round, experts in the field of health promotion, e-marketing and communication and technical implementation were selected. In the field of health promotion, a distinction was made between experts working as researchers and those who had a more applied background. This difference is between those working at universities or other research institutes and those working in health promotion institutes. For the selection of relevant experts in the field of health promotion, experts who previously participated in qualitative interviews with regard to promotion of healthy behaviour through the Internet were screened [1]. Furthermore, experts from editorial boards of leading journals in the field of health promotion and the Internet were selected. To select relevant experts from other disciplines, the researchers in this study mainly used their international network in the field of health promotion research. They were asked to mention experts in health promotion institutes or ICT (Information and Communication Technology) companies with whom they had previously cooperated. Experts in e-marketing and communication were not necessarily specialized in behaviour change interventions, but they had knowledge in other areas of e-marketing and communication which could also be appropriate for Internet-delivered behaviour change interventions. With regard to technical implementation, for example, web developers and designers were invited to take part.
In an invitation letter, sent by e-mail in September 2006, with an explanation about the study goals and procedure, 62 experts were asked to participate in all three rounds of the Delphi study (Table I). In this e-mail, there was a link to the website where they could complete the first-round questionnaire. Invitees received a reminder 2 weeks afterwards and had a 5-week period to complete the first-round questionnaire. Thirty experts (48% response rate) responded to the first-round questionnaire (Table I).
|
Questionnaire
This first-round questionnaire consisted of four open-ended main questions. In line with the aim of this study, these questions were (i) What are, according to your expertise, the factors that determine whether an adolescent will visit an Internet-delivered behaviour change intervention for the first time? (ii) What are, according to your expertise, the factors that will determine whether an adolescent will stay on an Internet-delivered behaviour change intervention long enough to engage actively in and process the educational content provided in the intervention? (iii) What are, according to your expertise, the factors that will determine whether an adolescent will revisit an Internet-delivered behaviour change intervention? (iv) What are, according to your expertise, the factors that are essential for successful dissemination of an Internet-delivered behaviour change intervention aimed at adolescents?
In the questionnaire instructions, the interest in factors associated with dissemination of and exposure to Internet-delivered behaviour change interventions was explicitly defined (as in the introduction section of this paper). Furthermore, it was stressed that the focus was on (i) interventions aimed at promoting behaviours related to the primary prevention of chronic diseases and therefore encouraging a healthy lifestyle with regard to health risk behaviours and (ii) adolescents as a specific target group since they use the Internet differently compared with adults and the expectation was that they are susceptible to other factors associated with dissemination of and exposure to interventions. Finally, participants were asked to think of factors on several levels (e.g. user, intervention and source).
Data analysis
The answers to each of the questions resulted in a list of potential factors associated with dissemination and exposure. Two researchers analysed this list of potential factors and merged factors that were semantically similar into one factor. After discussion about semantic similarity, both researchers fully agreed about all factors that were included in the second-round questionnaire (Tables III–VI). Two other researchers approved this questionnaire.
|
|
Second round
Procedures and participants
Experts who participated in the first round were also invited to participate in the last two rounds. Additional experts from all disciplines were selected for the last two rounds. Several databases (PubMed, PsycINFO and Medline) were searched for literature on Internet-delivered behaviour change interventions (scientific papers published between 2000 and 2006). Corresponding authors of journal articles were screened for relevant publications in the field of Internet-delivered behaviour change interventions. The same procedure was carried out with abstract books from relevant conferences [conferences of the Society for the Internet in Medicine (MEDNET), International Society of Behavioral Medicine, national conferences]. Furthermore, abstract books from conferences not purely focused on improving public health ‘through the Internet’ (International Society for Behavioral Nutrition and Physical Activity, European Health Psychology Society) were screened. An invitation letter, sent by e-mail in October 2006, with an explanation about the study goals and procedure asked 233 experts to participate in the last two rounds of the Delphi study (Table II). Invitees received two reminders and had a 4-week period to complete the second-round questionnaire. In total, 93 experts (40% response rate) responded to the second-round questionnaire (Table II).
|
Questionnaire
The factors that were mentioned by the experts in the first round formed the basis of the questionnaire for the second round. This second-round questionnaire consisted of 82 structured items (Tables III–VI) and experts were asked to indicate to what extent they thought each factor was important on a seven-point Likert scale ranging from 1 (not important at all) to 7 (extremely important).
Data analysis
Median scores and interquartile deviations (IQDs) were used to summarize the extent to which participants agreed with the importance of the factor under consideration [27]. The median score can be defined as the score that falls exactly in the middle of a group of scores, the 50th percentile. In this way, it is possible to determine the agreement on the importance of a factor. The IQD is the absolute value of the difference between the 25th and 75th percentiles of the responses of the experts to an item, with smaller values indicating a higher degree of consensus among the experts. On a seven-point Likert scale, an IQD 
1 can be considered as good consensus and means that >50% of all opinions fall within one point on the scale [25]. A distinction between importance and consensus among experts was made since it is possible that there is consensus among experts (IQD 1) that the factor under consideration is not important (low median). The cut-off point for importance of potential factors was a median score 
6. In combination with an IQD 1, this represents factors that are rated as ‘very important’ (Mdn = 6) or ‘extremely important’ (Mdn = 7) by >50% of the experts.
Third round
Procedures and participants
All the experts who completed the second-round questionnaire were invited for the third and final round. Exceptions were two experts who could not be identified since they did not provide their name and/or e-mail address and three experts who explicitly stated in advance that they were not able to participate in the third round owing to time constraints. An invitation letter, sent by e-mail in November 2006, asked 88 experts to participate in the third round (Table II) and procedures were comparable to the second round. Fifty-nine experts (67% of second-round participants, 25% response rate) completed the third-round questionnaire (Table II).
Questionnaire
A questionnaire, with feedback on the second-round group results (median and IQD) for each item, was presented to third-round participants. In accordance with the Delphi methodology, experts were asked to re-rate their answers for items on which no consensus was obtained in the second round (IQD > 1). Items on which consensus was obtained were excluded, resulting in a third-round questionnaire of 63 items.
Data analysis
Group median ratings and IQDs were again analysed. All statistical analyses were conducted by means of SPSS 13.0.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
The items of the second-round questionnaire represented the factors that were identified during the first round. Items were split up with regard to: (i) a first-time visit (Table III, 26 items), (ii) whether a person will stay long enough to engage actively in and process the educational content provided (Table IV, 34 items), (iii) revisiting (Table V, 15 items) and (iv) successful dissemination (Table VI, seven items).
|
|
Consensus
In the second round, consensus was obtained on 19 items (IQD
1). After experts re-rated their answers (during the third round) based on the feedback of the second-round group results, consensus was obtained on another 56 items. All together, consensus was obtained on 75 items (91%).
Importance
A total of 30 of these items had a median score 6. Three (out of 26) of these items were related to first visit (12%) [e.g. a positive recommendation by word of mouth (WOM) and an attractive interface at first sight], 19 (out of 34) to staying on an intervention (56%) (e.g. interactive features and an easy to follow navigation structure) and 8 (out of 15) to revisiting an intervention (53%) (e.g. commitment to revisit and new content on a regular basis). Although experts obtained consensus about factors that potentially determine dissemination of Internet-delivered interventions among adolescents, none of these factors had a median score 6. Providing the Internet intervention in a controlled setting (e.g. school curriculum), instruction of executives, an appealing name, assuring a high search-engine ranking, embedding the Internet intervention in other (health) programmes and co-operation with commercial partners were all factors with a median score of 5 on a seven-point Likert scale.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
To date there is a lack of evidence-based insight into dissemination of and exposure to interventions delivered through the Internet and encouraging a healthy lifestyle for adolescents. This Delphi study, with experts from several disciplines, provided an overview of factors that potentially determine dissemination of and exposure to Internet-delivered interventions aimed at promoting healthy lifestyles in adolescents and the extent to which experts agree on these factors. These factors relate to the dimension of implementation in the RE-AIM model. Notably, most of the factors of which experts agreed on the importance were related to staying on and revisiting the intervention. The results of this exploration are based on the opinions of experts and can now be used for the formation of further questions to be tested and answered in research among adolescents. As a first step, focus group interviews with adolescents are to be conducted and will be reported elsewhere (R. Crutzen, J. De Nooijer, W. Brouwer, A. Oenema, J. Brug and N. K. De Vries, in preparation). A next step would be to conduct observational, experimental and longitudinal studies to test ‘if’ and ‘how’ these factors improve dissemination and exposure.
According to the experts included in our Delphi panel, a positive recommendation by WOM is an important trigger for adolescents to visit an Internet-delivered intervention for the first time. Information received through WOM is considered more credible than information from, for example, the mass media [28] since it is both tailored and independent [29]. Furthermore, an attractive interface at first sight is a key issue on the first visit. At ‘first’ sight should be taken literally since previous research has shown that people are very quick (50 ms) in forming an opinion about web page visual appeal [30]. Besides these two factors, perception of the Internet intervention as relevant was the only other factor with regard to a first-time visit that had a median score above the cut-off score. A good place to start would be to identify health-related topics that are of interest to adolescents and on which they might look for information, for example, weight loss instead of fruit consumption or increasing physical activity.
With regard to staying on the intervention, experts indicated a few factors that should be taken into account in the design of Internet-delivered interventions. One of these is that adolescents can associate themselves with the look and feel of the intervention. For example, the intervention should come across as discreet [31] to increase perceived trustworthiness [32]. Another factor mentioned by the experts is the importance of the use of visual materials (e.g. graphs, videos, pictures) and interactive features (e.g. tests, forums, games, etc.). In other words, Internet-delivered interventions should utilize all features provided by the Internet [33, 34]. Interventions should also provide the opportunity for a visitor to stop at any moment and to resume at a later time. A last point related to the design of the intervention is an easy navigation structure.
Time was mentioned as an important factor. An intervention should not take much time to complete entirely, provide brief textual information (i.e. does not involve a lot of reading) and include a short questionnaire to provide tailored feedback. Additionally, it should be possible to use the intervention free of charge [35].
According to the experts, it is important that adolescents, at the end of their first visit, are committed to revisit the intervention (if revisiting the intervention is required to establish desired behaviour change). If the intervention requests adolescents to indicate a time when they want to revisit, an implementation intention could be formed. This is a powerful self-regulatory strategy that promotes the initiation of goal-directed behaviours [36] and increases the likelihood that adolescents will revisit the intervention [37]. Apart from a commitment to revisit the intervention, experts indicated that interventions should provide new content on a regular basis and the chance for adolescents to monitor their progress in changing a behaviour.
All factors with regard to dissemination had a median score of 5 on a seven-point Likert scale. It can therefore be concluded that experts do not agree on a gold standard for successful dissemination of an Internet-delivered intervention.
Besides factors with regard to Internet-delivered interventions, experts also confirmed the importance of factors that are generally applicable to behaviour change interventions. The intervention should provide tailored feedback [38, 39], it should be easy to understand [40] and perceived as personally relevant [41]. Furthermore, the intervention and its use should be perceived as rewarding, enjoyable and attractive.
Implications
Encouragement of WOM recommendations could be facilitated by the provision of ‘tell-a-friend’-services at such interventions, the online equivalent of WOM marketing (also ‘virtual word of mouth’ [42]). Virtual WOM facilitates communication by removing barriers created by spatial distance and time [43]. While early research on computer-mediated communication characterized this as impersonal, more current research has illuminated that communication via the Internet can be just as personal as face-to-face communication [44]. Further research should therefore concern the effectiveness of ‘tell-a-friend’-services to encourage WOM recommendations.
With regard to the design of an Internet-delivered intervention, there are three important issues. First of all, an attractive interface at first sight since people are very quick in judging the interface [30]. Second, further research should be conducted among adolescents to generate more detailed ideas about how to establish an intervention with a look and feel with which adolescents can associate themselves [45, 46]. Third, using formative evaluations, designers should examine if and how adolescents understand the navigation structure of given interventions [47]. Besides focus group interviews, other approaches like in-depth interviews and card sort methods are useful to improve web design with regard to these issues [48, 49].
Another point of interest is the utilization of all features provided by the Internet and how this should be operationalized. Although previous research advocates an option for adolescents to communicate with each other [50], a more thorough investigation is needed. Although online communication can create a more equal communication field because of the anonymity associated with it [51], there are also issues with regard to security and confidentiality [52]. A possible solution would be to monitor these activities but it is not yet clear exactly how this changes the communication process [53]. Future research should focus on possible solutions and their effect on the communication process online.
Since experts do not agree on a gold standard with regard to dissemination, further research is also needed to understand how to maximize the public health impact of Internet-delivered interventions through successful dissemination of those interventions.
Limitations
This Delphi study examined a very broad topic. Through subdivision of this topic into several subtopics, however (dissemination, first visit, stay long enough and revisit), several factors related to each subtopic are identified, as well as the extent to which experts agree on the importance of these factors. Despite the fact that the Delphi method itself leads to convergence, researchers in this study expected that more innovative ideas would be generated during the first round. Nevertheless, the results of this Delphi study are in line with many preconceived notions with regard to Internet-delivered interventions. Another way to generate more ideas would be to ask second-round participants to identify new factors but this is not in line with the Delphi methodology which is the base of this study [25].
Although response rates were non-optimal, they are not exceptional when unsolicited questionnaires are used [54]. Saturation, however, in the answers given to the open-ended questions of the first round indicates that—according to expert opinion—a broader spectrum of factors associated with dissemination of and exposure to Internet-delivered behaviour change interventions is identified, which was the aim of the present study. Furthermore, despite the heterogeneous sample of experts, a high rate of consensus (91%) was found in comparison with other Delphi studies [55, 56]. This implies that experts from different disciplines strongly agree on factors that are associated with dissemination of and exposure to Internet-delivered interventions.
|
Conclusion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
Although this qualitative exploration is only a first step, factors that potentially determine dissemination of and exposure to Internet-delivered interventions aimed at promoting healthy lifestyles in adolescents have been identified. This knowledge is crucial to improve the public health impact of such interventions. The results of this exploration therefore serve as a handle for the formation of further research questions to be tested and answered in research among adolescents.
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
ZonMw—the Netherlands Organization for Health Research and Development (4016.0017).
|
Conflict of interest statement |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
None declared.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionFundingConflict of interest statementReferences |
|---|
1. De Nooijer J, Oenema A, Kloek G, et al. Bevordering van gezond gedrag via internet: nu en in de toekomst [Promotion of Healthy Behavior through the Internet: Now and in the Future] (2005) Maastricht, The Netherlands: Universiteit Maastricht.
2. Evers KE, Cummins CO, Prochaska JO, et al. Online health behavior and disease management programs: are we ready for them? Are they ready for us? J Med Internet Res (2005) 7:e27.[Medline]
3. Bernhardt JM, Hubley J. Health education and the Internet: the beginning of a revolution. Health Educ Res (2001) 16:643–5.
4. De Nooijer J, De Vries NK. Monitoring health risk behavior of Dutch adolescents and the development of health promoting policies and activities: the E-MOVO project. Health Promot Int (2006) 22:5–10.[Web of Science][Medline]
5. Pérez-Rodrigo C, Wind M, Hildonen C, et al. The pro children intervention: applying the intervention mapping protocol to develop a school-based fruit and vegetable promotion programme. Ann Nutr Metab (2005) 49:267–77.[CrossRef][Web of Science][Medline]
6. Griffiths KM, Christensen H. Website quality indicators for consumers. J Med Internet Res (2005) 7:e55.[Medline]
7. Peterson G, Aslani P, Williams KA. How do consumers search for and appraise information on medicines on the Internet? A qualitative study using focus groups. J Med Internet Res (2003) 5:e33.[Medline]
8. Hong T. Contributing factors to the use of health-related websites. J Health Commun (2006) 11:149–65.[CrossRef][Web of Science][Medline]
9. Kelder SH, Perry CL, Klepp KI, et al. Longitudinal tracking of adolescent smoking, physical activity, and food choice behaviors. Am J Public Health (1994) 84:1121–6.
10. Tapscott D. Growing Up Digital: The Rise of the Net Generation (1998) New York: McGraw-Hill.
11. Leung L. Impacts of Net-generation attributes, seductive properties of the Internet, and gratifications-obtained on Internet use. Telemat Inform (2003) 20:107–29.[CrossRef]
12. Fox S. Online Health Search 2006 (2006) Washington, DC: Pew Internet & American Life Project.
13. Gross EF. Adolescent Internet use: what we expect, what teens report. J Appl Dev Psychol (2004) 25:633–49.[CrossRef][Web of Science]
14. Hansen DL, Derry HA, Resnick PJ, et al. Adolescents searching for health information on the Internet: an observational study. J Med Internet Res (2003) 5:e25.[Medline]
15. Van den Eijnden R, Vermulst A. Online communicatie, compulsief internetgebruik en het psychosociale welbevinden van jongeren [Online communication, compulsive Internet use and the psycho-social welfare of young people]. In: Jaarboek ICT en samenleving: De digitale generatie [Yearbook ICT and Society: The Digital Generation]—De Haan J, Van ‘t Hof C, eds. (2006) Amsterdam: Boom. 25–46.
16. Rogers EM. Diffusion of Innovations (2003) New York: The Free Press.
17. McGuire WJ. Attitudes and attitude change. In: The Handbook of Social Psychology—Lindsay M, Aronson E, eds. (1985) New York: Random House. 233–346.
18. Glasgow RE, Vogt TM, Boles SM, et al. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health (1999) 89:1322–7.
19. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med (2007) 32:S119–26.[CrossRef][Web of Science][Medline]
20. CBS. Onderzoek ICT gebruik bij personen [Survey ICT Use by Persons] (2006) Heerlen, The Netherlands: Centraal Bureau voor de Statistiek [Statistics Netherlands].
21. Internet World Stats. Countries with Highest Internet Penetration Rates. Available at: http://www.internetworldstats.com/top25.htm Accessed: 13 March 2007.
22. Zebrack JR, Mitchell JL, Davids SL, et al. Web-based curriculum. A practical and effective strategy for teaching women's health. J Gen Intern Med (2005) 20:68–74.[CrossRef][Web of Science][Medline]
23. Verheijden MW, Jans MP, Hildebrandt VH, et al. Rates and determinants of repeated participation in a web-based behaviour change program for healthy body weight and healthy lifestyle. J Med Internet Res (2007) 9:e1.[Medline]
24. Lenert L, Munoz RF, Stoddard J, et al. Design and pilot evaluation of an Internet smoking cessation program. J Am Med Inform Assoc (2003) 10:16–20.
25. Linstone HA, Turoff M. The Delphi Method: Techniques and Applications (1975) Reading, MA: Addison-Wesley.
26. Green LW, Kreuter MW. Health Promotion Planning: An Educational and Environmental Approach (1999) Mountain View, CA: Mayfield.
27. Jones J, Hunter D. Consensus methods for medical and health services research. Br Med J (1995) 311:376–80.
28. Bickart B, Schindler RM. Internet forum as influential sources of consumer information. J Interact Mark (2001) 1:31–40.
29. Silverman G. The Secrets of Word-of-mouth Marketing (2001) New York: American Management Association.
30. Lindgaard G, Fernandes G, Dudek C, et al. Attention web designers: you have 50 milliseconds to make a good first impression! Behav Inf Technol (2006) 25:115–26.[CrossRef][Web of Science]
31. Flicker S, Goldberg E, Read S, et al. HIV-positive youth's perspectives on the Internet and eHealth. J Med Internet Res (2004) 6:e32.[Medline]
32. Sillence E, Briggs P, Harris P, et al. A framework for understanding trust factors in web-based health advice. Int J Hum Comput Stud (2006) 64:697–713.[CrossRef][Web of Science]
33. Noar SM, Clark A, Cole C, et al. Review of interactive safer sex web sites: practice and potential. Health Commun (2006) 20:233–41.[CrossRef][Web of Science][Medline]
34. Rhee H, Wyatt TH, Wenzel JA. Adolescents with asthma: learning needs and Internet use assessment. Respir Care (2006) 51:1441–9.[Web of Science][Medline]
35. Barner JC, Mason HL, Murray MD. Assessment of asthma patients’ willingness to pay for and give time to an asthma self-management program. Clin Ther (1999) 21:878–94.[CrossRef][Web of Science][Medline]
36. Gollwitzer PM. Goal achievement: the role of intentions. In: European Review of Social Psychology—Stroebe W, Hewstone M, eds. (1993) Chichester, UK: Wiley. 141–85.
37. Gärling T, Fujii S. Structural equation modeling of determinants of planning. Scand J Psychol (2002) 43:1–8.[CrossRef][Web of Science][Medline]
38. Brug J, Glanz K, Van Assema P, et al. The impact of computer-tailored feedback and iterative feedback on fat, fruit, and vegetable intake. Health Educ Behav (1998) 25:517–31.[Abstract]
39. Kreuter MW, Strecher VJ. Do tailored behavior change messages enhance the effectiveness of health risk appraisal: results from a randomized trial. Health Educ Res (1996) 11:97–105.
40. Wilson FL, Brown DL, Stephens-Ferris M. Can easy-to-read immunization information increase knowledge in urban low-income mothers? J Pediatr Nurs (2006) 21:4–12.[CrossRef][Medline]
41. Petty RE, Cacioppo JT. Attitudes and Persuasion: Classic and Contemporary Approaches (1981) Dubuque, IA: William C. Brown.
42. Buttle FA. Word of mouth: understanding and managing referral marketing. J Strateg Mark (1998) 6:241–54.[CrossRef]
43. Rogers EM, Hart WB. New communication technology and the changing nature of conversation. In: The Changing Conversation in America—William E, Paul N, eds. (2002) Thousand Oaks, CA: Sage. 91–117.
44. Norman AT, Russell CA. The pass-along effect: investigating word-of-mouth effects on online survey procedures. J Comput Mediated Commun (2006) 11:1085–103.[CrossRef]
45. Morgan DL, Kreuger RA. The Focusgroup Kit (1998) Thousand Oaks, CA: Sage.
46. Van Assema P, Mesters I, Kok G. Het focusgroep-interview: een stappenplan [The focus group interview: a step-by-step plan]. Tijdschr Soc Gezondheidsz (1992) 70:431–7.
47. Borzekowski DLG, Rickert VI. Adolescents, the Internet, and health: issues of access and content. J Appl Dev Psychol (2001) 22:49–59.[CrossRef][Web of Science]
48. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. Br Med J (2002) 324:573–7.
49. Koyani SJ, Bailey RW, Nall JR. Research-based Web Design and Usability Guidelines (2006) Washington, DC: US Department of Health & Human Services.
50. Ybarra ML, Mitchell KJ, Wolak J, et al. Examining characteristics and associated distress related to Internet harassment: findings from the second Youth Internet Safety survey. Pediatrics (2006) 118:e1169–77.
51. Christopherson KM. The positive and negative implications of anonymity in Internet social interactions: "On the Internet, nobody knows you're a dog". Comput Hum Behav (2007) 23:3038–56.[CrossRef]
52. Rochlen AB, Zack JS, Speyer C. Online therapy: review of relevant definitions, debates, and current empirical support. J Clin Psychol (2004) 60:269–83.[CrossRef][Web of Science][Medline]
53. Smahel D, Subrahmanyam K. "Any girls want to chat press 911": partner selection in monitored and unmonitored teen chat rooms. Cyberpsychol Behav (2007) 10:346–53.[CrossRef][Web of Science][Medline]
54. Swanborn PG. Methoden van sociaal-wetenschappelijk onderzoek [Research Methods in Social Science] (1987) Amsterdam: Boom.
55. De Vet E, Brug J, De Nooijer J, et al. Determinants of forward stage transitions: a Delphi study. Health Educ Res (2005) 20:195–205.
56. Hahn EJ, Toumey CP, Rayens MK, et al. Kentucky legislators’ views on tobacco policy. Am J Prev Med (1999) 16:81–8.[Web of Science][Medline]
Received on April 10, 2007; accepted on November 24, 2007
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  R. Crutzen, J. de Nooijer, W. Brouwer, A. Oenema, J. Brug, and N. K. de Vries A conceptual framework for understanding and improving adolescents' exposure to Internet-delivered interventions Health Promot. Int., June 10, 2009; (2009) dap018v1. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||