Health Promotion International Advance Access originally published online on June 17, 2005
Health Promotion International 2005 20(3):306-315; doi:10.1093/heapro/dai008
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DEBATE |
The development of quality criteria for research: a Finnish approach
Ministry of Social Affairs and Health, PO Box 33, FIN-00023, Helsink, Finland, 1Finnish Centre for Health Promotion, Helsinki, Finland and 2Research Centre for Health Promotion, University of Bergen, Norway
Address for correspondence: Maurice B. Mittelmark, Research Centre for Health Promotion, University of Bergen, Christiesgt. 13 5015 Bergen, Norway E-mail: maurice.mittelmark{at}psyhp.uib.no
 |
SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONMETHODS AND PROCESSRESULTSDISCUSSIONREFERENCES |
|---|
Health promotion research is distinct in some respects from research within its contributing disciplines. A study was conducted in Finland to identify the special characteristics and distinctiveness of health promotion research, to develop a set of specific quality evaluation criteria for health promotion research, and to test the usefulness of such criteria in selecting research applications for funding. A wide range of discussions supplemented a systematic literature review with experts within and outside Finland. The review was conducted with regard to the content of health promotion, its basic principles, as well as theories and models used to steer practical health promotion activities. This resulted in a proposal for quality criteria for health promotion research. Key informants from Finnish universities and institutes evaluated the criteria in three Delphi rounds, and they were suitably revised. The utility of the revised criteria was tested using 31 research proposals. After further revision, the criteria were tested with a further 16 research proposals. Seven health promotion research quality criteria were developed by these methods, as were seven general criteria of research quality, applicable to all social research, whether in the health promotion arena or not. The 14 criteria will undergo continuous revision for improvement, but they are already at this stage thought to be suitable for use by (a) funding agencies in the design of calls for health promotion research proposals, and calls for tenders; (b) researchers who respond to calls for health promotion research proposals and/or calls for tenders; (c) evaluators and proposal review groups; and (d) reviewers and auditors of research progress.
Key words: health promotion; research; quality criteria
|
INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODS AND PROCESSRESULTSDISCUSSIONREFERENCES |
|---|
Health promotion is enjoying a period of expansion worldwide, as realization grows that multifaceted approaches to health improvement are essential. In a parallel development, the volume of research regarding health promotion is rising steadily. A cursory search of the US National Library of Medicine's PubMed database revealed 1189 citations in 1993, which almost doubled to 1966 citations in 2002. As the volume of research increases, those who make decisions concerning the funding and publishing of health promotion research have focused ever more attention on the quality of the work. This is consistent with a trend in the field of increasing emphasis on proving and improving the quality and effectiveness of health promotion (International Union for Health Promotion and Education, 1999
; McQueen, 2001).
The pursuit of quality in health promotion research is complex, because health promotion is a multifaceted arena. Health promotion research's problems, methodologies, standards and philosophical foundations stem from a number of disciplines, including psychology, pedagogy, nursing science, sociology, medicine, economics and theology (Ketonen, 1981; Bunton and MacDonald, 1992; Janhonen et al., 1993; Freudenberg et al., 1995; Macdonald, 2003a; Macdonald, 2003b). In concert with this, the aims and activities of health promotion, and also of health promotion studies, are varied and dependent on viewpoints (Naidoo and Wills, 1998). For example, the positivist tradition stresses the medical and behavioural aspects of health, while the critical-emancipatory perspective centres on empowerment processes.
In this light, it is not surprising that there are no widely acknowledged and accepted quality standards for health promotion research, aside from the standards of the various contributing disciplines. As a consequence, judges of the quality of health promotion research—scholars, funding agencies, journal editors and journal reviewers—apply the values and standards of their own disciplines, or some combination of disciplinary standards (National Advisory Board on Research Ethics, 2002; Valovirta, 2002). A practical consequence is that the criteria for evaluating the quality of health promotion research are not often specified.
Yet, health promotion research is distinct in some respects from research within its contributing disciplines. It is often interdisciplinary work, having the task of combining different scientific disciplines in a way that is in synchrony with health promotion aims and values. Consider, for example, the implications for research of health promotion in enabling people to gain control over their own health (WHO, 1986; Vertio, 2003). One of the ways this may be accomplished is to create opportunities for making choices and changes that produce better health, but also creating conditions that maintain positive factors, such as social capital. The overarching strategy of health promotion is to ensure an environment that supports health, adequate information, life skills and opportunities for making healthy choices, as well as health services that include health promotion among their priorities. Further, health promotion aspires to work with people in a participatory manner. This way of working is as applicable to health promotion research as it is to professional practice.
These issues have received attention in Finland, which in recent years invested about seven million Euros annually in health promotion activities. The funds are generated, among other ways, through levies authorized by the Act on Measures to Reduce Tobacco Smoking (Act on Measures to Reduce Tobacco Smoking, 1976) and by the Act on Measures to Reduce the Use of Alcohol (Act on Measures to Reduce the Use of Alcohol, 1982). Funds are allocated annually to development and education projects, as well as to research. No funds are allocated to commercial projects, or to statutory or standard functions of municipalities, organizations or other actors. More than 300 applications for all types of health promotion projects have been received annually in recent years, about half of which receive funding. A sixth of these are proposals for research.
The review and funding processes in the first years of the programme were handicapped because no formal criteria for assessing the quality of health promotion research applications had yet been articulated. As a result, review panels tended to place the greatest emphasis on classical quantitative study design issues, such as adequacy of the sampling plan and sample size, the quality of measurements, the degree of control, the extent of internal and external validity and reliability, and so on. This situation generated discussion about the possibility that innovative and highly relevant research proposals might not receive sufficient attention, or a high enough priority, because of departures from 'traditional' quality standards for quantitative public health research. There was concern that reviewers and funding decision makers needed the guidance of uniform and specific evaluation criteria, to increase the likelihood that the best quality health promotion research proposals would be prioritized. Disciplinary preferences and biases were acknowledged to have influence on the decision-making processes, and understandably so. It was hoped that these could be harnessed, with the help of standard quality criteria, such that quality and innovation would be recognized and rewarded.
With that background, the present project was launched. The aims were to: (1) define the special characteristics and differences/divergences of health promotion research compared to research in contributing disciplines, (2) develop a set of specific evaluation criteria for health promotion research, (3) define the values and criteria that steer research in general, and (4) test the usefulness of such criteria in selecting research applications for funding. Several principles were articulated to guide the development of the criteria. The criteria were to reflect the appreciation of the distinctive character of health promotion research, and the constructed nature of much of the knowledge in social research. The criteria were intended, also, to provide a framework to help identify what was health promotion research, and what was not. The criteria were to serve as tools to help evaluators produce the least biased evaluations possible. They were to include subsets appropriate to the various disciplines contributing to health promotion research.
|
METHODS AND PROCESS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODS AND PROCESSRESULTSDISCUSSIONREFERENCES |
|---|
The first aim was addressed by a systematic literature review supplemented by a wide range of discussions and consultations organized by the Finnish Centre for Health Promotion Research, involving Finnish and other European health promotion experts. The review was conducted with regard to the content of health promotion, its basic principles, as well as theories and models used to steer practical health promotion activities. It was intended that the results would feed into the methodology for the second aim, addressed in five stages. The first stage was the development of the criteria by the Delphi method with health promotion experts as the participants. In the second stage, the criteria were examined and discussed by the 30-member Scientific Advisory Committee of the Finnish Centre for Health Promotion Research, the Netherlands Institute for Health Promotion and Disease Prevention, and at an international seminar held in Helsinki. The third stage tested the criteria in practice. The fourth stage consisted of another round of discussions in the Finnish Scientific Advisory Committee. Finally, in the fifth stage, the criteria were revised.
In the first stage, the starting point was a list of 24 initial evaluation criteria (available from the authors) compiled by a specially constituted Finnish working group (Koskinen-Ollonqvist, 1999). The criteria were then evaluated in three Delphi rounds (Linstone, 1978), involving 18 health promotion experts from Finnish universities and institutes. During the first Delphi round, the experts were asked to evaluate and comment on the criteria for health promotion research still in a draft stage, and to make additional suggestions for criteria. These were categorized into five clusters: (1) framework criteria, having to do with the conceptual framework around which a study is built, (2) structural criteria, having to do with the practicality of completing a study as designed, (3) process criteria, having to do with the adequacy of a study's underlying logic model, (4) results criteria, having to do with how a study's effects are detectable and verifiable and (5) application criteria, having to do with the utility of the study's intervention approach in typical practice settings. The first round of the Delphi process yielded 21 criteria, reduced to 15 criteria by the end of the third round (Koskinen-Ollonqvist, 1999).
In the second stage, the Scientific Advisory Committee revised the 15 criteria into 14 criteria. In the third stage, the Finnish National Research and Development Centre for Welfare and Health (Stakes) examined the utility of the criteria using the 31 research proposals submitted for funding to the Ministry of Social Affairs and Health in 2003. In addition, the criteria were examined using all 16 health promotion research projects that had received funding from the Ministry of Social Affairs and Health in 2001. Each research application was examined by two expert raters working independently, who provided ratings of the degree to which the application met each criterion (‘criterion completely met’ to ‘criterion not met at all’). A third rater was engaged in cases where the first two raters were in disagreement. The evaluation criteria were fine-tuned based on the work of these raters.
At the fourth stage, the criteria were reviewed again by the Scientific Advisory Committee, and at the fifth stage, revised criteria were again tested in practice, using health promotion research projects that had received funding from the Ministry of Social Affairs and Health in 2002.
|
RESULTS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODS AND PROCESSRESULTSDISCUSSIONREFERENCES |
|---|
Using the methodology described, which involved a combination of literature review and consultation with experts, it was concluded that health promotion research is, indeed, distinguishable from research in the contributing disciplines, on three counts. First, health promotion research is research on action. Second, disciplinary research is guild-like, with more or less distinct boundaries regarding subject matter, theories, methods and the qualifications of its practitioners. Health promotion research is open and encouraging to multidisciplines, but does not go so far as to reject intra-disciplinary research. Third, health promotion research makes explicit efforts to reflect the core values of equity, participation and empowerment in decisions about how the research is conducted. Disciplinary research may be conducted in this manner as well, but this is an essential feature of health promotion research. These distinctive features of health promotion research are taken up in more detail below.
The distinctiveness of health promotion research relates in part to the action concept of health promotion, in which three types of action may be undertaken. Action can aim at: (1) making changes for better health, including change by individuals regarding behaviour, attitudes, etc.; (2) creating possibilities for change for better health, including environment change to support healthy choices; and (3) making changes at the individual and the environmental levels that are needed so people can maintain the good health they already enjoy. These types of action have in common the aim of assisting people to gain control over their own health. We refer to them collectively as control-enhancing actions, and research on such actions is control-enhancing research. Importantly, control-enhancing research may be entirely at the macro environmental level, such as research on healthy public policy. It may also involve research at the group and individual levels, such as health education interventions in schools.
Our emphasis that health promotion research is control-enhancing research is meant to make a distinction between research about action and descriptive research. Purely descriptive research, however relevant to health promotion, is thus distinguishable from health promotion research. Consistent with this distinction, the models and theories that are most relevant for health promotion research emphasis control-enhancing action. They assist the researcher to determine appropriate participants, interventions (micro to macro), processes and outcomes (Rotter, 1966; Bowlby, 1969; Antonovsky, 1987; Green and Kreuter, 1991; Glanz et al., 1997; Bartholomew et al., 2001; Rootman et al., 2001). The distinction between control-enhancing research and purely descriptive research blurs where the two types are very near each other. It may be difficult to classify research on the identification of protective factors, for example, using this distinction. Nevertheless, the idea of control-enhancement appears useful to distinguish health promotion research from purely descriptive research in many cases. It might be argued that because the distinction is subject to debate, it is not defensible. However, when the intention is to encourage control-enhancing research, emphasizing the action element sends the message that purely descriptive research is not prioritized.
The distinctiveness of health promotion research relies, as well, on the value placed on multidiscipline programmes of work, and the field's openness to the discourses and cultures of the various contributing disciplines (Macdonald, 2003a; Macdonald, 2003b). The methodological and theoretical foundation of health promotion makes simultaneous use of several different scientific disciplines, such as theology, biomedicine, behavioural sciences, pedagogy, nursing science, social sciences and economics. Table 1 summarizes this point through contrasts and comparisons of how various contributing disciplines conceptualize ideas about health, and the foundation, definition, content and goals of health promotion. It is an important point that a high quality health promotion research project has just a few well-defined questions, and is focused conceptually and methodologically. It is highly unlikely, therefore, for any single research study to reflect the distinctive multidisciplinary character described above. It is in the breadth of the projects within health promotion research portfolios that this multidisciplinary character should be manifest.
|
The third feature distinguishing health promotion research from research in contributing fields is the degree to which values related to inclusion and participation are explicitly woven into the research. Within the health and social sciences, health promotion is among the arenas most taken with the idea that science is not, should not and cannot be value free. To the contrary, there is insistence that values should be carefully considered and examined, and that the values of the practitioner should guide the researcher as well. Accordingly, health promotion practice and research emphasize the importance of working with people in respectful partnership, in contrast to a style in which ‘experts’ know best. In designing the best health promotion research, the scholar seeks approaches and study designs that exhibit real regard for the dignity of all participants. Related values are those of equality, sustainable development, transparency, empowerment and participation in all stages of the research. Similarly, appreciation of peoples' wisdom and experience, listening to people's expressions about needs and solutions, and respect for the significance of the community in people's lives, all have a special significance in health promotion research.
These conceptualizations about health promotion research's distinctiveness—related to enhancing peoples' control over their health, multidiscipline and the centrality of values—were applied in shaping a set of seven health promotion quality criteria. The project also identified seven general criteria of research quality, applicable to all social research, whether in the health promotion arena or not. The 14 criteria were intended as a guide to:
- Funding agencies in the design of calls for health promotion research proposals, and calls for tenders.
- Researchers who respond to calls for health promotion research proposals and/or calls for tenders.
- Evaluators and proposal review groups.
- Reviewers and auditors of research progress.
- Reviewers of completed research projects.
- Evaluators of research programmes consisting of a series of completed research projects.
Health promotion relevance. The rationale for the research addresses explicitly individual, social or societal level priorities for health promotion research as set forth by relevant policy documents, calls for proposals and calls for tenders, and/or the state-of-science in the relevant health promotion research arena.
Health promotion values. The research methodology addresses explicitly how health promotion values are incorporated in the research, including especially the values on citizen participation, partnership, fully authorized participation, open communication, sustainability, and empowerment.
Health promotion innovation. The research is innovative and distinctive, addressing explicitly its intentions to clarify and/or strengthen an important aspect of health promotion practice.
Health promotion discourse. The study question(s) are framed in a manner consistent with, and flowing from, clearly stated theory/model/rationale with a high degree of relevance to health promotion discourse.
Health promotion practice. The research has practical relevance for health promotion activities, and makes explicit reference to the arena(s) of practice to which it applies.
Health promotion action. The research addresses explicitly action for health promotion, including action for change, and/or action to create opportunities for choice, and/or action for maintenance of change/choice already achieved, at any level or combination of levels from the individual to the societal.
Health promotion context. The research demonstrates appreciation for the manner and degree to which it is embedded in a larger health promotion context, by reference to critical aspects of the problem that are not objects of study, e.g. systems, ecologies and/or processes of which the object of study is a part.
The seven general quality criteria are as follows.
Scientific quality. Within a specified disciplinary or multidisciplinary framework, the research is scientifically justified, the approach is sound, and there is coherence between the research problem(s), research question(s) and research methodology.
Defined scope. All key elements of the research are defined and delimited so that the scope of the phenomena under study are explicit [discipline(s), theory(ies), model(s), methods, analyses, interpretations].
Anticipated outcomes. The long-term consequences of the research are considered/estimated, including possible consequences related to health, welfare, social, economic, scientific and technical spheres, and including unexpected outcomes.
Operationalization. There is evidence of careful consideration of the approach to the research, the research strategy is operationalized, and rationales are presented for decisions involving choices among alternative approaches and alternative operationalizations.
Feasibility. Completion of the research is feasible within the limits of the time and resources described, and within the limits of the training and experience of the responsible researcher(s).
Process evaluation. There is a feasible and adequate plan for the regular monitoring, recording and analysis of research processes and activities, sufficient to permit open/independent inspection of the course of the research.
Documentation and dissemination. For completed research, the work and impacts/outcomes/outputs are thoroughly documented, the record of this is conserved and readily available for inspection and the work is disseminated via normal scholarly channels including written and oral communications.
Testing the criteria
At the various stages in their development, the criteria were tested using existing research applications and research final reports, as described in the Methods. The form and wording of the criteria were modified several times on the basis of these tests and advice from the consultative groups involved in the project. In a final test, 16 research projects (a mix of proposals and final reports) administered by the Ministry of Social Affairs and Health in 2002 were systematically evaluated using the 14 quality criteria in a form close to that described above. For each criterion, each project was rated on a four-point scale: criterion is not fulfilled at all; criterion is fulfilled to some extent; criterion is fulfilled completely; criterion cannot be assessed. The last rating was used when the report contained no reference at all to the subject of a criterion. Here, the first and last categories are combined, since the writers of the proposals and reports were not aware at the time of writing that these criteria would be used to evaluate their proposals in this test.
Figure 1 shows that the general research criteria related to project feasibility was the best satisfied among all 14 criteria. Among the health promotion quality criteria, two were at least partly met by all 16 projects. The criterion with the lowest level of fulfillment was that having to do with the framing of study questions in a manner flowing from health promotion discourse.
|
|
DISCUSSION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODS AND PROCESSRESULTSDISCUSSIONREFERENCES |
|---|
The results of the development process together with the information from the review of the literature show that a range of disciplines contributes to health promotion research. The paradigm of health promotion research is based on information, methodologies and research models originated from several different scientific disciplines. The paradigm of health promotion research entails a conscious creation and development of a culture of health promotion. The arena of health promotion research is fairly new, and its methods and paradigms are adapted from several different fields of science and research.
A main outcome of this project was the recognition that research funding programmes that are intended to support health promotion research should adopt explicitly quality criteria that will illuminate the best research in the arena, and not set (or signal) inappropriate standards for health promotion research. For example, proposals for descriptive epidemiological studies should not displace good health promotion research proposals merely on the basis of ‘superior’ methodological sophistication. Under the principle that study method should be dictated by study question, the action element in all health promotion research may require relatively ‘messy’ study designs that lack the quality trappings of classical public health investigations. Thus, if quality criteria for health promotion research are not made available to researchers and reviewers alike, even research funding programmes that aim explicitly to support health promotion research may end up funding other kinds of research instead.
The National Advisory Board on Research Ethics in Finland holds that each researcher and member of a research team is responsible for following good scientific practice, but also each research team as a whole, director of a research unit and the management of an organization doing research have responsibility for the quality of the science they help produce. Purposeful strengthening of good scientific practice is important in all scientific organizations, and functional evaluation criteria that are relevant to the aims of the research (or research programme) will assist all parties to agree on the principles of good science, given that aims are specified in advance.
In the Finnish exercise that has been described here, health promotion quality research criteria have been developed in a systematic process involving several networks of experts in the arena of health promotion research. Further, and continuous, development of the criteria is viewed as essential, as dialogue and debate among funders, researchers and evaluators leads to new insights about the essential nature of health promotion research. Thus, the criteria presented in this paper represent the current state of an evolving process, and one that will surely require adaptation, should similar efforts be undertaken outside Finland.
In the course of the work reported here, the impression was strengthened that describing the distinctiveness of health promotion is necessary for understanding the current status of research in this area and the conditions for its further development. The results of this work suggest that there is value to be gained from deep consideration of the principles and foundations of health promotion research and the methods appropriate to it. For the authors, this work highlighted the need to also consider quality issues for research aimed at the development of health promotion theory, and research into the preconditions for health promotion.
The testing of the quality criteria that we report here did not allow us to observe their functionality in all the different situations they are intended for. Continuous testing is needed, and planned, for this purpose. It is also uncertain if the approach to quality improvement described in this paper has a high degree of applicability in settings outside Finland, or if results would be similar in such research that might be undertaken elsewhere. What does seem clear is that this project at least demonstrates the feasibility of quality improvement processes in the health promotion research arena. It is hoped that means can be developed to engage the global health promotion research community in a dialogue about these matters. We can only assume that similar efforts are being mounted elsewhere, and therefore, that a discussion forum is needed where the dialogue on health promotion research quality criteria can be broadened.
|
ACKNOWLEDGEMENTS |
|---|
Financial support for the development process of evaluating health promotion research and the resulting set of evaluating criteria were performed from the Finnish Ministry of Social Affairs and Health.
|
REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODS AND PROCESSRESULTSDISCUSSIONREFERENCES |
|---|
Act on Measures to Reduce Tobacco Smoking, No. 693/1976.
Act on Measures to Reduce the Use of Alcohol, No. 828/1982.
Antonovsky, A. (1987) Unravelling the Mystery of Health: How People Manage Stress and Stay Well. Jossey-Bass, San Francisco, CA.
Argyris, C. (1992) On Organisational Learning. Blackwell, Oxford.
Bartholomew, L. K., Parcel, G. S., Kok, G. and Gottlieb, N. H. (2001) Intervention Mapping. Designing Theory-based and Evidence-based Health Promotion Programs. Mayfield, Mountain View.
Bowlby, J. (1969) Attachment and Loss. Vol. 1. Attachment. Basic Books, New York.
Bunton, R. and MacDonald, G (1992) Health Promotion. Disciplines and Diversity. Routledge, London.
Burrell, G. and Morgan, G. (1979) Sociological Paradigms and Organisational Analysis. Ashgate Publishing Company, Vermont.
Catford, J. and Nutbeam, D. (1984) Towards a definition of health education and health promotion. Health Education Journal, 38, 2–3.
Da Silva, A. and Eriksson, K. (1989) Vårdteologi som vårdvetenskapens deldisciplin, I Eriksson K. and Barbosa da Silva A. (red.) Vårdteologi. Vårdforskningar 3/1991. Institutionen för vårdvetenskap, Åbo Akademi. Vasa. 204–221.
Engeström, Y. (1987) Uusi oppimisnäkemys ja kehittyvä työntutkimus. Aikuiskasvatus, 4, 170–175.
Flynn, B. C., Ray, D. W. and Rider, M. S. (1994) Empowering communities: action research through healthy cities. Health Education Quartely, 21, 395–405.
Fogel, K. J. (1994) Teorianmuodostuksen termistöä. Teoksessa: Marriner-Tomey. Hoitotyön teoreetikot ja heidän työnsä. SHKS. Vammala: Vammalan kirjapaino Oy.
Freire, P. (1975) Utbildning för Befrielse. Gummessons Bokförlag, Stockholm.
Freudenberg, N., Eng, E., Flay, B., Parcel, G., Rogers, T. and Wallerstein, N. (1995) Strengthening individual and community capacity to prevent disease and promote health: in search of relevant theories and principles. Health Education Quarterly, 22, 290–306.[Web of Science][Medline]
Glanz, K., Lewis, F. M. and Rimer, B. K. (eds) (1997) Health Behaviour and Health Education, Theory, Research and Practise, 2nd edition. Jossey-Bass Inc., California, San Francisco.
Green, L. W. and Kreuter, M. W. (1991) Health Promotion Planning. An Educational and Ecological Approach, 3rd edition. Mayfield Publishing Company, Mountain View, California.
International Union for Health Promotion and Education (1999). The Evidence of Health Promotion Effectiveness: Shaping Public Health in a New Europe. Part Two. ECSC-EC-EAEC, Brussels, Luxembourg.
Janhonen, S., Kuokkanen, R., Kääriäinen, S. and Matikka, S.-L. (1993) Kehittyvä hoitotyö. WSOY Porvoo: WSOY:n graafiset laitokset.
Karisto, A., Lahelma, E. and Rahkonen, O. (eds) (1992) Terveyssosiologia. WSOY. Juva: WSOY:n graafiset painolaitokset.
Ketonen, O. (1981) Ihmisenä Olemisesta. Otava, Keuruu.
Koskinen-Ollonqvist, P. (1999) Developing quality criteria for health promotion programmes in Finland. In Best Practices. A Selection of papers on Quality and Effectiveness in Health Promotion. May 16–19, 1999. Helsinki, Finland and Tallinn, Estonia. Presented at the 4th European IUPHE Conference on Effectiveness and Quality of Health Promotion. Tallinn, April, 2000.
Krause, K. and Kiikkala, I. (1996) Hoitotieteellisen tutkimuksen peruskysymyksiä. Hygienia. Kirjayhtymä Oy. Tampere:Tammer-Paino Oy.
Labonte, R. and Robertson, A. (1996) Delivering the goods, showing our stuff: the case for a constructivist paradigm for health promotion research and practice. Health Education Quarterly, 23, 431–447.[Web of Science][Medline]
Lahtinen, E. (1996) Työikäisten terveyden edistäminen avoterveydenhuollossa. Kelan ja Turun terveyskeskuksen kehittämän terveyden edistämisohjelman toteutus ja tulokset. Kansaneläkelaitos. Sosiaali-ja terveysturvan tutkimuksia 16. Turku.
Leino, A.-l. and Leino, J. (1998) Kasvatustieteen Perusteet. Kirjayhtymä, Helsinki.
Leontjev, A. N. (1977) Toiminta, Tietoisuus, Persoonallisuus. Kansankulttuuri. Helsinki.
Linstone, H. (1978) The Delphi technique. In Fowles, J. ed. Handbook of Futures Research. Greenwood Press, London.
Macdonald, G. (2000) A new evidence framework for health promotion practice. Health Education Journal, 59, 3–11.
Macdonald, G. (2003a) Health promotion research. Quality criteria for health promotion research Seminar. Ministry of Social Affairs and Health, Helsinki, 25 April 2003.
Macdonald, G. (2003b) Realising the imagination; opportunities for alternative evidence base for health promotion. Nordic Health Promotion Research Conference. Västerås, Sweden, 10 June 2003.
McQueen, D. V. (2001) Strengthening the evidence base for health promotion. Health Promotion International, 16, 261–268.
Naidoo and Wills (1998) Health Promotion, Foundation for Practice. Bailliêre Tindall, Canada.
National Advisory Board on Research Ethics (2002) Hyvä Tieteellinen Käytäntö ja sen Loukkausten Käsitteleminen. Edita Prisma Oy, Helsinki.
Niinistö, K. (1981) Inhimillistä toimintaatarkasteleviin tieteisiin ja erityisesti kasvatustieteelliseen tutkimukseen soveltuvat toiminnallisen paradigman mukaiset tutkimusmallit ja—menetelmät, niiden filosofinen tausta ja valintaan vaikuttavat tekijät. Turun yliopiston kasvatustieteen laitos. Julkaisusarja, A:85.
Rauhala, L. (1985) Ihmiskäsitys empiirisessä ihmistutkimuksessa. Suomen Lääkärilehti, 1–2, 54–58.
Rauste-von Wright, M and von Wright, J. (1998) Oppiminen ja koulutus. WSOY.Juva:WSOY, Kirjapainoyksikkö.
Rootman, I. M., Goodstadt, B., Hyndman, D.V., McQueen, L., Potvin, J. Springett and Ziglio E. (eds) (2001) Evaluation in Health Promotion. Principles and Perspectives. WHO Regional Publications, European Series, No. 92, Copenhagen.
Rotter, J.-B. (1966) Generalized expectancies for internal versus external control of reinforcement. Psychological Monographs, 80, (Whole No. 609).
Schweiker, R. S. (1982) Disease prevention and health promotion. Journal of Medical Education, 57, 15–19.[Web of Science][Medline]
Tones, K. and Tilford, S. (1994) Health promotion: effectiveness, efficiency and aquity. Chapman & Hall, London.
Valovirta, V. (2002) Tieteenalojen arviointi Suomessa– Kehittämisen vai vakuuttamisen väline. Painopörssi Oy, Helsinki.
Vertio, H. (2003) Terveyden Edistäminen. Kustannusosakeyhtiö Tammi, Helsinki.
Weckroth, K. (1998) Toiminnan Psykologia. Hanki ja Jää, Helsinki.
WHO (1986) Ottawa Charter for Health Promotion. First International Conference on Health Promotion. Ottawa, Canada, November 17–21, Geneva.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. T. Wright, B. Roche, H. von Unger, M. Block, and B. Gardner A call for an International Collaboration on Participatory Research for Health Health Promot. Int., October 23, 2009; (2009) dap043v1. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. B. Mittelmark Setting an ethical agenda for health promotion Health Promot. Int., March 1, 2008; 23(1): 78 - 85. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. Kwak, S. P. J. Kremers, M. A. van Baak, and J. Brug Participation rates in worksite-based intervention studies: health promotion context as a crucial quality criterion Health Promot. Int., March 1, 2006; 21(1): 66 - 69. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. de Leeuw Who gets what: politics, evidence, and health promotion Health Promot. Int., September 1, 2005; 20(3): 211 - 212. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||