Health Promotion International Advance Access originally published online on September 16, 2005
Health Promotion International 2005 20(4):367-374; doi:10.1093/heapro/dai022
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
PERSPECTIVES |
Criteria for the systematic review of health promotion and public health interventions
1Cochrane Health Promotion and Public Health Field, Australia, 2School of Health & Social Development, Deakin University, Australia and 3L. Anderson (Centers for Disease Control and Prevention, USA); R. Bailie (Menzies School of Health Research, Institute of Advanced Studies, Charles Darwin University, Australia); G. Brunton [Evidence for Policy and Practice Information and Co-ordinating (EPPI) Centre, Institute of Education, University of London, UK]; P. Hawe (University of Calgary, Canada); E. Kristjansson (University of Ottawa, Canada); L. Naccarella (University of Melbourne, Australia); S. Norris (Agency for Healthcare Research and Quality, USA); S. Oliver (EPPI-Centre, Institute of Education, University of London, UK); M. Petticrew (MRC Social and Public Health Sciences Unit, UK); E. Pienaar (South African Cochrane Centre); J. Popay (Lancaster University, UK); H. Roberts (City University, UK); W. Rogers (Flinders University, Australia); J. Shepherd (University of Southampton, UK); A. Sowden (Centre for Reviews and Dissemination, University of York, UK); H. Thomas (McMaster University and the Effective Public Health Practice Project, Canada)
Address for correspondence: N. Jackson Cochrane Health Promotion and Public Health Field Australia E-mail: nickijackson{at}hotmail.com E. Waters School of Public Health Deakin University Australia E-mail: elizabeth.waters{at}deakin.edu.au
 |
SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONENSURING REVIEWS MEET THE...ISSUES TO ADDRESS WHEN...CONCLUSIONREFERENCES |
|---|
Systematic reviews of public health interventions are fraught with challenges. Complexity is inherent; this may be due to multi-component interventions, diverse study populations, multiple outcomes measured, mixed study designs utilized and the effect of context on intervention design, implementation and effectiveness. For policy makers and practitioners to use systematic reviews to implement effective public health programmes, systematic reviews must include this information, which seeks to answer the questions posed by decision makers, including recipients of programmes. This necessitates expanding the traditional evaluation of evidence to incorporate the assessment of theory, integrity of interventions, context and sustainability of the interventions and outcomes. Unfortunately however, the critical information required for judging both the quality of a public health intervention and whether or not an intervention is worthwhile or replicable is missing from most public health intervention studies. When the raw material is not available in primary studies the systematic review process becomes even more challenging. Systematic reviews, which highlight these critical gaps, may act to encourage better reporting in primary studies. This paper provides recommendations to reviewers on the issues to address within a public health systematic review and, indirectly, provides advice to researchers on the reporting requirements of primary studies for the production of high quality systematic reviews.
Key words: review; systematic; effectiveness; Public Health intervention
|
INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONENSURING REVIEWS MEET THE...ISSUES TO ADDRESS WHEN...CONCLUSIONREFERENCES |
|---|
The difficulty of conducting systematic reviews of public health interventions directly reflects the complexity of the interventions reviewed and the subsequent determination of effectiveness. Some of the key challenges in the public health field include the focus on populations rather than individuals, multi-component interventions, qualitative as well as quantitative approaches, an emphasis on processes of implementation, and the complexity and long-term nature of the interventions and outcomes (Jackson et al., 2001
). The challenges of reviewing the evidence in public health have been discussed previously (Jackson et al., 2004). Despite the many methodological challenges, many organizations are systematically reviewing the health promotion and public health literature and consequently contributing to the methodological knowledge of how systematic reviews should be conducted.
Users of health promotion and public health reviews have raised a number of criticisms of systematic reviews relating to the methodological criteria for inclusion of studies, insufficient attention to the quality of the interventions reviewed, and a lack of assessment of the theoretical foundation of the intervention and processes of implementation (Tilford, 2000). Furthermore, reviews have been criticized for their focus on limited individual health education interventions rather than complex environmental or structural interventions and the poor coverage of issues relating to the social determinants of health (Tilford, 2000). For reviews to be useful and relevant, reviewers need to address these concerns and therefore include information, which seeks to answer the broad questions posed by decision makers.
The appropriateness of applying systematic review methods to public health has been questioned (Tilford, 2000; Jackson et al., 2001). Criteria for evaluating public health interventions have been published (Rychetnik et al., 2002), with the authors arguing that there are important aspects of evidence related to public health interventions that are not covered by the established criteria for evaluating medically oriented evidence. This paper expands upon this article, and our previous article (Jackson et al., 2004) discussing the challenges of reviews in this area, by providing criteria to enable reviewers to produce high quality systematic reviews that meet the needs of users. These criteria or recommendations were formulated by a taskforce of international public health professionals (The Guidelines for Systematic Reviews in Health Promotion and Public Health Taskforce) experienced in the review or evaluation of public health interventions. The full-text version of the guidelines can be found on the Cochrane Health Promotion and Public Health Field website (http://www.vichealth.vic.gov.au/cochrane). The guidelines will be updated regularly and contribution will be sought from continents in which there are currently few contributors.
|
ENSURING REVIEWS MEET THE NEEDS OF THE USERS |
|---|
|
TOPSUMMARYINTRODUCTIONENSURING REVIEWS MEET THE...ISSUES TO ADDRESS WHEN...CONCLUSIONREFERENCES |
|---|
Systematic reviews are likely to be more relevant to the end-user and of higher quality if they are informed by advice from people with a range of experiences, in terms of both the topic and the methodology [Centre for Reviews and Dissemination (CRD), 2001
; Rees et al., 2004; Thomas et al., 2004a]. Choosing which interventions, outcomes, settings and populations to review relies on knowledge of current policy, practice and the views of the people targeted by the interventions.
Reviewers should establish an Advisory Group whose members are familiar with the topic area. It is usually useful to include the perspectives of policy makers, funders, practitioners and potential users/recipients. Given that public health questions of effectiveness are often broad in nature, a wide range of members will often be required. It is also important to incorporate the needs and views of resource poor countries in the review process to ensure that where possible (and appropriate) the interventions and outcomes reviewed have international relevance (Richards, 2004). The tasks of the Advisory Group may include:
- making and refining decisions about the scope of the review, i.e. which interventions, populations (including subgroups) and outcomes will be included in the review. This decision should be based on the intended use of results, the complexity of the review, and the time available;
- providing important background material that elucidates the issues from different perspectives;
- helping to interpret the findings of the review; and
- disseminating the review to relevant groups.
|
ISSUES TO ADDRESS WHEN REVIEWING PUBLIC HEALTH INTERVENTIONS |
|---|
|
TOPSUMMARYINTRODUCTIONENSURING REVIEWS MEET THE...ISSUES TO ADDRESS WHEN...CONCLUSIONREFERENCES |
|---|
Inclusion of study designs
The field of public health is characterized by a high degree of methodological pluralism (Nutbeam, 1999
), using methods, which include randomized controlled trials (RCTs), controlled before and after studies, uncontrolled studies, interrupted time series (ITS) designs, surveys and qualitative studies. The study designs to be included in a public health review should be dictated by the interventions being reviewed (methodological appropriateness), and not vice versa (Nutbeam, 1999; Petticrew and Roberts, in press). Although RCTs should be prioritized where possible for inclusion in systematic reviews of effectiveness, we recommend that other study designs such as non-randomized controlled before and after studies and ITS designs should also be considered where RCT evidence is lacking. Comparisons with historical controls or national trends should only be included when this is the sole type of evidence available (e.g. in reviews investigating the effectiveness of policies) and accompanied by an acknowledgement such evidence is necessarily weaker.
Qualitative research has an important role in systematic reviews to answer questions that go beyond effectiveness, including appropriateness of interventions to participants (Popay et al., 1998) and the factors that promote and/or impede the implementation of the intervention (CRD, 2001). The answers to these questions are seldom found in quantitative studies of effectiveness. Evidence on the factors that impinge on the implementation of interventions may be particularly important in the context of complex multifaceted public health interventions. Moreover, qualitative research can also contribute to reviews through assisting to identify or refine the questions to be addressed in a systematic review (Berkwits, 1998; CRD, 2001). Despite the importance of qualitative research it has tended to be omitted in systematic reviews (Dixon-Woods et al., 2001).
Searching for public health literature
Searching for public health studies can be very difficult and time-consuming. Reviewers should allocate sufficient time (up to 5 days) to develop, test and re-test the review search strategy to ensure that it captures all of the relevant studies. A number of electronic databases should be searched to cover the range of disciplines relevant to the topic area. A list of relevant databases is included in the full-text of the guidelines (http://www.vichealth.vic.gov.au/cochrane). We recommend that reviewers use care in applying study design filters to the search strategy if studies other than RCTs are to be included, as there are no validated filters for non-randomized studies.
Given that public health literature is likely to be widely dispersed (Peersman and Oakley, 2001) it is important that other retrieval methods, beyond electronic database searching, are utilized. This includes searching the Internet to find organizations that may hold studies, accessing government or public health service provision level reports, and asking experts in the area to identify studies that may have been missed. Hand-searching generalist public health journals or journals related to the topic may also result in additional citations to include in the review. Further information is included in the full-text guidelines.
Quality assessment
We recommend reviewers use the Quality Assessment Tool for Quantitative Studies (http://www.city.hamilton.on.ca/phcs/EPHPP/), developed by the Effective Public Health Practice Project, Canada. This tool was judged suitable to be used in systematic reviews of effectiveness (Deeks et al., 2003), and can be used for RCTs, quasi-experimental studies and uncontrolled studies. Content and construct validity have been established (Thomas et al., unpublished).
Particular sources of threats to the validity of public health studies include the data collection methods used, especially where outcomes are subjective (e.g. reported behaviour), and the potential for the control group to become ‘contaminated’. The recommended quality assessment tool assesses the following quality criteria: selection bias, study design, confounders, blinding, data collection methods, withdrawals and dropouts, intervention integrity, and statistical analyses. Information on ITS designs can be found on the Effective Practice and Organisation of Care website (www.epoc.uottawa.ca).
The quality appraisal of qualitative research is a much-discussed issue in relation to the role of qualitative research in systematic reviews (Popay et al., 1998). A number of checklists can be of use to the reviewer, including a framework for assessing qualitative research (Spencer et al., 2003) and the Critical Appraisal Skills Programme (CASP) appraisal checklist (CASP, 2004). The Cochrane Qualitative Research Methods Group (http://mysite.freeserve.com/Cochrane_Qual_Method/qmmodule.htm) also has a number of tools available to reviewers. We await the results of a number of methodological projects to further guide the appraisal of qualitative research.
In addition, reviewers may also assess whether the intervention itself meets quality standards. For example, in a review of educational interventions for chronic conditions (Mullen et al., 1985) the authors rated studies according to their methodological strength and the adherence to educational principles in their intervention.
Theoretical frameworks for interventions
Although many public health interventions are developed and implemented without explicit reference to theory, there is substantial evidence from the literature to suggest that the explicit use of theory will significantly improve the chances of effectiveness (Green and Kreuter, 1999; Nutbeam and Harris, 2004). The use of theory within systematic reviews may help to explain success or failure in different interventions, by identifying the key elements and highlighting the possible impact of differences between what was planned and what actually happened in the implementation of the programme (Nutbeam and Harris, 2004).
Incorporating theory into a review remains a challenge because many primary studies either do not have an explicit theoretical basis, use several theories, or describe a theory but do not appear to integrate it into the intervention. In addition, authors of primary studies may differ in their conceptualization of the theoretical basis for their intervention, as found in a review of stage-based interventions for smoking cessation (Riemsma et al., 2003). Despite these challenges, reviewers should seek to examine the impact of the theoretical framework on the effectiveness of the intervention. For example, when combining the findings from different studies, reviewers can group interventions by their theoretical basis. Alternatively, reviewers may consider grouping interventions depending on whether they seek to influence individual behaviour, interpersonal relationships or community or structural factors or whether they used a Program Logic or Program Theory approach (Patton, 1986; Wholey 1987; Funnel, 1997; Oliver et al., in press).
Systematic reviews would be greatly enhanced by attention to the theoretical coverage of interventions in their discussions, e.g. a large number of interventions seek to address individual change (behaviour, attitudes, etc.) but fail to incorporate theories, which seek to change the broader environment within which people make their choices.
Integrity of interventions
Assessing the degree to which interventions are implemented as planned is important in preventive interventions, which are often implemented in conditions that present numerous obstacles to complete delivery (Dane and Schneider, 1998). Reviewers should seek to determine whether findings of ineffectiveness within primary studies are simply due to incomplete delivery of the intervention (failure of implementation) or to a poorly conceptualized intervention (failure of intervention concept or theory). A review of smoking cessation in pregnancy (Lumley et al., 2004) found that in studies, which measured the implementation of the intervention, the implementation was less than ideal.
In order to provide a comprehensive picture of intervention integrity Dane and Schneider believe that five dimensions of the intervention should be measured (Dane and Schneider, 1998). These factors are adherence, exposure, quality of delivery, participant responsiveness and programme differentiation (to prevent contamination).
Heterogeneity
Variability in results between public health studies may arise due to differences in populations, settings, outcomes, interventions and study designs. For these reasons, reviewers should expect considerable heterogeneity across studies and should consider a priori the most appropriate method for synthesis (narrative synthesis or meta-analysis). Guidance on how to conduct narrative synthesis will be published at the end of 2005 (Popay, 2005).
Integrating qualitative and quantitative studies
Users of reviews require information on both the effects of interventions and which interventions will be most appropriate and relevant to people. The Evidence for Policy and Practice Information and Co-ordinating (EPPI) Centre has developed methods for synthesizing the findings from diverse types of studies within one review (Thomas et al., 2004b). These methods involve conducting three types of syntheses in the same review: (i) a statistical meta-analysis (or narrative synthesis) to pool trials of interventions tackling particular problems; (ii) a synthesis of studies examining people's perspectives or experiences of that problem using qualitative analysis (‘views’ studies); and (iii) a ‘mixed methods’ synthesis bringing the products of (i) and (ii) together.
Ethics and inequalities
It is important to determine whether public health interventions are effective in reducing inequalities and therefore improving the health of those who are disadvantaged and marginalized. However, most systematic reviews have focused on the population level effects despite the fact that even well-intentioned interventions may increase inequalities, and overall improvements in aggregate health behaviours may mask health differentials between groups (Macintyre, 2003; Petticrew, 2003).
Reviews that address inequalities are challenging for a number of reasons: (i) many studies are not indexed according to whether they conduct subgroup analyses; (ii) many primary studies are already small, thereby stratified analyses by level of disadvantage will result in small numbers in each sub-group; and (iii) difficulties in determining whether an intervention is effective in reducing inequalities. A Cochrane protocol for a review (Kristjansson et al., 2004) has addressed the latter concern by describing the differential effectiveness of school feeding interventions. This review describes an ‘effective’ intervention as one which is more effective for people in lower socio-economic groups, and a potentially effective intervention as one which is equally effective across the socioeconomic spectrum (due to the higher prevalence of health problems among the disadvantaged). Effective interventions targeted only at disadvantaged groups will be labelled as potentially effective. We await further methodological research before providing definitive recommendations on how to conduct reviews addressing inequalities.
Sustainability
The extent to which the intended interventions or outcomes are sustained should be an important consideration in systematic reviews as decision-makers are increasingly concerned with allocating scarce resources effectively and efficiently (Hawe et al., 1990). Reviewers should examine whether or not studies measure outcomes over multiple time points. Some programmes may be successful in changing behaviour during and immediately after the intervention but subsequently find that participants lapse into old behaviour patterns. Alternatively, it has been suggested by Green that some programmes need a certain time period after the programme before the effect ‘matures’ and before people have had enough time to practise the skills they have learned (Green, 1977).
Although the empirical knowledge-base about the determinants of sustainability is in its infancy, reviewers may refer to a number of frameworks, which assist in determining the potential sustainability of interventions and outcomes (Bossert, 1990; Shediac-Rizkallah and Bone, 1998; The Health Communication Unit, 2001). For example, it is believed that interventions, which isolate individual action from its wider social context would be unlikely to produce sustainable health gain in the absence of change to the organizational, community and institutional conditions that make up the social context (Swerrisen and Crisp, 2004).
Context
Interventions may be effective due to pre-existing aspects of the context into which the intervention was introduced. In this case, context acts as an effect modifier. Information on context within reviews can be of considerable value to people who are charged with implementing interventions in the ‘real world’. Where information is available in primary studies, reviewers should report on the presence of context-related information:
- aspects of the host organization and staff, e.g. number, experience, morale, expertise of staff, competing priorities to the staff's attention, the organization's history of innovation, size of the organization, the status of the programme in the organization, the resources made available to the programme (Hawe et al., 2004):
- aspects of the system, e.g. payment and fee structures for services, reward structures, degrees of specialization in service delivery; and
- characteristics of the target population (e.g., socioeconomic, cultural, place of residence).
Exposure in systematic reviews of the low level of reporting on local contextual factors may stimulate charting of systematic gaps in studies and draw attention to critical factors that should be reported, thereby alerting investigators of the need to qualify statements about ‘programme’ effects. It may also spawn more combined methods research (qualitative and quantitative) that may alleviate this problem in future (Hawe et al., 2004).
Applicability
Assessing the applicability of the findings of the review and the feasibility of replicating the interventions included in the review to other settings should be a key part of the process of summarising evidence. The full-text review guidelines (http://www.vichealth.vic.gov.au/cochrane) contain a detailed framework for reviewers to determine applicability. This framework is based on the RE-AIM model (Glasgow et al., 1999) for conceptualizing the public health impact of an intervention.
Limitations of the guidelines
While it may not be possible to elicit all of the recommended information from primary studies, for example, intervention context, theoretical frameworks and process data, we believe that it is important to report when data is not available in the primary research in order to further improvements in public health research and its publication. The TREND statement (Des Jarlais et al., 2004), developed to improve the reporting of public health research, is important in ensuring that future primary studies meet the reporting needs required for conducting high quality systematic reviews.
|
CONCLUSION |
|---|
|
TOPSUMMARYINTRODUCTIONENSURING REVIEWS MEET THE...ISSUES TO ADDRESS WHEN...CONCLUSIONREFERENCES |
|---|
Public health practitioners, policy makers, funders and potential recipients all need to have access to the findings of high quality systematic reviews to enable them to make informed decisions about whether or not to implement, or participate in, a specific intervention. This paper has sought to provide comprehensive criteria to enable reviewers to produce high quality systematic reviews that are relevant to the questions being asked by end-users. Reviewers should access the full-text of the guidelines for further information. However, for high quality reviews to be successfully completed we need a concurrent improvement in the reporting of public health studies, so that detailed information related to the intervention, context and study population is included.
|
ACKNOWLEDGEMENTS |
|---|
This project is grateful to the sources of support for each of the contributing authors. The Cochrane Health Promotion and Public Health Field acknowledges the generous support of the Victorian Health Promotion Foundation (VicHealth), the Australian Commonwealth Department of Health and Ageing and the Department of Health for England.
|
FOOTNOTES |
|---|
The online version of this article has been published under an open access model. Users are entitled to use, reproduce, disseminate, or display the open access version of this article for non-commercial purposes provided that: the original authorship is properly and fully attributed; the Journal and Oxford University Press are attributed as the original place of publication with the correct citation details given; if an article is subsequently reproduced or disseminated not in its entirety but only in part or as a derivative work this must be clearly indicated. For commercial re-use, please contact journals.permissions@oxfordjournals.org
|
REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONENSURING REVIEWS MEET THE...ISSUES TO ADDRESS WHEN...CONCLUSIONREFERENCES |
|---|
Berkwits, M. (1998) From practice to research: the case for criticism in an age of evidence. Social Science and Medicine, 47, 1539–1545.
Bossert, T. J. (1990) Can they get along without us? Sustainability of donor-supported health projects in Central America and Africa. Social Science and Medicine, 30, 1015–1023.[Medline]
CRD (2001) Undertaking Systematic Reviews of Research on Effectiveness. CRD's Guidance for those Carrying Out or Commissioning Reviews: University of York.
CASP (2004) Appraisal Tool for Qualitative Research. Available at: http://www.phru.nhs.uk/casp/qualitat.htm (last accessed 6 June 2005).
Dane, A. V. and Schneider, B. H. (1998) Program integrity in primary and early secondary prevention: are implementation effects out of control? Clinical Psychology Review, 18, 23–45.[CrossRef][ISI][Medline]
Deeks, J. J., Dinnes, J., D'Amico, R., Sowden, A.J., Sakarovitch, C., Song, F. et al. (2003) Evaluating non-randomised intervention studies. Health Technology Assessment, 7, 1–173.
Des Jarlais, D. C., Lyles, C. and Crepaz, N. (2004) Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. American Journal of Public Health, 94, 361–366.
Dixon-Woods, M., Fitzpatrick, R. and Roberts, K. (2001) Including qualitative research in systematic reviews: opportunities and problems. Journal of Evaluation in Clinical Practice, 7, 125–133.[CrossRef][ISI][Medline]
Funnel, S. (1997) Program logic: an adaptable tool for designing and evaluating programs. Evaluation News and Comment, 7, 5–17.
Glasgow, R. E., Vogt, T. M. and Boles, S. M. (1999) Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health, 89, 1322–1327.
Green, L. W. and Kreuter, M. W. (1999) Health Promotion Planning: An Educational and Ecological Approach, 3rd Edition. Mayfield Publishing Company, California, USA.
Green, L. W. (1977) Evaluation and measurement: some dilemmas for health education. American Journal of Public Health, 67, 155–161.
Hawe, P., Degeling, D. and Hall, J. (1990) Evaluating Health Promotion: A Workers Guide. MacLennan and Petty Pty Limited, Sydney Australia, pp. 101–127.
Hawe, P., Shiell, A., Riley, T. and Gold, L. (2004) Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. Journal of Epidemiology and Community Health, 58, 788–793.
Jackson, N. W., Waters, E., for the Guidelines for Systematic Reviews in Health Promotion and Public Health Taskforce. (2004) The challenges of systematically reviewing public health interventions. Journal of Public Health Medicine, 26, 303–307.
Jackson, S. F., Edwards, R. K., Kahan, B. and Goodstadt, M. (2001) An Assessment of the Methods and Concepts Used to Synthesize the Evidence of Effectiveness in Health Promotion: A Review of 17 Initiatives. Canadian Consortium for Health Promotion Research, Toronto.
Kristjansson, E., Robinson, V. A., MacDonald, B., Krasevec, J., Greenhalgh, T., McGowan, J. et al. (2004) School feeding for improving the physical and psychosocial health of disadvantaged elementary school children. (Protocol) The Cochrane Database of Systematic Reviews 2003, Issue 4. Art. No.: CD004676. DOI: 10.1002/14651858.CD004676.
Lumley, J., Oliver, S. S., Chamberlain, C. and Oakley, L. (2004) Interventions for promoting smoking cessation during pregnancy. The Cochrane Database of Systematic Reviews, Issue 3. Art. No.: CD001055. DOI: 10.1002/14651858.CD001055.pub2.
Macintyre, S. (2003) Evaluating the evidence on measures to reduce inequalities in health. In Oliver, A. and Exworthy, M. (eds) Health Inequalities: Evidence, Policy and Implementation. Proceedings from a Meeting of the Health Equity Network. The Nuffield Trust, London.
Mullen, P. D., Green, L. W. and Persinger, G. S. (1985) Clinical trials of patient education for chronic conditions: a comparative meta-analysis of intervention types. Preventive Medicine, 14, 753–781.[CrossRef][ISI][Medline]
Nutbeam, D. (1999) The challenge to provide ‘evidence’ in health promotion. Health Promotion International, 14, 99–101.
Nutbeam, D. and Harris, E. (2004) Theory in a Nutshell. A Practical Guide to Health Promotion Theories. McGraw-Hill Australia Pty Ltd, Sydney, Australia, vii-9.
Patton, M. (1986) The programs' theory of action: Conceptualising causal linkages. Utilization-Focussed Evaluation, 150–174.
Oliver, S., Harden, A., Rees, R., Shepherd, J., Brunton, G., Garcia, J. et al. (2005) An emerging framework for including different types of evidence in systematic reviews for public policy. Evaluation, the International Journal of Theory, Research and Practice, in press.
Peersman, G. and Oakley, A. (2001) Learning from research. In Oliver, A. and Peersman, G. (eds) Using Research for Effective Health Promotion. Oxford University Press, Buckingham, pp. 32–43.
Petticrew, M. and Roberts, H. (2006) Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing, Oxford, in press.
Petticrew, M. (2003) Presumed innocent. Why we need systematic reviews of social policies. Am J Prev Med, 24, 2–3.[CrossRef][ISI][Medline]
Popay, J., Rogers, A. and Williams, G. (1998) Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research, 8, 341–351.
Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodger, M. et al. (2005) Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. Lancaster University, UK, in press.
Rees, R., Oliver, S., Harden, A., Shepherd, J., Kavanagh, J., Burchett, H. et al. (2004) Use of an advisory group to ensure relevance: reflections on participation of stakeholders in a review of sexual health promotion for men who have sex with men (MSM). Paper presented at the XII Cochrane Colloquium, Ottawa, Canada, 2–6th October, 2004.
Richards, T. (2004). Poor countries lack relevant health information, says Cochrane editor. British Medical Journal, 328, 310.
Riemsma, R. P., Pattenden, J., Bridle, C., Sowden, A. J., Mather, L., Watt, I. S. et al. (2003) Systematic review of the effectiveness of stage based interventions to promote smoking cessation. British Medical Journal, 326, 1175–1177.
Rychetnik, L., Frommer, M., Hawe, P. and Shiell, A. (2002) Criteria for evaluating evidence on public health interventions. Journal of Epidemiol Community Health, 56, 119–127.
Shediac-Rizkallah, M. C. and Bone, L. R. (1998) Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Education Research, 13, 87–108.
Spencer, L., Ritchie, J., Lewis, J. and Dillon, L. (2003) Quality in Qualitative Evaluation: A framework for assessing research evidence. Government Chief Social Researcher's Office, Crown Copyright.
Swerrissen, H. and Crisp, B. R. (2004) The sustainability of health promotion interventions for different levels of social organization. Health Promotion International, 19, 123–130.
The Health Communication Unit (2001) Overview of Sustainability, University of Toronto, Centre for Health Promotion. Available from: http://www.thcu.ca/infoandresources/publications/SUS%20Master%20Wkbk% 20and%20Wkshts%20v8.2%2004.31.01_formatAug03.pdf [last accessed 6 June, 2005].
Thomas, B. H., Ciliska, D., Dobbins, M. and Micucci, S. (2004a) A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions. Worldviews on Evidenced-Based Nursing, 1, 176–184.[CrossRef]
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R. et al. (2004b) Integrating qualitative research with trials in systematic reviews. British Medical Journal, 328, 1010–1012.
Tilford, S. (2000) Evidence-based health promotion. Health Education Research, 15, 659–663.
Wholey, J. S. (1987) Evaluability assessment: developing program theory. In Bicklam, L., (ed.) Using Theory in Evaluation. New Directions for Program Evaluation. Jossey-Bass Publishers, San Francisco, CA, pp. 77–92.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. R. Aro, J. Smith, and J. Dekker Contextual evidence in clinical medicine and health promotion Eur J Public Health, September 11, 2008; (2008) ckn082v1. [Full Text] [PDF]
|
|
|
|
 |
|
 R. Armstrong, E. Waters, L. Moore, E. Riggs, L. G. Cuervo, P. Lumbiganon, and P. Hawe Improving the reporting of public health intervention research: advancing TREND and CONSORT J. Public Health Med., March 1, 2008; 30(1): 103 - 109. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Braveman When do we know enough to recommend action? The need to be bold but not reckless J. Epidemiol. Community Health, November 1, 2007; 61(11): 931 - 931. [Full Text] [PDF]
|
|
|
|
 |
|
 E. Mayo-Wilson Reporting Implementation in Randomized Trials: Proposed Additions to the Consolidated Standards of Reporting Trials Statement Am J Public Health, April 1, 2007; 97(4): 630 - 633. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. R Goldsmith, C. R Bankhead, and J. Austoker Synthesising quantitative and qualitative research in evidence-based patient information J. Epidemiol. Community Health, March 1, 2007; 61(3): 262 - 270. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. Young, E. Brunner, M Thorogood, K Rees, G Hewitt, W. Elizabeth, C. Summerbell, and H. Moore Cochrane Column Int. J. Epidemiol., June 1, 2006; 35(3): 538 - 540. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||