DEBATE |
Challenges in assessing the implementation and effectiveness of physical activity and nutrition policy interventions as natural experiments
1Department of Public Health Sciences, University of Toronto, Toronto, ON, Canada 2KR Allison Research Consulting, Toronto, ON, Canada 3School of Physical and Health Education, University of Toronto, Toronto, ON, Canada 4Department of Family Relations and Applied Nutrition, University of Guelph, Guelph, ON, Canada
* Corresponding author. E-mail: subha.ramanathan{at}utoronto.ca
 |
SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
The Ontario (Canada) government has instituted a policy requiring elementary schools to offer at least 20 min of daily physical activity for students in Grades 1–8 and replace non-nutritious vending machine foods with healthier choices. These policy interventions represent ‘natural experiments’ offering unique opportunities for conducting research and evaluation. The use of natural experiments to contribute evidence on the effectiveness of policy interventions is identified as an underused tool for public health [Tudor-Locke, C., Ainsworth, B. E. and Popkin, B. M. (2001) Active commuting to school: an overlooked source of children’s physical activity? Sports Medicine, 31, 309–313; Petticrew, M., Cummins, S., Ferrell, C., Findlay, A., Higgins, C., Hoy, C. et al. (2005) Natural experiments: an underused tool for public health? Public Health, 119, 751–757]. To date, some Canadian school-based food and nutrition policies are being monitored, but their impact on child and youth obesity is unknown [Canadian Institute for Health Information. (2006) Improving the Health of Canadians: Promoting Healthy Weights, Ottawa, ON]. There are a number of challenges to the evaluation of policy interventions as natural experiments. Often, there are little or no baseline data available to use as the basis for assessing change. Government policies that result in the adoption of particular approaches across large jurisdictions, such as provinces, may result in wide variation in the design and implementation of interventions. Thus, in some cases, natural experiments may be at risk of having low potential to be adequately evaluated on key outcomes. In this paper, we discuss the context of these challenges in relation to the Ontario government school physical activity and nutrition policies.
Key words: natural experiment; evaluation challenges; educational policy intervention
|
INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
When the purpose of a study is to collect convincing evidence on the effect of an intervention, randomized controlled trials are considered the ‘gold standard’ against which all other methods should be compared (Campbell and Stanley, 1963
; Abramson and Abramson, 1999; Rychetnik et al., 2002). For practical reasons, however, public health interventions can rarely be studied using randomized controlled trials. For example, in school-based youth physical activity and nutrition interventions, many uncontrollable factors contribute to the heterogeneity of intervention implementation including community involvement, financial support and school context (Veugelers and Fitzgerald, 2005). Furthermore, the heterogeneity of implementation may result in variable effectiveness of the intervention (Rychetnik et al., 2002).
Acknowledging that randomized controlled trials are difficult to conduct in human behavioural research, quasi-experimental designs have been used extensively, especially in public health (Abramson and Abramson, 1999). Quasi-experiments are used in social settings, where researchers cannot fully control the scheduling of the stimuli or intervention (Campbell and Stanley, 1963). Quasi-experimental designs are often used in public health because of their feasibility, acceptability and appropriateness to evaluate health interventions, though they are subject to bias and cannot directly determine causality (Rychetnik et al., 2002). In the case of natural experiments, researchers cannot differentiate between experimental and control groups by choosing who does and does not receive the intervention. Instead, allocation of the intervention occurs because of changes in the social and political world (Dunning, 2005), and there is a natural variation in intervention allocation (Cummins et al., 2005). A second defining characteristic of natural experiments is their opportunistic nature, i.e. sudden changes in policy or the built environment can be evaluated as natural experiments to inform future decisions. In this paper, we use the term to refer to the situations in which the treatment (policy) has been assigned to an entire population and there is a natural variation in the implementation of the treatment.
|
ROLE OF NATURAL EXPERIMENTS IN STUDYING HUMAN BEHAVIOUR |
|---|
|
TOPSUMMARYINTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
Natural experiments are generally associated with economics and epidemiology and to some extent political science (Dunning, 2005
). The interest in natural experiments is driven particularly by the need for population-level intervention research. Recently, Petticrew et al. (2005) advocated for the evaluation of natural experiments in public health and suggested that such experiments may provide evidence on the effectiveness of health interventions because of the inherent variation in how, where and to whom interventions are administered.
Although data from natural experiments may not stand alone as proof of success or failure for a given intervention, they represent one of several factors to be appraised in the process of intervention-based policies or operational decisions (Rychetnik et al., 2002). Since public health interventions tend to be programmatic and context-dependent (Rychetnik et al., 2002), natural experiments may provide unique opportunities for investigation. For example, if a natural experiment for a health intervention is discovered and assessed, multiple health outcomes and contributing factors may be explored simultaneously because it takes place within a specific context that is, in turn, constrained by the complex socio-political world. Although it is not possible to generate causal data, this potential to investigate interventions within the context of ‘real-life’ situations is an advantage of natural experiments (Petticrew et al., 2005; Tudor-Locke et al., 2001). Health policymakers inevitably stimulate natural experiments by creating policies that vary in allocation and implementation. As policymakers must make critical decisions and implement interventions to improve health and reduce health inequalities even in the absence of conclusive scientific data (Petticrew et al., 2004), there is a pressing need for stronger and more accepted research designs to evaluate natural experiments. As natural experiments and quasi-experiments share observational methods, it may be possible to infuse natural experiments with some features of quasi-experimental designs. We suggest that features of quasi-experimental designs (e.g. time-series and pre-/post-test methods) may be successfully applied to generate useful data on natural experiments. Regardless of the design used, researchers and policymakers must work together to identify relevant outcome measures and determine if a policy is effective.
|
STUDYING NATURAL EXPERIMENTS |
|---|
|
TOPSUMMARYINTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
Natural experiments are a different paradigm requiring an adjustment in the mindset among researchers, funding agencies and policy makers. Currently, there are a few examples of research on natural experiments that have been conducted in public health. Researchers have investigated outcomes following changes in smoking legislation (Sargent et al., 2004
), changes in local food infrastructure (Cummins et al., 2005), changes in political power (Alvarez-Dardet and Franco-Giraldo, 2006) and changes in health educational policies (Veugelers and Fitzgerald, 2005).
Such natural experiments are likely to be ‘messy’ in comparison with the ideal research design, but may provide valuable information. ‘Canada on the Move’ (COTM) provides a relevant case study (Dietz, 2006). From the end of 2003 to 2004, Kellogg Canada initiated a media campaign to encourage Canadians to walk more and distributed 2 million pedometers in specially marked cereal boxes. To track the impact of the campaign and pedometer use on walking behaviour, consumers were encouraged to visit a website created for COTM. Research following the campaign found that there was a lack of clarity in the objectives of the project represented in the news media (Faulkner and Finlay, 2006) and the pedometers did not accurately capture physical activity levels (Tudor-Locke et al., 2006). Despite these challenges, it was argued that COTM was a beneficial model of practice-based research that led to valuable insights for practitioners, researchers and the general population. For example, COTM showed how partnerships may be formed between commercial marketing and public health research to maximize resources (Cameron et al., 2006); it used an innovative research platform at the population level (Cameron et al., 2006); finally, data provided insight into how environmental variables may impact the likelihood of walking for exercise in a Canada-wide (convenience) sample (Dietz, 2006). Overall, studying COTM as a natural experiment opened up a new research area to address and assess population-level health in Canada.
Natural experiments on walking behaviour have also been evaluated in Australia (Krizek, 2000) and the USA (Boarnet et al., 2005). In 1998, the state government in Western Australia created Liveable Neighbourhood (LN) guidelines to create safe and pedestrian-friendly neighbourhoods and convenient access to shops, public transit and parks that would in turn encourage more walking, cycling and use of public transport (Giles-Corti et al., 2008). In 2003, the Residential Environments Project (RESIDE) was devised to measure the longitudinal impact of the LN Guidelines as a natural experiment (Giles-Corti et al., 2008). RESIDE is currently ongoing and will use a quasi-experimental design over 5 years to examine survey and pedometer data among residents of 74 new housing developments. Currently, only baseline data are available, but these provide some insight into previous debates on self-selection of neighbourhood and physical activity (Krizek, 2000). It has been argued that changes to urban form have little impact on increasing individual walking behaviour, as those who walk regularly elect to live in walkable neighbourhoods (Krizek, 2000). RESIDE baseline data show that people who move into developments meeting LN Guidelines prefer walkable neighbourhoods more than those in hybrid or conventional neighbourhoods (less walkable neighbourhoods), and this drives their choice of new housing development (Giles-Corti et al., 2008). However, data show that the mean pedometer steps of individuals moving into liveable developments do not differ from those moving into hybrid or conventional developments. In the future RESIDE, as a natural experiment, will be able to address this self-selection debate by monitoring any changes in walking behaviour in response to changes in urban form and adjusting for baseline preferences for residential location.
In the USA, changes in the built environment have been studied as a natural experiment on children’s walking behaviour. In 1999, the state of California passed legislation that allocated funding for Safe Routes to School (SR2S) construction projects near schools with the aim of encouraging active transportation (walking and cycling) among school children. Recognizing this as an opportunity to study potential links between the built environment and physical activity, Boarnet et al. (2005) evaluated the effectiveness of SR2S projects (crosswalks, bike lanes, sidewalks, etc.) in promoting students’ active travel to and from school. A cross-sectional survey was administered to the parents of Grade 3–5 students from 10 schools between 1 and 18 months after the completion of SR2S projects. Parents answered retrospective questions to indicate whether their child walked or cycled to school more often after the completion of the SR2S project versus before the project. Comparison groups were students who would and would not pass a project on their usual route to school. Results showed that, among children who passed a SR2S project, 15.4% walked more often following construction compared with 4.3% of children who did not pass the project. Authors concluded that ‘small but strategic pedestrian or bicycle facility improvements may impact the propensity of children to walk or bicycle to school’ (Boarnet et al., 2005). Evaluation of SR2S as a natural experiment provided empirical support that changes in urban form can promote children’s active commuting to school and strengthened the case for state and federal funding of SR2S construction programmes.
While many health researchers and policymakers share a common goal to improve health and reduce inequalities at the population level, there is a large gap between the research that is conducted and the policies that are implemented (Petticrew et al., 2004). We suggest that natural experiments may have an important role in bridging this gap. According to Petticrew et al. (2004), ‘researchers have been castigated for their political naivety, for lacking understanding of how policy is made, and for having unrealistic expectations about what research can achieve’. There must be congruence between researchers’ questions and the answers required by policymakers, otherwise any generated evidence will not be valued or appraised when creating interventions and policies (Rychetnik et al., 2002; Mackenbach, 2003; Petticrew et al., 2004; Canadian Institute for Health Information, 2006). Moreover, if those with decision-making power do not value current evidence, then evidence-based policy cannot be realized (Petticrew et al., 2004).
In this paper, we suggest that two recent policies issued by the Ontario Ministry of Education: Healthy Foods and Beverages (HF & B) in Elementary School Vending Machines (Ontario Ministry of Education, 2004) and Daily Physical Activity (DPA) in Elementary Schools (Ontario Ministry of Education, 2005) may also be evaluated as natural experiments. In the case of HF & B, policymakers have shown interest in several health-related outcomes for school children including students’ nutritional knowledge, good eating habits, general learning abilities, and long-term health (Ontario Ministry of Education, 2004). The DPA memorandum indicates that specific desired outcomes include improvements in achievement and readiness to learn, increased self-esteem, decreased rates of childhood obesity and long-term adoption of regular physical activity behaviours. Taking both HF & B and DPA into account, it is clear that policymakers are interested in data that address the aforementioned health outcomes. Knowledge of these outcomes would assist policymakers to modify existing policies and develop future health targets and interventions. By generating and aligning evidence with health policies through natural experiments, the impact of research can be maximized (Petticrew et al., 2004).
|
CHALLENGES OF EVALUATING NATURAL EXPERIMENTS |
|---|
|
TOPSUMMARYINTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
While valuable information can be learned by exploiting the natural experiments resulting from health policies, evaluating these experiments is challenging. At a fundamental level, further thought is required in developing terms that are distinct from experimental or quasi-experimental designs to more appropriately describe features of natural experiments, and in developing evaluation criteria suitable to the process and products of natural experiments. Practically, researchers face two main challenges when designing an evaluation of natural experiments: how to assess change without baseline data, and how to evaluate key outcomes when interventions are implemented in diverse contexts. Both challenges are outlined below, with suggestions on how they may be dealt with in the cases of HF & B and DPA in Ontario elementary schools.
Assessing change without baseline data
Monitoring systems are one of the major ways for researchers to move forward with evaluating natural experiments. Opportunities to generate evidence on school-based policies are lost when programmes like HF & B and DPA are mandated province-wide without an embedded surveillance system, i.e. a mechanism for tracking changes. As schools by nature are not uniform across the province with respect to several factors including funding, ratio of teachers to students and general infrastructure, it follows that any interventions, including HF & B and DPA, are neither uniformly adopted nor consistently effective. Although inherent variations are not a criticism of either policymakers or interventions, they must be appreciated in any outcome evaluations. For example, with respect to DPA, it is important to ask how teachers and students have dealt with scheduling challenges—how was the programme implemented into an already full class schedule? How many teachers and/or students took charge in running the programme? What incentives were in place to encourage participation or programme adherence? In an ideal research setting, a few schools would be randomly selected to implement the policies within a pilot study, and school administrators, teachers, parents and students (key stakeholders) would be recruited to provide feedback on the programmes over time (a longitudinal panel study design). Participants’ perspectives in various stages of implementation are integral sources of outcome measures (Allison and Rootman, 1996). However, HF & B and DPA policies were released across Ontario without having generated baseline data and without identifying formal measures of success. Thus, any future outcome evaluations will be limited because there is no mechanism to comprehensively examine the effectiveness of either HF & B or DPA.
While an absence of baseline data is a limitation, there may be ways to creatively construct baseline measures. For example, with respect to DPA, there is an existing database with several physical health outcome measures for students in Grades 7–12: the Ontario Student Drug Use Survey (OSDUS) (Adlaf et al., 2006). OSDUS is the longest ongoing school survey in Canada and has been conducted every 2 years since 1977. Starting in 1997, OSDUS included questions on frequency of physical activity for at least 20 min both in and outside of school. If OSDUS data for students in Grades 7 and 8 are considered ‘pre-intervention’ data and are compared with data on physical activity frequency among Ontario students after the implementation of DPA, this will provide us with some indication as to the effectiveness of this policy. With respect to evaluating HF & B, baseline measures of the frequency and nature of vending machine use may be creatively constructed from databases generated by vending machine providers. For example, inventory and sales reports are likely to contain information on the frequency of vending machine use and the popularity of individual snack and beverage items. These measures could then be compared with the frequency of use and popularity of items following HF & B to identify if this policy has been effective in promoting healthy eating among schoolchildren. However, this requires the full cooperation of vending machine providers.
Another creative approach to providing baseline and ongoing surveillance data in evaluating natural experiments is the School Health Action Planning and Evaluation System (SHAPES). Developed by the Centre for Behavioural Research and programme Evaluation (CBRPE) at the University of Waterloo, SHAPES has recently expanded its capability to assess student and school-level information on health behaviours. In addition to their earlier smoking module, they have recently validated an assessment instrument for physical activity and are also developing a module focussing on eating behaviours (Cameron et al., 2007).
Variation in the design and implementation of policies
Evaluations of public health intervention outcomes, unless tied to well-funded research studies, are uncommon; thus, policymakers and practitioners are often left without robust and current evidence of policy effectiveness (Petticrew et al., 2005). To rectify this, the Canadian Institutes of Health Research (2008) (CIHR) has recently partnered with several national organizations to create a funding programme specifically for intervention and evaluation research on healthy living and/or chronic disease prevention at the population level. CIHR recognizes the importance of monitoring existing and future programmes and policies as natural experiments and considers them ‘a rich but untapped data source that could benefit from researcher expertise and involvement’ (Canadian Institutes of Health Research, 2008). The first grants awarded under this competition were to support the collection of baseline data before the re-launch of the Canadian physical activity social marketing organization ‘ParticipACTION’ in September, 2007.
Although CIHRs funding opportunity holds great promise for generating evidence of effective public health interventions, currently there are several knowledge gaps. In particular, few school-based interventions have been demonstrated as effective (Veugelers and Fitzgerald, 2005). Mackenbach (2003) confirms that it is difficult to find ‘good’ evidence that shows whether a policy or intervention effectively reduces inequalities in health and speaks to the necessity of conducting evaluations throughout the implementation period. Although there is no variation in the allocation of HF & B and DPA (as they have been mandated across Ontario), the main challenge is that there is natural variation in the timing and the extent of implementation for these (and all) school-based interventions. Unlike a randomized controlled experiment, any evaluation of these policies as natural experiments must be designed to account for schools in various stages of policy implementation.
To deal with this challenge of variable implementation for each policy, we suggest that Ontario public schools could be matched with out-of-province schools for treatment-control comparisons and indicators of overall policy effectiveness. We believe that these policies present opportunities for ‘within treatment’ comparisons, i.e. opportunities to (i) evaluate schools in various stages of programme adoption, (ii) report any outcomes and (iii) share knowledge between Ontario school boards. Evaluations must take place at two levels: process and outcome (Green and Kreuter, 1991; Posavac and Carey, 2005). Process evaluations are done first after implementation of the policy and may look at the programme structure, goals and implementation successes and challenges. Outcome evaluations would then look at immediate effects of the intervention on target behaviours and environmental influences (also referred to as ‘impact’ evaluations) as well as long-term changes in physical and social indicators that in turn influence health status and quality-of-life. Overall, researchers must design methods of evaluating schools that can distinguish between the relative success or failure of achieving policy outcomes and the relative success or failure of the policy itself (Rychetnik et al., 2002).
Veugelers and Fitzgerald (2005) conducted a relevant example of research on the effectiveness of healthy eating and physical activity interventions in schools in Nova Scotia, Canada. Outcome measures included overweight and obesity [determined by body mass index (BMI) data], fruit and vegetable consumption and overall diet quality. Schools were grouped into three categories: those with policy and practices to offer healthy foods, those participating in the Annapolis Valley Health Promoting Schools Project (a coordinated programme) and those without nutrition programmes. Results showed that diet and activities were similar among students attending schools with and without nutrition programmes, and there was no statistical difference in overweight and obesity rates in schools with a nutrition programme. Students participating in the Annapolis Valley project had significantly lower rates of overweight and obesity than students from schools with and without a nutrition programme. Annapolis Valley students also reported significantly higher consumption of fruits and vegetables and foods of better diet quality than all other students.
In the case of Ontario’s HF & B and DPA, individual school boards are accountable for informing parents and their respective district office of the Ministry of Education of their progress of achieving policy standards (Ontario Ministry of Education, 2004, 2005). This is an example of a process evaluation embedded within the policy framework but does not generate data on the short- and long-term effectiveness of HF & B or DPA. Outcome evaluations are important to further refine these policies so that all schools, despite their inherent variation, can maximize benefits from participation. The Ontario Ministry of Education recently commissioned a study to evaluate DPA implementation in schools. At this stage, however, the Ministry has not documented the specific approach to be used in this evaluation. It is our assertion that HF & B and DPA can be studied as natural experiments to provide insight into the immediate and long-term outcomes of these interventions across various contexts (schools). Through a natural experiment framework, convincing evidence of policy successes may be generated if investigators can replicate positive outcomes in diverse schools across Ontario. If an intervention is unsuccessful in a particular school setting, then researchers and participants can work together to distinguish whether the programme was poorly planned, poorly executed or a combination of both (Rychetnik et al., 2002). By studying the intervention as a natural experiment, researchers may be able to detect interactions between the intervention and context and determine if the policy is particularly effective in certain types of schools but ineffective in others. Therefore, instead of a blanket policy, we might be able to match certain interventions to schools. Overall, natural experiments that include process and outcome evaluation designs should be prudently and strategically linked to policy interventions in all stages of development (Rychetnik et al., 2002).
|
CONCLUSIONS |
|---|
|
TOPSUMMARYINTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
We have shown some approaches that can be used in overcoming two challenges that exist when evaluating HF & B and DPA as natural experiments. We believe that by exploiting the natural experiments presented by educational policies, we can close the gap between what is currently being done in Ontario schools and what can potentially be done to improve healthy eating and regular physical activity among school children. Although we applaud the Ontario Ministry of Education for placing healthy eating and physical activity on their agenda, evidence-based policies must be designed, implemented and evaluated on an ongoing basis. Presently not enough resources are devoted to outcome evaluations of health and educational policies, but if policy-mandated interventions were first piloted in schools and revised accordingly, costs associated with distributing programme materials and then distributing revisions province-wide may be contained. As stated by Veugelers and Fitzgerald (2005
), ‘the effectiveness of school-based healthy eating and physical activity programmes is critical to evidence-based health policy and to justify broader implementation of successful programmes’. We feel that it would be beneficial to prospectively couple policy implementation and research from natural experiment perspectives because this may facilitate the generation of ‘best practice’ guidelines, and minimize redundant challenges experienced by individual schools. We have shown that natural experiments can provide highly relevant data on intervention effectiveness and may even provide strong evidence if elements are borrowed from quasi-experiments and results are repeated across diverse schools and combined with other data sources. We recognize that natural experiments are not without challenges and have suggested specific ways that two of these challenges may be handled. Although policies often create natural experiments before baseline data are collected, researchers can creatively draw upon existing databases to track baseline measures of specific health outcomes like BMI. Researchers should develop logic models demonstrating the theorized chain of events—clearly displaying the sequence of actions that describe what an intervention is (e.g. a policy) and will do (e.g. change in school environment), and by using natural variation, we can see whether the sequence is working as planned (e.g. in changing behaviour and ultimately BMI). In Canada, we need a comprehensive surveillance system so that if and when natural experiments come along, we have data to help evaluate them. We believe that by exploiting natural experiments presented by educational policies, researchers can contribute to the evidence base and inform future school-based policy interventions. Overall, we urge researchers worldwide to consider and exploit current or potential natural experiments in their respective countries.
|
Funding |
|---|
|
TOPSUMMARYINTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
Support provided by a grant (HBR4860) from the Heart and Stroke Foundation of Ontario.
|
REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONROLE OF NATURAL EXPERIMENTS...STUDYING NATURAL EXPERIMENTSCHALLENGES OF EVALUATING NATURAL...CONCLUSIONSFundingREFERENCES |
|---|
Abramson J. H., Abramson Z. H. Survey Methods in Community Medicine (1999) Toronto, ON: Elsevier Churchill Livingstone.
Adlaf E. M., Pagua-Boak A., Beitchman J. H., Wolfe D. The Mental Health and Well-Being of Ontario Students 1991–2005: OSDUS Highlights. In: CAMH Research Document (2006) Toronto, ON: Centre for Addiction and Mental Health.
Allison K. R., Rootman I. Scientific rigor and community participation in health promotion research: are they compatible? Health Promotion International (1996) 11:333–340.
Alvarez-Dardet C., Franco-Giraldo A. Democratisation and health after the fall of the wall. Journal of Epidemiology and Community Health (2006) 60:669–671.
Boarnet M. G., Anderson C. L., Day K., McMillan T., Alfonzo M. Evaluation of the California Safe Routes to School legislation - Urban form changes and children’s active transportation to school. American Journal of Preventive Medicine (2005) 28:134–140.[CrossRef][Web of Science][Medline]
Cameron R., Bauman A., Rose A. Innovations in population intervention research capacity. Canadian Journal of Public Health (2006) 97:S5–S9.
Cameron R., Manske S., Brown S., Jolin M. A., Murnaghan D., Lovato C. Integrating public health policy, practice, evaluation, surveillance, and research: the school health action planning and evaluation system. American Journal of Public Health (2007) 97:648–654.
Campbell D. T., Stanley J. C. Experimental and Quasi-Experimental Designs for Research (1963) Boston, MA: Haughton Mifflin Company.
Canadian Institute for Health Information. Improving the Health of Canadians: Promoting Healthy Weights (2006) Ottawa, ON.
Canadian Institutes of Health Research. Operating Grant: Intervention Research (Healthy Living and Chronic Disease Prevention) (2007–2008) (2008) http://www.researchnet-recherchenet.ca/rnr16/viewOpportunityDetails.do?prog=399&&view=currentOpps&org=CIHR&type=AND&resultCount=25&sort=program&all=1&masterList=true.
Cummins S., Petticrew M., Higgins C., Findlay A., Sparks L. Large scale food retailing as an intervention for diet and health: quasi-experimental evaluation of a natural experiment. Journal of Epidemiology and Community Health (2005) 59:1035–1040.
Dietz W. H. Canada On the Move. Canadian Journal of Public Health (2006) 97:S3–S4.[Medline]
Dunning T. Improving Causal Inference: Strengths and Limitations of Natural Experiments (2005) Washington, DC: American Political Science Association.
Faulkner G., Finlay J. Canada on the Move: an intensive media analysis from inception to reception. Canadian Journal of Public Health (2006) 97:S16–S20.
Giles-Corti B., Knuiman M., Timperio A., Van Niel K., Pikora T. J., Bull F. C. L., et al. Evaluation of the implementation of a state government community design policy aimed at increasing local walking: Design issues and baseline results from RESIDE, Perth Western Australia. Preventive Medicine (2008) 46:46–54.[CrossRef][Web of Science][Medline]
Green L. W., Kreuter M. W. Health Promotion Planning: An Educational and Environmental Approach, (1991) Toronto, ON: Mayfield Publishing Company. 215–261.
Krizek K. J. Pretest-posttest strategy for researching neighborhood-scale urban form and travel behavior. Transportation Research Record (2000) 1722:48–55.[CrossRef]
Mackenbach J. P. Tackling inequalities in health: the need for building a systematic evidence base. Journal of Epidemiology and Community Health (2003) 57:162.
Ontario Ministry of Education. Policy/Program Memorandum No.135: Healthy Foods and Beverages in Elementary School Vending Machines. (2004) http://www.edu.gov.on.ca/extra/eng/ppm/135.html.
Ontario Ministry of Education. Policy/Program Memorandum No.138: Daily Physical Activity in Elementary Schools, Grades 1–8. (2005) http://www.edu.gov.on.ca/extra/eng/ppm/138.html.
Petticrew M., Cummins S., Ferrell C., Findlay A., Higgins C., Hoy C., et al. Natural experiments: an underused tool for public health? Public Health (2005) 119:751–757.[CrossRef][Web of Science][Medline]
Petticrew M., Whitehead M., Macintyre S. J., Graham H., Egan M. Evidence for public health policy on inequalities: 1: the reality according to policymakers. Journal of Epidemiology and Community Health (2004) 58:811–816.
Posavac E. J., Carey R. G. Program Evaluation: Methods and Case Studies (2005) Thousand Oaks: Sage Publications.
Rychetnik L., Frommer M., Hawe P., Shiell A. Criteria for evaluating evidence on public health interventions. Journal of Epidemiology and Community Health (2002) 56:119–127.
Sargent R. P., Shepard R. M., Glantz S. A. Reduced incidence of admissions for myocardial infarction associated with public smoking ban: before and after study. British Medical Journal (2004) 328:977–980.
Tudor-Locke C., Ainsworth B. E., Popkin B. M. Active commuting to school: an overlooked source of children’s physical activity? Sports Medicine (2001) 31:309–313.[CrossRef][Web of Science][Medline]
Tudor-Locke C., Sisson S. B., Lee S. M., Craig C. L., Plotnikoff R. C., Bauman A. Evaluation of quality of commercial pedometers. Canadian Journal of Public Health (2006) 97:S10–S15.
Veugelers P. J., Fitzgerald A. L. Effectiveness of school programs in preventing childhood obesity: a multilevel comparison. American Journal of Public Health (2005) 95:432–435.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||