Health Promotion International, Vol. 14, No. 4, 307-316,
December 1999
© Oxford University Press 1999
Measuring the impact of a school food programme on food sales in New Zealand
The National Heart Foundation of New Zealand, 17 Great South Rd, PO Box 17 160, Auckland, New Zealand
Address for correspondence: Mary-Ann Carter, The National Heart Foundation of New Zealand, 17 Great South Rd, PO Box 17 160, Auckland, New Zealand
 |
SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
The School Food Programme of the Heart Foundation of New Zealand is a health promotion programme which aims to improve the health of the school community by increasing children's access to foods which are nutritious, safe and sufficient in quantity. The programme was introduced to New Zealand primary and secondary schools in 1989. Outcome evaluation undertaken in 1992 concluded that the programme had a positive impact in creating healthier school environments. As a result, programme implementation continued. The objective of this study was to identify whether the school food service has altered, as a result of programme implementation. The main outcome measures were food service staff reports on changes in sales of specific food items. Two hundred schools out of a total of 2730 schools in New Zealand were successful in achieving Heartbeat Awards by meeting all the programme criteria between January 1996 and December 1997. Thirty-two schools achieved awards in successive years, and data from all 232 awards were included in the analysis. Schools were categorized according to the number of years that they had participated in the programme. Increasing participation in the programme was found to be significantly associated with a reduction in the sales of doughnuts and cream buns (p = 0.01), pies and sausage rolls (p = 0.009), crisps (p = 0.0065) and sweets (p = 0.004), and an increase in sales of sandwiches and filled rolls (p = 0.0005). Other foods all showed changes in a favourable direction, although the proportions did not change significantly over the years. Limitations of this study include self-selection bias and the use of self-reported data. However, the results of the evaluation indicate that the School Food Programme is successful in achieving its aim of influencing the school environment by improving healthy food choices, and that improvements have continued over 7 years in the programme.
Key words: health promotion; nutrition; schools
|
INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
The Heart Foundation's School Food Programme was introduced to New Zealand schools in 1989. Its aim is to improve the health of the school community by increasing children's access to foods that are nutritious, safe and sufficient in quantity, and by influencing policy development which promotes healthy food choice. These aims are similar to those of a number of programmes described in the literature which seek to influence the school food service environment (Butcher et al., 1988
; Simons-Morton et al., 1991; Snyder et al., 1992; Raizman et al., 1994; Luepker et al., 1996).
Evaluating the outcomes of such health promotion programmes is difficult. Standard experimental designs are usually not applicable, because of the constraints on community-based interventions. Even if random controlled trials were possible, the information obtained would often not be useful or applicable for health promotion practitioners. Non-experimental designs, on the other hand, are subject to substantial bias (Nutbeam et al., 1990; McDonald, 1996; Rychetnik et al., 1997).
From a practitioner's perspective, the most valuable outcome measures are those that faithfully track progress towards the programme's goals and provide a practical guide for decision-making, programme design and management. Unfortunately, the literature provides limited information to guide programme and evaluation design. Evaluations which provide valuable information for a practitioner may not reach academic publication, because their designs do not allow for statistical analyses (Nutbeam et al., 1990). Nutbeam et al. suggest that when assessing the outcome of an intervention, two basic questions must be answered: i) can change be observed in a variable? and ii) can this observed change be attributed to the intervention? This paper attempts to answer these questions through an evaluation of a school food programme.
|
PROGRAMME DESCRIPTION |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
In its widest sense, the Heart Foundation's School Food Programme attempts to influence key structures affecting nutrition in schools. This includes working with food manufacturers to influence the formulation, marketing and distribution of food for children. At an individual school level, the programme encourages schools to implement nutrition policies and to introduce a wider range of food choices for students. The programme emphasis is on encouraging schools to change the range and nutritional content of the food served in the school food service.
An objective of the School Food Programme has been to provide motivation and support for schools wishing to make changes in the school food service. Schools that register with the programme are encouraged to work towards the achievement of a Heartbeat Award. This certificate provides formal recognition to a school food service that meets set criteria, and the process of achieving an award provides an incentive for schools to make changes to their food service. The award criteria address food policy, food choice, food safety and a smoke-free food preparation environment. Each year, schools are encouraged to apply for a successive Heartbeat Award. The same criteria apply, but to be eligible for a new award schools are asked to identify changes that have occurred in the sale or supply of food items. A school will be eligible for a successive award if the school has deleted or limited the supply of high-fat, high-sugar items, and if the school has extended its range of healthy food choices. Promotion of healthy food items including pricing structure, physical layout of the food service and visual promotions, are also considered when determining a school's eligibility for an award.
|
PREVIOUS OUTCOME EVALUATION |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
In 1992, an outcome evaluation was undertaken to assess what changes had occurred in participating schools, following at least 1 year of participation in the programme. The School Food Programme has been an example of a programme that was implemented nationwide, without the collection of baseline data or the establishment of a control group (Peach et al., 1996
). Failure to incorporate evaluation measures within the original programme design limited the type and validity of data that could be obtained in retrospect.
The outcome evaluation results showed that participation in the programme had influenced food sales, with schools reporting increased sales of low-fat, low-salt, low-sugar items and decreased sales of high-fat, high-sugar, high-salt items. It was concluded that the programme had an impact in creating healthier school environments (Peach et al., 1996).
This evaluation gave the Heart Foundation of New Zealand sufficient confidence to continue the programme, however, no further monitoring measures were introduced at that time. In 1995, consideration was again given to developing and implementing an evaluation process. By then 318 schools had achieved a Heartbeat Award, with some schools receiving their fifth award. The literature was reviewed to identify evaluation measures used in other programmes but provided limited information, with only one paper describing in detail the process evaluation measures used to assess the programme (Raizman et al., 1994). The evaluation measures described in this report from the United States were not considered applicable to the New Zealand School Food Programme because of the differences in the food services provided. The decision was taken to develop an ongoing outcome-based data collection system throughout the School Food Programme.
|
METHODS |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
Changes in food sales were identified as the key measure to monitor the impact of the programme on pupils' eating patterns. In designing the data collection methods, the following factors were considered: simplicity; cost effectiveness; and the respondent burden on food service staff, many of whom were volunteers. The numbers of schools involved in the programme, as well as budget considerations, precluded the use of nutrient analysis of foods served, a method used by some overseas studies (Simons-Morton et al., 1991
; Snyder et al., 1992; Luepker et al., 1996). Schools were asked to provide baseline data about the foods sold when they registered with the programme. Each time they applied for a Heartbeat Award, the main outcome measures of the monitoring programme were staff reports on changes in sales of specific food items. The changing trends in food sales were then compared between schools with successive numbers of Heartbeat Awards. Categorizing schools by the number of Heartbeat Awards achieved, enabled the researchers to use the data obtained from all schools involved in the programme, as well as providing projections about the effects of ongoing involvement.
When schools applied for their annual Heartbeat Award they were asked to rate the changes in food sales since their previous award or, for first year awards, since registering. They were asked to indicate whether item sales had increased (more), decreased (less), not changed (same), or state if a food item was not sold.
The schools self-determined their suitability to apply for a Heartbeat Award. The assessment of an award was based on the written documentation provided to programme management. A system of random audits was used to validate the award process, with 30 schools audited annually. The audits were completed by Heart Foundation staff who used a standard form. Audits compared the foods offered for sale with Heartbeat Award criteria and the menu the school provided when applying for an award. The audit also assessed how the school was promoting healthier food choices, the smoke-free and safe food criteria. In essence, the audit provided an evaluation of the process of school self-assessment.
The New Zealand Ministry of Education decile rankings of schools are designed to indicate educational disadvantage. Decile rankings are calculated from a range of social, economic, ethnic and housing indicators. Schools are ranked from one to 10, with a decile one indicating the most disadvantage. Decile rankings were used to identify whether the programme was used by schools across the range of socio-economic status.
|
STATISTICAL ANALYSES |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
Data were collected from all schools achieving Heartbeat Awards from January 1996 to December 1997 on sales of 20 items, and the results presented here represent 2 years of monitoring. The data from a small number of schools (32, 14%) were counted twice, as these schools achieved successive Heartbeat Awards during the data collection period. Data were grouped according to a school's level of participation in Heartbeat Awards, from Year 1 to Year 7, and used to identify trends of food sales for specific items.
Food items excluded from analysis were those which 75% of schools applying for a Year 1 Heartbeat Award did not sell. Items excluded were croissants, hot chips, fried food other than chips, ice cream, chocolate ice cream and water.
The dependent variables were the percentage increases in sales of sandwiches and filled rolls, muffins and scones, low-fat pies, American hot dogs, popcorn, fruit, yoghurt and milk, and also the percentage decreases in the sales of cakes and biscuits, doughnuts and cream buns, pies and sausage rolls, crisps, sweets and soft drinks. The independent variable was the year of award. The data were analysed in PC SAS (Cary, NC), using a linear regression model (PROC REG). Because the statistical tests were applied to self-reported, biased data, there are limitations in evaluating how much change occurred as a result of programme implementation.
|
RESULTS |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
Two hundred and thirty-two Heartbeat Awards were achieved between January 1996 and December 1997. Of these, 32 were successive Heartbeat Awards within the study period. One hundred and thirty schools operated a canteen and 72 schools a lunch order food service. Details of school type and the food service offered are outlined in Table 1
.
|
Food sales trends were analysed using simple linear regression to identify an association between the changes in food sales and the number of Heartbeat Awards a school obtained (Table 2
). The number of Heartbeat Awards, therefore, provided a measure of the length of time a school had participated in the School Food Programme.
|
The percentage of schools reporting more sales of sandwiches and filled rolls increased significantly with the number of Heartbeat Awards (p = 0.0005), while the increase in sales of muffins and scones did not quite reach statistical significance (p = 0.0618) (Figures 1 and 2
). The percentage of schools reporting fewer sales of doughnuts and cream buns (p = 0.01) (Figure 3), pies and sausage rolls (p = 0.009) (Figure 4), crisps (p = 0.0065) (Figure 5) and sweets (p = 0.004) (Figure 6) increased with the number of Heartbeat Awards. Most of the variance in changes of sales for sandwiches and filled rolls, doughnuts and cream buns, pies and sausage rolls, crisps and sweets could be explained by Heartbeat Awards (R2 = 0.76–0.92).
|
|
|
|
|
|
Mean percentages for each category of sales for each food item were calculated. The ratio of per cent reporting an increase to per cent reporting a decrease was compared (Table 3
), changes were in a favourable direction for all foods: fruit (43.2% more : 2% less), low-fat pies (20.5% more : 4.7% less), popcorn (22.4% more : 2.4% less), milk (59.6% more : 1.8% less), yoghurt (34.4% more : 7.9% less), hot dogs (49.9% more : 6.1% less) and soft drinks (7.2% more : 10.4% less).
|
Thirty schools were audited between September 1996 and August 1997, 14 secondary schools, seven intermediate and nine primary schools. An analysis of the results of the audits found all of the schools successfully met the requirements for a Heartbeat Award.
The results of a comparison of decile rankings of socio-economic status of schools achieving Heartbeat Awards are presented in Table 4, and show a relatively even spread.
|
|
DISCUSSION |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
These results suggest that the School Food Programme was able to influence the sale of foods available in school food services. Ongoing participation in the programme by schools achieving up to seven Heartbeat Awards was associated with more schools reporting a reduction in the sales of doughnuts and cream buns, pies and sausage rolls, crisps and sweets. Similarly, more schools reported increasing sales of sandwiches and filled rolls. Even items which showed no trends over time showed important overall changes towards healthier choices.
Data were collected on 20 food items. Six food items were excluded from analysis on the basis of small numbers of schools attaining Year 1 Heartbeat Awards selling these. Of the remaining 14 food items, five showed statistically significant changes in sales in a healthy direction. No information was collected which identified the amount of change in the food sales, only the direction of change. Nor has it been established what combination of foods students purchased for lunch. However, anecdotal information suggested that those items found to be statistically significant in this project were those that con-tributed a significant proportion to a student lunch. Of these, four out of the five were fresh food items and measurement by canteen managers may be more accurate for these high-turnover, fresh food items.
One of the main advantages of the method of our data collection system was its simplicity, which allowed easy compliance. Respondent burden was low, as the data could be supplied quickly and did not require detailed calculations of sales figures. Respondents could simply compare changes in the quantities of food ordered since their previous Heartbeat Award. The resulting disadvantage from the data collection method lies in not being able to quantify the extent of change that occurred and that recall bias may have influenced the results.
Categorizing schools according to the number of Heartbeat Awards achieved suggests that continued positive changes occurred from schools receiving their first Heartbeat Award to schools receiving their seventh Heartbeat Award.
The data from a small number of schools that applied for successive Heartbeat Awards during the data collection period were counted twice. Although this was a cross-sectional study, it was decided to include these data, as the comparison was not between individual schools but rather the impact of successive Heartbeat Awards on the sale of food items. This method has been successful in identifying changes in high-volume items, e.g. increased sales of sandwiches and filled rolls, and reduced sales of pies and sausage rolls. Data collected implied that ongoing changes in behaviour were occurring among the pupils, however, no qualitative or quantitative data have been collected to confirm this. Identifying and reporting statistical significance without the collection of more robust data may have provided a false impression of programme effectiveness.
While these results identified strong trends in the sales of some food items, bias was a potential problem in this study because of the self-reported nature of the data collected. Random audits of 30 schools occurred each year, and analysis of audit outcomes supported the general validity of the information provided on the self-assessment forms. All schools audited had achieved the required criteria for a Heartbeat Award, however, exaggerated reported changes in sales of food items are still possible.
Self-selection bias is likely to be important in this study. Schools obtaining a Heartbeat Award and those which continue to achieve successive Awards are likely to have a higher degree of commitment to making healthy changes than schools which drop out of the programme after achieving fewer awards. However, this study was not attempting to evaluate award schools compared with non-participating schools. Instead it was exploring the impact of long-term participation in the programme and, in this instance, the Year 1 Heartbeat Award schools formed the baseline for comparison. Each year, they judged their sales against their own sales the year before.
Analysis of socio-economic status using decile rankings suggested that socio-economic disadvantage was not a barrier to programme implementation. A comparison of socio-economic status and implementation of a programme addressing school food services was not described in the literature.
Interventions described in the literature (Butcher et al., 1988; Simons-Morton et al., 1991; Snyder et al., 1992; Raizman et al., 1994; Whitaker et al., 1994) involved up to 96 elementary schools within a defined urban location and a maximum of 4 years intervention (Luepker et al., 1996). In three of these studies, complete menus were modified to ensure levels of nutrients were within defined levels. These studies provided intensive training and support for food service staff, including assistance with modifying recipes and food preparation techniques (Simons-Morton et al., 1991; Snyder et al., 1992; Luepker et al., 1996). The main measure was nutrient analysis of foods served. While successful in achieving their goals, measures of change in the food service were independent of student food selections. No information was provided on the impact of these changes on student food selections.
A randomized intervention to increase selection of low-fat items, counted the number of low-fat entrees selected by students, when these items were labelled and promoted (Whitaker et al., 1994). While this intervention was successful in increasing the selection of low-fat items, the intervention was for a short duration (4 months).
The School Food Programme described in the current report provided only a low level of support to food service staff and relied on simple written guidelines to support staff in making changes to the variety of foods offered. Because New Zealand school food services were not able to withstand financial losses, the changes in food items offered had to be translated into the students' selection patterns, otherwise they could not be maintained. Change was not imposed on food selections but was supported by students altering their food selections to choose healthier items.
In conclusion, the School Food Programme was successful in achieving its aim of increasing the access of children to food which is nutritious, safe and sufficient in quantity. The low level of support provided to schools and simplicity of measurements enabled the programme to be implemented and monitored nationally. Socio-economic status did not appear to be a limiting factor for schools implementing the programme. Despite problems with bias, the evaluation measures identified continued change occurring in schools as a result of the intervention. The extent of change was not quantifiable, and further evaluation is needed to determine the impact of the programme on students' daily nutrient intake.
|
ACKNOWLEDGEMENTS |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
The School Food Programme was funded by the Health Funding Authority of New Zealand and the Heart Foundation. The authors wish to acknowledge Rodger Peach, Mary-Ann Dehar and Sally Casswell on whose work the model is based. The authors would also like to acknowledge the work of all the people who have worked on the programme since its inception.
|
REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONPROGRAMME DESCRIPTIONPREVIOUS OUTCOME EVALUATIONMETHODSSTATISTICAL ANALYSESRESULTSDISCUSSIONACKNOWLEDGEMENTSREFERENCES |
|---|
Butcher, A., Frank, G., Harsha, D., Sherpas, D., Little, S., Nicklas, T., Hunter, S. and Berensen, G. (1988) Heartsmart: a school health program meeting the 1990 objectives for the nation. Health Education Quarterly, 5, 17–34.
Luepker, R., Perry, C., McKinlay, S., Nader, P., Parcel, G., Stone, E., Webber, L., Elder, J., Feldman, H., Johnson, C., Kelder, S. and Wu, M. (1996) Outcomes of a field trial to improve children's dietary patterns and physical activity. Journal of the American Medical Association, 275, 768–775.[Abstract]
Mcdonald, G. (1996) Where next for evaluation, Editorial. Health Promotion International, 11, 171–173.
Nutbeam, D., Smith, C. and Catford, J. (1990) Evaluation in health education. A review of progress, possibilities and problems. Journal of Epidemiology and Public Health, 44, 83–89.
Peach, R., Dehar, M., Casswell, S. and Gourley, G. (1996) Heartbeat Awards in intermediate and secondary schools in New Zealand: outcomes of the first two years. Health Promotion Journal of Australia, 6, 42–50.
Raizman, D., Montgomery, D., Osganian, S., Ebzery, M., Evans, M., Nicklas, T., Zive, M., Hann, B., Snyder, M. and Clesi, A. (1994) CATCH: food service program process evaluation in a multicenter trial. Health Education Quarterly, Supplement 2, S51–S71.
Rychetnik, L., Nutbeam, D. and Hawe, P. (1997) Lessons from a review of publications in three health promotion journals from 1989 to 1994. Health Education Research, 12, 491–504.
Simons-Morton, B., Parcel, G., Baranowski, T., Forthofer, R. and O'Hara, N. (1991) Promoting physical activity and healthful diet among children: results of a school based intervention study. American Journal of Public Health, 81, 986–991.
Snyder, M., Story, M. and Trenker, L. (1992) Reducing fat and sodium in school lunch programs: the Lunchpower! Intervention study. Journal of the American Dietetic Association, 92, 1087–1091.[ISI][Medline]
Whitaker, R., Wright, J., Koepsall, T., Finch, A. and Psaty, B. (1994) Randomized intervention to increase children's low-fat foods in school lunches. Journal of Paediatrics, 125, 535–539.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M.-A. Carter and B. Swinburn Measuring the 'obesogenic' food environment in New Zealand primary schools Health Promot. Int., March 1, 2004; 19(1): 15 - 20. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||