Health Promotion International Advance Access originally published online on April 7, 2006
Health Promotion International 2006 21(2):130-135; doi:10.1093/heapro/dak005
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Evaluating the effectiveness of health promotion policy: changes in the law on drinking and driving in California
1 Department of Applied Statistics and Economics, University of Pavia Italy 2 Centers for Disease Control and Prevention Atlanta, GA, USA 3 emme & erre, Management and Research Padua, Italy
Address for correspondence: Stefano Campostrini Department of Applied Statistics and Economics, University of Pavia, Corso Strada Nuova, 65 27100 Pavia, Italy E-mail: stefano.campostrini{at}unipv.it
 |
SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
The purpose of the study was to determine the utility of general population health surveillance data for evaluating broad policy changes that relate to health promotion. Data were drawn from the United States (US) Behavioral Risk Factor Surveillance System (BRFSS) for one US state, California. Because these data are collected frequently and continually, a quasi-experimental approach to the evaluation was possible using a type of interrupted time series analysis or longitudinal impact analysis. A statistically significant decrease in the number of declared episodes of drinking and driving was found after enactment of new state policy. These findings were compared and found consistent with another study in California that examined the effect of changes in the law on alcohol-related traffic accidents. Our findings suggest that data from a behavioral surveillance system, in this case the BRFSS, are useful to evaluate the effect of a health promotion intervention. Further, the study demonstrates the utility of comparing different data sources when assessing a population-wide change in health promotion policy.
Key words: evaluting health promotion policy; surveillance; risk factors; drinking and driving
|
INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
This article evaluates a complex phenomenon, namely how broad changes in policy at the population level can be monitored in order to determine whether such policies are effective. The broader ramifications of such an evaluation, in light of recent attention to the production of evidence for the effectiveness of health promotion interventions, are also addressed.
Data collection systems that assess population morbidity and mortality, and more recently, systems that assess lifestyle or behavioral factors that contribute to morbidity and mortality are important in many ways. Such systems that collect data frequently and continuously are especially useful for analyzing the dynamic features of population health. The ideal model to monitor human behavior is a regularly repeated time series of cross-sectional surveys. However, it is quite clear that an actual continuous observation of the same individuals over time cannot be performed to assess human attitudes, opinions and behavior. Rather, a continuous data collection is defined as the product of a sequence of repeated surveys, or more precisely, it is characterized by multiple independent samples drawn sequentially over time, with observations of the same population not involving the same subjects. Further, the observations must be sufficiently near one another over time, avoiding gaps of such a consistency that the comprehension of what has happened between two observations would be uninterpretable. Data produced in this way can be used to estimate the prevalence of health-related behaviors and practices, identify sociodemographic variations, provide information to target health programs and services, address critical health issues in a timely manner, guide legislation and policy, measure health objectives, and assess the impact of policies and programs at various levels on the population. One such data system is the US Behavioral Risk Factor Surveillance System (BRFSS). Our purpose here is to assess the usefulness of this data collection system for evaluating the impact of a particular state policy.
In the United States, driving under the influence of alcohol (DUI) remains a major cause of crashes and injuries, particularly among people under 34 years of age (National Center for Health Statistics, 1994
). For example, in 1994, 
41% of the more than 40 000 traffic fatalities were related to alcohol (National Highway Traffic Safety Administration, 1995). During the last decade of the 20th century, changes in the drinking and driving laws were introduced at the state level (Hingson et al., 1996), including two different changes introduced in California. The first, implemented in January 1990, changed the legal definition of DUI, lowering the allowable blood alcohol limit from 0.10 to 0.08%. The second, implemented 6 months later, immediately suspended the license of any person who violated the new limit.
The purpose of the study, conducted by the Department of Statistical Science at the University of Padova, Italy, in collaboration with the US Centers for Disease Control and Prevention (CDC), was 2-fold, and used data from the BRFSS, a continuous survey carried out by all states in the United States, through the support and coordination of the CDC. We were interested foremost in the methodological issue of investigating whether data from a health surveillance system were useful for evaluating the effect of a policy intervention. Secondarily, we wanted to examine the substantive issue of assessing the impact of the changes in the law on drinking and driving among California adults.
|
METHODS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
The BRFSS is a unique, state-based surveillance system (McQueen, 1996
; Holtzman, 2003), currently active in all 50 states, the District of Columbia, Puerto Rico, Guam and the Virgin Islands. The system was designed to gather information on health behaviors and preventive practices primarily related to chronic disease and injury. Every month, a representative sample of persons 18 years or older is selected for interview in each participating state and territory. Because the system has been operating for two decades, it approximates a continuous data stream. It is the availability of this continuous data stream that allows us to assess the potential of these data for policy or intervention evaluation. For our analysis, we used data from California for the 3 years before and the 3 years after the year in which the interventions took place: a total of about 20 000 interviews and 84 points of observation (from January 1987 to December 19931). We examined monthly mean values of the responses to the following question: ‘During the past month, how many times have you driven when you've had perhaps too much to drink?’ The question wording remained the same over the 7-year period.
Because the BRFSS was designed to collect data continuously and frequently, a quasi-experimental approach to the evaluation was used (Campbell, 1963; Campbell and Ross, 1968; Caporaso and Pelowski, 1971). This is the so-called interrupted time series analysis or longitudinal impact analysis, introduced by the work of Box and Tiao (Box and Tiao, 1975; Box and Jenkins, 1976; McDowall et al., 1980). We wanted to observe whether the changes in the law produced a shift in the time series or any other change in its evolution; a method previously employed in studies using BRFSS or other data (Glass, 1968; Tyler and Brown, 1968; Ross et al., 1970; Deutsch and Alt, 1977; Escobedo et al., 1992). The model considers:
- a stochastic component (usually an ARIMA model) that takes into account all the dependencies among the observations and
- an intervention component of which we want to evaluate the significance.
Formally, the model is:
 |
The second component will be equal to zero until the observation corresponds to the intervention, at which point it will assume a non-zero value. If this value is statistically different from zero, we can conclude that the intervention had a significant effect.
The intervention component can be modeled according to three different combinations of effects that capture the duration of the effect (temporary or permanent) and the onset of the effect (gradual or immediate). Thus, we distinguish:
- an immediate and permanent impact;
- an immediate and temporary impact;
- a gradual and permanent impact.
 |
is the parameter measuring the effect on the observed series. We tested three models (Table 1): the effect of the January 1990 intervention [reduction of the allowable blood alcohol concentration (BAC) from 0.10 to 0.08%]; the effect of the July 1990 intervention (license suspension); and the effect of the combination of both interventions.
|
We also indirectly assessed validity by comparing our findings with those of another study conducted in California (Rogers, 1995
). Using intervention time series analysis, Rogers evaluated the influence of the new DUI law among the general population of DUI offenders, as measured by the number of alcohol-related traffic accidents. The California Highway Patrol (CHP) provided the accident data from the Integrated Traffic Records System (SWIRTS); these data are not self-reported and in view of how they are collected are assumed to be valid. |
RESULTS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
In all three cases, neither autocorrelation nor trend was detected, indicating a white noise model. Of the three, the third was found to be the best fit. Table 1 reports the numerical results.
The first parameter (1), which considered the effect of the first intervention (BAC reduction), was positive, i.e. it seems that the introduction of the law has increased the number of people declaring episodes of drinking and driving in the month previous to the interview. A possible explanation for the positive value is that the new law did not change the actual number of declared episodes of drinking and driving, but instead influenced (to a lesser extent) only the number of people considering themselves to have drunk perhaps too much. However, the parameter was not significant (P > 0.2), indicating no statistically significant effect of the first law. The other parameter is higher in absolute value 
and statistically significant. The real change occurred only after the second law was enacted, which produced a substantial decrease in the number of declared episodes of drinking and driving (Figure 1).
|
Results of the CHP comparison were consistent with those from our study, in that both alcohol-related accidents and declared drinking and driving episodes decreased with the enactment of the second law, license suspension, but less so with the first law which lowered the BAC limit. Moreover, the analysis conducted on the time series of accidents (Rogers, 1995
) with an identical approach to that in our analysis, led to the same conclusions: the first ‘intervention’ did not have a statistically significant effect on the time series, while the second one produced a highly significant change. |
DISCUSSION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Our findings suggest that changes in the drinking and driving law, specifically suspending the license of those who break the law, decreased drinking and driving behavior among adults in California. Our findings also suggest that data from a behavioral surveillance system, such as the BRFSS, can be useful to evaluate the effect of such a change. One often cited limitation of this type of survey is that it gathers self-reported information. Some behaviors, however, are not easily observable and self-reported information may be the only option. Furthermore, over or under estimates of behaviors that may result from self-reports would not influence our findings with respect to trends. Quite often in public health, the primary interest is on whether and how change occurs, i.e. is there an increasing or decreasing trend for a particular phenomenon; generally there is less interest in the actual value of a specific variable. For example, knowing that the percentage of smokers among an adolescent population over a certain time period is increasing may be much more useful than having a precise estimate of that percentage. The methodological consequence is that less attention can be paid to biases in the estimated value of a variable, since, if we can assume that these are stable over time (as quite often is reasonable in repeated measurements based on independent samples), the possible bias will not influence trend estimates.
In addition, a number of studies have shown that estimates from BRFSS for selected health-related behaviors are similar to those based on in-person interviews or observations (Gentry et al., 1985; Anda et al., 1989; Nelson, 1996; Nelson et al., 2001). We were also able to compare our results with another California study that carried out analyses using ‘actual’ data from alcohol-related traffic accidents, rather than data from self-reports. The similarity of our results with this study lends support to the validity of our conclusions. More generally, we believe that in many cases it is possible to use behavioral surveillance data for evaluation purposes; the study here is one example. In these analyses, of course, validity issues must be always addressed, but quite often data for analysis of an intervention's effect are difficult or impossible to collect, for ethical, feasibility or cost reasons. Furthermore, when ad hoc data are collected, selection bias, attrition and other problems intervene, severely jeopardizing the reliability and the validity of the results. Therefore, the continuous data stream offered by surveillance systems, such as the BRFSS, are often a reasonable alternative, certainly much less demanding of resources and sometimes as valid, or even more so.
Most importantly, our approach addresses the fundamental challenge for health promotion of how to show, in a ‘scientifically acceptable’ way that a policy change, stemming from a health promoting ideology, can be evaluated. Furthermore, it addresses the issue of complexity which is always present in assessing such broad population effects. Finally, it illustrates that we must have and use institutionalized systems of monitoring (Campostrini and McQueen, 2005) if we are to attribute cause to large scale policy changes that only manifest their effects over longer periods of time. Many of the challenges related to evaluating policy have been addressed in recent years (Campostrini, 2003). Conversely, McQueen and Springett (McQueen and Springett, 2001) have shown, reflecting many of the discussions of the WHO European Working Group on Health Promotion Evaluation, that there are a dearth of ready strategies to evaluate the effectiveness of health promotion policies at the population level. This article provides one such approach.
|
ACKNOWLEDGEMENTS |
|---|
The authors would like to acknowledge the state of California BRFSS staff.
|
FOOTNOTES |
|---|
1For 2 months, September and October 1987, the relevant questions for this study were not included in the BRFSS questionnaire in California.
2Three parameters characterize an ARIMA model: (p) the AutoRegressive parameter (the process is considered as a linear function of the p preceding values); (d) the order of Integration (typically not more than 2 for modeling purposes; it is convenient to eliminate the non-stationarity by means of d differences of the original series); and (q) the Moving Averages parameter (a linear combination of q random processes).
|
REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Anda R. F., Dodson D. L., Williamson D. F. (1989) Health promotion data for state health departments: telephone versus in-person survey estimates of smoking and alcohol use. American Journal of Health Promotion 4:32–36.
Box G. E. P. and Jenkins G. M. (1976) Time Series Analysis: Forecasting and Control (Holden-Day, Oakland, CA).
Box G. E. P. and Tiao G. C. (1975) Intervention analysis with applications to economic and environmental problems. Journal of the American Statistical Association 70:70–79.[CrossRef][Web of Science]
Campbell D. T. (1963) From description to experimentation: interpreting trends as quasi-experiments. In Harris C. W. (Ed.). Problems of Measuring Change (University of Wisconsin Press, Madison, WI).
Campbell D. T. and Ross H. L. (1968) The Connecticut crackdown on speeding: time series data in quasi-experimental analysis. Law and Society Review 3:33–53.
Campostrini S. (2003) Surveillance systems and data analysis. In McQueen D. V. and Puska P. (Eds.). Global Behavioral Risk Factor Surveillance (Kluwer, New York) pp. 47–55.
Campostrini S. and McQueen D. V. (2005) Institutionalization of social and behavioral risk factor surveillance as a learning system. Social and Preventive Medicine 50:S9–S15.
Caporaso J. A. and Pelowski A. L. (1971) Economic and political integration in Europe: a time series quasi-experimental analysis. American Political Science Review 65:418–433.[CrossRef][Web of Science]
Deutsch S. J. and Alt F. B. (1977) The effect of Massachusetts gun control law on gun-related crimes in the city of Boston. Evaluation Quarterly 1:543–568.[CrossRef][Web of Science]
Escobedo L. G., Chorba T. L., Remington P. L., Anda R. F., Sanderson L., Zaidi A. A. (1992) The influence of safety belt laws on self-reported belt use in the United States. Accident Annuals and Prevention 24:643–653.[CrossRef]
Gentry E. M., Kalsbeek W. D., Hogelin G. C. (1985) The behavioral risk factor surveys: II. Design, methods, and estimates from combined state data. American Journal of Preventive Medicine 1:9–14.[Medline]
Glass G. V. (1968) Analysis of data on the Connecticut speeding crackdown as a time series quasi-experiments. Law Society Review 3:55–76.
Hingson R., Heeren T., Winter M. (1996) Lowering state legal blood alcohol limits to 0.08%: the effect on fatal motor vehicle crashes. American Political Science Review 86:1297–1299.
Holtzman D. (2003) The analysis and interpretation of data from the Behavioral Risk Factor Surveillance System (BRFSS). In McQueen D. V. and Puska P. (Eds.). Global Behavioral Risk Factor Surveillance (Kluwer, New York) pp. 35–46.
McDowall D., McCleary R., Meidinger E. E., Hay Jr R. A. (1980) Interrupted Time Series Analysis (Sage Publications, Beverly Hills, CA).
McQueen D. V. (1996) Surveillance of health behavior. Current Issues of Public Health 2:51–55.
McQueen D. V. and Springett J. (2001) Introduction to part 4: policies and systems. In Rootman I., Goodstadt M., Hyndman B., McQueen D., Potvin L., Springett J., Ziglio E. (Eds.). Evaluation in Health Promotion: Principles and Perspectives (WHO Regional Publications, European Series) 92: pp. 337–340.
Health, United States, 1993 National Center for Health Statistics. (1994) National Center for Health Statistics, Hyattsville, MD.
Traffic Safety Facts, 1994 National Highway Traffic Safety Administration. (1995) US Department of Transportation, National Highway Traffic Safety Administration, Washington, DC, DOT HS 808 292.
Nelson D. E. (1996) The validity of self-reported data on injury prevention behavior: lessons learned from observational and self-reported surveys of safety belt use in the United States. Injury Prevention 2:67.
Nelson D. E., Holtzman D., Bolen J., Stanwyck C., Mack K. (2001) Reliability and validity of measures from the Behavioral Risk Factor Surveillance System (BRFSS). Social and Preventive Medicine 46:1–42.
Rogers P. N. (1995) The general deterrent impact of California's 0.08% blood alcohol concentration limit and administrative per se license suspension laws:. Volume 1 of an evaluation of the effectiveness of California's 0.08% blood alcohol concentration limit and administrative per se license suspension laws. (Report No. 158). Sacramento: California Department of Motor Vehicles.
Ross H. L., Campbell D. L., Glass G. V. (1970) Determining the effects of a legal reform: the British "breathalyzer" crackdown of 1967. American Behavioral Scientist 13:493–509.[CrossRef][Web of Science]
Tyler V. O. and Brown G. D. (1968) Token reinforcement of academic performance with institutionalized delinquent boys. Journal of Educational Psychology 59:164–168.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||