Health Promotion International Advance Access originally published online on October 17, 2007
Health Promotion International 2007 22(4):299-306; doi:10.1093/heapro/dam024
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The development of measures of community capacity for community-based funding programs in Canada
1Public Health Agency of Canada, Edmonton, Alberta, Canada 2Centre for Interdisciplinary Research in Rehabilitation and Social Integration (CIRRIS-IRDPQ), Université Laval, Quebec, Canada 3Centre for Health Promotion Studies, University of Alberta, Edmonton, Alberta, Canada 4Faculty of Health and Social Development, University of British Columbia Okanagan, Kelowna, British Columbia, Canada
* Corresponding author. E-mail: mfran1971{at}yahoo.ca
| SUMMARY |
|---|
|
|
|---|
Improving community capacity for influencing actions on the determinants of health is an immediate outcome of many Public Health Agency of Canada-funding community-based programs. Despite the importance of this outcome, it has been difficult to measure and describe the contribution of funding programs to improving community capacity. This paper reports on a study conducted to develop and establish the psychometric properties of scales that measure community capacity to address health issues in the context of federally funded community-based programs. A literature review and national think tank with 21 experts informed the development of the first draft of the scales that outlined nine key domains of community capacity. Two focus groups with community practitioners provided information on the face and content validity and general usability of this draft instrument. The revised instrument was sent for pilot testing to 114 community organizations. Qualitative and quantitative analyses were performed to assess the validity, reliability and usability of the instrument. Twenty-nine organizations returned a completed instrument (25% response rate). Principal Component Analysis confirmed scale unidimensionality for eight multi-item scales: all of the component loadings were considered good with all scales loading between 0.60 and 0.92. Scale internal consistency was also considered high with alphas between 0.72 and 0.86 for six of these eight scales. Spearman's correlations were significant for the remaining two multi-item scales (composed of two items each), indicating that the two items for each scale were significantly correlated to each other. One scale could not be analyzed quantitatively, as it contained only a single item. Triangulation of qualitative and quantitative results found consistency in interpretations of scale response sets. Feedback on the instrument indicated interest in using it for project planning and evaluation. Psychometric analyses and triangulation provided evidence of the construct validity and reliability of the instrument. The final instrument covers 9 domains and has a total of 26 items, each with a four-point rating scale and a section for qualitative contextual comments. The instrument provides quantitative and qualitative information on community capacity within the context and scope of community-based funding programs.
Key words: community capacity; government programs; evaluation methodology
| INTRODUCTION |
|---|
|
|
|---|
Building community capacity can influence community health, the sustainability of community initiatives and communities' abilities to respond to emerging health issues (Torjman, 1999
| METHODS |
|---|
|
|
|---|
A six-step process was used to develop the community capacity scales. A similar process was used by several of the authors of the current paper in the development and validation of individual and organizational level scales to measure capacity for health promotion within health organizations (Anderson et al., 2004
Instrument development
Literature review
A literature review was conducted to explore how community capacity has been defined and measured and to identify validated community capacity measurement instruments. The following search terms were used: measurement, community empowerment, social capital, instrument and community capacity, in SilverPlatter databases (Ageline, CAB Health, CINAHL, Current Contents, Econlit, ERIC, IPA, Medline, PAIS, PsycInfo) and in gray literature.
We reviewed definitions of community capacity. Some authors treat community capacity as a generic attribute or generalized orientation (Hawe et al., 1997
; Bowen et al., 2000
; Veazie et al., 2001
; Gibbon et al., 2002
; Labonte et al., 2002
). Others argue that it is only appropriate to assess capacity in relation to a specific object or objective of change; that is, capacity for what. It is critical that community capacity assessment work not be undertaken in the abstract, but rather through a participatory process involving community stakeholders and a linking of capacity assessment with ongoing community transformation work (Bopp and Bopp, 2004
). Markey and Vodden (Markey and Vodden, 2000
) and Cheers et al. (Cheers et al., 2005
) also look at capacity for specific purposes, such as community economic development.
Capacity building is observed in the development of a set of assets and attributes that enable a community to take action (Jackson et al., 1999
; Bopp et al., 2000
; Thompson et al., 2000
; New South Wales Health Department, 2001
; Labonte and Laverack, 2001a
; Labonte and Laverack 2001b
; Gibbon et al., 2002
). Casswell (Casswell, 2001
) explicitly distinguishes two approaches, which she terms community development and community action. In the former, capacity building could be an end in itself. Deliberate effort can be invested in a parallel track, with one track representing the goal of capacity building, and the other goal being that of tangible community improvement (see also Hawe et al., 1998
; Labonte et al., 2002
). In this way, community capacity building can serve as a proxy for longer term outcomes that often cannot be measured within projects' immediate funded period (Easterling et al., 1998
; Smith et al., 2001
; Burdine et al., 2003
). In the community action approach, the achievement of a specific change is intended and capacity building is a by-product of successful (or even unsuccessful) collaborative efforts around this particular issue (Brownson et al., 1996
, as cited in Labonte et al., 2002
). For some, it is always both means and end (Laverack, 2005
).
The following health-oriented definition of community capacity building was selected for its relevance to the context of this research project: an approach to the development of sustainable skills, organizational structures, resources and commitment to health improvement in health and other sectors, to prolong and multiply health gains many times over (New South Wales Health Department, 2001
).
Many writers define community capacity in part through a list of elements or domains (Easterling et al., 1998
; Goodman et al., 1998
; Bopp et al., 2000
; Chaskin, 2001
; Laverack, 2001
; New South Wales Health Department, 2001
). The literature review revealed substantial overlap regarding the nature of these domains, and there is no evidence to date that any single conceptualization is demonstrably superior to the others (Labonte and Laverack, 2001a
). Nine domains were identified by the research team as being important in the context of PHAC-funding programs; that is, they were areas in which local efforts could conceivably have an observable and measurable impact within a relatively short time frame. These domains include participation; leadership; community structures; asking why; resource mobilization; links with others; role of external supports; skills, knowledge and learning; sense of community. On the basis of the literature, a representative list of potential indicators for each of the nine domains was developed (e.g. 30 possible indicators were listed for the domain links with others).
Within the literature, a total of seven instruments were found that could be accessed in their entirety and reviewed (Aspen Institute, 1996
; Alberta Heart Health Project, 1999
; Jackson et al., 1999
; Bopp et al., 2000
; Hawe et al., 2000
; Thompson et al., 2000
; Bush et al., 2002
; Gibbon et al., 2002
). Of these instruments, only four were established as valid and reliable (Hawe et al., 2000
; Thompson et al., 2000
; Chaskin, 2001
; Laverack, 2001
). None were designed specifically for the context of community-based funding programs of interest to this study. For example, none of the instruments collected project-level data that could be analyzed both quantitatively (using statistical analyses) and qualitatively on the basis of a consistent set of indicators.
National think tank
A group of 21 people from across Canada attended the think tank in January 2003. The research team selected invitees on the basis of recognized expertise and experience in the measurement of community capacity and involvement in one of the following sectors: academia (n = 7), community (n = 7) or PHAC (n = 7). Think tank participants reviewed the domains and indicators identified in the literature and concluded that all nine domains were relevant for measurement in the context of PHAC's community-based programs. Participants identified indicators for each of the nine domains on the basis of the longer lists mentioned earlier, thus contributing to the content validity of the ensuing instrument (Streiner and Norman, 2003
). Participants also provided recommendations on the instrument design.
Instrument drafting
The research team designed a draft instrument that was divided into nine domains. Each domain had a number of indicator items (range 1–4, with a total of 26 items). Each item had four response options (just started, on the road, nearly there, we're there) and opportunity for the project to add contextual descriptive information. The contextual descriptive information provided allowed for an elaboration on why the response option was chosen and on the unique conditions in which the project was operating. For each domain, two additional items were included: an overall rating of the project's progress within the domain (using the same response set) and an open-ended question to capture information on areas the project would like to strengthen. The draft instrument thus had a total of 44 questions. To establish face validity, this draft was sent to all think tank participants for review. Minor revisions to question wording were made to the instrument on the basis of this review.
Focus testing
Two 3-h focus groups were held in February 2003, with a total of nine health promotion and community development practitioners residing in Edmonton, Alberta, who were reflective of the intended end-user population. Participants reviewed the instrument instructions, questions and formatting while responding to a series of questions designed to ensure that the instrument and instructions were free of ambiguities; to obtain comments on how to improve the instrument to make it user-friendly and relevant and to further establish and verify face and content validity. Minor revisions were made to the instructions and questions on the basis of the feedback from focus group participants.
Pilot testing
The revised instrument was sent to the key contact person in 114 PHAC-funded projects in the Alberta/NWT, Manitoba/Saskatchewan and Atlantic Regions for pilot testing. Projects were included if they were receiving community-based population health promotion-related funding in 2003. Completion of the instrument was voluntary. Four questions were added to the end of the instrument for the pilot phase that assessed respondents' (1) experience in completing the instrument, (2) suggestions for improving the instrument, (3) suggestions for making it available to projects and (4) views on how the instrument fits with their project planning and evaluation currently underway. Although the instrument was considered to be valuable for use in a project group setting, most of the instruments returned for analysis were filled out by one person, as other project stakeholders were busy with year-end reporting and other wrap-up activities. Of the 114 community-funded projects mentioned earlier, 29 returned the completed tool.
Data analysis
Quantitative data were analyzed using Statistical Package for Social Sciences version 11.0 (SPSS). The Kaiser Meyer Olkin test of sampling adequacy and Bartlett's test of sphericity determined that the sample and correlation matrix were satisfactory for factor analysis. Principal Component Analysis (PCA), an examination of the interrelationships among variables to determine which variables go together as unified concepts (Tabachnick and Fidell, 1996
), was used to assess the construct validity of eight scales. Cronbach's Alpha was used to assess internal consistency of items within scales (reliability) comprising three or more items. For scales with two items only, Spearman's correlation was performed to establish the strength of the relationship between the two items. Thus, PCA was used to assess the unidimensionality of each scale, and Spearman's correlation or Cronbach's Alpha was employed to assess how strongly items on scales were grouped together. The last scale, a sense of community, had one item only and could not be assessed quantitatively. Qualitative data were analyzed using content analysis to assess whether the contextual descriptive information provided by projects for each item related consistently to the selected response option. Content analysis was also used to assess feedback on the instrument overall.
We recognized that the validity of performing factor analysis on such a small sample is opposed to certain statistical rules such as the rule that requires a ratio of four subjects per item. With this in mind, we emphasize the exploratory nature of these analyses. In response to the small sample size, we decided to examine the unidimensionality of each scale reducing the limits and influence of the sample size on analysis, an approach successfully adopted by some of the authors in other psychometric papers (Anderson et al., 2004
; Anderson et al., 2005
; Barrett et al., 2005
; Plotnikoff et al., 2005
).
| RESULTS |
|---|
|
|
|---|
A total of 29 project sites returned a completed instrument in the pilot-testing phase (25% response rate). Descriptive statistics for each of the nine scales, including the number of items per scale, mean scale scores and standard deviations, item factor loading range, percent of variance accounted for by each scale and the scale alpha, are presented in Table 1. PCA confirmed scale unidimensionality for each of the eight multi-item scales; all of the component loadings were considered good, with all scales loading between 0.60 and 0.92. Scale internal consistency was also considered high with alphas between 0.72 and 0.86 for each of the scales with three or four items. Spearman's correlations were significant at the p
0.05 level for both of the two-item scales, indicating that the two items for each scale were significantly correlated to each other. The sense of community scale comprised one item only and therefore did not undergo any psychometric testing. Triangulation of qualitative data and the selected response option for each item found consistency in interpretations of the scale response sets.
|
Qualitative responses to each domain's item on the overall rating for progress made to date within the domain often repeated information provided on individual items. Thus, the research team determined that the overall rating items should be removed from each domain in future versions of the instrument, reducing the total number of items by nine and consequently reducing respondent burden.
The qualitative data on projects' experiences with instrument completion revealed several ways in which participants thought the user-friendliness of the instrument could be improved. Suggestions included reducing the length of the instrument and increasing the precision of the wording of definitions for domains and items. Qualitative feedback on the instrument indicated considerable interest in using it for project planning and evaluation. A number of respondents recommended that the instrument be applied in a group setting to generate group discussion and consensus on what stage a project is at in developing community capacity. This suggestion is consistent with the intent of the instrument. However, sufficient time must be set aside to organize the group for this purpose. Respondents' recommendations on how the instrument should be used included using the instrument at various stages (i.e. project planning, implementation and evaluation stages) of a project, as an evaluation instrument, and using it as a guide to build capacity by helping to identify project strengths and weaknesses.
Perhaps most importantly, the instrument appeared to have value in project development for community practitioners. Some may question the value to communities of employing formal and standard instruments or mechanisms to assess change, arguing that it imposes onerous and irrelevant burden. In contrast, our projects endorsed such work, as illustrated in the following comments:
- [The instrument] was helpful in helping us to reflect on the project and clarify our next steps.
- The questions stimulated reflection—either to remind me to focus more on an area ... or to celebrate what we have done.
- [The instrument] served as a refresher, as it raises many issues which ultimately enhance a project's success.
| DISCUSSION |
|---|
|
|
|---|
Building community capacity is an important strategy enabling communities to respond to emerging health issues and to improve health. According to the Bangkok Charter, support for capacity building is important because well organized and empowered communities are highly effective in determining their own health, and are capable of making governments and the private sector accountable for the health consequences of their policies and practices (World Health Organization, 2005, p. 2). Yet it can be difficult to justify investment in community capacity-building strategies when measuring community capacity building itself or the outcomes of this investment can be inconsistent and challenging. Prior to this study, PHAC lacked any instrument that could be appropriately used by funded projects to assess their efforts to build community capacity. As a result of this study, a valid and reliable instrument has been developed for this purpose. The final instrument covers 9 domains using 26 items, each with a four-point rating scale. Each item has a section for qualitative contextual information. The instrument is available at http://www.phac-aspc.gc.ca/canada/regions/ab-nwt/download.html.
There were some limitations in the process of developing the instrument. As a result of time and resource constraints (pilot testing occurred at the end of the fiscal year, which is a particularly busy time for project participants), the pilot test instrument was completed by individuals instead of groups of individuals or committees involved in projects. Pilot-testing the instrument in a group setting might have provided greater insight into how the instrument instructions could have been revised to be more effective for the group experience. Further testing in a group setting is recommended. Psychometric testing of the Sense of Community domain was not possible, as it contained one item. Further research on appropriate items for this domain is required; one option is to collect and analyze qualitative results from future projects to determine commonalities and develop additional items. Finally, the sample size for conducting factor analysis was smaller than recommended. Hence, research on a larger sample of community organizations is recommended to allow for additional confirmatory factor analytical work required to further establish construct validity of the measures. A test-re-test research design should be undertaken to assess instrument stability.
Potential uses of the instrument are many. Within the PHAC context, it is being used as a means to track progress in building community capacity and as a teaching tool to introduce the concepts and measurement of community capacity. The ability to quantify projects' work to build community capacity while also capturing important contextual information through qualitative data addresses the need for measurable results while remaining sensitive to local realities. The tool may also help both project stakeholders and PHAC better respond to community needs and build upon existing community capacities. Instrument users in community settings have indicated that the instrument is useful for project planning and evaluation at various stages. Projects using the instrument in a group setting may benefit from engaging stakeholders in a dialogue about what is going on within the project. The instrument could be adapted for other contexts where health promotion and disease prevention are being undertaken. To date, it has been used for educational purposes in workshops and university courses, and it has been adapted for assessing community capacity in the context of community economic development.
| FUNDING |
|---|
|
|
|---|
Funding for this research was provided by the Health Canada Office of the Chief Scientist. Funding to pay the Open Access publication charges for this article was provided by the Public Health Agency of Canada.
| ACKNOWLEDGEMENTS |
|---|
The authors acknowledge the valuable contribution of participants involved in the think tank, focus-testing and pilot-testing processes. The opinions expressed in this article are those of the authors and do not reflect the official opinions of Health Canada or the Public Health Agency of Canada.
| REFERENCES |
|---|
|
|
|---|
Alberta Heart Health Project. Handbook for Assessing Community Capacity, Heart of the Land Project (1999) Alberta: David Thompson Health Region, Red Deer.
Anderson D., Plotnikoff R., Raine K., Cook K., Smith C., Barrett L. Towards the development of scales to measure will to promote heart health within health organizations in Canada. Health Promotion International (2004) 19:471–481.
Anderson D., Plotnikoff R., Raine K., Barrett L. Development of measures of individual leadership for health promotion. International Journal of Health Care Quality Assurance Leadership in Health Services (2005) 18:i–xii.
Aspen Institute. Measuring Community Capacity Building: A Workbook-in-Progress for Rural Communities (1996) Aspen, CO: The Aspen Institute.
Auditor General of Canada. (2001) Report of the Auditor General of Canada. Chapter 9( Health Canada, a proactive approach to good health (http://www.oag-bvg.gc.ca/domino/reports.nsf/html/0109ce.html retrieved September 2002).
Barrett L. L., Plotnikoff R., Raine K., Anderson D. Development of measures of organizational leadership for health promotion. Health Education and Behavior (2005) 32:195–207.
Bopp M., Bopp J. Welcome to the swamp. Addressing community capacity in Ecohealth research and intervention. EcoHealth (2004) 1:24–34.
Bopp M., GermAnn K., Bopp J., Baugh Littlejohns L., Smith N. Assessing Community Capacity for Change (2000) Red Deer, Alberta: David Thompson Health Region and Four Worlds Centre for Development Learning.
Bowen G. L., Martin J. A., Mancini J. A., Nelson J. P. Community capacity: antecedents and consequences. Journal of Community Practice (2000) 8:1–20.[Medline]
Brownson R. C., Smith C. A., Pratt M., Mack N. E., Jackson-Thompson J., Dean C. G., et al. Preventing cardiovascular disease through community-based risk reduction: five-year results of the Bootheel Heart Health Project. American Journal of Public Health (1996) 86:206–213.
Burdine J. N., Felix M. R. J., Wendel M. L., Somachandran S. The political and policy implications of a rating system of community capacity to improve population health. Family Community Health (2003) 26:254–267.[Web of Science][Medline]
Bush R., Dower J., Mutch A. Community Capacity Index, Version 2 (2002) Brisbane: University of Queensland, Centre for Primary Health Care.
Casswell S. Community capacity building and social policy—what can be achieved? Social Policy Journal of New Zealand (2001) 17:22–35.
Chaskin R. J. Building community capacity: a definitional framework and case studies from a comprehensive community initiative. Urban Affairs Review (2001) 36:291–323.
Cheers B., Cock G., Keele L., Kruger M., Trigg T. Measuring Community Capacity: An Electronic Audit Tool (2005) Paper presented to the Second Conference on the Future of Australia's Country Towns, Bendigo, 11–13 July: Victoria. (retrieved from http://www.latrobe.edu.au/csrc/fact2/refereed/cheers_et_al.pdf.
Easterling D., Gallagher K., Drisko J., Johnson T. Building Health by Promoting Community Capacity: Summary (1998) Denver, Colorado: The Colorado Trust.
Ebbesen L., Heath S., Naylor P. J., Anderson D. Issues in measuring health promotion capacity in Canada: a multi-province perspective. Health Promotion International (2004) 19:85–94.
Gibbon M., Labonte R., Laverack G. Evaluating community capacity. Health and Social Care in the Community (2002) 10:485–491.[CrossRef]
Goodman R. M., Speers M. A., McLeroy K., Fawcett S., Kegler M., Parker E., et al. Identifying and defining the dimensions of community capacity to provide a basis for measurement. Health Education & Behavior (1998) 25:258–278.
Hawe P., Noort M., King L., Jordens C. Multiplying health gains: the critical role of capacity-building within health promotion programs. Health Policy (1997) 39:29–42.[CrossRef][Web of Science][Medline]
Hawe P., King L., Noort M., Gifford S. M., Lloyd B. Working invisibly: health workers talk about capacity-building in health promotion. Health Promotion International (1998) 13:285–295.
Hawe P., King L., Noort M., Jordens C., Lloyd B. Indicators to Help with Capacity Building in Health Promotion (2000) Sydney: New South Wales Health Department. State Health Publication No. (HP) 990099.
International Federation of Red Cross and Red Crescent Societies. World Disasters Report: Focus on Community Resiliency (2004) Bloomfield, CT: Kumarian Press Inc.
Jackson S., Cleverly S., Burman D., Edwards R., Poland B., Robertson A. Towards Indicators of Community Capacity. National Health Research and Development Program Project No. 6606-6084-002, Final Report (1999) Toronto, Ontario: Centre for Health Promotion.
Jamieson K., Simces Z. Connecting the Dots: Reflections from a Community Perspective on Population Health Policy Issues (2002) Vancouver, British Columbia: Social Planning and Research Council of British Columbia.
Labonte R., Laverack G. Capacity building in health promotion, part 1: for whom and for what purpose? Critical Public Health (2001) 11(a):111–127.[CrossRef]
Labonte R., Laverack G. Capacity building in health promotion, part 2: whose use and with what measurement? Critical Public Health (2001) 11(b):129–138.[CrossRef]
Labonte R., Bell Woodard G., Chad K., Laverack G. Community capacity building: a parallel track for health promotion programs. Canadian Journal of Public Health (2002) 93:181–182.[Web of Science][Medline]
Laverack G. An identification and interpretation of the organizational aspects of community empowerment. Community Development Journal (2001) 36:134–145.[Abstract]
Laverack G. Evaluating community capacity: visual representation and interpretation. Community Development Journal (2005) 41:266–276.[CrossRef]
Markey S., Vodden K. Success Factors in Community Economic Development: Indicators of Community Capacity (2000) British Columbia: Community Economic Development Centre, Simon Fraser University, Burnaby.
New South Wales Health Department. A Framework for Building Capacity to Improve Health (2001) Sydney: New South Wales Health Department. State Health Publication, no (HP) 990226.
Plotnikoff R., Anderson D., Raine K., Cook K., Smith C. Scale development of individual and organization infrastructure for heart health promotion in Regional Health Authorities. Health Education Journal (2005) 64:256–270.
Smith N., Baugh Littlejohns L., Thompson D. Shaking out the cobwebs: insights into community capacity and its relation to health outcomes. Community Development Journal (2001) 36:30–41.[Abstract]
Smith N., Baugh Littlejohns L., Roy D. Measuring Community Capacity: State of the Field Review and Recommendations for Future Research. Health Policy Research Program Project no. 6795-15-1001/4440001, Final Report (2003) Alberta: David Thompson Health Region, Red Deer.
Streiner D., Norman G. Health Measurement Scales (2003) 3rd edition. Oxford: Oxford University Press.
Tabachnick B., Fidell L. Using Multivariate Statistics (1996) 3rd edition. New York, NY: HarperCollins College Publishers.
Thompson D., Baugh Littlejohns L., Smith N. Caught in the web: piloting a methodology to assess community capacity in a rural heart health project. Canadian Journal of Program Evaluation (2000) 15:35–55.
Torjman S. Are Outcomes the Best Outcome? (1999) Ottawa, Ontario: Caledon Institute of Social Policy.
Veazie M. A., Teufel-Shone N. I., Silverman G. S., Connolly A. M., Warne S., King B. F., et al. Building community capacity in public health: the role of action-oriented partnerships. Journal of Public Health Management and Practice (2001) 7:21–32.[Medline]
World Health Organization and the Thai Ministry of Health. Bangkok charter for health promotion in a globalized world (2005) Sixth Global Conference on Health Promotion. Policy and partnership for action: addressing the determinants of health, 7–11 August, 2005: Bangkok, Thailand. Available at http://www.who.int/healthpromotion/conferences/. Accessed 5 January 2006.
![]()
CiteULike
Connotea
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||