Health Promotion International Advance Access originally published online on June 21, 2005
Health Promotion International 2005 20(4):391-397; doi:10.1093/heapro/dai013
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DEBATE |
Responsive evaluation in health promotion: its value for ambiguous contexts
Healthcare Ethics and Philosophy, University of Maastricht, The Netherlands
Address for correspondence: Ms T. A. Abma, University of Maastricht, Healthcare Ethics and Philosophy, PO Box 616, 6200 MD Maastricht, The Netherlands, E-mail: t.abma{at}zw.unimaas.nl
 |
SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONRESPONSIVE EVALUATIONCONTRIBUTION TO HEALTH PROMOTION...DISCUSSIONREFERENCES |
|---|
Responsive evaluation offers a perspective in which evaluation is reframed from the assessment of program interventions on the basis of policymakers' goals to an engagement with all stakeholders about the value and meaning of their practice. This article argues for this perspective both generally and more particularly in relation to health promotion. Responsive evaluation is especially appropriate in health promotion contexts characterized by a high degree of ambiguity. Ambiguity refers to the absence of or contradictory interpretations about what needs to, can and should be done, when and where. Ambiguity is high in the case of non-routine programs, lack of knowledge about success indicators, collaborative and communitybased programs and the absence of consensus among stakeholders. In health promotion contexts marked by a low degree of ambiguity random controlled trials (RCTs) and quantitative methods are to be considered. This implies the evaluators should assess the degree of ambiguity of a situation before deciding about an appropriate design.
Key words: ambiguity; health promotion; responsive evaluation
|
INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONRESPONSIVE EVALUATIONCONTRIBUTION TO HEALTH PROMOTION...DISCUSSIONREFERENCES |
|---|
‘Evidence-based policy’ and ‘evidence-based medicine’ have become catchwords in many countries around the world. The evidence-based movement draws on experimental methods advocating the wider application of random controlled trials (RCTs) and methodological rigour. There is, however, an increasing uneasiness among health researchers about the fact that RCTs and meta-analyses have become the norm in the field of health promotion (Nutbeam, 1997
; Tones, 1997; Nutbeam, 1998; McQueen, 2000; Koelen et al., 2001; Morse et al., 2001). Qualitative methods, process evaluations and more participatory forms of research are considered as an alternative (Kahan and Goodstadt, 2001; Springett, 2001; Wallerstein et al., 2002).
In this article I propose responsive evaluation as a perspective to evaluation that generates qualitative evidence about the value and meaning of programs. Evaluation criteria to assess the program's effectiveness are not only derived from the goals and intentions of policymakers but include a wide range of issues of as many stakeholders as possible (Stake, 1975; Guba and Lincoln, 1989). Responsive evaluation is a disciplined form of inquiry that enhances the understanding of human behaviour, promotes holistic thinking, offers contextual information and brings in the perspective of the community or target group. Responsive evaluation is especially appropriate in health promotion contexts characterized by a high degree of ambiguity (Abma and Noordegraaf, 2003).
|
RESPONSIVE EVALUATION |
|---|
|
TOPSUMMARYINTRODUCTIONRESPONSIVE EVALUATIONCONTRIBUTION TO HEALTH PROMOTION...DISCUSSIONREFERENCES |
|---|
Responsive evaluation has been developed in the field of education (Stake, 1975
; Guba and Lincoln, 1989; Abma and Stake, 2001) as an alternative for the shortcomings related to an over reliance on experimental methods, the lack of attention to the process and implementation of programs and the distanced relationship with stakeholders and subjects. The term ‘negotiation’ (versus measurement, description or judgement) characterizes the essence of responsive evaluation. Criteria for the evaluation are derived from the issues of various stakeholders and gradually emerge in conversation with stakeholders. These issues should be related to underlying value-systems in order to facilitate the negotiations and mutual understanding among stakeholders.
Stakeholders are groups of people whose interests are at stake. In a responsive evaluation stakeholders should actively participate in the evaluation process; they are involved in the formulation of questions, the selection of participants and the interpretation of findings (Greene, 1997). Stakeholders become partners (versus information-givers) in the evaluation process. Deliberate attention should be paid to the identification of ‘victims’ or ‘silenced voices’ (Lincoln, 1993), because they are often hard to find, for example, because they want to remain anonymous or fear sanctions.
Methodologically plurality implies that the ‘design’ gradually emerges in conversation with the stakeholders. Metaphorically one may compare the designing process in a responsive evaluation with improvizational dance (Janesick, 2000). Whereas a minuet prescribes definite steps, turns and movements, improvization is spontaneous and reflexive of social conditions. The evaluator charts the progress and examines the study as it proceeds by keeping track of his/ her role in the process.
Besides the identification of issues the evaluator should create conditions for the interaction between stakeholders. This is a dialogical process and includes deliberation. Participants will explore each other's beliefs and persuasions. Listening and probing characterize this process, rather than confronting, attacking and defending. Central features of dialogue are openness, respect, inclusion and engagement (Abma et al., 2001; Greene, 2001). Dialogue may lead to consensus. Absence of consensus is, however, not problematic, because differences stimulate learning processes (Widdershoven, 2001). Conditions for dialogue are the willingness of stakeholders to participate, to share power and to change in the process (Abma et al., 2001). Responsive evaluation requires a certain power balance to give all stakeholders a fair chance in the process. If these conditions are not met evaluators should invest time in developing conditions of trust and safety.
In the exploration of issues the evaluator will concentrate on controversies, and end the evaluation with an ‘agenda for negotiation’ that covers controversies. The evaluator will not formulate conclusions or recommendations, because this prevents the input and interpretation by stakeholders. The evaluation report is a vehicle for dialogue (Abma, 1998). This ‘working document’ will portray the existing diversity.
In a responsive evaluation evaluator roles include those of interpretator, educator, facilitator and Socratic guide. As interpretator the evaluator has to endow meanings to issues. The role of educator refers to the creation of understanding by explicating various experiences to involved groups. Facilitator refers to the organization of the dialogue and the creation of required conditions. As Socratic guide the evaluator will probe into taken for granted ideas, final truths and certainties, and bring in new perspectives (Schwandt, 2001).
Philosophical perspectives to knowledge
Responsive evaluation is grounded in a social constructivist perspective to knowledge. In this perspective human beings are considered as active interpreters of their world. In the interpretation people will bring their own biography, training and pre-assumptions into play. Given this diversity reality is in principle multiple; different meanings may exist side-by-side. Meaning construction is a dialogical process; subject and object mutually influence each other and will change in the process. Thus, evaluators cannot be objective and will inevitably bring in their prejudices. Prejudices are, however, not negative, but a prerequisite for understanding if they are confronted with other, strange experiences (Koch, 1996; Schwandt, 1999; Widdershoven, 2001). This confrontation stimulates a learning process to probe into the differences of insight. If evaluators and participants know more about the evaluated program than before, this is an indication of the productive use of their prejudices.
Validation strategies
A responsive approach uses mixed methods (Greene et al., 2005; Stake and Abma, 2005). Qualitative methods are appropriate to gain an insight in the experiences and complexity of our social world. In order to guarantee the quality of knowledge the following criteria and validation strategies have been introduced.
The credibility of interpretations in the eyes of stakeholders as a validation strategy requires that respondents receive interpretations of (group) interviews with the question if they recognize the analysis (so called ‘member check’). Triangulation of sources and methods helps to include different perspectives and to prevent biases. Evaluators should critically and openly reflect on the main filters that are influencing their interpretations. Keeping a reflexive logbook or journal proves to be a good way to keep track of the process and the evaluator's role in it.
Responsive evaluation results in context-bound knowledge. This local knowledge can be generalized from the studied context to the context of readers of the evaluation report if it contains ‘thick descriptions’. ‘Thick descriptions’ not only reveal factual details, but also include meanings of experiences and events. Whether or not the results can be transferred to other situations is to be decided by readers.
The quality of the process is partly dependent on the created power balance: all participants should be able to have ‘a say’. Authenticity refers to the enhancement of personal and mutual understanding, changes in perspectives, and increased opportunities to act. In responsive evaluation one especially has to be aware of power relations (Koch, 2000). One should try to find means to give voice to less powerful people and groups, for example, via in-depth interviews or focus groups. Active engagement of as many stakeholders as possible and deliberation minimizes the chance of bias and domination of one party. Afterwards, it needs to be checked whether the dialogical process was really open. A careful reading of the transcript can do this.
|
CONTRIBUTION TO HEALTH PROMOTION CONTEXTS |
|---|
|
TOPSUMMARYINTRODUCTIONRESPONSIVE EVALUATIONCONTRIBUTION TO HEALTH PROMOTION...DISCUSSIONREFERENCES |
|---|
Responsive evaluation offers a unique vision on evaluation given the link between an interpretive methodology and a democratic and emancipatory ideology. Responsive evaluation has been implemented in various policy sectors, including the field of health care. Case examples include evaluation of injury prevention programs for performing art students (Abma, 2001b
), palliative care programs and units (Abma, 2000b; Groen, 2000; Abma, 2001a), elderly care (Koch, 1994; Koch, 1996; Koch, 2000), nursing curricula (Koch, 2000), rehabilitation programs for psychiatric patients (Abma, 2000c; Wadsworth, 2001) and supported employment programs for mentally handicapped (Widdershoven and Sohl, 1999). Responsive evaluation was also used to plan positive youth development programs (Huebner and Betts, 1999) and to formulate quality criteria in psychiatric care (Berghmans et al., 2001). Finally, responsive evaluation has been applied in the context of medical technology assessment (Van der Wilt and Reuzel, 1998).
Responsive evaluation is especially appropriate in health promotion contexts characterized by ambiguity. Ambiguity refers to the absence of or contradictory interpretations about what needs to, can and should be done, when and where (Weick, 1995; Abma and Noordegraaf, 2003). High degrees of ambiguity can be found in the following situations.
Non-routine interventions. Interventions, which are not standardized, but incidental. Algorithms—shared understanding of how to act in certain situations—break down or are absent. There is a lack of knowledge about the effectiveness and mechanisms of interventions.
Collaborative interventions. Interventions, which are community-based are the joint product of an interaction between participants and cannot always be predefined and specified.
Absence of consensus among stakeholders. The processes and interventions, which are disputable and lead to conflicts, such as the forced use of anti-conception among mentally handicapped.
Non-routine interventions in health promotion are hard to measure, because the means and outcomes, and causal relationships are unknown. When algorithms are absent it is difficult to attribute outcomes to aspects of the intervention. It is hard to find out why certain objectives and standards are not met. Lack of knowledge and empirical evidence on the effects of health promotion interventions requires systematic observation of community-based clinical practice (Lincoln, 1992). In the case of non-routine interventions there is a need for qualitative data to gain an understanding in the question of why and how interventions work. The effectiveness of interventions are related to human understandings, beliefs, fears, attitudes, prejudices, hopes, dreams and aspirations. This human, cultural and social side of health promotion can be understood and assessed well in all its complexity following a responsive approach.
Collaborative interventions in health promotion are characterized by a high degree of ambiguity, because the intervention is developed in interaction with the population or target-group. They are involved in the diagnostic process, the preparation and further development of interventions. This kind of approach is difficult to incorporate in a study design drawn up in accordance with customary ideas, because the purpose and content of the intervention are not fully established at the start of the activity, because the time schedule is undetermined and unpredictable and because only time will tell what outcome measurements are suitable. The flexible ‘design’ of responsive evaluation allows it to be more responsive to these problems. The ‘design’ emerges on the basis of the ‘issues’ that appear to be important in daily practice (versus intentions). Responsive evaluation acknowledges that relevant outcomes cannot always be preordained and that practitioners who implement health promotion plans are confronted with the difficult task adjusting the design to the local context. It also acknowledges that during the implementation process human, social and behavioural and cultural factors intervene.
Absence of stakeholder consensus refers to situations characterized by confusion and conflicts between stakeholders. Various stakeholders will have various interests and diverging, sometimes conflicting, ideas about the most appropriate interventions and standards to assess programs (Ray and Mayan, 2001). In the medical sector eight stakeholder parties can be distinguished, among them insurers, regulators, health care organizations, health professionals, the research community, the medical–industrial complex, the legal system and consumers. These parties have various agendas, such as fiscal accountability, professional effectiveness, quality of care, safety and personal needs. Ray and Mayan argue that the general public has a small power base with regard to the production and the use of evidence (Ray and Mayan, 2001). Responsive evaluation offers an approach to deal with various stakeholder interests and to restore the power balance given the engagement, inclusion and active participation of various stakeholders, including the local community and target groups.
Responsive evaluation is not only responsive to health promotion contexts characterized by a high degree of ambiguity; the perspective is also synergistic with health promotion. Below three parallels between responsive evaluation and health promotion are listed.
- The move from passive constructions of health to active and meaningful participation in the diagnostic process, in setting up and further development of the intervention, from absence of disease to wellness, from sickness prevention to health promotion, reflects new understandings of the move from being a research object to a respondent and active participant in the evaluation process.
- The move from single causes to multiple, mutually interacting factors and the need for co-ordination from different angles and sectors reflects responsive evaluation's embrace of contextual interaction, mutually shaping forces and webs of influence in human life and health.
- The move from a professional posture, which focuses on disease, treatment and patient freed of accountability, to a professional posture where responsibility is taken for a whole patient and shared equally between practitioner and patient. This reflects the move to shared decisions, shared constructions and dialogue in responsive evaluation.
In short, the characteristics of responsive evaluation allow it to be responsive to health promotion contexts characterized by a high degree of ambiguity. Moreover, the emerging ideas in health promotion are more congruent with responsive evaluation.
|
DISCUSSION |
|---|
|
TOPSUMMARYINTRODUCTIONRESPONSIVE EVALUATIONCONTRIBUTION TO HEALTH PROMOTION...DISCUSSIONREFERENCES |
|---|
Implementation of responsive evaluation requires that both the evaluator and policymaker are willing to share power and control with other stakeholders and to engage in more horizontal collaborations with other stakeholders. Besides analytical skills a responsive evaluator requires interpersonal, communication and negotiation skills (Guba and Lincoln, 1981
). These skills can be learned best by doing it, preferably as an ‘apprentice’ to an empathic and knowledgeable evaluator, in a climate of support and encouragement (Swenson, 1991).
A barrier in the implementation of responsive evaluation relates to the perceived weakness in qualitative methods. Evidence is often restricted to quantitative facts derived from large sample, randomized experimental designs (McQueen, 2000). Madjar and Walton argue that a broad notion of evidence also includes qualitative evidence in the form of lived experiences, case histories and stories (Madjar and Walton, 2001). This kind of evidence is important because it enhances the understanding of human behaviour, it promotes holistic thinking, offers contextual information and brings in the perspective of the community or target group. Qualitative data are more than just ‘mere opinions’ when generated in a systematic way and according to internal verification and validation strategies.
Still another barrier for implementation is a concern about the practical application of such work. One of the strengths of responsive evaluation is that practitioners do not need to wait for results until the evaluation is completed, but can in fact begin to use findings during the process given the frequent stakeholder communication and participation. This is known as ‘process use’: the acceptance of knowledge and personal and organizational learning processes occurring during the evaluation (Shulha and Cousins, 1997). Stakeholder participation, engagement and communication promote ‘process use’ (Greene, 1988). It gives participants confidence in the quality of information and in the ability to use the information. Responsive evaluation does not only deliver evidence in time, but produces context-bound knowledge that enables practitioners to use results on the specific needs, life-styles, preferences, problems and other particularities of their community or target group.
Some suggest that responsive evaluation is only feasible within a relatively closed system, such as a school community. Good experiences have been noticed, however, about the possibility of responsive evaluation within the context of social renewal in the city of Rotterdam (Fortuin, 1993; Fortuin, 1994; Abma, 1997). In three ‘backward’ neighbourhoods the evaluator spent six months gaining an insight in the actors developing authority, resistance and kinds of social action undertaken. Then stakeholders (citizens, civil servants, social welfare organizations, business people and politicians) were invited to list projects that envisaged social renewal. In each of these neighbourhoods the evaluator selected three of the nominated projects for further investigation. This example demonstrates the possibility to conduct responsive evaluation within an open environment.
Finally there is the issue of how the evaluator proceeds in situations marked by the absence of consensus and diverging perspectives on the situation. In these situations the intervention cannot be meaningfully defined or reduced to simple endpoints, because there are many, diverse and sometimes contradictory endpoints or a vacuum of meaning. A responsive evaluator will then not produce unilateral conclusions and recommendations how to improve the practice, but organize a dialogue between stakeholders to assist them to gain more insight in the different perspectives and existing confusion. This may not lead to a decision, but often does enhance the mutual learning and understanding between stakeholders as a vehicle to practice improvement.
In other fields responsive evaluation has led to paradigm debates (Guba et al., 1990). Although it is important to discuss philosophical perspectives on knowledge, this article has emphasized that every evaluation approach has its own weaknesses and strengths in certain situations. Balanced evaluation acknowledges the apparent fuzziness of certain health promotion contexts, and adjusts the design to the degree of ambiguity.
|
ACKNOWLEDGEMENTS |
|---|
Edited text of a paper presented at the ‘Evaluation in Health Promotion’ conference, Rotterdam, 17 February 2003. I would like to thank Willy de Haes, Hans Saan, David McQueen and the reviewers for their constructive feedback on the earlier version of this article.
|
REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONRESPONSIVE EVALUATIONCONTRIBUTION TO HEALTH PROMOTION...DISCUSSIONREFERENCES |
|---|
Abma, T. A. (1997) Playing with/in plurality: revitalizing realities and relationships in Rotterdam. Evaluation, 3, 25–48.
Abma, T. A. (1998) Writing for dialogue, text in an evaluation context. Evaluation, 4, 434–454.[Abstract]
Abma, T. A. (2000b) Responding to ambiguity, responding to change. The value of a responsive approach to evaluation. Evaluation and Program Planning, 23, 461–470.[CrossRef]
Abma, T. A. (2000c) Stakeholder conflict: a case study. Program Planning and Evaluation, 23, 199–210.[CrossRef]
Abma, T. A. (2001a) Evaluating palliative care. Facilitating reflexive dialogues about an ambiguous concept. Medicine Health Care and Philosophy, 4, 261–276.
Abma, T. A. (2001b) Reflexive dialogues: a story about the development of injury prevention in two performing art schools. Evaluation, 7, 238–252.[Abstract]
Abma, T. A. and Stake, R. E. (2001) Responsive evaluation, roots and evolution. New Directions for Evaluation, 92, 7–22.
Abma, T. A., Greene, J., Karlsson, O., Ryan, K., Schwandt, T. S. and Widdershoven, G. (2001) Dialogue on dialogue. Evaluation, 7, 164–180.[Abstract]
Abma, T. A. and Noordegraaf, M. (2003) Public managers amidst ambiguity, towards a typology of public management settings. Evaluation, 9, 285–306.[Abstract]
Berghmans, R., Elfahmi, D., Goldsteen, M. and Widdershoven, G. (2001) Kwaliteit van dwang en Drang in de Psychiatrie. Eindrapport. Utrecht/Maastricht: GGZ Nederland; Cluster Zorgwetenschappen, sectie Gezondheidsethiek en Wijsbegeerte, Universiteit Maastricht; Instituut voor Gezondheidsethiek, Universiteit Maastricht.
Fortuin, K. (1993) Gezonde stad stuit op adoptieproblemen. Tijdschrift Voor Gezondheid en Politiek, 11, 6–8.
Fortuin, K. (1994) Evaluatie-onderzoek in een paradoxale beleidscontext. In Francke, A. and Richardson, R. (eds) Evaluatie-onderzoek, Kansen voor een Kwaliatieve Benadering. Bussem, Coutinho, pp. 155–179.
Greene, J. C. (1988) Stakeholder participation and utilization program evaluation. Evaluation Review, 12, 91–116.
Greene, J. C. (1997) Participatory evaluation. In Mabry, L. (ed.) Evaluation and the Post-modern Dilemma. Advances in Program Evaluation. JAI Press, Greenwich, Vol. 3, pp. 171–189.
Greene, J. (2001) Dialogue in evaluation; a relational perspective. Evaluation, 7, 181–203.[Abstract]
Greene, J. C., Kreider, H. and Mayer, E. (2005) Combining qualitative and quantitative methods in social inquiry (274–281). In Somekh, B. and Lewin, C. (eds) Research Methods in the Social Sciences. Sage, London.
Guba, E. G. and Lincoln, Y. S. (1981) Effective Evaluation. Sage, Beverly Hills, CA.
Guba, E. G. and Lincoln, Y. S. (1989) Fourth Generation Evaluation. Sage, Beverly Hills, CA.
Guba, E. G. (1990) The Paradigm Dialog. Sage, Newbury Park.
Huebner, A. J. and Betts, S. C. (1999) Examining fourth generation evaluation: application to positive youth development. Evaluation, 5, 340–358.[Abstract]
Janesick, V. J. (2000) The choreography of qualitative research design: minuets, improvisations and crystallisation. In Denzin, N. and Lincoln, Y. S (eds) Handbook of Qualitative Research. Sage, Thousand Oaks, CA, pp. 379–400.
Kahan, B. and Goodstadt, M. (2001) The interactive domain of best practices in health promotion: developing and implementing a best practices approach to health promotion. Health Promotion Practice, 2, 43–67.[Abstract]
Koch, T. (1994) Beyond measurement: fourth generation evaluation in nursing. Journal of Advanced Nursing, 20, 1148–1155.[CrossRef][ISI][Medline]
Koch, T. (1996) Implementation of a hermeneutic inquiry in nursing: philosophy, rigour and representation. Journal of Advanced Nursing, 24, 174–184.[CrossRef][ISI][Medline]
Koch, T. (2000) ‘Having a say’: negotiation in fourth generation evaluation. Journal of Advanced Nursing, 31, 117–125.[CrossRef][ISI][Medline]
Koelen, M. A., Vaandrager, L. and Colomer, C. (2001) Health promotion research: dilemmas and challenges. Journal of Epidemiololgy and Community Health, 55, 257–262.
Lincoln, Y. S. (1992) Fourth generation evaluation, the paradigm revolution and health promotion. Revue Cannadienne De Sante Publique, 83, 6–10.
Lincoln, Y. S. (1993) I and thou: method, voice, and roles in research with the silenced. In McLaughlin, D. and Tierney, W. (eds) Naming Silenced Lives, Routledge, New York, pp. 29–47.
Madjar, I. and Walton, J. A. (2001) What is problematic about evidence? In Morse, J. M., Swanson, J. M. and Kuzel, A. J. (eds) (2002) The Nature of Evidence in Qualitative Research. Sage, Thousand Oaks, CA.
McQueen, D. V. (2000) Perspectives on health promotion: theory, evidence, practice and the emergence of complexity. Health Promotion International, 15, 95–97.
Nutbeam, D. (1997) A Conceptual Model for Health Promotion, and the Consequences for Research. Report of the Expert Meeting Beyond RCT towards Evidence Based Public Health, February 13, Rotterdam.
Nutbeam, D. (1998) Evaluating health promotion—progress, problems and solutions. Health Promotion International, 13, 27–44.
Ray, L. D. and Mayan, M. (2001) Who decides what counts as evidence? In Morse, J. M., Swanson, J. M. and Kuzel, A. J. (eds) The Nature of Evidence in Qualitative Research. Sage, Thousand Oaks, CA.
Schwandt, T. S. (1999) On understanding understanding. Qualitative Inquiry, 5, 451–464.[Abstract]
Schwandt, T. S. (2001) A postscript on thinking about dialogue. Evaluation, 7, 264–276.[Abstract]
Shulha, L. M. and Cousins, J. B. (1997) Evaluation use: theory, research, and practice since 1986. Evaluation Practice, 18, 195–208.[CrossRef]
Stake, R. E. (1975) To evaluate an arts program. In Stake, R. E. (ed.) Evaluating the Arts in Education: A Responsive Approach. Merrill, Colombus, OH, pp. 13–31.
Stake, R. E. and Abma, T. A. (2005) Responsive evaluation. In Mathison, S. (ed.) Encyclopaedia of Evaluation. Sage, Thousand Oaks, CA.
Springett, J. (2001) Participatory approaches to evaluation in health promotion. In: Rootman, I. et al. (eds) Evaluation in Health Promotion, Principles and Perspectives. WHO regional publications, European Series, 92, 83–106.
Swenson, M. M. (1991) Using fourth generation evaluation in nursing. Evaluation and the Health Professions, 14, 79–87.
Tones, K. (1997) Research for Health Promotion Practice. A Tale of Three Errors, Illumination and the Judicial Principle. Report of the expert meeting beyond RCT towards evidence based public health, February 13, Rotterdam, 46–60.
Van der Wilt, G. J. and Reuzel, R. (1998) Assessment of health technologies, which issues should be addressed? Evaluation, 4, 351–358.[CrossRef]
Wadsworth, Y. (2001) Becoming responsive—and some consequences for evaluation as dialogue across distance. New Directions for Evaluation, 92, 45–58.
Wallerstein, N., Polascek, M. and Maltrud, K. (2002) Participatory evaluation model for coalitions: the development of systems indicators. Health Promotion Practice, 3, 361–373.[Abstract]
Weick, K. E. (1995) Sensemaking in Organizations. Sage, Thousand Oaks, CA.
Widdershoven, G. A. M. (2001) Dialogue in evaluation: a hermeneutic perspective. Evaluation, 7, 253–263.[Abstract]
Widdershoven G. A. M. and Sohl, C. (1999) Interpretation, action and communication: four stories about a supported employment program. In Abma, T. A. (ed.) Telling Tales. On Narrative and Evaluation, Advances in Program Evaluation. JAI Press, CT., Vol. 6, pp. 109–130.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||