New Working Paper Slovenian Evaluation Society, 1/XVII (2024), announcement

The traditional result-based approach to policy impact evaluation, which compares actual achievements with initial plans, proves inadequate for evaluating complex interventions (Powel et al.; Deprez; Willson-Grau; Davies & Dart) with uncertain or indeterminate impacts. This inadequacy arises from the traditional approach’s limited inclusiveness of atypical, indirect, or unexpected impacts. There is a need for new evaluation approaches that are both rational and democratic. This demands researching evaluation problems within the context of the collective choice theory (Condorcet; Arrow) and its inherent complexity.

Complex interventions are specific because of an incomplete understanding of the evaluated object: their impacts often lack a common unit for comparison (they are incommensurable), sometimes they are immeasurable, or they also produce a multitude of effects that cannot be definitively attributed to specific causes or beneficiaries. This is particularly true when dealing with human and social behaviours, for interventions aiming to drive social change, enhance interaction between social groups, or develop internal group dynamics (Wilson-Grau) – where outcomes are emergent and may not be predictable in advance. In such a situation, evaluators can see the evaluated object only as epistemic blind and biased.

Over the past few decades, a new generation of evaluative approaches has emerged (Guba, Lincoln; Weiss), moving beyond the traditional rationalist logic of analytical evaluation. This new paradigm shifts the focus to more stakeholder-driven and design-based constructivist approaches leading to the development of dialogical forms of policy impact evaluation (Patton, 2002). Stufflebeam defines participatory evaluation as a collaborative assessment process. It evaluates complex interventions by involving a broad range of stakeholders with their diverse viewpoints. It also considers various data types, quantitative and qualitative. On the one side, this approach strives for the inclusivity of multiple perspectives. On the other, it is criticised for lacking a robust theoretical frame with inconsistent enhancement of collective rationality.

The theory of collective choice tackles the challenges of collective decision-making. It examines how societies, communities, or organisations navigate conflicting options and make collective sense of fragmented individual inputs. The conflict between individual autonomy and the common good lies at the core of collective choice theory. It studies the inherent conflict between democracy’s goal of inclusivity for all society’s members and the need for rational decision-making in identifying the best among the available options for the collective. The theory of collective choice faces a fundamental limitation framed by Arrow’s impossibility theorem. This theorem states that no aggregation method can perfectly satisfy inclusivity and rational choice, which means reflecting all individual preferences and simultaneously selecting the most advantageous option for the collective. This impossibility also highlights the serious limitations of democratic and simultaneously rational impact evaluation or collective sensemaking in general.

The concept of democracy frames efforts for resolving the inherent tension between demos and kratos in societies. Demos represent the collective body of citizens, who value inclusivity and a diversity of views often at the expense of choosing the best available option. Kratos, on the other hand, refers to the practical governance that enables these decisions. Kratos seeks to ensure that choices are coherent, just, and functional, often at the expense of the inclusion of diverse viewpoints. The tension is evident for instance in debates around freedom of speech on social media platforms. While demos value the free exchange of ideas, kratos necessitate measures to prevent the spread of misinformation or hate speech. The democratic legitimacy of political authority depends on finding a middle ground that intersects the aspirations of citizens and the need for efficient and effective governance. Participatory evaluation assists the demos in participating in kratos. It also facilitates the kratos to become decreasingly exclusionary in enforcing unity over diversity simply by a more connective understanding of complex contradictions in the collective choice.

Policy impact evaluation emerged as a critical response to the limitations of conventional approaches to collective choice. Participatory evaluation includes various tools. This paper uses two criteria for assessing their success in achieving inclusivity and collective rationality. The inclusivity will be determined by how they incorporate epistemic blindness (Fricker) and resolve bias in the participatory evaluation. Epistemic blindness is the opposite of epistemic certainty. It may arise from ignorance or lack of knowledge, resulting in the exclusion of anything that does not match established patterns of understanding social phenomena (Kahneman). However, epistemic blindness cannot always be eliminated. Our knowledge is inherently limited or bounded (Simon). This is particularly true in complex evaluations which contain uncertainty. Uncertain things involve the void in their core. These things can be better understood by someone blindsighted, with no preconceptions, than by a knowledgeable and enlightened scientist. ‘Blindsighted’ means that understanding of complex issues is crafted from ‘the empty middle’ as indeterminate.

The second criterion examines how effectively from the aspect of the collective these tools aggregate the fragmented contributions gathered through the participatory process.  Various aggregation procedures exist, ranging from micro-level (focusing on individual data points) to macro-level (looking at broader concerns) and meso-level (observing intermediate processes and correlations) to meta-level (on the overlap between shared concerns). Various aggregation procedures are designed with dissimilar logics of synthesis yielding strikingly different results (Radej, 2021a). This suggests that the selection of aggregation method should not be arbitrary, but must closely correspond to the complex nature of evaluated interventions.

To explore both the inclusiveness and collective rationality of design-based approaches, this paper examines four popular tools in participatory evaluation: SenseMaker by Cognitive Edge, Outcome Harvesting by Wilson-Grau, Most Significant Change by Rick Davies, and Causal Mapping (Copestake et al., Goddard, Powel).

The paper begins by introducing four tools. It then develops two key assessment criteria: epistemic blindness (limitations in knowledge) and aggregation problems (summing up fragmented and diverse viewpoints). The core analysis delves into how well these tools satisfy two criteria. While the paper acknowledges the tools’ strengths and positive contributions, it argues that they fail to meet selected criteria in complex circumstances. It concludes with a call for the antipost-modern turn in evaluation theory that achieves inclusiveness and collective rationality by intersecting them in the empty middle and then reading evaluation messages blindsighted.

The four selected tools were presented and discussed at the 5th Western Balkans Evaluator’s Network (WBEN) conference in Ljubljana in late September 2023. The Slovenian Evaluation Society hosted the event on behalf of WBEN.

This is the opening chapter of a forthcoming working paper.

Chapter II

Chapter III.1