|Slovenian Evaluation Society|
1000 Ljubljana, Slovenija
I. Basics about SDE - Slovenian evaluation society
II. Vision and Mission of SDE
III. Activity of SDE
IV. Initiative to the Government
V. Methods: Meso-matrical Impact Assessment
VI. Curriculum Vitae of serving Chairman
I.3 Formal structure
SDE (ID code 2376784000/AJPES) has been established on 7. January 2008 in Ljubljana. SDE is non-governmental, not-for profit and non-professional organisation comprised of academics, practitioners and users of program and project evaluation in a variety of fields-regional and rural development, spatial, urban and landscape planning, education, social services, business, and others. Activity of SDE is based on voluntary contribution of its members (10+) and non-members (350+).
SDE is a member of National Evaluation Societies of Europe. SDE is a member of Slovenian spatial network and Slovenian Forum for Social Economy. SDE is also member of local civil initiatives.
Activity of the Society are organised in three permanent commissions:
1. Commission for ethical codex and evaluation standards (COD) - its tasks are to work on evaluation standards and ethical codex of evaluators;
2. Commission for meta-evaluation (MET) - its tasks are to evaluate results of evaluation processes from the aspect of codex and standards;
3. Commission for evaluation studies (STU) - its tasks is to collect fundamental and also intermediate new knowledge in the area (not only in evaluation but also in complexity theory, system theory, formal system, anthropology that is relevant for evaluation etc).
Society operates as a platform of the civil society aimed at reconstruction of the administrative evaluation system and at promotion of independent but effective evaluation of public policies in Slovenia and in European union. The Society does not limit its activity to members and their immediate concerns but operates as an ‘incubator' of evaluation efforts for members and non-member alike. In line with this the Society established a network of evaluators, it is publishing working paper series and newsletter as a free of charge service to members and non-members; it is serving as a facilitator of activities and processes in the area of evaluation.
I.3 Formal structure of the Society
Slovenian Evaluation Society has been established with the vision to affirm neutral evaluation of social phenomena, observed as complex. Social complexity means that any individual person or theory is able to fully comprehend social complexity - such as sustainable development - and its multiple meanings are persistently disagreed collectively. What one sees and evaluates is always predefined in scope and scale of his or her observation.
There are incommensurable viewpoints and values with regard to different social realities, such as economic, social and environmental, or local and global, which provide us with very different ‘numeraires' and views of the world, and they are not reducible to one common denominator. The fact that variegated forms of welfare need to be taken into account in evaluation implies that ‘different principles of social primacy and legitimacy must be reckoned with and reconciled' (Wacquant). When confronted with multidisciplinary of social issues involving scientific controversy, even competent, honest and disinterested scientists may arrive at different conclusions because of systematic differences in the way they summarise available information of one and the same world (Kuhn). Kuhn concludes that theories and scientists are unable to truly reflect reality. When values are based on incommensurable oppositions, there is no objective basis for rational choice between theories and than no neutral observation of social reality is possible.
This considered, the evaluator would be seen as a kind of social conflict accountant and the assessment is his or her tool for diagnosing the ‘dark matter' of the universe of social oppositions and a tool for re-processing them into a potential energy for progress. Evaluator needs to be equipped for manipulation of weak and strong oppositions with approaches that go beyond simple minded reductionism that can "increase capacity for thought and increase pool of possible reasons" (Habermas). In this way evaluator can invoke "the Foucault effect", depriving binary and antagonistic social practices of their exclusive necessity, extending the bounds of the thinkable and progressing of reason to permit the invention of others, and therefore become a real factor of change. In this way, an evaluator can make use of conflicts instead of trying to ignore or eliminate them (Donzelot).
III. Activities of the Society
III.1 Round tables & Public Lectures
SDE organised the following round tables & Public Lectures:
III.2.1 The Working papers series
III.2.2 The Newsletter
III.2.3 Occassional papers and articles (in Slovenian language)
III.2.4 Reference library
III.2.1 The Working papers series (WP)
The Working papers series of the SDE (WP/SDE): It publishes scientific and technical papers from the aspect of evaluation of public policies and from related disciplines. Working paper series is the first serial publication on the impact assessment in Slovenia. It is freely accessible on the internet. Papers are reviewed and bibliographic catalogued (CIP) ensuring that information is available in all catalogues in Slovenia and elsewhere. Already published in WP/SDE, in Slovenian language:
III.2.2 The Newsletter:
SDE publishes approx. 20 issues of the Newsletter in Slovenian language annually.
An English version of SDE Newsletter is dispached once or twice a year.
III.2.3 Occassional papers, articles, miscellaneous (in Slovenian language)
In perspective, SED will set up a reference library with links only to downloadable documents relating to the fundamental issues for policy evaluators, comprising not only impact assessment literature but also broader issues important for the future development of evaluation theory such as complexity, fuzzy sets, multi-criteria and mixed-methods research, soft-system methodology, etc. STU has already collected and arranged the first entries to the library (here).
SDE has composed a network of evaluators and those concerned for evaluation; when launched this network consisted of more than 350 subscribers plus more than 300 institutional addresses that received regular information about SDE activity, including governmental institutions and media.Slovenain Evaluation Society is a mamber of several networks on the European, national and on the local level.
Slovenian Evaluation is member of National Evaluation Societies Network
In February 2009, SDE invited a group of contributors, non-members and members indiscriminately, to convene to prepare a text of a joint evaluators' statement and initiatives to the Government regarding state and perspectives of policy evaluation in Slovenia. Professional and practicing evaluators as well as all other closely concerned for the situation in the area were invited to support the statement. In five weeks the statement has been signed by 170 individuals and ten collectives, including the largest trade union in the country and several scientific institutes (March 2009). This statement has been also supported by approx. 30 academic professors and 30 government officials that are active in some 30 different professional areas in which evaluation takes place. The received support for the proposed statement has been assessed by the editorial board as more than encouraging for further activities.
Signatories proposed to the Government the following:
In 2009 a series of meetings has been organised with high level governmental officials and ministers (Secretay general of the government, Ministry of finance, Ministry for public administration, Ministry for environment and spatial planning, Ministry for education and science, Ministry for culture and several governmental agencies).The purpose of the meetings is launching a series of evaluation initiatives in cooperation with the stakeholders in the bottom up as well as in the top down perspective.
V.1 About MIA - in general
V.2 MIA - history of application, publishing, presentations and editorial reviews
V.3 Ad hoc peer to peer assessment of MIA
Slovenian evaluation society (SDE) has been developing new impact assessment method - Meso-matrical impact assessment (MIA) since 2006. This method is a response to observed and experienced difficulties of evaluators to provide policy-makers' with useful (clear and applicable) conclusions from the assessment. This difficulty is in large part a consequence of evaluators' inability to summarise and synthesise their observations into conclusions that usually go beyond surface description of the assessment results. MIA looks below the surface of the assessment table where the system as a whole becomes visible.
MIA is derived from the most conventional ('Leopoldian') impact assessment matrix approach, but strictly based on the definition of complexity that puts forward two dimensions of irreducible social conflicts in policy evaluation: (i) between issues that are totally different but equally legitimate (such as economic, social, nature scope of development or of evaluation); (ii) one and the same issue, such as climate change or globalisation or ‘economic meltdown' is perceived entirely differently in scale such as from the micro or from the macro perspective.
MIA ‘knows' that what evaluator is expected to evaluate is always predefined in scopes and in scales of social complexity. Conventional evaluative approaches to impact assessments take into account only two scopes, one as cause and the other as the effect, such as economy and environment, and only one scale - either micro or on macro, depending on the size of the assessed programme and impact area related to it. Instead, MIA puts forward a procedure that always requires assessing at least three (but for a practical reasons not more than four) scopes and three scales (micro-meso-macro, such as in Dopfer, Potts and Fosters; see also Simon, Easterling, Geels, project Matisse, Munda , Bar-Yam, Heyligen, Sankey, Weaver ...).
Social complexity is not easily assessable because of the fact that judgements that are obtained at different scopes are between each other incommensurable; the same holds true for different scales. This raises question on how to aggregate and summarise assessment observations in the impact assessment into the overall conclusions that are handed over to the policy-makers? MIA solves this problem so as to abandon derivation of evaluation conclusions exclusively from their primary meanings; when impact assessment derives summary conclusion exclusively from the observation of impacts on the assessment criteria these conclusions could be too easily manipulated. In deriving summary conclusions MIA relies essentially on secondary and tertiary overlaps between direct observations - these are positioned on the margin of the overall system hierarchy and produce more subtle meaning in hybrid content. MIA thus instructs evaluator to focus the assessment on the issues where marginal areas of crucial issues in strong oppositions overlap. This places evaluator in the un-excluded middle of meso-matrical assessment perspective.
Another lesson learned using MIA is that the standard multi-criteria evaluation may produce insufficient and misleading resutls. Standard impact assessments finish their otherwise respectful efforts exactly where the summative evaluation in meso-matrical context really sets off. Assessment of ‘specific impacts' (intersection between particular policy measure and particular assessment criteria) shall be seen only a preparatory phase for the complex evaluation - it only constructs the micro base that informs evaluation with quantification of causal relations between individual policy measures and individual assessment criteria (Leopold's causal view). Results obtained at this level of assessment do not yet enable evaluative learning and do not inform policy makers holistically. Bare facts do not speak for themselves in complexity, but they need first to be inter-related such as in correlation matrix (derived from the Leontief's view which is not only causal but also relational) where they obtain their meaning through a multi-relations (say, impact of economic measures on environment criteria of the assessment, relative to the opposite impact of environmental measures on the economic criteria).
Hence, introduction of assumption of incommensurability does not result in incompatibility of different aspects of social reality (welfare, wellbeing), but only point to the exclusive or two-valued treatment of a complex social world in standard impact assessments. The meso-matrical approach reverses the social incommensurability problem in policy evaluation. Impossibility to create consent between deep oppositions should not be seen as a threat to the indiscriminate assessment of welfare concerns from individual and collective viewpoints. The post-modern loss of a social or political common ground as the condition of unity and cohesion is not an irresolvable challenge for evaluators, with the meso-matrical perspective in mind. When social incommensurability is seen in meso-matrical frame, it can be understood in the coexistence without contradiction because it is grounded in multi-valued logic. MIA can be seen as a principle that is indiscriminate between system unity and diversity which is crucially important for addressing conflicts between individual and collective (social causality; Coleman) that are not black-and-white issues but complexities that prevail in governance.
(1) Versions: First (Nov. 2006 - Final project report), second (May 2007 - published as Working paper), third (Jun'08 - the first published in Slovenian language; Dec'08, in English, accepted for publishing in JMDE; Jan.2009 accepted for the Conference of the European Society for Ecological Economics - ESEE).
(2) Peer reviews: Universite de Versailles - Saint Quentin-en-Yveline (2006, EU's 6thFP, research team), Faculty for Social Sciences Ljubljana (2007, to the students of policy studies), Institute for Economic Research Ljubljana (2007, to Slovene academics and consultants), Slovenian Evaluation Society (2008, to Slovene experts); ad hoc peer review (see below); International seminar at University of Valencia, 28-29 January 2010.
(3) Reviewed by editors: Institute for economic Research, Occasional Papers Series (2007); Economic Faculty of Ljubljana, Working paper series (2007, 2009); Casopis za kritiko znanosti (2007); Revija 2000 (2008), Journal of Multidisciplinary Evaluation - JMDE (2008); ESEE (2009).
(4) Practical applications of MIA: (i) an ex-ante impact assessment of the Regional development programme for Pomurje region 2007-2013 in Slovenia (Radej, 2008); (ii) the impact of the national energy program on the spatial cohesion of Slovenia for the Ministry of Environment (Radej, 2008).
(5) Financing: Applicative part of this research project has been financially supported by the 6thEU research programme (project acronym: SRDTOOLS). It has been also supported as a part of Target Research Programme »Competitiveness of Slovenia 2006-2013«, focus 5 (Connecting the measures for better implementation of sustainable development), thematic group 5.2 (Coherent regional development and improvement management of the territory), the theme 5.2.8 (Monitoring and territorial impact assessment of sector policies). Research has been financed by the Slovene ministry of environment and carried out by the Slovene Urban Planning Institute.
Bojan Radej is an independent researcher from Ljubljana.
Education: Master degree in macroeconomics has been earned at the University of Ljubljana - Faculty of Economics (1993); free researcher at the LICOS (1993), specialisation in environmental management in Japan (JICA, 1994).
Professional experience record: employed at the Governmental Institute of Macroeconomic Analysis and Development (1987-2004; served as under-secretary to the government), areas of work: sustainable development (1998-04), chief manager of the modelling department (1993-5); initiator and the first editor of Slovenia Economic Mirror (1995-8); Co-Editor of IB journal (2001-04).
University of Ljubljana, Faculty of Economics (January 1996-2009), assistant lecturer in Economics of Environmental Protection.
(1) Founder of Slovenian Evaluation Society (2008);
(2) Co-founder of NGO Slovenian E-forum on economy, environment and energy (1995-02);
(3) Membership of professional bodies: Slovene commission on sustainable development (2001-2); National climate committee (2001-4); Supervisory board of Slovene environmental development fund, member (2000-4); Slovene statistical council on environmental statistics (2001-4), member.
(4) Activist devoted to supporting autonomous creativity as a lifestyle and a world-view.
SDE is supported by the Institute for economic research, Ljubljana (free access to their conferencing facilities).
A logo of SDE has been designed by Naja Marot.
SDE is recognised as a partner by government.
SDE is supported by the largest network of evaluators in Slovenia.