Slovenian Evaluation Society

Address: Tabor 7, c/o Radej Bojan,

1000 Ljubljana, Slovenija

Ta e-poštni naslov je zaščiten proti smetenju. Če ga želite videti, omogočite Javascript. ; web page: http://www.sdeval.si

Content of this page

I. Basics about SDE - Slovenian evaluation society

II. Vision and Mission of SDE

III. Activity of SDE

IV. Initiative to the Government

V. Methods: Meso-matrical Impact Assessment

VI. Curriculum Vitae of serving Chairman

I. Basics about Slovenian evaluation society

I.1 Administration

I.2 Activity

I.3 Formal structure

I.1 Administration

SDE (ID code 2376784000/AJPES) has been established on 7. January 2008 in Ljubljana. SDE is non-governmental, not-for profit and non-professional organisation comprised of academics, practitioners and users of program and project evaluation in a variety of fields-regional and rural development, spatial, urban and landscape planning, education, social services, business, and others. Activity of SDE is based on voluntary contribution of its members (10+) and non-members (350+).

SDE is a member of National Evaluation Societies of Europe. SDE is a member of Slovenian spatial network and Slovenian Forum for Social Economy. SDE is also member of local civil initiatives.

I.2 Activity

Activity of the Society are organised in three permanent commissions:

1. Commission for ethical codex and evaluation standards (COD) - its tasks are to work on evaluation standards and ethical codex of evaluators;

2. Commission for meta-evaluation (MET) - its tasks are to evaluate results of evaluation processes from the aspect of codex and standards;

3. Commission for evaluation studies (STU) - its tasks is to collect fundamental and also intermediate new knowledge in the area (not only in evaluation but also in complexity theory, system theory, formal system, anthropology that is relevant for evaluation etc).

Society operates as a platform of the civil society aimed at reconstruction of the administrative evaluation system and at promotion of independent but effective evaluation of public policies in Slovenia and in European union. The Society does not limit its activity to members and their immediate concerns but operates as an ‘incubator' of evaluation efforts for members and non-member alike. In line with this the Society established a network of evaluators, it is publishing working paper series and newsletter as a free of charge service to members and non-members; it is serving as a facilitator of activities and processes in the area of evaluation.

I.3 Formal structure of the Society

Chairman of the COD - Jože Grabar Kos, spatial planner, publicist and activist.
Chairman of the MET - dr. Majda Černič Istenič, professor at the faculty for biotech.
Chairman of the STU - dr. Pavle Gmeiner, retired evaluator.
Arbitrator and Conflict Mediator - prof. dr. Darko Štrajn, scientific councillor.
Supervisor - mag. Metod Dolinšek, International Consultant.
Chairman of the SDE - mag. Bojan Radej, independent researcher.

Back to the top

II. Vision and mission of the Society

Slovenian Evaluation Society has been established with the vision to affirm neutral evaluation of social phenomena, observed as complex. Social complexity means that any individual person or theory is able to fully comprehend social complexity - such as sustainable development - and its multiple meanings are persistently disagreed collectively. What one sees and evaluates is always predefined in scope and scale of his or her observation.

There are incommensurable viewpoints and values with regard to different social realities, such as economic, social and environmental, or local and global, which provide us with very different ‘numeraires' and views of the world, and they are not reducible to one common denominator. The fact that variegated forms of welfare need to be taken into account in evaluation implies that ‘different principles of social primacy and legitimacy must be reckoned with and reconciled' (Wacquant). When confronted with multidisciplinary of social issues involving scientific controversy, even competent, honest and disinterested scientists may arrive at different conclusions because of systematic differences in the way they summarise available information of one and the same world (Kuhn). Kuhn concludes that theories and scientists are unable to truly reflect reality. When values are based on incommensurable oppositions, there is no objective basis for rational choice between theories and than no neutral observation of social reality is possible.

This considered, the evaluator would be seen as a kind of social conflict accountant and the assessment is his or her tool for diagnosing the ‘dark matter' of the universe of social oppositions and a tool for re-processing them into a potential energy for progress. Evaluator needs to be equipped for manipulation of weak and strong oppositions with approaches that go beyond simple minded reductionism that can "increase capacity for thought and increase pool of possible reasons" (Habermas). In this way evaluator can invoke "the Foucault effect", depriving binary and antagonistic social practices of their exclusive necessity, extending the bounds of the thinkable and progressing of reason to permit the invention of others, and therefore become a real factor of change. In this way, an evaluator can make use of conflicts instead of trying to ignore or eliminate them (Donzelot).

Back to the top

III. Activities of the Society

III.1 Round tables & Public Lectures

III.2 Publishing

III.3 Networking

III.1 Round tables

SDE organised the following round tables & Public Lectures:

Lecture - Presentation of methodology: »Meso-matrical Impact Assessment« (24. Apr 2008)
Round table: »Ethical codex - between principles and practice« (in Slovene, 20. May 2008)
Presentation of evaluation study results: »Impact of National Energy Programme on the Spatial Cohesion of Slovenia and its regions« (long summary in English, 3. Jul. 2008)
Round table: »Understanding of public interest in evaluation« (in Slovene, 28. Oct. 2008)
Round table: »Mercantilism in spatial planning« (in Slovene, 3. Feb. 2009)
Round table: »Evaluation in the intersection between civil society and government« (in Slovene, 8. Apr. 2009)
Lecture: »Financial crisis, economic meltdown and evaluation« (in Slovene, 29. Apr 2009, in cooperation with YHD-Association for the theory and culture of handicap, Ljubljana)
Round table: »Data demands in evaluation of social programmes« (in Slovene, 12. Apr. 2009, in cooperation with the Faculty of social work and with the Institute of Republic of Slovenia for social safety)
Workshop: "Obstacles to inter-sectoral coordination", 8 July 2009, in Slovene, in cooperation with Government Office for Development and European Affairs and Ministry of Finance.

Workshop: "More for Less? Self-evaluation in public administration withCAF", 9. July 2009, in Slovene, preparred in cooperation with theSchool of Advanced Social Studies in Nova Gorica.
Round table: "Performance Based Budgeting", 23. October 2009, in Slovene, Preparred together with Ministry of Finance.
Round table: "Annual Programing Conference of Slovenian Evaluation Society 2009", 7. December 2009.
Lectures: "Synthesis in Policy Impact Evaluation", Unversity of Valencia, 28. January 2010; and University of Ljubljana (Annual Congres of European Association for Environmental Economics), 2. July 2009.
Educational programme: "Evaluation for occasional users" (in Slovene, permanent programe, accompanied with a book, since 2010)
NEW: Out of the Box Seminar, University of Maribor, Lecture (24. april 2011) "Multi-level social complexity", Lecture: http://videolectures.net/outofthebox_radej_vidik/; gradivo: http://www.sdeval.si/Publikacije-za-komisijo-za-kodekse-in-standarde/Vec-nivojski-vidik-druzbene-kompleksnosti.html; Paper http://www.sdeval.si/Publikacije-za-komisijo-za-kodekse-in-standarde/Vec-nivojski-vidik-druzbene-kompleksnosti.html
NEW: Six predictions about the future of evaluation,
NEW: Creative destruction of the present imperfect ,
NEW: Free lunch for evaluator? Yes, if you work for CSO!

III.2 Publishing

III.2.1 The Working papers series

III.2.2 The Newsletter

III.2.3 Occassional papers and articles (in Slovenian language)

III.2.4 Reference library

III.2.1 The Working papers series (WP)

The Working papers series of the SDE (WP/SDE): It publishes scientific and technical papers from the aspect of evaluation of public policies and from related disciplines. Working paper series is the first serial publication on the impact assessment in Slovenia. It is freely accessible on the internet. Papers are reviewed and bibliographic catalogued (CIP) ensuring that information is available in all catalogues in Slovenia and elsewhere. Already published in WP/SDE, in Slovenian language:

WP/SDE no. 2/2008: »Sinteza vplivov nacionalnega energetskega programa na prostorsko kohezijo Slovenije« (Synthesis of Territorial Impact Assessment for Slovene Energy Programme; Bojan Radej, 47 pp., only long summary in English)
NEW version WP/SDE no. 3/2008: Radej B. Synthesis in policy impact assessment. Sage: Evaluation 17/2(April 2011):133-50 (Bojan Radej, 21 pp; Published in Evaluation (Sage), 17/2(April 2011)
WP/SDE no. 1/2009: »Meso-matrical Impact Assessment - peer to peer discussion of the working paper 3/2008, report for the period 8/08-2/09« (Bojan Radej, editor, 30 pp; forthcoming in april 2009; in English)
WP/SDE no. 2/2009: »Anti-system matrix« (Bojan Radej, 12 pp; april 2009; in English)
WP/SDE no. 3/2009: »Turisticna regionalizacija Slovenije« (Tourist regionalisation of Slovenia; Jože Kos Grabar; in Slovenian language, with English long summary)
WP/SDE no. 1/2010: "Beyond New Public Management doctrine in policy impact evaluation" (B. Radej, M. Golobic, M. Cernic Istenic, 25 pp)
Objective Gap Analysis in Quality of Life (In English; B. Radej, 26 pp.)
NEW Alternative social welfare measures justify alternative welfare doctrines (January 2011, 5 pp.)
NEW WP/SDE 1/2011:Primary and secondary perspective in policy evaluation (August, 2011)
NEW WP/SDE 1/2012: Excelence Squarred (Self-evaluation in public organisations, Jan. 2012)

III.2.2 The Newsletter:

SDE publishes approx. 20 issues of the Newsletter in Slovenian language annually.

An English version of SDE Newsletter is dispached once or twice a year.

III.2.3 Occassional papers, articles, miscellaneous (in Slovenian language)

Bojan Radej, Nepristransko gledano: vse v redu! (Independently observed: everything OK; Feb. 2009)
Bojan Radej, Vrednost vrednotenja javnih ucinkov vladnih ukrepov (Evaluation of evaluation of public impacts of public policies; Feb. 2009)
Mojca Golobic: Vec prostora za (kriticno) presojo - boljši prostor za življenje (More space for critical judgement - more place for better living; Feb. 2009)
Bojan Radej, Od post-razvoja k drugotnemu razvoju (From post-development to secondary development; Apr. 2009)
Bojan Radej, Ukradena blaginja (Wellbeing that is stolen; Mar. 2009)
Matjaž Majeric, Planinska vas - ko zadošca aroganca (When arrogance is sufficient: The Planina Village; Jan. 2009)
Ivan Marušic, Teze za razpravo (discussion thesis; Mar. 2009)
Radio 3 Slovenia; Interview on evaluation (2.Mar.2009)
Primitive simplicity of politics against social complexity (2012)

III.2.4 e-Library

In perspective, SED will set up a reference library with links only to downloadable documents relating to the fundamental issues for policy evaluators, comprising not only impact assessment literature but also broader issues important for the future development of evaluation theory such as complexity, fuzzy sets, multi-criteria and mixed-methods research, soft-system methodology, etc. STU has already collected and arranged the first entries to the library (here).

III.3 Networking

SDE has composed a network of evaluators and those concerned for evaluation; when launched this network consisted of more than 350 subscribers plus more than 300 institutional addresses that received regular information about SDE activity, including governmental institutions and media.Slovenain Evaluation Society is a mamber of several networks on the European, national and on the local level.

Slovenian Evaluation is member of National Evaluation Societies Network

Back to the top

IV. Activity of SDE - Evaluators' joint initiative to the government of Slovenia

In February 2009, SDE invited a group of contributors, non-members and members indiscriminately, to convene to prepare a text of a joint evaluators' statement and initiatives to the Government regarding state and perspectives of policy evaluation in Slovenia. Professional and practicing evaluators as well as all other closely concerned for the situation in the area were invited to support the statement. In five weeks the statement has been signed by 170 individuals and ten collectives, including the largest trade union in the country and several scientific institutes (March 2009). This statement has been also supported by approx. 30 academic professors and 30 government officials that are active in some 30 different professional areas in which evaluation takes place. The received support for the proposed statement has been assessed by the editorial board as more than encouraging for further activities.

Signatories proposed to the Government the following:

to invite representatives of the initiative to the meeting with the purpose of setting up a plan for future activities;
to establish a governmental consultative council on evaluation as a body for exchange strategic opinions between financiers of evaluation (government), providers of evaluation services (consultancy) and final ‘users' of results such as general public (including SDE) concerning ethic and standards of evaluation;
to provide for a fully fledged system of evaluation in Slovenia, comprising also issues such as education of professionals, certification of professionals, promotion of evaluation culture.

In 2009 a series of meetings has been organised with high level governmental officials and ministers (Secretay general of the government, Ministry of finance, Ministry for public administration, Ministry for environment and spatial planning, Ministry for education and science, Ministry for culture and several governmental agencies).The purpose of the meetings is launching a series of evaluation initiatives in cooperation with the stakeholders in the bottom up as well as in the top down perspective.

Back to the top

V. Meso-matrical Impact Assessment - MIA

V.1 About MIA - in general

V.2 MIA - history of application, publishing, presentations and editorial reviews

V.3 Ad hoc peer to peer assessment of MIA

V.1 Meso-matrical Impact Assessment in general

Slovenian evaluation society (SDE) has been developing new impact assessment method - Meso-matrical impact assessment (MIA) since 2006. This method is a response to observed and experienced difficulties of evaluators to provide policy-makers' with useful (clear and applicable) conclusions from the assessment. This difficulty is in large part a consequence of evaluators' inability to summarise and synthesise their observations into conclusions that usually go beyond surface description of the assessment results. MIA looks below the surface of the assessment table where the system as a whole becomes visible.

MIA is derived from the most conventional ('Leopoldian') impact assessment matrix approach, but strictly based on the definition of complexity that puts forward two dimensions of irreducible social conflicts in policy evaluation: (i) between issues that are totally different but equally legitimate (such as economic, social, nature scope of development or of evaluation); (ii) one and the same issue, such as climate change or globalisation or ‘economic meltdown' is perceived entirely differently in scale such as from the micro or from the macro perspective.

MIA ‘knows' that what evaluator is expected to evaluate is always predefined in scopes and in scales of social complexity. Conventional evaluative approaches to impact assessments take into account only two scopes, one as cause and the other as the effect, such as economy and environment, and only one scale - either micro or on macro, depending on the size of the assessed programme and impact area related to it. Instead, MIA puts forward a procedure that always requires assessing at least three (but for a practical reasons not more than four) scopes and three scales (micro-meso-macro, such as in Dopfer, Potts and Fosters; see also Simon, Easterling, Geels, project Matisse, Munda , Bar-Yam, Heyligen, Sankey, Weaver ...).

Social complexity is not easily assessable because of the fact that judgements that are obtained at different scopes are between each other incommensurable; the same holds true for different scales. This raises question on how to aggregate and summarise assessment observations in the impact assessment into the overall conclusions that are handed over to the policy-makers? MIA solves this problem so as to abandon derivation of evaluation conclusions exclusively from their primary meanings; when impact assessment derives summary conclusion exclusively from the observation of impacts on the assessment criteria these conclusions could be too easily manipulated. In deriving summary conclusions MIA relies essentially on secondary and tertiary overlaps between direct observations - these are positioned on the margin of the overall system hierarchy and produce more subtle meaning in hybrid content. MIA thus instructs evaluator to focus the assessment on the issues where marginal areas of crucial issues in strong oppositions overlap. This places evaluator in the un-excluded middle of meso-matrical assessment perspective.

Another lesson learned using MIA is that the standard multi-criteria evaluation may produce insufficient and misleading resutls. Standard impact assessments finish their otherwise respectful efforts exactly where the summative evaluation in meso-matrical context really sets off. Assessment of ‘specific impacts' (intersection between particular policy measure and particular assessment criteria) shall be seen only a preparatory phase for the complex evaluation - it only constructs the micro base that informs evaluation with quantification of causal relations between individual policy measures and individual assessment criteria (Leopold's causal view). Results obtained at this level of assessment do not yet enable evaluative learning and do not inform policy makers holistically. Bare facts do not speak for themselves in complexity, but they need first to be inter-related such as in correlation matrix (derived from the Leontief's view which is not only causal but also relational) where they obtain their meaning through a multi-relations (say, impact of economic measures on environment criteria of the assessment, relative to the opposite impact of environmental measures on the economic criteria).

Hence, introduction of assumption of incommensurability does not result in incompatibility of different aspects of social reality (welfare, wellbeing), but only point to the exclusive or two-valued treatment of a complex social world in standard impact assessments. The meso-matrical approach reverses the social incommensurability problem in policy evaluation. Impossibility to create consent between deep oppositions should not be seen as a threat to the indiscriminate assessment of welfare concerns from individual and collective viewpoints. The post-modern loss of a social or political common ground as the condition of unity and cohesion is not an irresolvable challenge for evaluators, with the meso-matrical perspective in mind. When social incommensurability is seen in meso-matrical frame, it can be understood in the coexistence without contradiction because it is grounded in multi-valued logic. MIA can be seen as a principle that is indiscriminate between system unity and diversity which is crucially important for addressing conflicts between individual and collective (social causality; Coleman) that are not black-and-white issues but complexities that prevail in governance.

V.2 Meso-matrical Impact Assessment - history of peer reviews

(1) Versions: First (Nov. 2006 - Final project report), second (May 2007 - published as Working paper), third (Jun'08 - the first published in Slovenian language; Dec'08, in English, accepted for publishing in JMDE; Jan.2009 accepted for the Conference of the European Society for Ecological Economics - ESEE).

(2) Peer reviews: Universite de Versailles - Saint Quentin-en-Yveline (2006, EU's 6^thFP, research team), Faculty for Social Sciences Ljubljana (2007, to the students of policy studies), Institute for Economic Research Ljubljana (2007, to Slovene academics and consultants), Slovenian Evaluation Society (2008, to Slovene experts); ad hoc peer review (see below); International seminar at University of Valencia, 28-29 January 2010.

(3) Reviewed by editors: Institute for economic Research, Occasional Papers Series (2007); Economic Faculty of Ljubljana, Working paper series (2007, 2009); Casopis za kritiko znanosti (2007); Revija 2000 (2008), Journal of Multidisciplinary Evaluation - JMDE (2008); ESEE (2009).

(4) Practical applications of MIA: (i) an ex-ante impact assessment of the Regional development programme for Pomurje region 2007-2013 in Slovenia (Radej, 2008); (ii) the impact of the national energy program on the spatial cohesion of Slovenia for the Ministry of Environment (Radej, 2008).

(5) Financing: Applicative part of this research project has been financially supported by the 6^thEU research programme (project acronym: SRDTOOLS). It has been also supported as a part of Target Research Programme »Competitiveness of Slovenia 2006-2013«, focus 5 (Connecting the measures for better implementation of sustainable development), thematic group 5.2 (Coherent regional development and improvement management of the territory), the theme 5.2.8 (Monitoring and territorial impact assessment of sector policies). Research has been financed by the Slovene ministry of environment and carried out by the Slovene Urban Planning Institute.

VI. Curriculum Vitae of the Society's serving chairman:

Bojan Radej is an independent researcher from Ljubljana.

Education: Master degree in macroeconomics has been earned at the University of Ljubljana - Faculty of Economics (1993); free researcher at the LICOS (1993), specialisation in environmental management in Japan (JICA, 1994).

Professional experience record: employed at the Governmental Institute of Macroeconomic Analysis and Development (1987-2004; served as under-secretary to the government), areas of work: sustainable development (1998-04), chief manager of the modelling department (1993-5); initiator and the first editor of Slovenia Economic Mirror (1995-8); Co-Editor of IB journal (2001-04).

University of Ljubljana, Faculty of Economics (January 1996-2009), assistant lecturer in Economics of Environmental Protection.

Social engagement:

(1) Founder of Slovenian Evaluation Society (2008);

(2) Co-founder of NGO Slovenian E-forum on economy, environment and energy (1995-02);

(3) Membership of professional bodies: Slovene commission on sustainable development (2001-2); National climate committee (2001-4); Supervisory board of Slovene environmental development fund, member (2000-4); Slovene statistical council on environmental statistics (2001-4), member.

(4) Activist devoted to supporting autonomous creativity as a lifestyle and a world-view.

Back to the top

ACKNOWLEDGEMENTS:

SDE is supported by the Institute for economic research, Ljubljana (free access to their conferencing facilities).

A logo of SDE has been designed by Naja Marot.

SDE is recognised as a partner by government.

SDE is supported by the largest network of evaluators in Slovenia.

Prijava / Odjava »člani SDE«

Prijava na e-novice

Pregled zadnjih objav

Ključni dokumenti

Kaj je SDE?

SDE je civilna platforma neodvisnih strokovnjakov in strokovnjakinj s širokim profilom za doseganje sinergijskih ucinkov vrednotenja za trajnostni napredek skupnosti.

Slovensko društvo evalvatorjev

Prijava / Odjava »člani SDE«

Prijava na e-novice

Ključni dokumenti

Kaj je SDE?

Menu