SDE je civilna platforma neodvisnih strokovnjakov in strokovnjakinj s širokim profilom za doseganje sinergijskih ucinkov vrednotenja za trajnostni napredek skupnosti.
Izmišljevanje vedno novih vprašalnikov, izdelave novih spletnih strani, evidenc in baz, mapiranje…, medtem ko se resnim vprašanjem vrednotenja učinkov politik na širko izogibajo – na primer o stanju evalvacijskega sistema EU v novem proračunskem obdobju do leta 2020… Poročilo: EES – NESE workshop on the project “Strengthening Cooperation among Evaluation Societies in Europe: Identifying and Implementing Synergies”,Warsaw, 23-24 January 2014, Draft meeting minutes
Najprej naš odgovor, spodaj poročilo s srečanja EES.
Dear all,
we are rather dissapointed with the outcomes of the meeting & in particular of hte outcomes of a project “Strengthening Cooperation among Evaluation Societies in Europe…”. Again it is proposed conducting new surveys, new database, new evidences, sites…
As it has been confirmed in your previous notes, neutral needs assessment for this project has not been accomplished, so it is really not surprising that outcomes are not convincing. We, as an evaluators should be rather worried about such erosion of professional standards and ignorance of some of the basic imperatives of our profession. In our view, EES is really working hard to position itself in the center and provide for avoiding the reall and important questions such as acceptability of changes in the evaluation system in EU’s structural policy by 2020, etc.
For your info: we have just completed the second regional meeting of evaluators network in the Balkan region (comprising 8 evaluation comunities in 7 countries: Bosnia&Herzegovina, Croatia, Serbia, Montenegro, Macedonia, Romania, Slovenia; 31. Jan 2014, Ljubljana, Slovenia). Regional community has been established and is operationg with the aim to adress real problems and successfully avoiding superficial tasks, like those involved in the project “Strengthening Cooperation among Evaluation Societies in Europe…”. Our experience is that if a project is adressing real problems in the field and searching for real solutions for these problems, such process enables much more creative and pleasant environment for constructive work, and work is accomplished in more effective and efficient way, where project finances are only a leverage for a complex of new activities (partly in voluntary sector, partly in business sector, in governments…), otherwise it is not clear to us how to provide for sustainable project impacts and how to avoid waste of energy and time for project that only serve short term needs of those directly involved.
Kind regards,
Bojan Radej,
Slovenian Evasluation Society
Meeting notes:
DAY 1
Welcome and introduction:
Derek Poate from the UKES, the NESE coordinating partner since 2013, welcomed the participants and opened the workshop.
Maciej Szalaj, the President of the Polish Evaluation Society, welcomed the participants on behalf of the Polish Evaluation Society as the hosting association.
Leszek Korporowitz, the PES member, welcomed the participants with highlighting the workshop as an opportunity for exchanging the experience and knowledge; as intercultural seminar with respect of dignity of each culture; he is very delighted the meeting can be held in Warsaw.
Barbara Befani introduced briefly the origin of the grant idea of IOCE, and expressed hope that this event will be repeated. At least the cooperation will continue within the NESE collaboration.
Roundtable of introduction of the EES – NESE workshop participants followed.
Derek Poate, the workshop chairman, presented the 5 themes/topics identified for this workshop and proposed a working structure:
– A follow up discussion in more small working groups on 2nd day;
– All topics may not cover the interest of everybody, other proposed topics by the participants to be discussed during the AOB session;
– The regular NESE meeting at the upcoming EES 11th Biennial Conference, Dublin, October 2014 to be considered as the continuing process of this workshop, where the decision what will be done next year should be taken.
Short presentation of the initiative and other EvalPartners-funded initiatives:
– Innovation Challenge of National Evaluation systems by Liisa Horelli
“Engaging Parliamentarians for an Evaluation Culture”; Parliamentarian Forum on Development Evaluation, project was briefly presented (cooperation of Pakistan Evaluation Network, Parliamentarians Forum On Development Evaluation in South Asia, Community of Evaluators-South Asia and European Evaluation Association). More on what are the national evaluation culture/policy/systems and existence of the relevant concepts for the evaluation in each country to be discussed on Friday.
– P2P on Professionalization by Claudine Voyadzis
Workshop organized by Helen Simons and Robert Picciotto from UKES in cooperation with EES, to take place before the UKES Annual conference, on 8 April; piloting of VEPR system and debate on professionalization. Based on the two surveys launched among EES and UKES members there is some interest and the VEPR will be discussed more in details at the workshop.
Session 1: Strategies to increase membership
A Skype conf call with some of the NESE members was held on this topics, however a specific proposal was not developed. The conf call minutes were circulated before this workshop.
Discussion and roundtable presentation by each society:
DeGeval:
– 752 members (approx. 150 institutional members for EUR 350,- fee, individual members for EUR 85,- fee), student membership also implemented
– Automatic renewal ensures the consistency in the membership
– Not all members are active, however DeGeval tries to motived them via working groups, 2 working group meetings per year (spring time, annual conference)
– Some members are from Austria and Switzerland
Ukrainian Evaluation Society:
– New society, recently founded, as of October 2013 until now 25 official members
– EUR 30,- individual membership fee
– 237 Facebook fans
– 140 people mailing list
– Internal discussion how to motivate people to become members
Romanian evaluation society:
– Established as informal network in 2006, in 2008 the Association was set up
– National evaluation culture not enough developed now, the civic society and culture sector is only being developed as well as the NGOs
– 45 members, not all of them are paying on regular basis, the automatic renewal is not desired in Romania
– Activities of the society are increasing
– Two other association on evaluation created in Romania, who impose themselves as the main voice for non-governmental initiative, they are not perceived as competitors but they are not supporting the joint activities
Polish Evaluation Society:
– 150 members, only 1/3 pays regularly, annual fee EUR 40,-
– There are no institutional members
– Significant increase of membership in 2011/2012, probably thanks to the policy and application rules changes, the application rules are not so strict anymore
– 174 Facebook page fans – as one of the main source of the new membership
– Portfolios of members created
– There were some projects (e.g. on ethical standards) which helped in increasing the membership
– Local administration entities organizes evaluation events – it is a great support
– Governmental regulations that makes evaluation as obligatory strategy in the field of quality assurance in education, which makes extremely difficult to conduct evaluation in this field – it is a big demand, not possible to make in a good quality all the evaluation
– The dialog with the government
Finnish Evaluation Society
– 1999 founded
– Internal discussion, how to do the self-organization of the members to start the activities
– an issue with the professionalization, a lot of people do not consider themselves as evaluators
UKES
– 200 individual members for the fee EUR 100,- and 30 institutional members for the fee EUR 570,-
– Direct debit is not allowed by our bank which limits automatic repeat membership
– Incentive for individual members – 135 EUR for 2-year membership
– Internal discussion how to avoid a high fluctuation of membership and try to make membership more stable
– An original network basis still alive – afternoon, lunch sessions, etc.
– there is quite an active evaluation community in UK, a lot of development evaluation, on education and health – but experts who are members of their own professional societies on the specific sector rather than UKES members
– no restrictions for applying for membership, the UKES’s aim is rather to promote and foster the knowledge about the evaluation
Norwegian Evaluation Society:
– 40 institutional members, there is no limitation for the amount of people benefiting from the institutional membership, EUR 120,- fee for individual, EUR 500,- for institutional membership fee
– Main focus on the institutional members
– Focus on to create more activities to increase the membership (social media activities, newsletter, short meetings in Oslo on different subjects several times a year)
Italian Evaluation Society
– Recently there was an investment in the communication strategies
– Newsletter not only for members but for a broader audience
– Annual conference
– EUR 150,- for individual membership fee, EUR 118,- for students, institutional members can contribute on a voluntary basis
– 250 members in total
Czech Evaluation Society
– 25-30 members
– Strict application conditions: 3 recommendations from the current members, detailed review of the application
– No institutional members
– EUR 70,- membership fee
– Some small fluctuation but more or less stable
– Annual conference for about 80 people
– Peer review journal on evaluation as a tool for the membership increase – the only professional journal on evaluation in Czech, distributed in the Czech Republic
– Persuade the commissioners to use the evaluation standards
– No working groups
– Evaluation culture promotion with the peer review – reports are not only for members, they are considered as a motivation to join the society
Discussion:
How to draw all this information together to enhance the membership?
– There is a big diversity in terms of amount of membership, approaching the membership etc.
How we might stimulate the membership?
Further points to be included into the working group:
– Problem of ethic and professionalization
– Engagement of academic societies (universities, colleagues) as a strategy of PES for the membership increase – but how to engage them, how to make the support more effective
– Representativeness of societies in terms of membership development, the prominent evaluator might not be always members of the society
– Attractiveness for the existing members, sustainability of the societies, low activities of current members, lack of sharing the own experiences (competition)
Session 2: Membership data
This session is linked to the session 1 with idea to increase the membership.
– A draft proposal on the mutual benefits and membership increase developed by EES was presented earlier to some NESE members; the proposal is based on the EES conference fee discount provided to a specific amount of people based on the country. Unfortunately there are no reliable data of other national societies to make the proposal alive and more concrete.
– Proposal of having a common template for collecting the data of the national societies in order to make the collaboration easier and profit from mutual benefits and from the complex data
Discussion:
– About the sample EES questionnaire to be used by other societies, or the modified version
– About concerns, willingness and way of sharing the data, data are supposed to be a public database not a property of any of the association
– Even if the data can be comparable, as the structure of membership varies throughout the Europe, the data collection might be difficult
– About the real purpose of the data collection, concerns with creating a database reaching the people on the European level, who benefits – the individuals or the associations?
– About the legal process and confidentiality of information provided (sharing among the societies/to publicise)
– About updating the data
Discussion about belonging to the national/European evaluation community:
– Clear incentives are needed in order to join a professional organization independently if it is national or European
– What is the benefit to become a member of EES/national society?, Question of benefits that members can get from any of the evaluation society for the same money
Proposal:
– To create a voluntary database of all members (of national societies) and to share, however the decision will be on the side of the members; there is no legal problems with sharing the data (keep bureaucracy to the lowest level as possible and to increase the activity)
– A general database can be very useful from the European perspective for e.g. an international research, it can help in finding partners, but the questionnaire should be then more extended on subject, experience, methodology preferences, etc…(however is this not only an alternative to LinkedIn?)
Decision:
– To report to the national Board first if it is worthwhile to share a common minimum set of data for a membership base (anonymous), limited set of information or what kind of information the national societies are willing to share and make available
o After collecting the answers to be able to fill up the database
– To report to the national Board if there will be interest and willingness to be involved in construction of the voluntary database of members/evaluators (directory), what information can be shared, what kind of template to be used?
o This might be considered as database of members of the national societies, which can help to increase the membership.
o Various concerns – not all members are real evaluators
– To get a feedback from the national societies 2-3 weeks before the next NESE meetings
Session 3: Website and Internet tools
The new EES website presented by Barbara Befani as well as the new possible features for the NESE collaboration.
Discussion:
– About the way how the NESE is presented on the website
– About the NESE discussion group, the roles of the NESE members, interactivity, coordinator/moderator/leader of the discussions, to avoid having sleeping members
– Discussion group as a tool to interact virtually and continuously, one meeting a year is not enough
– To create something for the EvalYear, EES website as a Wikipedia of evaluation…, EES website as a platform, everybody can contribute, to engage people e.g. from different DGs.
Decision:
– NESE materials on the website to be uploaded via EES Secretariat
– To share the national conferences topics
– To share the calendar of events of NESE members (google calendar), open to all, but to be edited only by a restricted amount of people and to be posted on the EES website
Decision on the accessibility of the discussion group and the NESE discussion forum topics:
– To set up a NESE discussion forum (first for NESE board only) and to create a status of NESE board members on the EES website
– To set up email alerts to the NESE board members when new topic is created
– To create 5 discussion topics (as follow – up of the EES-NESE workshop)
– To send to EES Secretariat up to 5 contact persons from each society – the rights for the NESE board members only forum to be assigned
– To create a public forum of NESE activities that can be accessed by all
Further internet tools to be developed:
– To create an interactive map incl. basic data about the membership (rather than having a list of NESE members on the EES website)
– To make an evaluation survey after the features being settled on the website
– Linking the forums on the same topics to be assessable for public
Session 4: The International Year of Evaluation
What is the aim of the International Year of Evaluation – Raising awareness on building the capacity of evaluation?
Will there be any NESE engagement in the coordination or event/action planning in 2015?
Roundtable of the societies on own plans and further ideas:
Czech Evaluation Society:
– Not any plan yet how to tackle this issue
– To use the Annual Conference (June, May) to highlight the EvalYear
– Journal – 2 issues planned and more copies to be distributed to increase awareness on evaluation
– To educate the media and to explain to the opinion makers what is the evaluation about
– Increasing awareness on the national policies (environment, health, education, transport, etc.)
Norwegian Evaluation Society:
– To make more interesting the annual conference in 2015 via a special keynote speakers – prime minister
– To raise awareness also via publishing the articles in Journal, Newspapers
UKES:
– National conference to be held
– Possibly to look for events to engage people in different sectors
– To engage young people/students
Finnish Evaluation Society:
– NESE can bring out some ideas
– Parliamentarian proposal
Polish Evaluation Society:
– At working groups will be set up
– 2015 is a 15 Anniversary of the PES, there will be joint efforts
– Proposal by Leszek Korporowicz: What is an international evaluation about?
EES:
– Not decided about concrete events
– There might be an event in the 2015 on the new trends and methods in evaluation as a follow up of the Seminar “vision and logic of evaluation” from 2011
– A potential event in cooperation with the Council of Europe on Evaluation on human rights
– Follow up of the event EES held at the European Parliament “Public hearing” – to try to bring parliamentarians around the table within the EU
Romanian Evaluation Society:
– To maximize the audience
– Partnership project with the Swiss evaluation society
Ukrainian Evaluation Society:
– Limited financial resources
– Training courses (ABC of evaluation)
– Workshop in partnership with National academy of public education
DeGeval:
– To hire a PR agency with the promotion strategy
– Working groups with a planning by every team leader individually – to coordinate these working groups in 2015 – idea to connect them internationally
– In 2014 conference is held together with Swiss Evaluation Society
Serbian Evaluation Society:
– Increase awareness about the evaluation and to promote evaluation stronger
– To contact the NGOs
Further ideas discussed:
– To create a dedicated website: central European place for the events, documents sharing, focusing on Europe
– To create a medium term task force of NESE for the International Year of Evaluation to discuss and plan the activities for the 2015
– Joint financial facilities within NESE to develop further ideas and activities and raise awareness
– To organise a seminar or workshop which can help sharing opinions
– To create working groups to activate small communities of evaluators in small cities
– To focus on how evaluation can improve the public life (good practices, examples to make evaluation more popular and adaptable)
– To promote evaluation media
– Giving voice to others who are interested in evaluation. Evaluation is not only for assessing the national or international policies but it is a systematic approach to improve our life
– Idea of doing a study on evaluation in London (summer school)
– To get really good speakers on EvalYear events
– Training for the media on what is evaluation in a way of a discussion, what they want to from the evaluators, etc.
– To publish some publications as a good memorial, to use the potential of collaborating together
Session 5: National Evaluation Conferences
The participants completed data of their national conferences, they will be circulated and shared via joint calendar.
DAY 2
Session 6: working groups
The participants were divided into two working groups:
– Strategies to increase membership
– International Year of Evaluation
Following questions were assigned to each working group:
1) To appoint a chair who will feedback to the audience + scribe
2) Discuss and summarise ideas to take back to national societies
3) Is there scope to use the web forum or promote partnership
4) What are the process steps up to the next meeting (next meeting at the EES Dublin Conference)
Working group on “Strategies to increase membership”
The summary of the problems discussed at the 1 DAY:
– Fluctuation
– Free riders
– Data collection on members inaccurate
– Inactive members (in good standing)
– Troublemakers/sabotages – opportunistic with hidden agendas
Solutions/strategies:
– Encourage continuing commitment or enlarge? Keeping existing members or attracting new ones?
– Clarifying fee/subscription policy and members level*
– Discounted two-year membership
– Focus on institutional membership
– Satisfactory survey
– Follow-up on termination (why did you leave?)
– Edit a newsletter, send it to membership and invite members to contribute actively
– EES to send you the WG points to raise with Board
– Discussion of the points with the Board at the next meeting
o If points are relevant for the national society and share/collect info/ideas on what is actually being done
– To prepare a report 2-4 weeks before the EES Conference in Dublin, the next NESE meeting to share a feedback from national board
– Attend the NESE meeting
– To circulate a note/report before the next NESE meeting
Discussion:
– Discussion about a possible discount offering to the larger group registering for the conference; However there are people interested only in the EES or in the national societies
– Too democratic decision process (the board meetings to be open to a wider audience) might raise problems at some societies
– How do you attract the audience who are still not the members, how to wider the audience?
o To promote student membership, training offers
o To increase the amount of institutional members (lower fluctuation compering to individuals), they can even contribute to the annual conference in terms of developing the programme and quality assurance (Norway’s case) – but differences in the countries in terms of organising the conference and about the fee level, big demand on the hosting role
Working group on “International year of Evaluation”:
The levels of thinking about the EvalYear:
– Most of the activities will be on the national level
– On the European level – inspired by the best practices from the countries, coordination by EES
– Common actions (small meetings to be organized through working groups, forums)
– European wide actions – to invite European Commission to cooperate in promotion to the wide public
Idea of “tailored messages”:
– To identify the EvalYear through simple ‘standardised’ or consistent messages about evaluation
– Interesting stories to promote evaluation in a wide public using short messages that are clear and understandable for media, for stakeholders, public, decision makers, students etc. (who is evaluator, how the evaluation started, short movies on evaluation to be wide spread (e.g. through EES website)
– First to identify:
o Target
o Message + title
o Communication tool
– Themes:
o What is evaluation, what is the role of evaluation, evaluation cultures, etc.
o How do we speak inside/outside?
Proposal:
– To create a matrix: some examples of problems/questions to be summarized and circulated among evaluation societies to find out, what is the understanding of evaluation in the particular countries, what kind of thinking on EvalYear is on national level, and how the activities on the international level can be coordinated.
– February – Matrix and questions to be circulated (how do we communicate with whom) in two weeks, to be sent for comments to the whole NESE
– May – to analyse the data and coordinate further within TWG on EvalYear on national and international level
– October – special panel on the EvalYear at the Conference (Remark: 1 March is the deadline for the abstract submission for the Conference!)
Session 7: AOB
Exploring national evaluation systems, by Liisa Horelli
Purpose of IOCE; map of VOPEs briefly presented.
To enhance the supply and demand side of evaluation, also through EEEs?
Activities:
– EvalYear
– P2P
– VOPE toolkit
– Innovation Challenge Competition (Evaluation Stories around the world – to be circulated to NESE members, Engaging the parliamentarians)
Engaging the parliamentarians project described:
– PEN, CoE, EES = survey on EEEs via national evaluation policies and systems
– Panel on EES conference
– Proposals for interventions = international EvalYear events and manifestations
– From USA, Africa, Asia to get the parliamentarians
Critical questions:
What is a national evaluation culture? (International atlas on the evaluation cultures defined by 9 indicators)
What is a national evaluation policy? (Mapping the status of National Evaluation policies)
What is a national evaluation system? (Critical systems heuristics (CSH))
The ideal evaluation system of the Finnish parliament presented (auditing committee)
Discussion:
– Is a NEP necessary for every context or is evaluation readiness/culture more important?
– Does National Evaluation System (real/ideal) help to understand better the context?
– What are the implication?
– Evaluation culture/ national evaluation policy / evaluation system (real/ideal) (e.g. Canada has very strict evaluation approaches and rules), different systems and approaches in different countries
Closing:
Derek Poate highlighted that the next NESE meeting will be held in Dublin, Ireland on occasion of the EES 11th Biennial Conference, the meeting to be scheduled longer than usually to have it also as a follow-up of this EES-NESE workshop
The participants were asked to collect the feedback from their national societies and report to NESE back 2-3 weeks before the NESE meeting is held.
Derek Poate thanked to all participants for a fruitful and interesting meeting and closed the workshop.
List of Participant (in alphabetical order of the societies)
Jiri Remr Czech Evaluation Society (CES)
Stefan Silvestrini DeGeval
Barbara Befani European Evaluation Society (EES)
Eva Petrova European Evaluation Society (EES)
Claudine Voyadzis European Evaluation Society (EES)
Liisa Horelli Finnish Evaluation Society (FES)
Erica Melloni Italian Evaluation Society (AIV)
Harald Furre Norwegian Evaluation Society
Beata Ciezka Polish Evaluation Society (PES)
Weronica Felcis Polish Evaluation Society (PES)
Monika Bartosiewicz-Niciolek Polish Evaluation Society (PES)
Joanna Hofman Polish Evaluation Society (PES)
Leszek Korporowitz Polish Evaluation Society (PES)
Jacek Pokorski Polish Evaluation Society (PES)
Jerry Radziwill Polish Evaluation Society (PES)
Agniezska Rudolf Polish Evaluation Society (PES)
Maciej Szalaj Polish Evaluation Society (PES)
Agniezska Szczurek Polish Evaluation Society (PES)
Roxana Mihalache Romanian Evaluation Society (Evalrom)
Michal Gozdzik Serbian Evaluation Association
Derek Poate UK Evaluation Society (UKES)
Natalya Yakymchuk Ukrainian Evaluation Society (UES)
