Monitoring and Evaluation Methodology of
National Programmes/Strategies for Roma or Travellers:

A Brainstorming Seminar between Key Practitioners to Share Ways Forward


Strasbourg, Palais de l’Europe, Room 10,
2-3 September 2004

MEETING REPORT
 

Seminar organised by

The Council of Europe, Roma and Travellers Division



Monitoring and Evaluation of national programmes/strategies for Roma or Travellers: brainstorming seminar between key practitioners to share ways forward, Strasbourg, Council of Europe, 2-3 September 2004

 

The background:

This Seminar is a follow-up of the European Conference on Policies towards Roma, Gypsies and Travellers, which took place in Granada, Spain, on 19 and 20 May 2003. It results from a proposal made by INTRAC, a UK-based International NGO Training and Research Centre specialised in monitoring and evaluation, following that Conference. The report of the Granada Conference, which had been organised with the Authorities of Andalusia and the Centro de Investigaciones Legales y Sociales (CILS) – is available in Spanish, English and French on the Council of Europe Roma Division website: http://www.coe.int/T/E/Social_Cohesion/Roma_Gypsies

The objectives of this seminar were stated to be:

• To begin identifying the needs of governments and Roma in monitoring and evaluating Roma strategies, policies, programmes and projects

• To specify where advice and support from the Council of Europe and INTRAC may be of value in monitoring and evaluating Roma strategies, policies, programmes and projects.

The need for the Seminar:

We will be asking all participants – though an evaluation questionnaire - how far they believed the seminar met these objectives, how the exchange of information can be enhanced and what measures should be taken in each country to advance the proposals of the two workshops.

This presentation is a set of person reflections, which with the questionnaires answered and the many papers that were presented, which will be placed on the Council of Europe Website, may help act as a record of the plenary debate.

The very interest of this seminar lay in the exchange of information, the pooling of good practices and warnings concerning the harmful effects of some approaches. Quite rightly as a brainstorming seminar, beginning the process of dialogue with the Council of Europe, there were more questions than answers. The need for monitoring and evaluation was not contested however questions were posed on what was appropriate and necessary, and the motivation driving those engaged in programmes and strategies. There was little evidence of widespread monitoring and evaluation taking place, particularly at internal level from policy-makers and, from the data available, there was little use of indicators to measure progress. Consequently the question was posed “are we too early, anticipating monitoring and evaluation that is not in place”, conversely the question was asked “are we too late as programmes and strategies and programmes are already in place”. The seminar found that the same answer could be given to both questions; it was never too early and never too late to implement useful monitoring and evaluation mechanisms.

The challenges and the scope of activities:

There was a debate on where different actors should be involved. Whether it was more advantageous to be involved in policies, strategies, programmes or projects. It was agreed that it was important to understand the reality for individuals, families and communities and how action really impacted on their lives. Consequently good quality information on how needs were being met locally and through projects were important building blocks for moving from the micro to the macro perspective. A proposal was made that NGO and IGO programmes should also be evaluated to avoid double standards and to ensure that public confidence was maintained through similar transparent processes.

The seminar heard that 20 European countries have now adopted a programme or a strategy for Roma or Travellers, or are in the process of doing so. They include 11 EU member states, 3 countries acceding to the EU, 4 Balkan countries and two CIS countries.

A number of participants had described the circumstances of Roma and Travellers within their own countries pointing to the very low levels of employment, education, housing and social services such as health care. This was all set in the general context of individual and systemic discrimination against Roma and Travellers, but the lack of high quality research and in depth understanding of the complex circumstances was highlighted, alongside the general comment that monitoring and evaluation are the poor relations in these programmes There were many challenges ahead and yet some experienced speakers spoke of the considerable improvements in Europe over the last decade in combating Roma exclusion and that where these had occurred they should be analysed, evaluated and celebrated, working closely with good officials to enhance and promote successful models of good practice.

Tackling the issues of Roma social exclusion and discrimination is not negotiable in the 35 States that had ratified the Framework Convention for the Protection of National Minorities (FCNM). These States are legally bound to adopt adequate measures to promote full and effective equality between persons belonging to a national minority and those belonging to the majority in all areas of economic, social political and culture life. What is negotiable is how these economic, social, cultural and political rights are gradually realised.

Differentiation, data collection, and relevant indicators:

The seminar was reminded by a number of Roma and Traveller delegates of the differences in circumstances that there were between countries, between communities called Roma, with differences in age groups, gender, education and literacy, which are also accompanied by geographical variations. These different circumstances often implied different needs and different priorities in the nature, the quantity and the quality of the response. It was agreed that it was important to collect reliable quantifiable data to help establish good sets of baseline data, it was also strongly recommended that it was important to have quality data and ensure that quality indicators, that measured the real impact on lives, were developed and maintained at all levels of strategies and programmes. The seminar heard from UNDP of its plans to collect quantifiable data to support the World Bank and it programme for the Roma Decade. Experience had shown that it was crucial that the person and the organisation collecting the data was trusted and there was transparency and guarantees on how the data was used, noting that every individual has the right to be recognised or not to be recognised as a member of a national minority. Very substantial concerns were expressed by many participants on using the census or adjusted data from the census as it was seen in many countries to be both a political and technical instrument that had been manipulated and notoriously misused in an number of cases in the past.


Forms of monitoring and evaluation:

It was recognised that there were many forms of monitoring whether they be internal or external, local or international and there could be some confusion over the different usages of this word. The seminar compared and contrasted the important monitoring work on the FCNM, ECRI, the Social Charter, and non-governmental bodies such as ERRC and EUMAP/OSI. It was agreed that cooperation should be encouraged between different monitoring bodies to promote synergies. A clear definition was given for the purposes of this seminar of monitoring being the systematic and continuous assessment of the progress of a piece of work overtime, which checks that things are going to plan, enabling adjustments to be made in a methodical way.

An evaluation is quite different from monitoring. The working definition used was that evaluations are the periodic assessment of the relevance, performance, efficiency and impact of a piece of work with respect to its stated objectives. Evaluations could be conflictual, when based on different needs and taken from only one perspective. It was important to include the donors, those implementing the programme and beneficiaries in a participative discussion to go well beyond checking the implementation of inputs and outputs but to review the short term outcomes and the longer term impacts from different view points. These can then be constructive tools for learning and strengthening further programmes and strategies.

Programming tensions:

There can be tensions between programming needs and methods of government. Roma programming rarely attract widespread public support and there can be hostility and prejudice against them. As democratic governments are regularly facing elections and they are accountable to the majority electorate, politicians are sensitive to this. This emphasises the need for good public information and education on Roma issues but also transparency in monitoring progress and evaluating successes of programmes and strategies. Additionally the government may not be strongly motivated to support a programme or to use the results of evaluation to enhance programming, prioritise funding or to reflect on problem issues.

Decisions of governments may be based on manifestos or lobbying that is followed by parliamentary resolutions and ministerial statements. It is often a top down mechanism rather than a participatory process. Programmes are then usually managed by specific ministries or state bodies that can limit a holistic approach. Furthermore there will be a debate and a tension between integrated mainstream programming and separate provision. The seminar reached no conclusion on the advantages and disadvantages of each approach but it became clear that research on this might be particularly helpful.

Participation:

Participation of Roma is not an optional extra. It is a legal obligation for all 35 states that have ratified the FCNM, as they have agreed to the effective participation of persons belonging to national minorities in cultural, social and economic life and in public affairs. Furthermore, one of the fundamental principles underlying the Council of Europe's approach to policies for Roma or Travellers is participation of the communities concerned in all stages of the process.

There are many challenges in ensuring effective participation and many forms in which it can manifest itself, however it must always be promoted in good faith. The participation may be in parliament, in local councils, in government ministries, in local government, as employees in programmes or on monitoring committees. It requires both a breadth and a depth of participation, which may in turn require resources, training and experience in monitoring and evaluation. It can place a great burden on Roma participants to represent the diversity within their communities effectively and resources are required, so that they can bring people together to consult on policy issues and on service delivery.

Workshop Recommendations:

The two workshops (see below workshop reports) have come forward with a number of generic proposals for the future, while there have been specific requests made in response to the questionnaire and in conversations in the margins of this meeting. Clear needs have been identified and new partnerships are needed to bring good key actors together to strengthen local activities. As series of pilot projects are likely to be developed to share models of good practice, to ensure an interaction of local and international expertise and to strengthen the capacity for monitoring and evaluating programmes and strategies within countries. This beginning has also opened the door for discussions and agreements in the Council of Europe on standards for monitoring and evaluating programmes and strategies.

It has been a good beginning of a complex process that will be with us for the next decade and I congratulate you and particularly the Council of Europe for this important initiative, where you have shown your commitment by dedicating so much of your valuable time. We look forward to INTRAC and the Council of Europe working together with you on practical initiatives in the future.


Alan Phillips INTRAC,
Seminar Rapporteur
5 September 2004
 

Monitoring and Evaluation Seminar - Workshop 1
Chair: Brian Pratt, Rapporteur: Vera Klopcic

Discussion concerning the replies to the questionnaire:

1.a General remarks
In the workshop there was some discussion on whether Roma representatives were informed about the answers to the questionnaire that the Council of Europe had circulated for this meeting. It was suggested that this should have happened and that specific questions on perception of Roma should have been included rather than referring only to external experts. It was proposed that the data should be updated by countries.

1.b Information provided by different countries
Programmes for Roma were adopted under international pressure particularly in the accession process of the candidate countries in the region. Money was spent. Monitoring and evaluation was conducted mostly by the governments, which at the same time planning, implementing and monitoring the implementation. In most countries the mixed committees were established, but there is a lack of transparency or evaluation of the real impact and changes in the life of Roma.

Some suggested that new ways of monitoring were needed accompanied by much greater awareness raising about the issues.

Some of the workshop members asked if it was not somewhat premature to discuss the substance and methodology of monitoring and evaluation at this stage as so little was being undertaken at present.

Others argued that it is never too early to start the design and introduction of monitoring or too late for those who have not done so.

1.c Obstacles
There were many obstacles including lack of political will, lack of financial and human resources and resources not being spent properly.

It was noted that particular problems occur within planning and implementation of the national strategies for Roma in one country because some Roma leaders are interested only for the concrete improvements at local level, which can show immediate results.

Furthermore it was recognised that specific problems within implementation at local level were mentioned, because of the prejudices among those who have to implement projects and programmes at local level.


2. Substantial questions
A number of key substantial questions were introduced into the discussion:
- What is the aim of the programmes for Roma: Integration or further marginalisation and segregation?
- Should there be any special strategy for Roma at all?
- Should there be special programmes or only measures within sectoral policies?

In the past Roma were treated as a social and vulnerable group. Now they need recognition as a key actor in the decision-making processes and not only as passive recipient in these processes.

It was pointed out that special programmes should not be used as an excuse for failing to address of Roma issues in mainstream politics.


3. Political representation of Roma in general
There were different opinions debated on whether Roma should they establish their own parties or should they integrate their activities within mainstream parties. It was agreed, however, that Roma should strengthen their political influence to ensure that the monitoring and evaluation of programmes should occur and results be acted upon.


4. Do we need international support for training?
It was suggested that different ministries are not ready to cooperate at horizontal level in government, they always ask about financing of concrete projects. Consequently training for them is needed.

In general the Workshop saw that different bodies, ministries, Roma NGOs, local authorities may have different training needs, according in their level of concern and involvement in strategies, policies, programmes, or projects.

One possible way forward would be to experiment with pilot studies, tailor made locally, for monitoring with the wider participation of Roma communities and population.


5. Participation
It is clear that good monitoring and evaluation should be participatory and include a wide selection of stakeholders including different Roma groups, with involvement across gender and generation etc. It was important to review different levels of participation within different types of programme.

Some participants expressed the wish for additional training. Furthermore there was an expressed need for guidelines for monitoring, which should be adopted at an international level.


6. Accountability
There was discussion about monitoring and evaluation contributing towards greater accountability of governments to the Roma as citizens. Requiring participatory methods to collect views and information from the Roma citizens on programmes designed to assist them. Participation in monitoring and evaluation can provide the strength of the voice of citizens, by providing more legitimacy for those speaking on behalf of Roma, while helping learning from experience.


7. Level of monitoring
Many noted the need for regional not just national focus on monitoring, as there is often a gap between national policy and local implementation, and there can be many local prejudices. Furthermore there is a need to include a wide range of Roma voices, perspectives and views from the grass roots.

Monitoring and Evaluation Seminar - Workshop 2
Chair: Henry Scicluna Rapporteur: Nikolay Kirilov


The Workshop discussion was broken into two distinct parts. In the first part the participants concentrated on the needs for accepting a strategy for work with the Roma population, while the second part of the discussion emphasised on how the Council of Europe could help in the preparation, implementation, monitoring and evaluation of the strategies and programmes.

1. General discussion about needs

In this Workshop participants first discussed the difficulties that were faced in the preparation and monitoring of the different programmes. A number of participants spoke positively about the role that the Council of Europe had been able to play in developing strategies and programmes and how it may now continue with assistance on monitoring and evaluation.

The participants agreed that one of the main difficulties was the problem of the quality of the information and data that was available to work with. Here some of the difficulties spring from the fact that the different sources give different data, this creates difficulties in planning but also for the evaluation of the different programmes. It was also emphasised that when figures are given they often do give a true representation of the real situation, which can be highly misleading. Unemployment was given as an example. People may speak of 90% unemployment, for example but these figures do not give information on the nature of the unemployment and do not differentiate diverse circumstances. Consequently this makes it difficult to envisage the precise measures needed for the best responses to the range of particular problems. The Workshop agreed that data should not be taken as dogma.

It was emphasised that governments and inter-governmental bodies should not use the problems of inadequate information as an excuse for delays on much needed programmes. It is important to start working with what data there is and ensure that precise needs are clarified in the process of work through effective monitoring. This should then have an impact on the development of the programme.

The participants agreed that one of the main motives for governments accepting Roma strategies was the pressure exerted by Roma organisations, the pressure of the international organisations and the political will of the governments. However the workshop recognised that the initiatives first came from the Roma organisations.

2. The role of the Council of Europe

The second part of the discussion was focused on how the Council of Europe can help in the preparation, implementation, monitoring and evaluation of the strategies and programmes.

The participants in the meeting discussed a range of different possibilities of how the Council of Europe could help. At the end of the wide discussion they agreed on 6 particular recommendations recognising that there might also be other specific measures needed in selected countries:

1. Facilitating the exchange of accurate information about the difficulties and the achievements in the implementation of strategies related to Roma and Travellers.

2. Training of teams who will carry out monitoring and evaluation, with differentiated approaches to respond to the specific needs of each country.

3. Training, including training of trainers, in monitoring and evaluation according to the needs of specific countries, who will then help in the monitoring and evaluation of the different programmes.

4. Training of representatives of different state institutions and NGOs who will work on the strategy or on programmes related to the Roma and Travellers.

5. Stimulate the participation of the local authorities in monitoring and evaluation in a variety of ways using the Council of Europe networks and mechanisms.

6. Advocate, together with other international organisations, ways of resolving the problems of Roma and Travellers.

Additionally there were a number of specific agreements. The Bulgarian Government stated that they intended to organize a meeting, in cooperation with the Council of Europe, and with representatives of Roma organizations and NGOs and of local authorities. The title of the meeting will be “Participation of the local authorities in the preparation, implementation, monitoring and evaluation of strategies for the integration of the Roma communities”.

Similarly the Croatian and Macedonian government representatives extended invitations to the Council of Europe to work with it on specific issues of monitoring and evaluation.