Monitoring and Evaluation
National Programmes/Strategies for Roma or Travellers:
A Brainstorming Seminar between Key Practitioners to Share Ways Forward
Strasbourg, Palais de l’Europe, Room 10,
2-3 September 2004
Seminar organised by
The Council of Europe, Roma and Travellers Division
Monitoring and Evaluation of national programmes/strategies for Roma or
Travellers: brainstorming seminar between key practitioners to share ways
forward, Strasbourg, Council of Europe, 2-3 September 2004
This Seminar is a follow-up of the European Conference on Policies towards Roma,
Gypsies and Travellers, which took place in Granada, Spain, on 19 and 20 May
2003. It results from a proposal made by INTRAC, a UK-based International NGO
Training and Research Centre specialised in monitoring and evaluation, following
that Conference. The report of the Granada Conference, which had been organised
with the Authorities of Andalusia and the Centro de Investigaciones Legales y
Sociales (CILS) – is available in Spanish, English and French on the Council of
Europe Roma Division website: http://www.coe.int/T/E/Social_Cohesion/Roma_Gypsies
The objectives of this seminar were stated to be:
• To begin identifying the needs of governments and Roma in monitoring and
evaluating Roma strategies, policies, programmes and projects
• To specify where advice and support from the Council of Europe and INTRAC may
be of value in monitoring and evaluating Roma strategies, policies, programmes
The need for the Seminar:
We will be asking all participants – though an evaluation questionnaire - how
far they believed the seminar met these objectives, how the exchange of
information can be enhanced and what measures should be taken in each country to
advance the proposals of the two workshops.
This presentation is a set of person reflections, which with the questionnaires
answered and the many papers that were presented, which will be placed on the
Council of Europe Website, may help act as a record of the plenary debate.
The very interest of this seminar lay in the exchange of information, the
pooling of good practices and warnings concerning the harmful effects of some
approaches. Quite rightly as a brainstorming seminar, beginning the process of
dialogue with the Council of Europe, there were more questions than answers. The
need for monitoring and evaluation was not contested however questions were
posed on what was appropriate and necessary, and the motivation driving those
engaged in programmes and strategies. There was little evidence of widespread
monitoring and evaluation taking place, particularly at internal level from
policy-makers and, from the data available, there was little use of indicators
to measure progress. Consequently the question was posed “are we too early,
anticipating monitoring and evaluation that is not in place”, conversely the
question was asked “are we too late as programmes and strategies and programmes
are already in place”. The seminar found that the same answer could be given to
both questions; it was never too early and never too late to implement useful
monitoring and evaluation mechanisms.
The challenges and the scope of activities:
There was a debate on where different actors should be involved. Whether it was
more advantageous to be involved in policies, strategies, programmes or
projects. It was agreed that it was important to understand the reality for
individuals, families and communities and how action really impacted on their
lives. Consequently good quality information on how needs were being met locally
and through projects were important building blocks for moving from the micro to
the macro perspective. A proposal was made that NGO and IGO programmes should
also be evaluated to avoid double standards and to ensure that public confidence
was maintained through similar transparent processes.
The seminar heard that 20 European countries have now adopted a programme or a
strategy for Roma or Travellers, or are in the process of doing so. They include
11 EU member states, 3 countries acceding to the EU, 4 Balkan countries and two
A number of participants had described the circumstances of Roma and Travellers
within their own countries pointing to the very low levels of employment,
education, housing and social services such as health care. This was all set in
the general context of individual and systemic discrimination against Roma and
Travellers, but the lack of high quality research and in depth understanding of
the complex circumstances was highlighted, alongside the general comment that
monitoring and evaluation are the poor relations in these programmes There were
many challenges ahead and yet some experienced speakers spoke of the
considerable improvements in Europe over the last decade in combating Roma
exclusion and that where these had occurred they should be analysed, evaluated
and celebrated, working closely with good officials to enhance and promote
successful models of good practice.
Tackling the issues of Roma social exclusion and discrimination is not
negotiable in the 35 States that had ratified the Framework Convention for the
Protection of National Minorities (FCNM). These States are legally bound to
adopt adequate measures to promote full and effective equality between persons
belonging to a national minority and those belonging to the majority in all
areas of economic, social political and culture life. What is negotiable is how
these economic, social, cultural and political rights are gradually realised.
Differentiation, data collection, and relevant indicators:
The seminar was reminded by a number of Roma and Traveller delegates of the
differences in circumstances that there were between countries, between
communities called Roma, with differences in age groups, gender, education and
literacy, which are also accompanied by geographical variations. These different
circumstances often implied different needs and different priorities in the
nature, the quantity and the quality of the response. It was agreed that it was
important to collect reliable quantifiable data to help establish good sets of
baseline data, it was also strongly recommended that it was important to have
quality data and ensure that quality indicators, that measured the real impact
on lives, were developed and maintained at all levels of strategies and
programmes. The seminar heard from UNDP of its plans to collect quantifiable
data to support the World Bank and it programme for the Roma Decade. Experience
had shown that it was crucial that the person and the organisation collecting
the data was trusted and there was transparency and guarantees on how the data
was used, noting that every individual has the right to be recognised or not to
be recognised as a member of a national minority. Very substantial concerns were
expressed by many participants on using the census or adjusted data from the
census as it was seen in many countries to be both a political and technical
instrument that had been manipulated and notoriously misused in an number of
cases in the past.
Forms of monitoring and evaluation:
It was recognised that there were many forms of monitoring whether they be
internal or external, local or international and there could be some confusion
over the different usages of this word. The seminar compared and contrasted the
important monitoring work on the FCNM, ECRI, the Social Charter, and
non-governmental bodies such as ERRC and EUMAP/OSI. It was agreed that
cooperation should be encouraged between different monitoring bodies to promote
synergies. A clear definition was given for the purposes of this seminar of
monitoring being the systematic and continuous assessment of the progress of a
piece of work overtime, which checks that things are going to plan, enabling
adjustments to be made in a methodical way.
An evaluation is quite different from monitoring. The working definition used
was that evaluations are the periodic assessment of the relevance, performance,
efficiency and impact of a piece of work with respect to its stated objectives.
Evaluations could be conflictual, when based on different needs and taken from
only one perspective. It was important to include the donors, those implementing
the programme and beneficiaries in a participative discussion to go well beyond
checking the implementation of inputs and outputs but to review the short term
outcomes and the longer term impacts from different view points. These can then
be constructive tools for learning and strengthening further programmes and
There can be tensions between programming needs and methods of government. Roma
programming rarely attract widespread public support and there can be hostility
and prejudice against them. As democratic governments are regularly facing
elections and they are accountable to the majority electorate, politicians are
sensitive to this. This emphasises the need for good public information and
education on Roma issues but also transparency in monitoring progress and
evaluating successes of programmes and strategies. Additionally the government
may not be strongly motivated to support a programme or to use the results of
evaluation to enhance programming, prioritise funding or to reflect on problem
Decisions of governments may be based on manifestos or lobbying that is followed
by parliamentary resolutions and ministerial statements. It is often a top down
mechanism rather than a participatory process. Programmes are then usually
managed by specific ministries or state bodies that can limit a holistic
approach. Furthermore there will be a debate and a tension between integrated
mainstream programming and separate provision. The seminar reached no conclusion
on the advantages and disadvantages of each approach but it became clear that
research on this might be particularly helpful.
Participation of Roma is not an optional extra. It is a legal obligation for all
35 states that have ratified the FCNM, as they have agreed to the effective
participation of persons belonging to national minorities in cultural, social
and economic life and in public affairs. Furthermore, one of the fundamental
principles underlying the Council of Europe's approach to policies for Roma or
Travellers is participation of the communities concerned in all stages of the
There are many challenges in ensuring effective participation and many forms in
which it can manifest itself, however it must always be promoted in good faith.
The participation may be in parliament, in local councils, in government
ministries, in local government, as employees in programmes or on monitoring
committees. It requires both a breadth and a depth of participation, which may
in turn require resources, training and experience in monitoring and evaluation.
It can place a great burden on Roma participants to represent the diversity
within their communities effectively and resources are required, so that they
can bring people together to consult on policy issues and on service delivery.
The two workshops (see below workshop reports) have come forward with a number
of generic proposals for the future, while there have been specific requests
made in response to the questionnaire and in conversations in the margins of
this meeting. Clear needs have been identified and new partnerships are needed
to bring good key actors together to strengthen local activities. As series of
pilot projects are likely to be developed to share models of good practice, to
ensure an interaction of local and international expertise and to strengthen the
capacity for monitoring and evaluating programmes and strategies within
countries. This beginning has also opened the door for discussions and
agreements in the Council of Europe on standards for monitoring and evaluating
programmes and strategies.
It has been a good beginning of a complex process that will be with us for the
next decade and I congratulate you and particularly the Council of Europe for
this important initiative, where you have shown your commitment by dedicating so
much of your valuable time. We look forward to INTRAC and the Council of Europe
working together with you on practical initiatives in the future.
Alan Phillips INTRAC,
5 September 2004
Monitoring and Evaluation
Seminar - Workshop 1
Chair: Brian Pratt, Rapporteur: Vera Klopcic
Discussion concerning the replies to the questionnaire:
1.a General remarks
In the workshop there was some discussion on whether Roma representatives were
informed about the answers to the questionnaire that the Council of Europe had
circulated for this meeting. It was suggested that this should have happened and
that specific questions on perception of Roma should have been included rather
than referring only to external experts. It was proposed that the data should be
updated by countries.
1.b Information provided by different countries
Programmes for Roma were adopted under international pressure particularly in
the accession process of the candidate countries in the region. Money was spent.
Monitoring and evaluation was conducted mostly by the governments, which at the
same time planning, implementing and monitoring the implementation. In most
countries the mixed committees were established, but there is a lack of
transparency or evaluation of the real impact and changes in the life of Roma.
Some suggested that new ways of monitoring were needed accompanied by much
greater awareness raising about the issues.
Some of the workshop members asked if it was not somewhat premature to discuss
the substance and methodology of monitoring and evaluation at this stage as so
little was being undertaken at present.
Others argued that it is never too early to start the design and introduction of
monitoring or too late for those who have not done so.
There were many obstacles including lack of political will, lack of financial
and human resources and resources not being spent properly.
It was noted that particular problems occur within planning and implementation
of the national strategies for Roma in one country because some Roma leaders are
interested only for the concrete improvements at local level, which can show
Furthermore it was recognised that specific problems within implementation at
local level were mentioned, because of the prejudices among those who have to
implement projects and programmes at local level.
2. Substantial questions
A number of key substantial questions were introduced into the discussion:
- What is the aim of the programmes for Roma: Integration or further
marginalisation and segregation?
- Should there be any special strategy for Roma at all?
- Should there be special programmes or only measures within sectoral policies?
In the past Roma were treated as a social and vulnerable group. Now they need
recognition as a key actor in the decision-making processes and not only as
passive recipient in these processes.
It was pointed out that special programmes should not be used as an excuse for
failing to address of Roma issues in mainstream politics.
3. Political representation of Roma in general
There were different opinions debated on whether Roma should they establish
their own parties or should they integrate their activities within mainstream
parties. It was agreed, however, that Roma should strengthen their political
influence to ensure that the monitoring and evaluation of programmes should
occur and results be acted upon.
4. Do we need international support for training?
It was suggested that different ministries are not ready to cooperate at
horizontal level in government, they always ask about financing of concrete
projects. Consequently training for them is needed.
In general the Workshop saw that different bodies, ministries, Roma NGOs, local
authorities may have different training needs, according in their level of
concern and involvement in strategies, policies, programmes, or projects.
One possible way forward would be to experiment with pilot studies, tailor made
locally, for monitoring with the wider participation of Roma communities and
It is clear that good monitoring and evaluation should be participatory and
include a wide selection of stakeholders including different Roma groups, with
involvement across gender and generation etc. It was important to review
different levels of participation within different types of programme.
Some participants expressed the wish for additional training. Furthermore there
was an expressed need for guidelines for monitoring, which should be adopted at
an international level.
There was discussion about monitoring and evaluation contributing towards
greater accountability of governments to the Roma as citizens. Requiring
participatory methods to collect views and information from the Roma citizens on
programmes designed to assist them. Participation in monitoring and evaluation
can provide the strength of the voice of citizens, by providing more legitimacy
for those speaking on behalf of Roma, while helping learning from experience.
7. Level of monitoring
Many noted the need for regional not just national focus on monitoring, as there
is often a gap between national policy and local implementation, and there can
be many local prejudices. Furthermore there is a need to include a wide range of
Roma voices, perspectives and views from the grass roots.
Monitoring and Evaluation Seminar - Workshop 2
Chair: Henry Scicluna Rapporteur: Nikolay Kirilov
The Workshop discussion was broken into two distinct parts. In the first part
the participants concentrated on the needs for accepting a strategy for work
with the Roma population, while the second part of the discussion emphasised on
how the Council of Europe could help in the preparation, implementation,
monitoring and evaluation of the strategies and programmes.
1. General discussion about needs
In this Workshop participants first discussed the difficulties that were faced
in the preparation and monitoring of the different programmes. A number of
participants spoke positively about the role that the Council of Europe had been
able to play in developing strategies and programmes and how it may now continue
with assistance on monitoring and evaluation.
The participants agreed that one of the main difficulties was the problem of the
quality of the information and data that was available to work with. Here some
of the difficulties spring from the fact that the different sources give
different data, this creates difficulties in planning but also for the
evaluation of the different programmes. It was also emphasised that when figures
are given they often do give a true representation of the real situation, which
can be highly misleading. Unemployment was given as an example. People may speak
of 90% unemployment, for example but these figures do not give information on
the nature of the unemployment and do not differentiate diverse circumstances.
Consequently this makes it difficult to envisage the precise measures needed for
the best responses to the range of particular problems. The Workshop agreed that
data should not be taken as dogma.
It was emphasised that governments and inter-governmental bodies should not use
the problems of inadequate information as an excuse for delays on much needed
programmes. It is important to start working with what data there is and ensure
that precise needs are clarified in the process of work through effective
monitoring. This should then have an impact on the development of the programme.
The participants agreed that one of the main motives for governments accepting
Roma strategies was the pressure exerted by Roma organisations, the pressure of
the international organisations and the political will of the governments.
However the workshop recognised that the initiatives first came from the Roma
2. The role of the Council of Europe
The second part of the discussion was focused on how the Council of Europe can
help in the preparation, implementation, monitoring and evaluation of the
strategies and programmes.
The participants in the meeting discussed a range of different possibilities of
how the Council of Europe could help. At the end of the wide discussion they
agreed on 6 particular recommendations recognising that there might also be
other specific measures needed in selected countries:
1. Facilitating the exchange of accurate information about the difficulties and
the achievements in the implementation of strategies related to Roma and
2. Training of teams who will carry out monitoring and evaluation, with
differentiated approaches to respond to the specific needs of each country.
3. Training, including training of trainers, in monitoring and evaluation
according to the needs of specific countries, who will then help in the
monitoring and evaluation of the different programmes.
4. Training of representatives of different state institutions and NGOs who will
work on the strategy or on programmes related to the Roma and Travellers.
5. Stimulate the participation of the local authorities in monitoring and
evaluation in a variety of ways using the Council of Europe networks and
6. Advocate, together with other international organisations, ways of resolving
the problems of Roma and Travellers.
Additionally there were a number of specific agreements. The Bulgarian
Government stated that they intended to organize a meeting, in cooperation with
the Council of Europe, and with representatives of Roma organizations and NGOs
and of local authorities. The title of the meeting will be “Participation of the
local authorities in the preparation, implementation, monitoring and evaluation
of strategies for the integration of the Roma communities”.
Similarly the Croatian and Macedonian government representatives extended
invitations to the Council of Europe to work with it on specific issues of
monitoring and evaluation.