RSS|
MOSAIC
External evaluation report by Péter Inkei (Consultant, Director of the Budapest Observatory, Hungary) Table of contents Methods of the evaluation
The main virtues of MOSAIC, as extracted from the written national evaluation reports
The main criticism of MOSAIC, as extracted from the written national evaluation reports
The full series consists of seven reports: 1. Final report for the MATRA Programme of the Netherlands Ministry of Foreign Affairs
The Board of Advisers of MOSAIC discussed the issue of the final evalution of the activities between 1998-2000 at its meeting on 18 October 2000. The Board decided that besides an evaluation group of their own, and besides pertinent activity of the Secretariat, an external expert should also be involved. This led to the assignment to make this report. The previous version, dated 28 October, was discussed at the MOSAIC Evaluation Seminar held in Innsbruck on the 14th of November, 2001. The present text incorporates modifications, prompted by that discussion. At its October 2000 meeting the Board of Advisers agreed, that the evaluation should investigate the following: · impact in the region in general
The Board also called for a quantitative analysis. The project has been duly documented by the Secretariat and they have been willing to share all information. They produced the necessary lists and tables for a qualitative analysis of what happened. The main focus of my evaluation has been the impact of MOSAIC, for which these input figures provide little basis. The questionnaire prepared by the Secretariat for the participating ministries asked for quantified output data, but the responses failed to produce them in a required amount. This is why this report is predominantly based on opinions and observations, collected from documents, reports, interviews as well as personal participation at MOSAIC events. The primary sources were of course the documents which the Secretariat produced during the preparation, the implementation and the conclusion phases of the project. I believe I have seen and used all of them. National evaluation reports have been received from seven countries. The reports were addressed to the Secretariat (and presumably to the Board of Advisers) and the author of this "independent evaluation" received copies for information; I took the liberty to add them to my sources. During the evaluation process visits were made to 5 countries and about 40 people were consulted about the MOSAIC project; of these 22 could be labelled as formal interviews on the subject. The composition more or less reflected that of the participants in the various events: the majority were civil servants; practitioners from the field constituted the smaller part, whether from the public or the third sector. Finally, I can be taken as an "independent, external" evaluator in relation to the participating governments, the Board of Advisors and the Secretariat only. I have had the privilege of attending a number of events myself. Since these lasted several days, I had lots of opportunities to collect spontaneous observations from many participants, not knowing how precious they would be for me one day. (See list of contacts annexed). Such kind of evaluation is worth little if it is not critical, not for its own sake but in a constructive way. This has been my intention, and this is why a larger, and perhaps more stimulating part of this text is critical. It is very important to set down right at the beginning, that any criticism contained here - or praise, for that matter - should be addressed to all parties concerned: to the participating countries, mostly represented by their ministries of culture, to the Board of Advisers of the project, to the experts involved and to the Secretariat. At an even more general level, the focus of the evaluation is the potentials and constraints of assistance given through projects generated and coordinated by intergovernmental organisations to solving the problems in transition countries; and with limited financial resources at that. Instead of going through the presentation of the history, constituents and dimensions of the project, which can be consulted in other documents created in this closing phase of the 3-year period, I would prefer to pass to evaluation proper – right in medias res. The main virtues of MOSAIC, as extracted from the written national evaluation reports The national evaluation reports are composed by administrators, who were the most closely attached to the project, both professionally and in personal relationship; some of them have been in charge of the execution in various roles. Most of these reports fall short of the level of concreteness implied by the questionnaire of the Secretariat. One can establish, nevertheless, a fair picture of the virtues of the MOSAIC project, in the eyes of national administrations. An important, hardly disputable statement acknowledges that the launching of MOSAIC was a proof of good intentions on behalf of the member countries of the Council of Europe with regard to this region and to the role of culture. Another statement (from another country) declares that the project has contributed to confidence and stability in the region (which is less obvious to prove than the previous). One country remarked the strengthening of attachment to the region. Finally, the sincere - and repeated - remark from the probably most committed recipient country: MOSAIC has been a very successful project, conceived at the right time and at the right place. The reports emphasise that communicating and promoting European values and principles ("principles of the Council of Europe") has been a major function of the project. These are at some points specified also as norms and standards, contained in conventions etc. The process of national cultural policy reviews in general, and the national debates in particular have been described as successful, open, fruitful, which provided basis for further actions; they were well received by the public, and thus prepared the field for the MOSAIC project ("legitimised" it). The hitherto non-existing agenda of writing reports on areas of cultural policy was also appreciated by some. Concerning subject matter, decentralisation and related concepts have been mentioned as areas where the project appears to have influenced thinking and practices the most; the notion of civil society receives smaller emphasis in this context. New ways of financing (e.g. sponsorship) have also been identified as important and successful subject areas; cultural centres were mentioned, too. Turning to the ways how the project has produced effects, the new orientations in policy making are singled out in the national reports: MOSAIC has influenced the way problems are defined and solutions are sought. Often the project incited the need to draw up strategies of their own. Legislation appears to be the area where the effect has been the most tangible: apart from general assistance, the majority of countries report on direct influence on law making, concrete participation in drafting laws. Sometimes the direction of the influence is described: MOSAIC experts (apparently successfully) emphasised the applicability of laws and warned against excessive legislation. Large numbers of participants (ranging between 150 and 220 by country, where figures were given) are reported to have attended the various events: seminars, training courses, workshops. The national evaluators identified the gains of these people in the following (approximate) order. Participants acquired a higher level of awareness of the goals and instruments of cultural policy; they gained familiarity with experiences and models elsewhere, providing an extended reference basis for their own work; participants established professional links and contacts; they underwent experiences of realistical, open, critical and analytical treatment of problems; acquired certain essential skills and techniques; these people received higher motivation for change. Among the virtues in the execution some mentions were made about good preparation of events; especially when work was done on concrete tasks or cases; or when foreign experiences where adequately interpreted.
The re-plays of the seminar on local cultural administration, held in one capital city1, in seven provincial centres, involving altogether some 500 people, appear to be the most direct and successful example of follow-up and of multiplying impact. The main virtues of MOSAIC, as extracted from the interviews During the evaluation process visits were made to 5 countries and about 40 people were consulted about the MOSAIC project; of these 22 could be labelled as formal interviews on the subject. The composition more or less reflected that of the participants in the various events: the majority were civil servants; practitioners from the field constituted the smaller part, whether from the public or the third sector. Some, perhaps the majority of the statements (both here, and in the next section, containing the critical remarks) overlap with the observations made in the written national reports. One obvious explanation for this coincidence is that several of the authors of the reports have been interviewed and their oral opinion is in accordance with the written statements. The interviewed persons have established that the project was instrumental in breaking the isolation of the region which was due to economic reasons, and where there was such need, in breaking political isolation. At the outset it was probably not obvious, how important it would become for the participating countries that the project brought together ministry and civil sector; ministry and cultural institutions; different sectors; colleagues in the same field, sometimes in the same country. MOSAIC events provided opportunity for conceptual discussion with outside experts and among themselves, and contributed to cohesion within the sector. An important function of MOSAIC was to emphasise horizontal links, also over the border, in addition to established hierarchies. What, on the other hand, was consciously planned from the beginning, was the role of MOSAIC in subregional communication, exchange of ideas and points of views; in the promotion of co-operation in south-eastern Europe: the interviewed persons duly acknowledged and greeted this function. There is general agreement about the enormous benefits gained from the opportunities to meet with internationally renowned experts; from encountering concrete ideas and samples; local professionals found it significant that MOSAIC opened doors and windows on the wider world, useful especially for civilians and even more for local administrators. The importance of broadening horizons and creating bonds was generally greeted, regardless of the content ("it does not even matter on what topic", as one of them said). Some people put special emphasis on the role of MOSAIC in supporting local progression, in creating platforms against provincialism and national introversion. On the whole, everyone seems to agree that the main role of MOSAIC, in the broadest sense, is to promote European principles. With regard to the actual methods, as to how MOSAIC appears to have achieved positive impact, the interviews confirmed the beneficial role of the national debates, emphasising of the positive effect of outside analysts and the significance of the openness of these meetings. In some cases the sectorial analyses were also positively referred to. Some people made references to the good preparation of the various meetings. Also, publications were welcomed, especially those in vernacular (the 21 Dilemmas is the "best seller"); but other "western" publications distributed at the various events were appreciated as well. The main virtues of MOSAIC, as seen by the author of this report My own list of merits reflects a hierarchy, whose logic tends to correspond to some order of significance. The list is partly a personal synthesis of the views presented in the preceding two sections (the written reports and the oral interviews); partly qualifies as one more interview, reflecting my impressions collected during the various encounters with the project over the last three years; finally, the list is also the result of the study of a number of documents. From these, I have spent the most attention to the original Proposal from March 19982. I know that above all a checklist would be welcomed. A grid, with cells for each country, the major instruments like national debates, regional seminars, training workshops, publications etc. and an assessment, or even an evaluation mark on a scale in each cell. Instead of that, the evaluator is making an attempt to give assessment on a higher level of abstraction. MOSAIC must be credited for the following: a) A statement for the role of culture
c) Regional cohesion
e) Change agent MOSAIC (the Council of Europe, the international experts) fulfils the role of change agent, the "neutral" outsider, necessary for de-freezing established situations and conditions, the main rationale of the existence of independent counselling agencies. This condition has been the secret of the huge success of the cultural policy reviews of the Council of Europe from the start. In any community, the initiative to come together and exert self-observation does not come easy, even the need is not identified easily. Without MOSAIC most of those opportunities to take one step backwards and examine what we are doing and what we could or should be doing, and the sharing this experience with others, would not take place. The Council of Europe must be credited for the recognition of such need both inside countries and on a regional level, and for responding to this need. f) Technical assistance
The secretariat showed evidence of flexibility, which was manifested in an evolution throughout the project. Although the less than desirable feedback from the countries and other technical difficulties wore down the management to some extent in later phases, the dedication and attention has been felt all through. Special appreciation is due for the laborious tour in eight participating countries, made by members of the secretariat during the winter of 1999, when these countries failed to come forward with their demands in the pace and form as supposed in the plan. These positive features of the project need to be appreciated particularly in the light of the considerable difficulties. The difficulties which the project has been facing It was certainly clear from the outset, that the implementation of the project would be seriously hampered by circumstances. Last June the Secretariat listed the main difficulties as follows:
· The Kosovo war;
This is a correct list, and the short comments by which the Secretariat explained these items do not exaggerate. In my own view, the first issue of the above list has been the most decisive. The organisers of the project have been wrestling with the challenge of the following contradiction. They did mean to act in the spirit of avoiding top-down programming. This guiding principle was repeatedly confirmed by the Board of Advisers, which insisted that the yearly programme should be strictly proposal-driven. It is hard to conceive that anybody would seriously contest this principle. However, the very structure of the project made the smooth realisation of the demand-driven approach highly unlikely:
· The Secretariat itself felt an increasing need to reach beyond the ministries but this intention has met with obvious structural difficulties; · The ministries were unsure about what exact assistance they needed from the project for their cultural policies not only at the beginning: the most adequate nature of technical assistance has been being searched and looked for all along the three years (this, on the other hand, took place in a spirit of willingness to adjust and search together and not impose but cooperate, demonstrating a positive sign of flexibility); · The network of local contact persons, making up the bulk of the Board of Advisers, apparently failed to act as a reliable filter and transmitter of the demands arriving from such a complex clientele (which demands have still been scarce while dispersed in character), leaving most of the job of collecting and classifying queries to the Secretariat; · The mandates of these local contact persons were very different and after all, they were acting with all the limitations of civil servants. The list of difficulties, to my mind, needs to be complemented with yet another one. This concerns a conflict of interests in the countries. On the one hand, there is a recognised need for establishing or re-establishing links with other countries in the region. On the other hand, however, most of the countries are busy searching their new identities, which – among drastically altered circumstances – applies to those countries as well, whose boundaries and national symbols did not happen to change in the last decade. This drive for self-definition acts, usually in a latent form, counters the desire for forging links with other countries and hampers the search for shared values and interests. The partners lack the necessary momentum, and an organisation like the Council of Europe is ideal for removing some of the obstacles, especially the political and psychological ones. It was indeed a great experience for many professionals to plunge into these opportunities. In some instances, however, the weakening of cohesion became more explicit; according to the Secretariat, such signs of spirit of competition between nations was rarely felt with partners from the civil sector. The main criticism of MOSAIC, as extracted from the written national evaluation reports As a logical consequence of what was said about the background of the national evaluation reports, this list of critical remarks is much shorter than that of the merits therein. What is more, some of these "complaints" are in fact not addressed to the Council of Europe or the MOSAIC team. One report gives a critical list of deficits in the cultural policy in the country, which have been left in spite of MOSAIC (for example the failure to exploit the somewhat increased international interest towards small cultures.) Another report points out that many practitioners, especially in national institutions, expected immediate effects, financial and technical support, and in the absence of these were hard to persuade about the long term benefits of, say, upgrading management skills. (Some local participants had even hoped for reimbursement for their contribution to certain events.) Ministries often met with indifference from the civilian sector, which made recruitment of participants difficult. It was felt that direct involvement of MOSAIC management is detrimental to the required balance of participants. The greatest obstacle to identifying participants is the lack of language proficiency, also other engagements of many professionals, sometimes administrative problems (e.g. visas). In spite of the relatively large number of participants, whole regions in the countries had to be left out, due to limited resources. Also in the regions which were involved, it was very difficult to monitor if any follow-up took place. Going beyond the issue of participants, one report declares that no spectacular changes can be attributed to MOSAIC, but it is clear from the context that the author does not consider this fact a criticism. Another report is more outspoken about the (alleged) misunderstanding of historical experiences and superficial, paternalistic treatment of complex issues in a heterogeneous environment creates barriers and is counter-productive3, without specifying when and how deep MOSAIC has been infected by these sins. Reciprocal to what was positively said about some MOSAIC publications in the national languages, the too small number of translations was deplored. More sectorial seminars were also called for, where the accent is not on asking for more, but on the required nature of assistance. The main criticism of MOSAIC, as extracted from the interviews It was an easier job to pick out critical remarks from the notes made during the interviews than from the written reports. The reasons are evident. People are more at ease in conversation, especially if they are encouraged to formulate criticism as well. This is true of civil servants, too. As to people in cultural institutions, they have a natural scepticism about anything done by the administration, whether national of inter-governmental. This applies to ngo-s and "indies" even to a greater extent. Nevertheless, the rich collection of attributed merits, seen above, testifies that the overall balance is positive in these "extremist" segments as well. There were various critical remarks or concerns about the conceptualisation of the region. Especially serious is the anxiety that by emphasising the south-eastern cluster an alternative is developed to joining the European Union. Although there was no explicit resistance to using the word, the risk of the simplified use of the Balkan label has been voiced4. It came up in several interviews and in various countries that the project did not pay due attention to the differences in the phase of development between countries. In this context greater contextualisation was desired in the approach to respective countries and/or more explicit comparison between countries. Several people called for more of local analysis and less emphasis on treating general European trends; someone else put it as east-west comparison dominated the contents too much. As the other side of the coin, it was also observed that too much time was wasted on listening to local grievances. With regard to the general attitude of the project, someone blamed the so-called teacher-pupil relationship5. It was felt that the accent should shift from training to cooperation, involving a growing number of experts from the region itself. Concerning ways of action and instruments, the need for decomposing, analysing and comparing issues was expressed. There has been a general claim for more concrete case studies, the presentation and analysis of good practices, dynamic strategies; didactically effective, problem centred, practical, activating tasks during the various events. Some went further and claimed preference for lasting practical assistance, for model pilot projects, not once with explicit reference to successful Soros projects. Someone remarked, however, that too many cultural managers have been trained in relation to actual needs. One person stated that MOSAIC should be modified, acquire a more clear and intensive character or be stopped. As one way of increasing the relevance of MOSAIC to local needs, it was suggested to go fully receptive: develop sensors, local agents, who collect, filter and communicate specific queries to the project management. As to partners, constituency based, membership oriented civil ngo-s were advocated to the detriment of public administrations. In general, people from the third sector deplored the excessive influence of ministries. Although civils appreciated the forums which MOSAIC created, they complained about the often formal, orchestrated deliberations. As another line of thought, more concrete operationalisation of European values was called for: more exact standards or guidelines, based on, or derived from official conventions and recommendations. The PR aspect of MOSAIC was often criticised. Quoting from the remarks: the project has no mass basis, it has remained in the shadow, very limited dissemination and no visible follow-up exists. There are too few messages in the national languages. The main criticism of MOSAIC, as seen by the author of this report a) Mechanical approach Especially at the early phase of post-communist transition there was an enormous thirst for western, or just modern ideas and techniques, and in addition to the subjective drive, there has been an objective necessity for such know-how and skills ever since. Yet the impact of the transmission of know-how, of the delivery of advice is usually over-estimated. One cannot help sensing slight, distant kinship to campaigns of re-education, which aim at changing the mindset of people, in order to enable them to perform differently and run their social environment in a new way.
b) Transitional versus regional specificities
c) Internal political developments
d) Civil society
e) Formal solutions MOSAIC events were attended by a large number of participants. Besides numbers, the organisers watched for balances between countries, sectors etc. Both extremes of mistakes were committed in the recruitment: one could often see the same faces, and sometimes there were very ephemeric, occasional, out of place visitors.
Long-term, indirect goals are also served better, if an event is able to produce clear-cut direct results, based on concrete and relevant subject.
The remarkable thing about the funding if MOSAIC is its complexity. In addition to matching the Cultural Fund and other sources of the Council of Europe with the generous threefold (or bigger) contribution of the MATRA Foundation of the Netherlands, voluntary support arrived from 6 more member governments. The participating7 countries themselves have spent large amounts as well, the quantification of which will take place later. All of these resources came from public bodies. They are right to expect the project to be accounted for. No-one, not even the most critical persons, made mention of instances of blatant wastages: even those, who were sceptical about MOSAIC being the best way of spending the available international resources, formulated this view in a general way, with no specific accusations. In other words, the organisers avoided the traps of spending exceeding proportions on “luxury” or other irritating nonsense. On the other hand, examples of prudent economising could be cited. In the absence of indication of financial misuse, the evaluation of the general use of resources will follow the formula of “on the one hand – on the other”8. A high rate of the amounts spent went on travel expenses and hotel bills. This may appear to reflect low efficiency, especially if we take into account the critical remarks about the composition of participants. The expected rate of return is also low if we meticulously enlist the tangible changes reported in the countries. On the other hand, I look upon the MOSAIC project as a long-term investment indeed, and I consider that the share holders can count on decent, maybe often indirect benefits. By share holders are meant the Council of Europe and the other members of the funding consortium as well as the participating countries. It is good news, that at the time of writing these lines, the preparation of MOSAIC II is under way. That next phase can be constructed on the basis of the experiences with MOSAIC I. One of the main conclusions may be that the main emphasis should be on cooperation instead of transfer of knowledge. This means a different attitude, to basically the same task. People in the region know that they very much rely on the solidarity of more fortunate or experienced fellow-Europeans: the success of transfer is more certain though, if it takes place in the interaction of peers rather than listening to advisers or learning from instructors.9 Successful peer interaction requires even more careful preparation on behalf of the experts involved - even if it entails more expenses. Greater amount of exact subject matter, illustrating examples, case descriptions etc. are needed. There are a few more issues, where, I believe, MOSAIC II can capitalise on the accumulated experiences of MOSAIC I. With regard to the role of the civilian sector in cultural policy: instead of "involving them", the main goal should be to establish and follow such a cultural policy which is best suited to respond to the ever changing needs of the civil society. What makes a cultural policy stand this test? An important task for the Council of Europe is to answer this and to illustrate it with proven examples of other member states. On the whole, I feel the a need for more operationalising of some of the principles advocated by the Council of Europe. Where the nature of subject contradicts rigid standards contained in conventions or official recommendations, like e.g. in theatre policies, the financial support of ngo-s, measures for the conditions of artists etc., still sets of rules of game or conduct can be elaborated by teams invited by the Council of Europe. One way of tackling the issue of follow-up is to extend the scope of an event not only in advance, i.e. putting more emphasis on work prior to the actual event, but also to expect the experts (hosts, the secretariat…) to maintain some kind of follow-up activity over the Internet after the event is over. The ultimate synthesis of the evaluation of the project comes close to the brief self-picture made by the Secretariat on the project: namely that MOSAIC has been an important attempt at a consistent, value-oriented regional assistance project in the field of culture, with the merits and weaknesses of a pioneering undertaking. MOSAIC constituted part of the learning process in terms of international intergovernmental cooperation and as such, this project has accelerated the progress. Notwithstanding the validity of some of the critical remarks in this report, congratulations and expression of gratitude are due to all parties for the iniation and implementation of the MOSAIC project of the Council of Europe. 1. Evaluation reports have arrived from the following countries:
2. The following visits were made with the aim of collecting material for the evaluation report:
15-18 May, Sofia, Bulgaria
3. List of people interviewed or consulted during the evaluation phase: Ela Agotic, Tsveta Andreeva, Maria Berza, Ina Bojidarova, Geoffrey Brown, Stefan Capaliku, Raina Cherneva, Igor Chipev, Vesna Copic, Aura Corbeanu, Suzana Curin, Biserka Cveticanin, Boris Danailov, Deyana Danailova, Sanjin Dragojevic, Asaf Dzanic, Xhevdet Feri, Antony Galabov, Raina Gavrilova, Momchil Georgiev, Marko Hren, Dorota Ilczuk, Tzvetelina Iossifova, Valerij Juresic, Javor Kondakov, Enes Kujundzic, Charles Landry, Mioara Lujanschi, Dan Matei, Peter Miladinov, Pirro Misha, Davor Miskovic, Delia Mucica, Borut Penko, Oana Radu, Gonzalo Retamal, Pavle Schramedei, Vladimir Simon,, Krassie Teneva, Sreten Ugricic, Aleksandra Uzelac, Lidia Varbanova, Davor Zagar 4. List of events personally attended: Hearing on the law on publishing, February 1999, Bucharest
|
Print 
Send 
Français