Evaluation of the partnership ... - European Commission · training with the Council of Europe, in...

Deloitte & Touche Management Solutions NV Naamloze vennootschap Maatschappelijke zetel: Louizalaan 240, B-1050 Brussel HR Brussel 200.472 - BTW BE 403.500.697

Evaluation of the partnership covenant on training in youth work between the European Commission and the Council of Europe Final report European Commission DG Education and Culture

Deloitte & Touche March 2004

Tel: +32 2 800 28 50 Fax: +32 2 800 28 04 http://www.deloitte.be

Management Solutions Pegasus Park Berkenlaan 8b B-1831 Diegem Belgium

http://www.deloitte.be

D e l o i t t e & T o u c h e 1

Evaluation of the partnership covenant on training in youth work between the European Commission and the Council of Europe Final report - March, 2004

TABLE OF CONTENTS 1. Executive summary ....................................................................................................................... 3 2. Introduction ................................................................................................................................... 5 2.1. Framework of the Covenant............................................................................................................ 5 2.2. General objectives of the Covenant................................................................................................. 6 2.3. Specific aims of the Covenant......................................................................................................... 6 2.4. Purpose of the evaluation ................................................................................................................ 7 2.5. Content of the evaluation ................................................................................................................ 7

3. Research methodology ................................................................................................................ 10 3.1. Design of the research ................................................................................................................... 10 3.2. Design of questionnaires ............................................................................................................... 11 3.3. Work plan for the evaluation......................................................................................................... 13

4. Quantitative description of the training tools........................................................................... 15 4.1. Training courses on European Citizenship.................................................................................... 15 4.2. Advanced training for trainers- ATTE .......................................................................................... 16 4.3. Training-kits, T-kits ...................................................................................................................... 17 4.4. Coyote Magazine........................................................................................................................... 19 4.5. Overview of activities and outputs................................................................................................ 21

5. The evaluation criteria - Analysis of the Training Tools ......................................................... 23 5.1. Relevance of the Training tools of the Partnership ....................................................................... 23

5.1.1. Accuracy of needs analysis............................................................................................. 23 5.1.2. Appropriateness of instruments/methods in relation to the objectives ........................... 25 5.1.3. Complementarity ............................................................................................................ 26

5.2. Results ........................................................................................................................................... 28 5.3. Impact - What indications have we detected in our evaluation? ................................................... 37 5.4. Efficiency in the implementation of the Partnership..................................................................... 43

5.4.1. Management and decision-making within the Partnership............................................. 43 5.4.2. Maximising transparency and efficiency in selection of participants and trainers......... 45 5.4.3. Cost- efficiency and co-ordination of meetings.............................................................. 45 5.4.4. Using the potential of the Internet as a tool for dissemination ....................................... 46 5.4.5. Inclusion of National Agencies ...................................................................................... 46 5.4.6. Multiplication and transfer- dissemination of results from centre to periphery ............. 47 5.4.7. Recognition..................................................................................................................... 48

5.5. Effectiveness ................................................................................................................................. 49

6. Conclusions and recommendations ........................................................................................... 51 7. Annexes ........................................................................................................................................ 55



List of tables Table 1: Number of participants per country in TC I, II, and III combined ......................................... 15 Table 2: Distribution of T-kits .............................................................................................................. 18 Table 3: Distribution of the Coyote Magazine...................................................................................... 19 Table 4: Overview of activities and outputs ......................................................................................... 21 Table 5: Use of the tools by TC and ATTE participants....................................................................... 26



1 . EXECUTIVE SUMMARY

The evaluation of the Partnership Covenant on training on youth work between the European Commission and the Council of Europe took place from September 2003 to March 2004. The Covenant’s activities were reaching completion during the same time frame, having commenced in 2000. The evaluation featured substantial documentary research, attendance at the ATTE training course run under the Partnership, and questionnaire and interview research with Partnership activity participants, National Agencies and SALTO resource centres of the EU Youth Programme, and officials in both the Commission and the Council of Europe. In our analysis we have considered the following evaluation criteria:

• Relevance – the closeness of the Partnership’s activities and products to needs; • Impact – what was achieved? • Efficiency – how good was the process? • Effectiveness – were the objectives reached?

We conclude that the Partnership Covenant largely achieved the objectives and produced the results that were expected. A considerable volume of direct training activity has taken place and ten separate publications have been produced. A functioning website has been created and some flanking activities in relation to curriculum development and related areas have also been completed. In relation to “relevance”, the work has generally been highly relevant to the developing scene in European Youth work training today. Despite the absence of in-depth needs analysis, the content of the Partnership’s activities and the approach taken has been largely judged to be relevant to the needs of the sector. In this respect, however, a more directly practical orientation, focusing on the studied needs of Youth workers at all levels within the system, is recommended for the future. This will build on the success achieved to date and make the activities even more interesting to a wider audience. The impact of the activities is difficult to measure, partly because of the fact that they have only just been completed. Nevertheless, the ability of the Covenant’s managers to measure impact is limited, given the absence of a developed tool for tracking and monitoring the extent to which participants in the Covenant’s events or subscribers to its publications actually make use of them in order to develop new activities in their own context. Related to this is the finding that an explicit strategy for ensuring broader impact through a multiplier effect has not yet been developed within the Partnership structure, and is something which we recommend. In terms of the desirable synergies with the EU Youth programme and other Council of Europe youth activities, we believe that the Covenant did not succeed in securing optimal integration with the other parts of the Youth programme, but that its work benefited appropriately from the close liaison with the Council of Europe’s activities. The uneven involvement of Council of Europe and European Commission staff in delivering the activities of the Partnership (with a disproportionate involvement of the Council of Europe) also had



an impact on the degree of integration of the Covenant with the Youth programme. This can be addressed in the next phase. In relation to the publications and website, we call for a more sophisticated and integrated strategy, in order to move forward towards a situation where all relevant materials and activities in this key area of Youth training (both those run within the Covenant and outside it) are structured in such a way that the user can have access to as broad as possible a range of material in a well-organised format, both paper and electronic. The results in terms of effectiveness show a general compliance in terms of outputs being achieved in line with the established programme. However, this was achieved with a limited investment of human resources and there is evidence of this having had a negative impact on the ability of the Covenant to maximise its effectiveness through devoting attention to qualitative management, strategic planning and some of the medium or longer term elements of the overall programme. In terms of efficiency, the above comment is also relevant as the limited human resources that were available to run the Covenant, while producing the results described above, were inevitably stretched. There is also room for improvement in terms of the governance and strategic management of the Covenant, both in terms of the level of attention given to it by the two key institutions, and also in terms of its integration with the broader European Youth activities. One of the key areas for improvement here is in the relationship with the Youth programme’s National Agencies and SALTO centres, something which is clearly the intention of the institutions for the future. Our report concludes with a number of recommendations in line with the above comments. We believe that the degree of investment that has been made has been justified by the overall results that have been achieved, and acknowledge the commitment and efforts of those charged with the management and leadership of the Covenant’s activities. The opportunity exists to boost the visibility of the Covenant and thereby to contribute to the steady increase in awareness and appreciation of the contribution made by youth work training, and youth activities more generally, to education and development in Europe.



2. INTRODUCTION

2.1. Framework of the Covenant

This evaluation concerns the Partnership Covenant on training in youth work between the European Commission and the Council of Europe (2000-2003).

The partnership Covenant was launched by the European Commission and the Council of Europe in 1998. The subject of this evaluation is the third Covenant on training, concluded between the European Commission and the Council of Europe for the period 2000 to 2003, and still functioning during our evaluation. It is within this context that the Commission has concluded the Covenant on training with the Council of Europe, in order to promote active European Citizenship of young people by giving impetus to the training and information of youth leaders and youth workers at a European level.

The Covenant comprises the development of four training tools:

A. Training courses on European Citizenship in Youth work;

B. Advanced Training for Trainers in Europe, a high-level course for experienced professionals in the youth field;

C. Training kits (T-kits), thematic publications on specific training topics addressed to trainers and youth workers at European level; and

D. Coyote, a newsletter addressed to trainers and youth workers, which comes out twice a year.

The T-kits and Coyote have a much wider direct target audience than the training courses.

The European Commission developed the Covenant jointly with the Youth and Sport Directorate of the Council of Europe. The Covenant is based on the EU’s YOUTH programme adopted on 13 April 2000. The programme mainly targets the 15 EU Member States, the three EEA countries and 12 pre-accession countries, altogether called programme countries, but is also open to participation by other countries (“third countries”).

Beneficiaries of the Youth programme are young people aged from 15 to 25. The programme is directed in particular at those young people who have had few, if any, chances to benefit from national or international mobility activities, and fewer developmental opportunities in general. The programme is also directed at those responsible for youth organisations, at youth workers and youth leaders, and those in charge of youth work at local and national level.

The programme offers financial resources to develop different types of activities. Besides the actions targeting young people, the Programme provides funds for support activities (Action 5), namely activities aiming at further development of Youth co-operation, such as training of youth workers, exchange of information and know-how between youth organisations and youth structures, support for networking and partnerships, information, study visits and seminars for all those involved in youth work or youth policy at local, regional and national level.



2.2. General objectives of the Covenant

The general objective of the Partnership between the European Commission and the Council of Europe is the promotion of European Citizenship and of international co-operation between young people, youth workers/leaders and youth work structures in Europe. This aim is primarily pursued through the YOUTH programme of the European Commission and the programmes of the Directorate of Youth and Sport of the Council of Europe.

Another general objective of the Covenant is to put a major emphasis on the quality and content of projects, funded by the EU and the Council. Major emphasis is placed on projects promoting European Citizenship, the participation of young people in society and politics, intercultural learning, the fight against racism and xenophobia and the involvement of disadvantaged young people.

2.3. Specific aims of the Covenant

The specific aims of the Covenant on training are set out with regard to training courses and publications, as follows: Training courses, Tools A+B − Facilitating the training of youth workers and the development of youth work methods, − To extend the youth workers/leaders knowledge of Europe, citizenship and young people in

Europe, − To improve the quality of European level youth worker training activities and to establish

standards for them, − To develop a network of trainers on a European level in the youth field; − To enable the trainers to design and implement the methods necessary for European level training

activities in the youth/non formal education field; − To disseminate the outcome of the training courses funded by the Covenant to a larger public of

trainers and youth workers in Europe; − To develop multiplier effects with regard to the National Agencies and the resource centres

(SALTO) within the YOUTH Programme of the European Commission. In this context the two training course models should be differentiated, due to their different target groups and content. Publications, Tools C+D − To inform trainers and youth workers on training concepts, methodologies, practices and current

developments in training at European level; − To provide a forum for the exchange of information and best practices in training and for

discussion amongst trainers and youth workers/leaders on training related issues; − To assist trainers and youth workers/leaders to give a European dimension to their training

activities; − To disseminate new methodologies of non-formal education and youth work and good practices

to a larger public of trainers and youth workers in Europe; − To develop a multiplier effect with regard to the National Agencies and resource centres

(SALTO) within the YOUTH Programme of the European Commission.



2.4. Purpose of the evaluation

The main purpose of the evaluation was to provide orientations and recommendations for the new generation of programmes of DG Education and Culture, to be established as of 2007. The evaluation will also serve for the preparation of a fourth Covenant on training covering the period from 2004 to 2006 (as part of the second phase of YOUTH Programme).

The evaluation focused on a qualitative approach rather than on collection of exhaustive quantitative data. The conclusions cover the important aspects requiring action and most conclusions lead to a corresponding recommendation. The following questions were included by the Commission in the Terms of Reference: − To which extent have the general and specific objectives of the Covenant on training been

achieved? − What have been the main successes/failures of the Covenant to date? − In how far has the Covenant on training contributed to increase the quality of activities and

products realised under the YOUTH programme? − To what extent has the action brought an added value, i.e. in how far has it succeeded in

developing new activities/partnerships/methods? − In which concrete ways could the relevance, efficiency, effectiveness and impact of the Covenant

be increased? − What are the recommendations as to the visibility of the Covenant in the YOUTH Programme? − What are the recommendations as to the re-orientation of the action for the forth Covenant

(second phase of the YOUTH Programme)? − What are the recommendations for the preparation of the new programme? Should there be a

revision of the different activities? Which products should be maintained? Which products should be modified? Which products should be abandoned?

− What are the recommendations for design of the new programme? Should there be a revision of the objectives or of the budgetary allocation?

2.5. Content of the evaluation

We present below the basic elements of the various parts of the evaluation. The four training tools are firstly listed and presented alongside each key question to be analysed per tool. Secondly, we provide a brief description on how the outputs of the Covenant are presented and the key questions to be answered. This part is based on statistical data provided by the Council of Europe and forms the quantitative part of the evaluation. Thirdly, we present details on the perspectives/issues forming the qualitative evaluation criteria: relevance, results, impact, efficiency and effectiveness. The four training tools -Key questions Training tool Key question a) Training courses on European Citizenship in youth work

To what extent have the Training courses on European Citizenship in Youth Work supported the professional development of youth workers and youth leaders by extending their competencies to integrate European citizenship within their projects and practices?



b)The advanced Training for trainers in Europe

To what extent has the Advanced training for trainers in Europe contributed to the improvement of the training competencies of the participants to enable them to implement European Youth training activities of high quality?

c) The Training kits (T-kits) To what extent have the T-Kits been disseminated and used at European level?

d) Coyote To what extent has the Coyote magazine provided a forum for trainers and youth workers and to what extent has it provided information on specific topics related to training issues and current developments in this field?

Quantitative description of outputs

This point of our report is based on the material provided by the Commission/Council of Europe and includes a description of each of the four training tools.

We provide a general synthetic description of activities implemented in the course of the Covenant 2000-2003, in order to give further explanation to and deepen the understanding of the context of the evaluation.

Qualitative evaluation criteria We applied the following approach to the qualitative evaluation work:

Relevance Analysis of the relevance of the Covenant was conducted, taking into account the following perspectives: − Accuracy of needs analysis; − Choice of key issues/themes; − Complementarity; − Accuracy of instruments/methods.

Results Analysis of the results of the Covenant was conducted, taking into account the following perspectives: − Added value produced to the Youth programme and especially Action 5; − Increase in quality and quality awareness as regards training; − Level of innovation; − Incorporation of European citizenship as a theme in all training tools; − Increase in co-operation between National Agencies and SALTO resource centres and other

European level actors ; − Tendencies in the reception of and demand for the produced tools by the target groups; − Multiplier effect; − Development of quality standards for European level youth worker training; and − Development of a European level network of trainers.

Impact Analysis of the impact of the Covenant was conducted, taking into account the following perspectives:



− Indicators of financial support for international co-operation between young people, youth workers and youth work structures;

− Indicators of promotion of European citizenship, participation of young people in society and politics, intercultural learning, fight against racism and xenophobia and the involvement of disadvantaged youth groups as content in training tools;

− Indicators of dissemination of results/knowledge/skills/experiences; − Indicators of improved quality in European level youth worker training; − Indicators of visibility of the role of the European Commission and the YOUTH programme in

the training tools of the Covenant.

Efficiency Analysis of the efficiency of the Covenant was conducted, taking into account the following perspectives:

− Cost analysis in relation to activities performed and in relation to the overhead costs of the Covenant;

− Flexibility of management of the actions, in order to adjust to changes in the general context; − General assessment of value for money; − Efficiency in management and monitoring of the Covenant by the European Commission and the

Council of Europe; − The extent to which new technology was used in order to ensure cost efficiency and save time.

Effectiveness Analysis of the effectiveness of the Covenant was conducted, taking into account the following perspectives:

− Level of achievement of foreseen results and outputs; and − General assessment of the impact achieved compared to that envisaged.

Our programme of documentary research, surveys of key actors and interviews has allowed us to address and report on all of these issues in this final report.

We would like to emphasise that the training tools developed during these three years of activity are of high quality and that the commitment and ambitions of the staff, trainers, authors, participants and experts involved in the implementation of the activities and publications are also very high. Chapter 3 explains our research methodology, and in Chapter 4 we give a brief quantitative description of the results and outputs of the Third Partnership Covenant. Chapter 5 guides the reader through the evaluation criteria and our considerations of the results in relation to those. In the last chapter, Chapter 6, we give a summary of our findings and our main recommendations in relation to the future work to be developed within the Fourth Partnership Covenant and later on in the framework of the new Youth programme starting from year 2007.

D e l o i t t e & T o u c h e 1 0


3. RESEARCH METHODOLOGY

3.1. Design of the research

Our study follows basically the same lines as presented in our proposal for the evaluation. Below you will find the four training tools presented with each key question that we identified in our proposal for the evaluation. Training tool

Key question

a) Training courses on European Citizenship in youth work

To what extent has the Training courses on European Citizenship in Youth Work, supported the professional development of youth workers and youth leaders by extending their competencies to integrate European citizenship within their projects and practices?

b)The advanced Training for trainers in Europe

To what extent has the Advanced training for trainers in Europe contributed to the improvement of the training competencies of the participants to enable them to implement European Youth training activities of high quality?

c) The Training kits (T-kits) To what extent has the T-Kits reached out, been used and been largely disseminated at European level?

d) Coyote To what extent has the coyote magazine provided a forum for trainers and youth workers and to what extent has it reached out with information on specific topics related to training issues and current development in this field?

Each of the tools was analysed separately, but in the course of the evaluation, it became clear that there was also a need for a more integrated approach, in order to be able to come to global conclusions and recommendations. We therefore conducted the evaluation with an integrative approach, seeking to trace synergies that could be detected within the documentation provided and in the answers to our questions in the questionnaires. As we assumed that the Partnership would be finished by June 2003 and that our evaluation would thereby be conducted at a stage where all activities and publications would be finished, the fact that the Partnership was still running while we were conducting the evaluation made our study somewhat more complicated to conduct than we originally had foreseen in our proposal. In that sense it resembles an ex-post evaluation, but took place before all outputs had been completed and in a context where all outcomes and impacts have not yet occurred. We requested most of the documentation in relation to meeting reports, reports from expert groups and statistics on the dissemination of publications. As we were not provided with a synthetic description of the activities, meetings or reports and as we had to start our desk research by identifying and producing a chronological description of the activities implemented within the Third Covenant, our originally planned Desk research phase extended into the phase where we were planning to concentrate on the collection of complementary data by the survey we conducted. The last documents that are included in our list of documentation (Annex 7) were received in early December



2003. This circumstance is the reason for the overlapping of the originally planned research phases as presented in the Terms of Reference and in our proposal; − Preparatory meetings and briefing sessions;

− Study and analysis of the available documentation - Desk research;

− Gathering of information and assessment of results and impact, by means of − Questionnaires and possible additional telephone interviews; − Participation in Evaluation seminars; − Elaborations of conclusions and recommendations for the future.

By adding in activities as we have found them during our extended desk-research, we have been able to produce an overview of meetings, actions and outputs. (See Chapter 3:1 Overview of activities and outputs) We have studied the extensive documentation in relation to the work produced between 2000 and 2003. An approximation of the number of pages of written text produced by the Partnership would amount to around 2000 pages. (See annex 7: Overview of Documentation) In addition to the documentary analysis we conducted a survey with one questionnaire targeted at the participants in the 3 training courses on European Citizenship and the participants in the “ATTE” training course, and another questionnaire targeted at the network of National Agencies and SALTO resource centres. The surveys aimed to complement our desk research and give an indication of the participants’, National Agencies’ and SALTO resource centres’ appreciation of/opinions on the training activities and publications produced within the Partnership. In agreement with the Commission, we decided to target the Staff of the National Agencies that are responsible for the Training activities and Action 5 (TCP officers) and the responsible staff at the SALTO resource centres, as well as all participants in the three Training courses on European Citizenship (in total approximately 90 people) and the 30 participants in the ATTE course.

3.2. Design of questionnaires

The design of questionnaires and choice of questions were dependent on preparation meetings and desk research. The aim of the questionnaires was to bridge eventual gaps in the desk research material made available to the evaluators. Questions were developed to fit each target group. The respondents were made aware of the great importance of their role as suppliers of analysis material and that their replies were crucial to the evaluation. The three target groups were as follows:

i. Participants (Approximately 120) in Training courses on European Citizenship in youth work and in the Advanced Training for Trainers in Europe course;

ii. Representatives of 32 National Agencies and 4 SALTO Resource centres; iii. Other stakeholders.

The evaluators identified a number of measurable parameters (linked to the Covenant’s general and specific aims), with regard to the Training courses. The specific aims can be measured looking into the participant’s appreciation of increase in their competencies, after taking part in activities. Increases in their competencies/knowledge would concern:



− Implementation of European level youth activities; − A sense of participation in a European network of trainers; − A better understanding of European level youth work; − Understanding of European citizenship; − Policy and ideas, on which the Youth programme and Youth directorates activities are based; − The role as multipliers in the efforts of dissemination of results of the covenant and information

on European level youth activities; − The four different training tools made available to them through this Covenant; and − The importance of quality in European level youth work and projects financed by the European

Commission or the Council of Europe. The questions posed to participants in the training courses and to representatives of the National Agencies and SALTO resource centres, on their views and appreciation of the publications and their complementarity to general provision of the same kind, gave us guidance and indications for evaluation. In our questionnaire to representatives of the National Agencies, we explored their counselling role and how they have experienced the co-operation between the management of the Covenant and themselves. Through responses to the questionnaires we were enabled to evaluate: − the general target groups’ (of the Youth programme) awareness of the existence of the

instruments; − the general target groups’ (of the Youth programme) interest in the publications; − participants’ views on the complementarity between training courses and publications; − national agencies’ interest in, usage/willingness to use and willingness to recommend the

publications to the general public. Through participants’ answers regarding activities they have organised or prepared after taking part in the training courses, we gained insight into the efficiency and impact of the Covenant on European youth activities in general. In order to measure the accuracy of the Covenant managers’ analysis of needs (and other elements in the evaluation), we used second level outputs and results, i.e. projects, activities and work performed/created by the participants in the aftermath of the training courses. The quality of future European level projects and activities created by the participants in the Advanced Training for Trainers in Europe course, will serve as a good indicator of the extent to which the course gave real impetus to the training of European level youth workers and trainers. In our analysis we have, as stated in the Terms of Reference, considered the following evaluation criteria; Relevance, Results, Impact, Efficiency and Effectiveness. We have applied different perspectives in relation to each evaluation criterion as presented in our proposal and we have taken into consideration the questions posed in the Terms of Reference. In order for the reader to be aware of the different phases of the evaluation and the process, we explain below our work plan for the evaluation.



3.3. Work plan for the evaluation

• Study of first documents such as evaluation reports and documentation of Training Courses

(TCs), T-kits and Coyote issues 4-7 / Desk research

• Production of a chronological overview of activities and an overview of outputs / Desk research

• Production of questionnaires through identification of questions/parameters for measurement of level of attainment of general and specific objectives of the Partnership /Desk research

• Production of emerging conclusions (based mainly on study of documentation) for interim report. Analysis of preliminary results of the surveys in relation to emerging conclusions on which we based our interim recommendations.

• Production of draft final report. Conclusions and recommendations based on studied documentation and complementary results of the conducted surveys and interviews.

• Finalisation of report after study of Commission’s comments. The main part of the analysis that we base our recommendations on derive from the studying of the provided documentation and our surveys. However, we have also visited the fourth and final seminar within the framework of the ATTE course for 2 days taking the opportunity to make a series of interviews with participants, trainers, course directors and staff of the Partnership. We had foreseen in our proposal to become more involved in the preparation of evaluation seminars together with the staff of the Partnership and the two Institutions. However this activity did not fit into the framework of what was foreseen by the Institutions and our participation for two days in the 4th ATTE evaluation seminar became of a more observational type. In addition to the interviews/informal discussions we held at the 4th ATTE seminar, we interviewed a representative of the European Youth Forum, the former official of the European Commission involved in the set-up of the Third Covenant Partnership as well as staff of the Technical Assistance Office, European trainers and experts involved in the production of T-kits, all of whom are stakeholders in the European Youth training field and have been involved at some stage during the Third Covenant Partnership. The survey that we have conducted had an approximate general reply-rate of 59%. Looking into each group, the reply-rates vary considerably from 15% to 87%. (See annexes 2 and 4). The high reply rates from the participants in the ATTE course and from the participants in the third training course on European Citizenship can be explained by the fact that we were able to attend the last ATTE seminar and were provided with some 20 minutes of the seminar to hand out the form and take it in. We left forms in Budapest and the course-director distributed the questionnaires to the participants during the course and sent them to us by mail. This procedure gained a lot of time for us and it left us with a good reply-rate for those two courses. For the participants in the first and second training course on European Citizenship we had only a 15% reply rate, although we managed to get an updated e-mail list from the expert that conducted the Impact survey some six months prior to our survey. From the National agencies and SALTO resource centres we had a reply-rate of 53%. For the mailing to the



network of National agencies we decided to target this mailing towards the staff, responsible for Action 5, Support measures in the framework of the Youth programme. We considered that the validity of the survey conducted in relation to the National agencies would benefit from being targeted towards the staff that can be considered to potentially have most experience in relation to training in the youth field in general and in relation to training projects within Action 5 in particular. The low reply-rate from the participants in the first and second training course on European Citizenship should be seen in the perspective of the above mentioned Impact survey which was conducted only six months prior to our survey. It is reasonable to believe that participants are less inclined to respond to a second questionnaire in the time-span of six month. With the results of the Impact survey (and internal as well as external evaluations of the two courses), we considered that we are able to produce valid recommendations in relation to the first and second training courses. Before we turn to Chapter 4 – Quantitative description of the training tools, where we give a brief quantitative description of the results and outputs of the Third Partnership Covenant we would like to emphasise that the training tools developed during these three years activity are of high quality and that the ambitions of the staff, trainers, authors, participants and experts involved in the implementation of the activities and publications are also very high. Due to the emphasis in the evaluation on recommendations for the future 4th Covenant as well as the next Youth programme starting 2007, we have focused on how the full potential of the Partnership can be reached rather than limiting the evaluation to statements on achievements as such. For the interested reader of the report we would like to propose a visit to the Training-Youth website (www.training-youth.net), where all main documentation on the Partnership is available on-line. In Chapter 5 we guide the reader through the evaluation criteria and our considerations of the results in relation to those.



4. QUANTITATIVE DESCRIPTION OF THE TRAINING TOOLS

4.1. Training courses on European Citizenship

A total of three training courses on European citizenship have been implemented within the third Partnership Covenant. They involved 87 participants and there were a total of 511 individual applications to follow the three courses. Applications came from nearly all countries of the European Union, the candidate countries and the member countries of the Council of Europe. For details on the first two courses we have received details on the origin of the total number of applications as well as on the origin of the selected participants. Total number of applications Selected participants TC I 137 applicants from 39 countries 30 participants from 22 countries TC II 165 applicants from 39 countries 29 participants from 22 countries TC III 209 applicants from …. countries1 28 participants from 21 countries Table 1: Number of participants per country in TC I, II, and III combined

AL 2 LT 1 ARM 2 ML 1 AZE 1 MOL 2 BE 3 NL 4 Belarus 1 POL 3 BiH 2 PT 4 BG 2 ROM 2 CY 1 RUS 3 CZ 1 SL 1 EE 3 SK 2 FIN 1 ES 6 FR 8 SERB MGO 2 DE 4 SE 2 GEO 1 TUR 1 GR 3 UK 6 HU 2 UKR 3 IRE 1 IT 5 TOTAL participants 87 KOS 1 35 countries represented (12 EU, 23

Non-EU)

1 Number currently unknown to evaluators



Russia, Romania, Italy, Spain and France have by far the greatest number of applications for the two first courses (more than 10). There is no analysis to be found as to why there were so many more applications from these countries. The training courses were nine day courses taking place at either of the two European Youth Centres of the Council of Europe, two at the Budapest centre and one in Strasbourg. The training courses were of a pilot character, mainly with regard to their content and theme, European Citizenship. The format of the training courses followed to a certain extent the traditional format for training courses provided by the Council of Europe. Courses followed a curriculum agreed within one of the two sub-group of the CQDG- Curriculum and Quality development group. The CQDG was an expert group initiated in order to provide the Partnership with a curriculum for the training activities taking place within the framework of the Partnership and ideally, on a more general level, with basic principles for the work on common standards of quality, in European Training in the Youth field. Trainers’ teams for the three courses were relatively free to elaborate the courses within the provided framework and each course incorporated the experiences of the previous course/courses in the sense that evaluations of the courses and recommendations from the evaluators were considered during the production of the next course. 20% of the participants felt they already belonged to a European network of trainers prior to their TC. 40 % felt they belonged to such a network having had the experience of their course. 34 % still felt unfamiliar with European-level Youth work and training and would like to learn more in order to be able to feel that they belong to a network of European-level trainers.

4.2. Advanced training for trainers- ATTE

There were 30 participants, or participants-trainers, in the ATTE course. They came from 16 different countries, 9 EU Member states and 7 Non-EU. We have not received data on how many applications there were for the course or any documentation in relation to the pre-selection of applicants to the ATTE course, i.e. how the selection was organised prior to the Introductory Seminar. However, in the end there were a total of 30 participants and there were no drop-outs during the course. The ATTE course was a part-time training programme, based on open and distance learning punctuated by face to face intensive course modules. During the two-year duration of ATTE there were a total of 4 residential seminars. (January, May 2002, January, October 2003). All 30 participants and an additional 10-15 candidates also participated in the Introductory seminar (November 2001), after which the final selection of participants was made. In between the four seminars, two practice periods were introduced, where participants should practice their newly acquired knowledge and develop projects. Other elements in the ATTE course involved for example mentoring, peer-support groups, personal development plans, an open learning community and an optional Training Quality Product. The ATTE seminars took place in the European Youth Centre in Budapest (2 seminars), the EYC in Strasbourg (2 seminars) and in Radovljica Training Centre in Slovenia. Meetings between Mentors and “Mentees” as well as between the different peer support groups and project groups took place at various locations.



16 of the selected participants came from EU and/or EEA countries and 14 from other European countries. 72% of the ATTE participant-trainers have implemented or plan to implement an Action 5 project within the framework of the Youth programme of the European Commission. 44% of the participant –trainers felt that they already belonged to a European network of trainers prior to the ATTE course. 56% felt that they now belonged to such a network as a result the experience gained within the ATTE framework.

4.3. Training-kits, T-kits

Within the framework of the third Covenant, 5 T-kits have been produced. The last T-kit, on Funding and Financial Management, was produced after the start of our evaluation which is why we have not included that T-kit in our survey. The four T-kits that were included in our survey are:

T-Kit 5 International Voluntary Service (2002) T-Kit 6 Training Essentials (2002) T-Kit 7 European Citizenship (2003) T-Kit 8 Social Inclusion (2003)

An additional T-kit was published in November 2003: T-Kit 9 Funding & Financial Management (2003) The T-kits have been printed in 1000-2000 copies. They are produced in English and French, and will be translated into German within the framework of the Partnership programme. National initiatives have also been taken and a few of the T-kits have been translated into other languages such as Hungarian, Russian and Polish. A total of 25 people have been involved as authors T-kits 5-9, 16 males and 9 females. They came from a wide range of countries (normally one or two from each, except for UK and Germany, which had 4 people from each country.) Almost all 2000 printed copies were mailed out to the two mailing lists below. They may also be downloaded from the Partnership’s website. The follow-up and statistical monitoring on downloads of the website are not done separately for the T-kits. Therefore it is not possible to comment on the volume of dissemination of the T-kits through the website. We have received some statistics of downloads of the T-kits from the SALTO website. The possibility to download the T-kits from the SALTO website exists since 2002, but at that stage no calculation of downloads was introduced. Calculations of total downloads of T-kits on the SALTO website on 13 November 2003 comes to 728 downloads. Training essentials (T-kit 6) is the most popular one, followed by Organisational Management.



Table 2: Distribution of T-kits

European Commission 50 European Youth Forum 30 INGYOs 97 NYCs 44 NAs 340 SALTO 200 TEMPUS 13 CDEJ 50 Info & Documentation centres 38 EYCA 2 CoE offices 16 EYCS 500 EYCB 350 subtotal 1730 Teams 10 ATTE participants 30 TP DYS 80 CQDG 13 TC EC 1 30 TC EC 2 30 TC EC Teams 15 Quality group 12 Advisory council 31 subtotal 251 TOTAL 1981 INGYOs = International Non-Governmental Youth Organisations NYCs = National Youth Councils CDEJ = European Steering Committee for Youth NAs = National Agencies EYCA = European Youth Card Association CoE = Council of Europe EYCS/EYCB = European Youth Centre Strasbourg/Budapest TP DYS = Trainers’ Pool of the Directorate of Youth and Sport CQDG = Curriculum and Quality Development Group



4.4. Coyote Magazine

Within the framework of the Third Covenant, issues no. 4 to 7 of the Coyote magazine have been produced and printed. Another issue of the magazine is foreseen to be produced and distributed within the (extended) timeframe of the Third Covenant. The magazine is published twice a year by the Partnership Programme. Its mission is to provide a forum for exchange, discussion and networking for European youth trainers and youth workers. Coyote is produced by an international editorial board comprising representatives of the Council of Europe, the SALTO – National Agencies network, the Youth Forum and also freelance trainers. The Coyote Magazine focuses on issues in the area of “Youth, Training and Europe”. Coyote is published twice a year (January and July). Each issue includes various sections: training methodologies, interviews with trainers, news, and a Coyote-theme focussing on a topic of relevance and interest for the youth field (White Paper, European Citizenship, social inclusion, etc). The Magazine is also available online on the Partnership website. The distribution of Coyote is increasing steadily. The distribution of the Coyote Magazine is, at the moment of writing, at a level of more than 4,500 for subscribers. An additional 1,500 copies is retained at the European Youth Centres and distributed subsequently. For more detail on the distribution, see below. Table 3: Distribution of the Coyote Magazine

European Commission 100 European Youth Forum 300 INGYO 485 NYC 220 NA 340 SALTO 200 TEMPUS 26 CDEJ 250 Info & Documentations centres 380 EYCA CoE offices 130 EYCS 1000 EYCB 500 subtotal 3931 Website mailing list 749 Commission Mailing list 1200 Coyote Editors 120 CQDG 0 TC EC 1 0 TC EC 2 0 TC EC Teams 0 Quality group 0 subtotal 2069 TOTAL 6000



We understand that “Commission Mailing list” referred to above is a mailing list maintained by the Youth TAO, covering projects funded within the framework of the Youth programme.



4.5. Overview of activities and outputs

In the following table we present a majority of the preparation and evaluation meetings, training activities and publications implemented under the 3rd Partnership Covenant on Training in the European Youth field, in approximate chronological order.

Table 4: Overview of activities and outputs

8-10/02/2000 Evaluation seminar 1+ 2 covenant 22/05/2000 Prep meeting for instalment of Salto RC 23-24/05/2000 TWP 11 12-13/09/ 2000 CQDG 1st meeting Mid Sept 2000 T-kit 1+2 printed 16-17 September 2000

T-kit 5 Meeting number 1

18-19 /09/2000 Steering group for training (1st meeting) 19/09/2000 TWP 12 Early Nov 2000 Coyote 3 6-7/11/2000 Editorial meeting Coyote 27-28 /11/2000 CQDG 5-6/12/2000 TWP 13 31/1 – 3/2 2001 T-kit 6 Prep mtg 2 19-21/03/2001 CQDG April 2001 Coyote 4 2-3/05/2001 TWP 14+15 25-26/06/2001 TWP 16 4-5/07/2001 ATTE Prep mtg 1 8-9/07/2001 Prep mtg 1 TC I 15-19/09/2001 Bridges for training 21-23/9/2001 ATTE Prep mtg 2 23-26/09/2001 Prep mtg 2 TC I 25-26/10/2001 TWP 17 19-21/10/2001 Prep mtg 3 21/10/2001-01/11/2001

TC 1

07-11/11/2001 Intr. Seminar ATTE 11-13/11/2001 Prep mtg ATTE 30/11 – 2/12 2001 T-kit 7 EU CIT 1st Prep Mtg 9-10/12/2001 Prep mtg ATTE Jan 2002 Coyote 5 7-19/01/2002 First seminar ATTE 17-18/01/2002 TWP 18 8/03/2002 TWP 19 17-20 March 2002 T-kit 7 Prep mtg 2



21/03/2002 Prep mtg ATTE 25-27/03/2002 Evaluation mtg TC I 27-28/03/2002 Prep mtg 1TC II 3-4/04/ 2002

T-Kit on Social Inclusion 1st Experts’ Meeting EYC Strasbourg

6-17/05/2002 Second seminar ATTE 11/06/2002 TWP 20 15-16/06/2002 Prep meeting 2 TC II 16-27/06/2002 TC 2 July 2002 T-kit 5 Int. Voluntary Service 8-9/10/2002 TC Eval mtg 10-11/10/2002 Interim Evaluation Future Strategy (IEFS)

mtg TC I and II and ATTE 17/10/2002 TWP 21 Oct 2002 T-kit 6 Training Essentials Nov 2002 Coyote 6 and Coyote Supplement 14-16/11/2002 Trainers forum 2002 31/01/2003 Evaluation & Prep mtg ATTE Jan 2003 Third seminar ATTE 10-12/03/2003 1st mtg Expert group on Quality standards,

Evaluation and Validation 22/05/2003 TWP 22 May 2003 T-kit 7 European Citizenship June 2003 T-kit 8 Social Inclusion 16-18/06/2003 2nd mtg Expert group on Quality standards 11-12 /06/2003 Prep mtg 1 TC III July 2003 Coyote 7 (printed later) 26-28/08/2003 Prep mtg 2 TC III 2-5/09/2003 Prep mtg ATTE 16/09/2003 Kick off evaluation meeting D&T Sept-Oct 2003 T-kit 9 Funding and Financial management 07/10/2003 TWP 23 12-17/10/2003 Final seminar ATTE 19-30/10/2003 TC 3 18/11/2003 Presentation of Draft Recommendations by

D&T 03/12/2003 TWP 23



5. THE EVALUATION CRITERIA - ANALYSIS OF THE TRAINING TOOLS

We present hereunder our analysis and findings in relation to the qualitative evaluation criteria as foreseen in our proposal for this evaluation. In order for the report to be easily used in the preparation of future activities and programmes, we have decided to divide the presentation into different perspectives of the five key evaluation criteria. We could also have presented the four main training tools as individual parts, but since there is a constant overlapping of the four main tools in relation to their more general objectives, we decided to take a more global approach. Another reason for our choice in this respect is that most reports provided by the partnership secretariat follow the other line, favouring individual perspectives of the training tools and the reports of meetings etc, with few current reports and material from the third covenant presenting a global perspective and global analysis. The qualitative evaluation criteria that we have used in our evaluation study of the Third Covenant Partnership on Training in the Youth field are Relevance, Results, Impact, Efficiency and Effectiveness.

5.1. Relevance of the Training tools of the Partnership

5 . 1 . 1 . A C C U R A C Y O F N E E D S A N A L Y S I S

In the development of the Curricula for the TCs on European Citizenship and for the ATTE long term training of trainers, the analysis of needs of the target groups mainly relied on the knowledge of the experts involved in the CQDG, which was extensive and multifaceted. However, there is no direct reference made in CQDG’s report to the needs of the youth workers, youth leaders or for that matter the young people of Europe. It would have been beneficial for this report to have reflected the discussions of the group on the needs of the target groups of the activities within the framework of the Partnership. There was a Needs assessment and Vision building seminar in April 1999 where the Council of Europe, the European Commission and other actors in the field of European-level Training in the Youth field, took the first steps in trying to develop a common vision on how to go about a long-term co-operation between the two Institutions, starting from the respective understandings on Training in the European Youth field. It is clear at this point in time that all stakeholders in the Partnership have individual views on what needs there are amongst the target groups and that all stakeholders agree that there is a lot of work to be done in clarifying concepts and terminology, defining such concepts as “added value”, making use of the synergies between the actors, coming to a common understanding of quality standards for training in the Youth field and so on. The 1999 seminar did not result in common views on the needs of the target groups. The final statement of the seminar focuses on vision building rather than needs assessment. The seminar was later, in February 2000, followed by the evaluation seminar of the previous Partnership agreement, where the issues of responding to needs were again addressed. The issues of responding to needs were divided into participant/user needs and trainer/editor needs. It was concluded here again that responding to needs is a complex issue and would have to involve both transfer of shared values by the institutions and more practically oriented skills development. The



challenge of the trainers is to combine the needs of the participants and the Institutions in training tools that are also easily measurable. Using this same division of the target groups and adding the needs of the stakeholders we will further try to explore to which extent the needs analysis within the third covenant have been accurate. With regard to the training courses on European Citizenship and the ATTE course, 48% of the respondents to our survey agreed to the statement “The mix of methods of my TC/ATTE corresponded well to the needs of the participants in the activity”. We assume here that participants have discussed amongst themselves whether or not they felt that the courses provided what they needed and expected in terms of training. Only 15 % of the participants disagreed with the above statement and 37 % replied that they did not know. We can hereby conclude that while half of the participants agreed to the above statement, there is still significant room for improvement in clarification of needs. Around 80% of the participants of the TCs on European Citizenship agree to the statement that “I would have wanted more focus on how I can share/use my training experience back home”. Between 40 and 50 % “would have wanted more guidance than they got in how to implement their training in a European Youth project”. The corresponding figures for the ATTE course participants are much more favourable. For the first statement only 20% agree and for the second only 12% agree. Presumably we can, in those percentages, read the difference between the courses in including practice or practical (close to real) exercises. We can conclude that the wider target group for the partnership activities, as compared to the target groups of the usual Training courses of the Council of Europe, have more need for practically-oriented training. The analysis of needs in relation to the trainers (including the experts/staff of the partnership) can not be considered to have been optimal, since time pressure and a lack of resources, both human and financial, are repeatedly mentioned in more or less all reports of preparatory meetings and evaluations of Courses. The fact that the key staff position of the partnership was not clearly designed and was sometimes unclear in relation to the other actors/experts of the Partnership adds to our impression of an implementation of the partnership without an accurate analysis of needs of the trainers and staff. For any co-operation or project to be well managed and implemented, making full use of the internal potential, and having a clear understanding and division of roles is a prerequisite. Had there been a more clear design of the key position, the tasks of the experts/trainers would have been less time-consuming and easier to fulfil. If the analysis of needs in relation to trainers/staff/experts had been more fully taken care of, the individual roles of the actors in the partnership would have been clearer and the integration of the tools of the partnership more smooth. With regard to the analysis of needs of the National agencies of the Youth programme, we have found through our survey that 85% of the NAs “consider it helpful in the implementation of the Youth programme and dealing with the target groups at (our) NA/SALTO, to be able to refer to people with the experience/knowledge of the TCs/ATTE or to use them as trainers in their training activities”. Whether or not there was a perceived need for this beforehand is impossible to say, but once there, the partnership activities are considered useful and in the future they might also conclude that there is a need for the kind of competencies the Partnership can provide. Further the NAs generally consider further mostly that the “tools provided by the partnership assist youth workers and trainers to give a European dimension to their projects/training activities”. 15% replied “Very well”, 50% “to a good extent” and 30% “a little bit”. Only 5% of the NAs do not at all consider that the “tools provided by the partnership assist youth workers and trainers to give a European dimension to their projects/training activities”. Given the relatively “low profile” of the NAs in the Partnership, we assume that their needs were not in focus when developing and implementing the tools and activities. However, our survey shows that the Partnership activities are relatively well in line with their needs as



things stand now. The needs of National Agencies and their knowledge of the needs of the target groups of the Youth programme should, however, be more closely integrated in the implementation of the Fourth Covenant.

5 . 1 . 2 . A P P R O P R I A T E N E S S O F I N S T R U M E N T S / M E T H O D S I N R E L A T I O N T O T H E

O B J E C T I V E S

The tools developed in the framework of the Partnership are all designed to complement each other in order to reach the wider target groups of the two Institutions´ policies and programmes. In order to implement the general objective of the Partnership - The promotion of European Citizenship and of international co-operation between young people, youth workers/leaders and youth work structures in Europe by giving impetus to the training and information of youth leaders and youth workers working at European level - the four main training tools seem appropriate, provided that the coherence between tools is maximised. We have found, for example, that the Coyote magazine and the website are underused among the participants in TCs on European Citizenship. 65% of the participants in the Training courses on European citizenship are not subscribers to the Coyote Magazine, 37 % are not frequent visitors/have only visited the website 2-3 times and 6% have not visited the website at all. We can find indications in our survey that participants in the TCs only learned about the existence of all the training tools at the TCs themselves. It would have been advisable to brief participants beforehand on the existence of (for example) T-kit 7 (applicable only to the TC EC III) and even to send it to them as preparatory reading on the topic of European Citizenship. In relation to the ATTE participants, the integration of the tools is very good and the rates of Coyote subscription are high (80% yes, 20% no). Frequency of website visits is 28% 2-3 times a week and 68% once a month. Only 4% have only visited it 2-3 times. The instruments/tools seem less appropriate in relation to the quality aspect than to the content aspect2, of the general objective of “putting major emphasis on the quality and content of projects which are funded by the programmes of the two Institutions, specifically with respect to promoting European Citizenship, the participation of young people in society and politics, intercultural learning, the fight against racism and xenophobia and the involvement of disadvantaged youth groups”. In order to put an emphasis on quality in the training tools, there is a need for more high-level discussion on the setting of quality standards, a point we stress in other parts of this evaluation. Our survey indicates that 32% of all participants in TCs and ATTE agreed to the statement that “there was not enough focus on the development of concepts of quality”. Only 20% of ATTE participants agreed to this same statement, while 40% of the TC participants agreed to it. 43% disagreed with the statement (ATTE 68%, TCs 25%) and 23% replied that they did not know (ATTE 8% TCs 35%).

The above figures can be seen as an indication that the ATTE course is an appropriate instrument to implement the quality aspect of the above general objective. On the other hand the 9-day training course with an extensive and time-compressed curriculum can not yet be considered an accurate instrument in order to implement the quality aspect as above. We will come back to the adequacy of the instruments in relation to the content aspect of the above quoted general objective.

2 content understood in the sense of coverage of a wide range of topics



5 . 1 . 3 . C O M P L E M E N T A R I T Y

The issues of complementarity concern on the one hand the internal complementarity of the training tools of the Partnership and on the other hand the complementarity of the Partnership activities in relation to other existing training tools at European level Training in the Youth field. The tools of the partnership complement each other by addressing different needs such as:

o the need for training of youth workers and youth leaders in the area of European Citizenship – TCs on this subject;

o the need for advanced long-term training for trainers – ATTE; o the need for manuals / publications in relation to the specific issues prioritised in the policies

of the Institutions; o the need for a European forum for discussion of the issues at stake in European Youth training

- the Coyote magazine, the website and to some extent the Trainers Forum (1+2). We can see signs that there has been a certain lack of cross-referential use of the tools within the Partnership. One sign of this is the usage of the T-kits amongst the participants in the three TCs and ATTE (ATTE % in parentheses) For full details see Annex 2. Table 5: Use of the tools by TC and ATTE participants

T-kit 5 International Voluntary Service has been Read in full by Partly read by Not read at all by No reply 7% (12) 25% (40) 67% (48) 2% T-kit 6 Training Essentials has been Read in full by Partly read by Not read at all by No reply 20% (32) 52% (60) 27% (8) 2% T-kit 7 European Citizenship has been Read in full by Partly read by Not read at all by No reply 20% (24) 67% (68) 10% (4) 3%(4) T-kit 8 Social Inclusion has been Read in full by Partly read by Not read at all by No reply 18% (28) 33% (24) 47% (48) 2% Had the Partnership TCs and ATTE promoted and used the T-kits during the courses as the complementary tools they were designed to be, the rates of “read in full” or “read parts of” would presumably have been much higher for all 4 T-kits. We note that the more specific focus of T-kit 5 and T-kit 8 can be a reason for the relatively high rates of “Not read at all” for those publications. More “general training oriented” participants might not have a great interest in T-kits with a more narrow focus. Another interpretation could be that the team of trainers expressed a difficulty in the weighting of International or European Voluntary Service experience as compared to other International/European Youth project/training experience (which emerges from the evaluation meeting report of TC EC I). In addition to that the relevance of an EVS experience was questioned in terms of validity for selection of participants for the second TC (1st preparatory meeting report TC II). A general impression is that the needs for training in relation to Voluntary service projects were not significantly considered in the contexts of the TCs/ATTE.



Despite the fact that the participants did not read all the T-kits from the 3d Covenant, 93% of the respondents consider the T-kits as a good complementary tool to the participation in the training activity and 87% have recommended them to others. Moving to the complementarity of the Training tools of the Partnership in relation to other provision of training tools for European-level Training in the Youth field, we can report that all National Agencies/SALTOs responding to our survey consider the T-kits to be a good complementary tool to the information and training materials/activities that they provide, in relation to the development of Youth projects or training activities within the Youth programme. 30 % of them recommend the T-kits “all the time” to their target groups and 70 % recommend the T-kits “often” to the target groups. The rates of reading of T-kits amongst the network of NAs and SALTOs are more evenly distributed between the T-kits than is the case with the participants in TCs/ATTE. Between 10 and 25% have read the T-kits in full, 50-60% have read parts of them and between 15 and 25 % have not read them at all. For full details see Annex 4. Looking at the report of the Partnership Trainers Forum 2002 (the second Trainers’ Forum) the issues of complementarity are dealt with in Annex 2. It is clear here that the 19 participants at this meeting agreed that the training tools provided by the SALTO resource centres were complementary to the tools provided by the Partnership. What is unclear is why this paper with the important sub-heading “Who does what?” is only put as an annex/appendix to the report, rather than being a central element in the main text. This kind of open and public discussion, presentation and/or debates should merit a more forefront presentation in the materials of the Partnership. A “Who does what?” section on the website would be a great leap forward for the promotion of the importance of the co-operation established between the two Institutions. In the draft final report of the Partnership (p. 7) it says on the internal evaluation of the Training courses that: “There seems to be a generally shared impression following the three courses that the curriculum framework is too heavy for the nine-day training course format used. Two suggestions emerge from the evaluation reports and the feedback by the trainers and participants: a) Certain elements in the curriculum need to be given more attention while other elements deserve less attention (training modules); b) A long-term format may ensure that all important content elements can be addressed in detail (long-term training course). Reading the above, we assume that a general tendency of the Partnership is to try to fulfil the objectives of the Partnership within its existing human and financial resources, (a rather static approach) rather than to use the complementarity of the general training provision in the field and try to find solutions and ways of implementing the objectives by pooling resources from various stakeholders and making use of the synergy effects. It is important to look at, for example, the training courses within the Partnership as a part of the provision of training in the European field, ensuring that the objective of the courses is not to be an “all inclusive” remedy for securing the efficient implementation of all aims and objectives of the work of the different actors in the field. It is, as stated over and over again in reports and documents, important to make sure that the complexity of the issues in relation to European citizenship is included in the curricula of the courses. However, it should not be impossible to think of a framework for the training courses where all the different elements identified by the CQDG are included, but not necessarily provided by and for the Partnership exclusively.



An example of this could be to leave some of the parts relating to active participation of young people in society to courses provided by the European Youth Forum and the parts on youth work practice for the courses provided directly by the National Agencies/SALTOs. The perspective proposed above would give time to the participants to make use of their new knowledge in between the different parts and the different modules. They could receive “credits” which could be collected over a longer period and in the end lead to a “Certificate”, provided by the Partnership and recognised by both Institutions. Cutting out some parts of the Curricula of the TCs on European Citizenship (while ensuring participants are able to pick it up elsewhere on the European training arena), would enable courses of a shorter format to take place within the Partnership. It would also help to find the participants that are most motivated to become real actors on the European training arena, since getting “all parts of the whole” and the Certificate of the European Citizenship Curriculum, would need patience, stamina and decisiveness. A concluding remark on the relevance of the objectives and instruments of the Third Covenant on training is that the objectives in relation to:

promotion of European Citizenship; giving real impetus to training of youth workers/youth leaders at a European level; and putting major emphasis on the quality and content of youth projects

are relevant to continue the co-operation, as there is still a great need for initiatives in this direction. The work implemented during the Third Covenant has been a step in the right direction, but objectives are still not fully attained. Further progress can be made in terms of:

securing greater integration of the tools and actors/stakeholders’ needs; getting agreement on the definitions of quality standards among all actors in the field; addressing the challenge of getting the non-formal education, as practised in the Youth field,

recognised outside the Youth field. If these are achieved there will be raised awareness of and ultimately recognition from the outside of the importance of Youth work, Youth policy and the participation of young people in society and politics.

5.2. Results

In order to evaluate the results of the third Covenant we first consider the specific aims, agreed on by the two Institutions, with regard firstly to the training courses and then with regard to the publications. Specific aims - Training courses

a) To facilitate the training of youth workers and the development of youth work methods b) To extend the youth workers/leaders knowledge on Europe, citizenship and young people in

Europe c) To improve the quality of European level youth worker training activities and to establish

standards for them d) To develop a network of trainers on a European level in the youth field e) To enable the trainers to design and implement the methods necessary for European level

training activities in the youth non-formal education field f) To disseminate the outcomes of the training courses funded by the Covenant to a larger

public of trainers and youth workers in Europe g) To develop a multiplier effect with regard to the National agencies and the resource centres

(SALTO) within the Youth programme of the European Commission



To assess the results produced in relation to the training courses, (to measure if youth workers/leaders have extended their knowledge, skills and competencies as the specific aims envisage), we posed questions to the participants in the TC/ATTE on the degree of such progress. Our survey shows that 92% consider that they have increased their knowledge on European Citizenship, 57% to the extent that they “know more about it” and 35% to the extent that they can now train others on these issues. On European youth policy and European funding programmes, 75% consider that they have increased their knowledge, 47% know more about it and 28% feel they can now train others on these topics. For Intercultural learning, 80% have increased their knowledge (skills) and an encouraging 45% consider they know enough to train others. Another aspect on European Citizenship that we asked the participants about was if they considered that their participation in the Partnership activities had supported their professional development and increased their competence to integrate European Citizenship as a theme into their planned or implemented projects and/or daily work with young people. 95% gave a positive signal in that direction. 42 % answered “Yes definitely” and 53% answered “To some extent” and we consider therefore that the Partnership training activities have been very successful in that respect. In our survey we posed a question on whether the participants in the courses had used their newly acquired knowledge/professional development/information in their local/regional/national setting. Since the questionnaire was unfortunately for practical reasons distributed at the third training course, rather than afterwards, due to the timing of the course and the evaluation, we exclude responses of TC 3 participants from our analysis of this question. The survey shows that 44% have used their new knowledge in bringing the issues of European Citizenship into discussion. 76 % of the participants in ATTE / TC 1+2 have disseminated information on the existence of other training tools such as T-kits and Coyote. 53% have “implemented a project within the Youth programme” and 59% have “implemented a project within Action 5 of the Youth programme”. In our questionnaire we decided to include questions related to Action 5 specifically and to the Programme as a whole, in order to assess the interest in and openness of respondents to the broader work of the Programme. Of the ATTE participants, 72% filled in they had implemented an Action 5 project, but for the question on implementation of a YOUTH project as such, the percentage was only 56 %. This may indicate a low level of concentration or little emphasis in the framework of the ATTE on the whole of the YOUTH programme as compared to the attention paid to the Action 5 “Support measures” of the Youth programme. In the impact survey conducted between April and June 2003, followed up in June by an impact evaluation seminar, we have found that 64% of the participants in the first two training courses developed and implemented a project in the aftermath of their participation in the training course. This 64% includes 5 projects directly developed within the first and second training course on European Citizenship, and in addition to those five projects, the participants that responded to the survey were engaged in some 25 other projects. We consider that at least 30 projects developed by the total of 59 participants in the two Training courses is another very positive signal in terms of results of the courses. In the Interim Evaluation and Future Strategy meeting of October 2002, the participants in this meeting concluded that “It was felt that the definition of the course (Training course on European Citizenship) developed by the CQDG was an incomplete one, in the sense that no systematic follow-up to the courses was foreseen. This would especially have been necessary to follow the development of projects after the course. This aspect can be looked at to some extent at the Impact Evaluation seminar, planned for October 2003.” (Meeting report IEFS meeting p.8) This is an indication that actors, teams and staff were already aware of the fact that there was a certain lack of general follow-up mechanisms of participants and their projects/activities in October 2002. However, we have not



found evidence that this issue has been on the agenda since, except for the Impact Survey that only had a very narrow focus on the follow-up of the “course-projects”. It would be very interesting to have as much information on the 25 other projects as on the 5 implemented “course projects”. Due to this specific circumstance, we do not know to what extent the projects have been financed within the framework of the Youth programme of the European Commission, or financed by the programmes of the Council of Europe or by another funding mechanism. The impact survey had, as referred to above, a relatively narrow focus and was more concerned with the outcomes of the projects developed as part of the course, than with the more general development of projects in the European Youth field or European Youth training field. Had it had a wider focus, the survey might have proved more useful for the measurement of general outputs and multiplication of the training courses. Projects are only one way of making practical use of the newly acquired knowledge, skills and competencies. We included in our survey, for example, a question in relation to publishing/writing of articles for magazines, newsletters and/or newspapers. Our hypothesis was that amongst the participants in the courses, we would find people that would be interested in writing about their experiences and that quite a few of the participants would have published an article in one of the newsletters or magazines in existence in the European youth field or at national or even local level. However, 7 % of the participants in the Training courses have published an article on their experience and we consider that there are reasons for encouraging participants, both past and future, to share their experiences, thoughts and ideas on the topics covered in the training courses with a wider public. Articles in their mother tongue on their experiences of European level training would be extremely valuable. One of the key issues for the future is in fact the dissemination of results and publications to national/regional and local levels. We will come back to this, especially in the context of the evaluation criterion “Impact”. The participants of the TCs/ATTE replied to the following question in our survey. “Do you consider the training activities you have participated in as equally promoting and informing about different types of training activities and publications available in the European youth training field –such as publication and training possibilities within the Partnership, the European Commission, the National agencies/SALTO resource centres of the Youth programme, the Council of Europe or any other?” 75% of the respondents reply “yes” to this question. This can be interpreted as a good sign in terms of equal promotion of the possibilities for training activities and existence of publications, and should also lead to a greater awareness amongst the target groups of the different possibilities in the European Youth training field. Hopefully the trainers and youth workers/leaders will explore these possibilities further and contribute in the future to the diversification of training activities in the youth sector at European level. Such a diversification has been largely supported by the ATTE curriculum in general and by the ATTE participants in particular. This was made possible by the innovative aspects of the course, including peer-support groups, a system of mentoring, practice projects and the Training Quality product. There are some discrepancies in the documents provided to us as to whether some of the elements in the course had a compulsory or optional character. The issues around elements being compulsory or optional and to what extent demands could be made by the team of trainers on the participants in relation to the implementation of, for example, a TQP are connected to the fact that non-formal



education, as practised in the youth field usually does not include compulsory elements. The fact that the ATTE course is supposed to provide a certificate to the participants in the course enhances the importance of reaching a common position amongst the actors in the field on issues connected to the extent to which an element in a training course can be compulsory. The Open Learning Community, which we unfortunately have been unable to visit3 and therefore can not comment on, is in principle a good place to find evidence that will support the role of the course as a great contributor to the diversification of and raised level of innovation in European Training activities in the Youth field. We have understood that the site for the Open Learning Community on the website of the Partnership proved not to be good enough for the purposes of the ATTE course and that most exchanges and virtual reality discussions took place at another site provided by CEDEFOP, called the “Platform” within the ATTE framework. In the report of the preparation meeting for the 4th ATTE seminar in September 2003, the team discussed the OLC/Platform developments and concluded that: “Most of the active people of the Platform are the ones who have been most active on the OLC too. For the Platform to remain operational after the October seminar, a moderator would be necessary. However, the real purpose of it should be clarified / defined. Currently there seems to be limited interest among participants to use the Platform. Monthly team input on the OLC – this was planned and begun, but then later it stopped.” It is quite clear that the potential of new technologies (in this case an Open and distance learning tool), has not easily been fitted into the ATTE framework. However, not having been able to study the use of or exchanges within the OLC/Platform we can not draw further conclusion on this. However, the external evaluator of the ATTE long-term training course has been able to access the OLC/Platform and therefore the conclusions will be able to be drawn from that study. The ATTE participants have implemented a total of 30 training projects involving approximately 1000 participants, potential multipliers of these new training activities taking place in the European youth training field. As a thorough evaluation of the ATTE course is underway we have not gone into more detail of results produced in relation to the ATTE course. Before we move on to the results in relation to the development of a network of trainers at European level, the concepts of quality and quality awareness as well as setting of standards for quality in training activities at European level, we will look at the results produced in relation to the specific aims with regards to the publications produced within the third Covenant: Specific aims – Publications

A. To inform trainers and youth workers on training concepts, methodologies, practices and current developments in training at European level

B. To provide a forum for the exchange of information and best practices in training and for discussion amongst trainers and youth workers on training related issues

C. To assist trainers and youth workers/leaders to give a European dimension to their training activities

D. To disseminate new methodologies of non-formal education and youth work and good practices to a larger public of trainers and youth workers in Europe

E. To develop a multiplier effect with regard to the National Agencies and the resource Centres (SALTO) within the Youth programme

In relation to the informing of trainers and youth workers on training concepts, methodologies, practices and current developments in training at European level, the five T-kits ranging from

3 The participants were unwilling to allow access to the evaluators



International Voluntary service to Funding and Financial management complement well the four first T-kits in terms of themes. The publications are produced in order to be usable as reference material rather than as a book that needs reading from start to end. The level of “Read parts of it” replies in our survey for both National Agencies/SALTOs and for participants in ATTE/TCs confirms that between 1/3 and 2/3 of the respondents have used the T-kits as reference material rather than as a “book”: Partly read Fully read + Partly read T-kit 5 32% 43% T-kit 6 51% 72% T-kit 7 65% 88% T-kit 8 40% 56% The total reading rate of the T-kits range between 43% for the International Voluntary Service T-kit to 88% for the T-kit on European Citizenship. These figures can be interpreted as a good sign in terms of the results in relation to the specific aim “A”. The fact that 95% of NAs/SALTOs read the Coyote magazine and 80% of ATTE participants subscribe to the magazine, are very positive signals. For participants in Training courses on European Citizenship, between 56 % and 69 % do not subscribe to the Coyote magazine. These figures suggest that the reading rate is comparably lower than the other groups (NA/SALTOs + ATTE). As the question concerned subscription, but not reading, we can only interpret this as a sign of low rate of subscription, but we find it reasonable to believe that the reading rate is also comparably lower in this group as at least some participants add in their comments that :

they did not know that the Coyote magazine existed, and; some would like to become subscribers but did not know if they could.

It would be advisable to add the participants of all training courses to the mailing lists of the Coyote and the T-kits, making sure they have all the complementary training tools and encouraging them to use them and to feel more part of the European Youth training field. By making sure all former participants in the partnership activities get the magazine, the Partnership secretariat can secure that the specific aim B is implemented to its full potential. 75% of NAs/SALTOs consider the Coyote Magazine to provide a forum for discussion of training related issues amongst Youth workers/trainers. Several of the respondents comment that the magazine is only in English and that their target group does not have enough high level command of the English language to be able to use the Coyote as a forum. Another respondent considers that the potential of the magazine is not fully explored yet. Of the respondents to the survey amongst NAs/SALTOS, only 60% of them provide a link from their website to the website of the Partnership. This is a relatively low percentage and should not be too difficult to arise. Some of the respondents replied in the questionnaire that they would arrange for a link to the Partnership website at once. They had not thought about it before. Of the NA/SALTOs that provide a link, for 25% it takes 1-2 “clicks” from their start page and for 42 % it takes 3-4 “clicks”. If the website would be more used as a forum, it would need very frequent updating, but would provide an excellent means for the curious visitor to already take part in the “forum for exchange of information and best practices in training and for discussion amongst trainers and youth workers on training related issues”. As it looks at the moment, or at least on the occasions we visited the page (around 10 times from September to November 2003) the exchanges and comments in the discussion forum concern more



the search for partners or traineeships than the exchange of best practices and information on training related issues. On the password protected area for ATTE there have definitely been many more of the kind of exchanges that were foreseen, and it would be very interesting for the rest of the European Youth training field to be able to take part in these exchanges, or at least follow them. The reasoning behind the protection of the group processes could have been secured if certain parts of the exchanges had been internal and password-protected and others made public on the open part of the website. This could have been implemented by giving the responsibility to the participants to make sure that some of the exchanges were “published” on the open part of the website. The exchanges amongst ATTE participants would stimulate discussion among the rest of the actors in the European Youth training field. The Coyote magazines from the previous Covenant (up to number 3) contained a “Flip-chart” designed to provide a forum for exchange. The reasoning behind the abandoning of this section is mentioned in issue number 4. It is considered difficult to maintain a Flipchart section as there are long intervals between the issues and the exact publication date is difficult to foresee. It is also mentioned that the section should be maintained on the Internet version of the Magazine. However, the flipchart section on the website is not regularly updated and therefore not a good replacement of the section in the Coyote magazine. The training calendar of the SALTO website seems to have taken over the implementation of the objective of keeping the field updated on training possibilities. It is essential to keep readers updated on what is coming up in the field. It would be a good way of securing the image of the Magazine as the forum for keeping the field informed. The Coyote magazine has covered most of the themes relevant for the European Youth training field and since they are all posted on the website, the articles will be able to be read by many more people in the coming years. The articles range from in-depth reports of activities to articles of a more academic character and there is almost always also a more easily digested treatment of each issue. It should be noted however, that the publication, looking through them from issue 4-7, has a slight tendency to become more “academic”, with longer articles on complex issues targeting internal issues rather than going in the opposite direction and trying to “introduce” the European Youth training field to “newcomers”. It is important to remember that these things do not just “happen”, but derive from conscious decisions taken to promote the magazine. It is important to clarify the objectives of and main target groups for the Coyote magazine as well as coming to a decision on what should be the focus of the magazine, and possibly to clarify the individual responsibility of the members of the editorial team, the editor and the Partnership secretariat. We have not been able to study any reports of the editorial meetings and would encourage that these meetings are documented, in order for other actors/stakeholders to be able to follow the developments and discussions. Our recommendations for the improvement of the Coyote magazine concern mainly the development of the web-based part of the publication. The interaction and possibilities for exchanges of information and best practices as well as of general exchange of opinions in the field or in relation to certain aspects of the Partnership could benefit from being held in an electronic space. A development of the web-based part of the magazine would need more financial and human resources to be allocated to the Coyote publication. If the “Virtual reality” aspect of the Partnership is further explored, the cost efficiency for exchanges of information and best practice can be improved, compared to the exchanges of information and best practice that have taken place within the framework of different meetings such as;

Trainers´ Forum 2001, The bridges for Training seminar, September 2001 (150 participants of which 50 constituted “Partnership delegation” in terms of costs)



The Interim Evaluation and Future Strategy meeting, October 2002 (16 participants) Trainers´ Forum 2002, November 2002 (19 participants) Impact Evaluation meeting, June 2003 (14 former participants and 7 people of the trainers´

teams) Concerning the Interim Evaluation and Future Strategy meeting and the Impact Evaluation Seminar, there were high expectations as to what the outcomes in terms of strategies and evaluations of the meetings would be. In reality, those two meetings did not push the work within the Partnership forward to any great extent. We are unaware of the reasons for the much smaller format and limited conclusions of the meetings, compared to what was expected by the members of the TWP for example. It is reasonable to believe that more could have been produced if the meetings had been prepared with discussions and exchanges of information and opinions in an online forum prior to the meetings. In case of late cancellations of participants for meetings, their “input” could be secured and disseminated amongst the partners and eventual cancellations would be less harmful for the whole of the meeting. Many more people can be involved in the preparatory discussions before a meeting and the interest in the conclusions by the actors within the European Training field would be more significant. Real life personal meetings should of course not at all be abolished, but if basic exchanges of information and preparatory discussions would be held in an electronic forum, real life meetings could produce more results and be implemented more efficiently in terms of reaching conclusions and moving common positions forward. The aims of the Third Partnership Covenant in relation to the specific objective D “To disseminate new methodologies of non-formal education and youth work and good practices to a larger public of trainers and youth workers in Europe” must be considered to have been successfully implemented within the Coyote magazine. The above comments are rather to be seen as recommendations for how the work in this context can be improved. Coming back to the specific aim in relation to training courses on “Development of a network of trainers on a European level in the youth field”, we have asked the participants in the training activities in our survey, whether they felt they belonged to a European network of trainers/youth workers. They had four optional replies and the total percentages are indicated after each reply; Yes, but I already belonged to it before the TC/ATTE 30% Yes, with this experience I feel that I belong to a European network of trainers/ youth workers 47% No, I still feel unfamiliar with European level Youth work and training and I would like to learn more in order to be able to feel that I belong to a European network of trainers and youth workers 20% No, I am not interested in belonging to such a network 1.5% 1.5% chose not to reply These figures can be interpreted as a positive increase of perceived sense of belonging to a European network of trainers. We conclude that the selection of participants in training activities has managed to include a good mix of more experienced trainers (30%) and trainers that are about to take the step into a European network of trainers (47%). Given that there is no strict definition of a “European network of trainers”, this gives some freedom to the respondent in interpreting the question. Some of the respondents have used the possibility to comment on their reply and amongst the comments we find such as “I have extended my network”, “Before I only knew “EC trainers”, now I know also CoE trainers”, “I feel that somehow here it can



begin a network. I don’t feel like belonging to the network. I feel like we are just activating possible contacts” and “Yes, definitely, but networks still can have “clubs””. It is important to note that the definitions of a European network of trainers vary, reflecting varying needs and focus areas. The comments imply that there are different “networks” and “clubs” and it is important for the Partnership activities, publications and products not to be “branded” by one or a few of these formal or informal networks or “clubs”, which could reflect a division of EC versus CoE, NA or SALTO trainers’ pool versus other trainers’ pools, or for that matter Commission Youth programme versus other funding possibilities. It is extremely important to emphasise co-operation rather than competition amongst the stakeholders and actors in the Partnership activities, and to ensure that the roles are clarified and stakeholders play equal roles in terms of contributions, be it financial or human resources. The Partnership activities and publications need to be placed above the different divisions of groups for identification and serve a larger public of trainers, youth workers and youth leaders across Europe in the largest possible sense. These issues have been properly taken care of in the publications, especially in the T-kits, whereas the Training courses have had a more “traditional” Council of Europe training course orientation. Although the largest part of the expertise and experience that is available in the field derives from the Youth directorate at the Council of Europe, it is important that throughout the larger audience that is targeted by the Partnership activities, all feel welcome and that the publications as well as open calls for participants for activities are largely disseminated and reach also the “new” target groups (be it, for example youth workers in a local community wanting to make use of the European Commission Youth programme or a young person involved in setting up a new action group outside the traditional structures of youth organisations). The development of a multiplier effect with regard to the National Agencies and the resource Centres (SALTO) within the Youth programme, which is another specific aim stated in the Covenant, would benefit from a more inclusive dissemination of training activities, publications and results. The positive discrimination in favour of younger youth workers/leaders in the courses, both in terms of dissemination of the calls and in the selection of participants, leads to a limited presence of local/regional administrators of youth issues, due to the fact that such positions are in many cases held by people above 30 years. This group can thereby be somewhat forgotten, whereas in several cases there is a possibility that these people, targeted also by the Youth programme, could be key multipliers for the Partnership activities and publications towards youth workers and youth leaders active in a local community, with an interest in but little resources for implementation of a European dimension in their work. We will look more into the details on multiplication in the chapter on “Impact”. Amongst the network of National Agencies and SALTOs, 15% of the respondents consider that the tools provided by the Partnership assist Youth workers and trainers “very well” to give a European dimension to their projects/training activities while 50 % believe this “to a good extent”. These figures are very positive signals of appraisal of the tools of the Partnership amongst the network of NAs/SALTOs. We will now turn to the results of the Partnership activities and publications in relation to Curriculum development, the setting of quality standards, increased quality awareness and an increase in the quality of the activities and products of the Youth programme. We have researched the documentation provided in order to assess the foundation, developments and achievements in the area of Curriculum and Quality Development. The Curriculum and Quality Development Group - CQDG, which in several ways has contributed to the development of the Training tools of this third Covenant on training, can be seen as having been set up as a result of the evaluation and the evaluation seminar (February 2000) of the preceding Covenant on Training. The task of the working group was threefold:



creation of Curricula for Training courses on European Citizenship; Curricula for Long-term training of trainers; creation of quality standards for training in the youth field.

The result of the expert working group is compiled in a 7 page Final report in English and French. The report is very concise and sets a clear framework with basic principles for how the training courses on European Citizenship (TC EC) and the long term training of trainers (which became the Advanced training for trainers in Europe-ATTE) should be implemented. There are also some indications with regard to quality and recognition and how TCs should be implemented in order to ensure good quality. Basic general criteria for quality in training tools are divided into quality criteria for participants, trainers and organisers. There is little information in the material provided to us on how the work of the group was structured. Our main findings in this respect derive from reports of the Technical Working Party, TWP. In the material provided to the evaluators, the concept of this working group appears for the first time in the 10th TWP (April 2000). It is referred to here as the Curriculum Development Group, CDG, and the TWP decided to divide the group of 12 experts into two subgroups. Representatives of the Institutions should be present at (parts of) the meetings of the experts, being seen as contributors, but they should not be regarded as members of the group. In December 2000, TWP 13, the work of the CQDG was presented. The TC EC was underway, and the precise curricula for the course was due to be finalised for the third meeting of the group. There will be three TC in total. The LTTC for trainers is also underway. Some elements of the course are still to be finalised. Recommendations on quality standards in non-formal education/training are also subject to the work of the group and were due to be presented separately. In May 2001, at the TWP 14/15, a report from CQDG was presented. The report stated that “Curricula have been drawn up for the long-term “training the trainers” course and the one on European Citizenship” and it was also stated that “Criteria and quality standards had also been established for European Youth leader training.” In the final report of the “Bridges for Training” event in September 2001, several references are made to the Quality and Curriculum Development work of the Partnership. Except the above “Bridges for Training” Event, it however, was not until the report of the TWP 20 in June 2002, that any further reference was made to the Quality work of the Partnership. Key comments from a brainstorming and planning discussion for the year 2003 include a comment that “Instead of installing a Quality Development Group it was seen as a better solution to create a T-kit on Quality Standards, Validation-Recognition and Evaluation. This T-kit may require more meetings than the usual two. In addition the various meetings (Joint evaluation meeting for ATTE and the TCs on European Citizenship in October 2002 as well as the Bridges for Training II Event) could provide opportunities to take stock of the achievements and to bring the relevant experts together”(20th TWP, p. 4). We have learned that the Joint evaluation meeting took place on 10-11 October 2002 (scaled down from 5 days to two). Outcomes of the meeting in relation to issues of Quality, Evaluation, Recognition and Validation include a number of recommendations in relation to the training tools. In the report of TWP 21, October 2002, there is a reference made to quality issues in relation to the ATTE course. It says that those aspects are to be worked on “in the next phase”. Presumably the next phase implies the third year of the partnership. It says further that “Recognition is a question of major importance. Participants expect to be recognised and to receive a certificate at the end of the course.” (p 5, TWP 21) It is further noted that “The question of recognition is broader than a certificate.” (p 5,



TWP 21) This could indicate that the issues around quality, recognition and validation had slightly moved from being a general issue of the partnership towards being more of an issue for the ATTE course. There is also (in this 21st TWP report) a reference to one T-kit on Evaluation (a traditional T-kit) and one T-kit on Quality and Validation (a resource book), the T-kit on Evaluation to be produced during 2003 and production of the T-kit on Quality and validation to be started also. Finally, in the 22nd TWP, May 2003, one of the conclusions of the discussions is “The issue of validation and accreditation of non-formal education youth programmes remains a key issue for the Commission and the Partnership programme for the next years. The expert group on Quality Standards, Evaluation and Validation of the current Partnership programme as well as the external evaluation of the ATTE course contribute to the ongoing discussions of the topic. A document on “Quality” is expected towards the end of 2003”. ( p.5 TWP 22, conclusion 10). Somewhere in between October 2002 and May 2003, a decision was taken to install (or reinstall) the expert group on Quality Standards, Evaluation and Validation. It seems to be difficult to find a clearly coherent way of approaching the work of setting quality standards as well as the work on validation and recognition in the Youth field. The discussions that must have taken place between the TWP meetings among the members of the TWP, resulting for example in a proposal for instalment of (or reinstalment of) a Quality development group in the 20th TWP, later on changed to favour the production of T-kits, can be seen as an indication of a general consensus that the CQDG did not entirely fulfil its task of proposing quality standards for training in the youth field. It is very difficult to follow the discussions on the issues of Quality, Evaluation, Recognition and Validation and to identify results. There is no clear documentation on the overall strategy and results in relation to the setting of quality standards, recognition or validation to be found in the material provided to us. The absence of documentation on how these relatively important objectives of the partnership are handled can be seen as indicating that no strategic approach to them was followed in the management and follow-up of the implementation of the Partnership. Having said that, it must also be underlined that all objectives relating to setting of quality standards, validation and recognition of training in the youth field, both by the actors themselves, by related fields and by society as a whole are extremely difficult to fulfil and demand a huge amount of work by all actors. The work that remains to be done in this area concerns inter alia promotion of the work implemented in the field, research on and evaluation of measures taken in the wider youth field, and enhanced work in bringing the European policy down to national/regional and local level, just to mention a few challenges. We will come back to the issues of setting of quality standards, increase of awareness of quality, and knowledge of concepts of quality and recognition in our chapter in relation to the Evaluation criteria “Impact”.

5.3. Impact - What indications have we detected in our evaluation?

We are dealing here with identifiable influences or effects that have been created or supported by the Partnership activities. We have in the desk research and in our survey tried to pin down indications

o of support created for international co-operation between young people, youth workers and youth work structures



o of promotion of the themes of European Citizenship, participation of young people in society and politics, intercultural learning, fight against racism and xenophobia as content in the training tools provided by the partnership

o dissemination of results, knowledge, skills and experiences o of improved quality in European level youth worker training o of visibility of the role of the European Commission and the Youth programme in the training

tools of the Covenant. Most of the above points are also treated elsewhere in the report, but we will in this chapter summarise and put focus on the indications of created impact and also look at different ways that the impact of the results produced in this Partnership Covenant as well as the future results of the Fourth Covenant can be maximised and possibly also be more easily monitored. In the training courses on European citizenship approximately 90 young people, youth workers/leaders and trainers have made new acquaintances, acquired new knowledge and skills. They all consider that their participation in the ATTE/TC have made them feel more confident in participating in/working with Youth training at European level. The same is true for the participants in ATTE and looking at the figures for the total of the participants in TCs/ATTE, 67% consider themselves to have definitely gained confidence in those activities and 33% consider themselves to have gained confidence to some extent. A gained confidence amongst the participants will improve their motivation and capacity to engage in international co-operation. Thereby we can conclude that there are indications that the Partnership activities will, on a long-term basis, have created support for international co-operation between young people, youth workers and youth work structures. As regards the impact of the Partnership activities and publications in terms of promotion of themes as above, we have already looked at some of the indications detected in our survey on the level of gained knowledge in relation to those themes (see above section on Results) Our survey shows that 92% consider that they have increased their knowledge on European Citizenship, 57% to the extent that they “know more about it” and 35% consider to such an extent that they now can train others on these issues. On participation and inclusion of young people in society and politics, 48 % consider that they have increased their knowledge to a level where they “know more about it” and 38% feel they can now train others on these topics. For Intercultural learning, 35% have increased their knowledge to a level where they “know more about it” and 45% feel they can now train others. As regards the promotion of the fight against racism and xenophobia as a theme, 37% consider that they have gained an increased knowledge to the extent where they “know more about it” and 20% consider they have gained enough knowledge to train others on this thematic issue. The highest percentage of replies on not having gained any new knowledge, 38%, is found on the thematic of the “Fight against racism and xenophobia”, which confirms our impression from the study of the documentation in relation to the courses that the “Fight against racism and xenophobia” was not handled as a specific theme in the courses. It might be advisable for future courses to give this topic more emphasis in the courses within the different modules/blocs. As a relatively high percentage of the participants do not consider that they have gained knowledge/skills in the Fight against racism and xenophobia, it is unclear whether they have been able to fully understand and incorporate the knowledge/information on this topic that were to be found in several of the modules of the Training courses on European Citizenship (for example European Citizenship, intercultural learning and Multicultural society). We consider that the above figures give reason to believe that the Training activities provided by the partnership have promoted the above themes to a greater or lesser extent. Concerning the theme of



European Citizenship, the training courses, the T-kit and the issue number 7 of the Coyote Magazine have provided a real impetus to the training and information of youth workers/leaders and trainers on the issues of European Citizenship. According to our survey, 45% of the respondents consider themselves to be able, with the increased knowledge/skills gained in the TC/ATTE, to train others in Intercultural learning. This indicates to us that the training courses have both promoted Intercultural learning as a theme and also raised the ability amongst the participants to a level where they feel confident and able to train others. This fact indicates that the process of promotion of the theme will continue, provided the participants stay in the field and continue to be active as trainers in the European Youth training field. For those that feel they need more training themselves in order to train others, the T-kit on Intercultural learning provides a possibility to deepen their knowledge and with further training activities they will also in the long run become full-fledged multipliers. In our search for indications of impact created we have chosen to pay attention to the dissemination of results, knowledge, skills and experiences. As treated previously under results we sought evidence of what means of dissemination the participants in the TC/ATTE had chosen. In our survey we posed a question on whether the participants in the courses had used their newly acquired knowledge/professional development/ information in their local/regional/national setting. Since the questionnaire was unfortunately for practical reasons distributed at the third training course, rather than afterwards, due to the timing of the course and the evaluation, we exclude the responses of TC3 participants in our analysis of this question. The survey shows that 44% have used their new knowledge in bringing the issues of European Citizenship to discussion. 76 % of the participants in ATTE /TC 1+2 have disseminated information on the existence of other training tools such as T-kits and Coyote. 36% of the ATTE participants indicate that they have shared their experiences at ATTE with most of their colleagues and/or young people that they work with. 44% indicate that they have shared their experience with a few of their colleagues and/or young people that they work with. The replies from the groups of participants in TC 1+2 are not entirely reliable since there was only a 15 % reply rate for the TC 1+2. However, 44% of the TC 1+2 respondents report having shared their experiences with most of their colleagues and/or young people that they work with and 55 % report having done so with a few of their colleagues and/or young people that they work. This indicates that a major part of the participants are likely to have shared/ disseminated their experiences. The aspects (and importance) of dissemination of experiences and knowledge and tools for multiplication deserve more attention in the future. This was underlined by the section in our survey where participants could agree/disagree or reply “don’t know” to a number of statements. For the statement “I would have wanted more focus on how I can share/use my training experience back home”, 20% of ATTE participants agreed, 89% of TC 1+2 agreed and 81 % of TC 3 participants agreed. The ATTE participants had practice periods within their training, which explains the huge difference between the TCs and ATTE in this respect. Another statement can be used as another reference on the existence of practically oriented methods/tools in the training activities. For the statement “I would have wanted more guidance than I got in how to implement my training in a European Youth project” 12% of the ATTE participants agreed (68 % disagreed), 44% of the TC 1+2 agreed (33 % disagreed) and 50% of the TC 3 participants agreed (27 % disagreed). The practically oriented method in relation to project development is slightly more developed than the method in relation to implementation of the training in the participants’ general youth work setting. If the emphasis on dissemination and multiplication remains, it is of great importance that the practical



orientation of the courses is further developed. Several comments from the NAs/ SALTOs confirm the impression that the Training courses are often perceived as too theoretical by the target groups of youth workers that the Youth programme of the European Commission and the National agencies target. It seems to be too early to find indications of increased quality (as a result of the Partnership activities) in the applications submitted to the Youth Programme. 40% of the NA/SALTOs can not see an increase in quality of applications and 25% did not reply to this question (mainly because they did not have a participant of their nationality in the courses). However, 35% of the NA/SALTOs consider that the Partnership activities “To some extent” have contributed to an increase in the quality of the applications submitted to the Youth programme. The NA/SALTOs that do see an increase in quality refer mainly to the application/applications submitted by the ATTE participants from their country. Some NAs take the opportunity to express their frustration and feeling of not being consulted in the selection process of participants to the TCs/ATTE. In order to improve the impact of the Partnership activities on the Youth Programme and especially in relation to Action 5, it is very important to improve the relations and working climate between the National Agencies (on the one hand) the Partnership staff and trainers’ teams. (See also Annex 5: Comments from National Agencies and SALTO resource centres) Whether or not the working climate has been ideal between the NAs and the staff/trainers of the Partnership activities, more than half of the participants (53%) have managed to implement a project within the Youth programme. 59% of the participants have implemented a project within Action 5 of the Youth programme. Of the ATTE participants, 72% indicated that they had implemented an Action 5 project, but for the question on implementation of a YOUTH project as such, the percentage was only 56%. It is impossible to say if this circumstance indicates a low level of concentration or little emphasis in the framework of the ATTE on the whole framework of the YOUTH programme, compared to the attention paid to specific Action 5 “Support measures” of the Youth programme. Impact in relation to the specific aims of the Covenant concerning dissemination and development of a multiplier effect with regard to the National Agencies and SALTO resource centres, have been the most difficult area to find evidence. The time-frame of this evaluation and the fact that it was supposed to take place after the conclusion of the Third Covenant have certainly complicated the measurement of impact. During our evaluation of the Third Partnership Covenant we have repeatedly come back to the issues of multiplication. What is multiplication and how do the Institutions and the Partnership work to support this mechanism? We can not find evidence that such a key discussion has taken place within the Partnership activities and therefore we feel it appropriate to briefly look into the interpretations and mechanisms of the concept. Multiplication is often referred to as a means for dissemination of information, skills, knowledge and experiences gained by participants/project organisers in one setting to other individuals or groups of people within or outside the participants/project organisers’ own organisation/network or project. This is usually understood as a good way to reach out to the target groups with information on how to use existing possibilities in the field of European youth work. Within the framework of the Partnership multiplication is supposed to take place in and be a result of all activities and publications produced. The concept of multiplication is taken for granted and the participants in the training activities are thought of as “natural multipliers”. However, our desk research and survey indicate that the various means for multiplication of the experiences and skills receive little focus in the training activities. In order for the participants to feel ready and confident as multipliers and to perform multiplication actions in their local/regional or national setting, they need to bring ideas on how they can work as multipliers with them from the courses or find out about this in the publications produced. We need a



short comparison between the Institutions’ view of multiplication in order to find out how a common understanding of the different means of multiplication can be used and how multiplication can be secured and the information, knowledge, skills and thereby experiences reach the various target groups of the Partnership. Let us first have a look at the Council of Europe framework for multiplication.

The Youth and Sports Directorate of the Council of Europe has a very long experience in using multiplication as a means of dissemination of information, skills, knowledge or for that matter values, or the specifically developed methods used in non-formal education as practised in the Training courses of the Council of Europe. Multiplication in the framework of the CoE has, to a significant extent, relied on existing well-functioning structures and channels for information, such as INGYOs and National Youth Councils. These organisations are well suited and easily used structures for multiplication of knowledge, skills and information. A large part of the participants in CoE training activities have a key position in their organisations. Their role and responsibility within their organisation involves sharing experiences and the information they receive in various European activities with youth leaders/workers in their own and other neighbouring organisations/networks. The structures of most youth organisations have thereby facilitated both the recruitment of new participants, and dissemination of information through magazines of their organisations or through other means of forwarding information. The skills and knowledge gained have been used within their organisations and thereby multiplication effects have been established.

Within its different actions and programmes, the European Commission always has an aim to inform and disseminate knowledge and skills. The rationale is that common European resources should be of benefit for all European Citizens. Multiplication is desired to take place in the framework of the YOUTH programme. The assumption is that the experiences, skills or knowledge gained will be discussed by those participating in an activity and thereby also reaching those that did not participate. Within the Youth programme the target group is much less homogenous than the target group of the activities of the Council of Europe, they might not necessarily have a structure or organisation to act within and their profile can range from an individual young person participating in a youth exchange organised by their Youth Club, via a student in adult education, studying to become a youth worker to an administrator of Youth issues at local/regional level. The multipliers in those three examples need very different means for and support in their multiplication and their target audiences might not be as interested in sharing the experiences as the fellow-members of a youth organisation. The partnership activities are tailored to benefit from decades of work and experience gained in the field of youth policy and training within the Council of Europe. The range of activities within the Partnership has however been developed to benefit both Council of Europe activities and the activities of the European Commission, especially in the framework of the Youth programme. The target groups towards which the Partnership activities are aimed, have also, as stated above, become broader compared to the traditional activities in the Council of Europe. The multiplication effect has not been evaluated in either setting, but it is likely that it could be improved through an analysis of the needs for methods and instruments for multiplication amongst new target groups. The format of publications could be more customised for broader distribution. The T-kits for example are very much customised for European Youth trainers with some experience, on their way to develop their experiences further. For a hands-on youth worker, wanting to develop the European dimension of a planned youth exchange, the T-kits are much too long and theoretical to serve as a means for the youth worker to make direct use of them. This fact has been pointed out by National Agencies in our survey. A way forward in this direction would be for example, to develop an “Anthology-format” T-kit, summarise parts of the T-kits and put them together in a Supplement edition of the Coyote Magazine. This should use an easily digested language and format and try to



make it very user-friendly for those target groups of the Partnership activities who have little experience of European level activities. A publication of this kind could be used both for promotion and to serve as an appetiser for the “not-yet-convinced”. Translation of the publication into as many of the European languages as possible, to be made available to the public in electronic versions, could for example be arranged by the National Agencies. The Partnership activities have, as we can see, not fully used the potential of the cooperation between existing structures. Notably the connection with National Agencies of the YOUTH programme has not been used extensively. The multiplication of the Partnership activities could also benefit from development of activities for the national/regional/local levels. These activities could, for example, involve short training activities for identified representatives of the target groups at a regional or local level. Through co-operation between NAs and former participants in TCs/ATTE, the activities would incorporate the experiences acquired by participants in TCs/ATTE (see also the chapter on Efficiency) and a creation of links between the different actors in the Partnership would take place. The structure for multiplication of the experiences, skills and knowledge gained by participants would be the existing channels of information of the NAs, and the target groups of the Youth programme would benefit from the Partnership activities in a more direct way. The real evidence of impact of both the training activities and the publications will be possible to detect at full range only several months after the end of the third Covenant (as it stands now the end is foreseen for the end of February 2004). Our general impression of the impact that the Partnership Covenant on Training has created on European Training in the Youth field is that European Citizenship as a theme in Youth projects and Youth work, be it within or outside the Youth programme of the European Commission, has begun to interest youth worker/leaders and trainers across Europe in its widest sense. A process of adaptation to a more integrative approach and more co-operation amongst the major stakeholders in the field has started and the importance of promoting non-formal education/learning as it is practised in the youth field to other related fields and institutions has become clear to most actors involved in the European Youth Training field. It seems therefore fully rational and logical to move into a Fourth Covenant and that the focus of this one should be on Multiplication and Transferability. Developing a proper dissemination strategy is of great importance to the initial stage of the Fourth Covenant. The impact created on the general quality of training activities in the European youth field is, as stated above, very unclear. We have looked into how participants in the TCs/ATTE have perceived the developments of quality concepts and their own perception of increased knowledge on concepts of quality. 32% of the participants in TCs/ATTE agree with the statement “There was not enough focus on the concepts of quality”. 43% disagreed with the statement and 23% answered they “did not know”. 68% of the ATTE participants disagreed with the above statement, which can be interpreted that participants in the ATTE course to a large extent considered that their expectations for the course in terms of development of concepts of quality were largely fulfilled. Approximately 40% or 2/5 of the participants in the TCs agreed to the statement, which indicates that a bit less than half of the participants had more expectations for developments of concepts of quality than what was fulfilled. The (in general relatively high) expectations by the participants in the Training course on European Citizenship, were also commented on by the course director as surprising, in an interview with him in October 2003. What was the outcome then in terms of increased knowledge on the concepts of quality? 12% of all participants in TCs/ATTE considered they had “Not at all” increased their knowledge on concepts of quality in European Youth work. All of those 12% were participants in TCs. 18% considered that they had gained enough knowledge in order to be able to train others on this topic. The major part of the



participants, 63%, considered they had gained knowledge to a level where they “Know more about it”. In terms of indications of impact on the general quality of projects and activities in the European Youth training field, we can only conclude that participants in the TCs/ATTE have to a relatively large extent gained knowledge on the concepts of quality and that they seem to have an increasing interest for the subject. They are likely therefore to pay more attention to the development of quality aspects in their future activities and projects. With the intention amongst the actors in the Partnership to forward the work and common positions in relation to the setting of quality standards for the European Youth training field, it should in the near future be easier for training organisers to implement high-quality training activities.

5.4. Efficiency in the implementation of the Partnership

In our evaluation of efficiency we have taken into account various perspectives of the implementation of the Partnership Covenant. We have, through reports of meetings, TWP reports and other evaluation reports, tried to follow developments and trace processes and progress that have led to the results/products of the implementation of the Partnership programme. In terms of efficiency we have mainly looked into the general management of the partnership programme and its training tools, efficiency of dissemination and efficiency in the monitoring/follow-up of results and publications. The guiding questions that we have used in our desk-research are as follows: To what extent is the current management of the Covenant appropriate to allow efficient development of the action? To what extent is the management and monitoring of the Covenant efficient as far as the role of both partners, European Commission and Council of Europe is concerned? To what extent have the means been efficiently transformed into results?

5 . 4 . 1 . M A N A G E M E N T A N D D E C I S I O N - M A K I N G W I T H I N T H E P A R T N E R S H I P

The management and decision-making aspects of the Partnership are not entirely clear to everybody involved in the various activities of the Partnership. For the actors involved, as trainers, resource persons in TCs, authors of T-kits or in any other position, the mechanisms in relation to decision-making (the TWP) and the key management position (the educational advisor position within the Partnership) do not seem clearly outlined or understood. The role of the Technical working party, TWP, is formally described in the Agreement between the two institutions. It states as follows “The TWP referred to in Article 3 is set up to monitor the implementation of the Covenant and to provide a forum for discussion. In addition the TWP shall make if necessary proposals for amendments regarding the implementation of the programme and measures with financial implications.” It is important that the respective roles of the two Institutions and the TWP are clearly stated and described and that this information is publicised, for example on the website. Our desk research suggests that the TWP does not use its full potential of monitoring, nor does it tend to propose amendments regarding the implementation of the programme. One possible explanation of



the low involvement of the TWP in the monitoring of the implementation of the programme concerns the reporting to the TWP. The reporting made to the TWP is of course crucial for its ability to make full use of the potential given to the decision-making body by the Agreement. The reporting to the TWP that is referred to in the reports of the meetings does not seem to include much in relation to strategies, monitoring and follow-up of the activities or the implementation of the programme. The reporting to the TWP seems to have more the character of updating on the achievements than on the processes of the achievements. In general it would be advisable to include in the reporting to the TWP also the less successful parts of the implementation of the programme or more problematic aspects in the course of implementation of the specific aims of the Covenant, in order to make full use of the TWP in the efficient implementation of the Partnership programme. Interviews or informal discussions with trainers and course directors indicate that they have sometimes perceived difficulties in adaptation of the different training tools to changes in the surrounding field or problems met in the course of implementation. Partnership management and the TWP have shown little flexibility in this area. In several preparation or evaluation meeting reports, efforts have been made to identify strategies and “inventions” for a better dissemination of publications/products (for example a prep meeting for T-kit on European Citizenship, March 2002), follow-up of participants and results (for example the IEFS meeting Oct. 2002) and for the internal efficiency of the management of the Partnership (for example proposals for how the website can be used more interactively). It is not entirely clear if these and other suggestions for improvements ever reached the TWP level. It is important that a sense of the energy, creativity and ambition devoted by various stakeholders to different activities or products, reaches the TWP, in order for them to be able to adapt the programme or financial framework, if they consider it appropriate, in order to maximise the results of the common effort by the stakeholders. To secure that the actors involved in the implementation of the Partnership programme remain active in the field, it is important that the Institutions and the TWP acknowledge their efforts and commitment and make use of their work and conclusions by taking it into consideration. One possible reason that the reporting to the TWP has focused on achievements rather than on the processes of achievements within the Partnership can be found in the limited resources in terms of Partnership staff. As it stands now the role of the educational advisor, the only senior officer employed entirely by the Partnership, is very broad and mixed and the focus has not been on analysis and co-ordination of the work by all actors, but rather being involved at operational level in the work. The role of the educational advisor is less that of a representative of the Institutions and more that of an educational senior adviser supervising the experts contracted as authors of T-kits and at the same time being responsible for the administration of the Partnership. This degree of involvement, which extends to the detail of the implementation of the activities, makes it difficult to keep a global analytical perspective, draw conclusions and produce reports for preparation of the decision-making. A recommendation for the 4th Covenant is to gear this position towards more of a coordinating role, making sure all actors, perspectives and objectives are considered and included in the process of implementation of the Covenant. The position should involve only to a limited extent direct work with the publications and products, leaving that to the trainers or experts contracted for the task. A more formal way of monitoring of progress through written reports on the developments of the partnership would certainly be of benefit to the whole process and all involved actors. More focus should be put on follow-up and monitoring of the process as well as making sure that the administrative work is



properly taken care of. This would for example include a procedure to post all finalised documents and reports on a password protected area of the website, to secure maximum transparency and sharing of experiences for all involved actors/stakeholders. It would be the responsibility of the co-ordinator to make sure that reports with proposals for improvements, actions and strategies, deriving from reports of the trainers and experts, are produced and that they are discussed and considered by the TWP. A structured and well functioning management with clear roles and responsibilities for all involved, is of obvious importance for the efficient management of the 4th Covenant.

5 . 4 . 2 . M AX I M I S I N G T R A N S P A R E N C Y A N D E F F I C I E N C Y I N S E L E C T I O N O F P A R T I C I P A N T S

A N D T R A I N E R S

The selection procedures for participants in TCs/ATTE have been very time-consuming and efforts have been made to adopt clear selection criteria for the pre-selection of participants. The pre-selections have been dealt with by course directors and Partnership staff and the final selections have been made by the teams of trainers. Provided the Partnership activities continue to grow in terms of applications it is advisable to develop a more rational and less time-consuming procedure. There is a lot of experience in managing large-scale selections amongst the NAs and TAO and this know-how should be used within the Partnership selections. It is important to assure a transparent and open climate in the selection of experts for various tasks and of trainers for the activities. This is especially important considering the fact that there will in the future most certainly be more individuals applying for the contracts as a result of the general increase in individuals with the right profile, due to the past years’ training activities, for example the ATTE course participants. In the selection of trainers and resource people for the activities the knowledge and experience of working with the Youth programme of the European Commission must be ensured. The presence of representatives of National Agencies or other people with a good extent of knowledge of the Youth programme have increased during the Third Covenant. It is important that this tendency continue, that the potential of the Youth programme be presented to the target groups in the Partnership activities and that the participants understand how they can make use of the Youth programme or for that matter other funding possibilities.

5 . 4 . 3 . C O S T - E F F I C I E N C Y A N D C O - O R D I N A T I O N O F M E E T I N G S

In contrast to the training courses, for which the participants were required by the organisers to achieve a high level of preparation, little preparation of meetings was demanded in other settings of the Partnership. Several meeting reports are unavailable (for example in relation to preparation of T-kits, Coyote editorial meetings and the meetings of expert groups). Even if meeting reports exist, it is not always clear who was present, who should have been present and reasons for absences. International meetings consume a great deal of the Partnership budget and it is important that the Partnership secretariat identifies easily managed systems for spreading information prior to meetings as well as sharing information of results of meetings. Absences must be kept to a minimum, as they tend to create a low level of cost-efficiency. At some stage the TWP clearly used its potential to propose action for improved implementation of planned activities by proposing to combine meetings in order to use synergies between two processes and increase cost-efficiency. This way of working needs to be used more often and in order to implement it, the planning and co-ordination of meetings is crucial. Meetings of expert groups also show to some low cost-efficiency, due to weak preparation, high travel costs and little guidance on what is really their task! A more structured framework for the expert groups is advisable.



5 . 4 . 4 . U S I N G T H E P O T E N T I A L O F T H E I N T E R N E T AS A T O O L F O R D I S S E M I N A T I O N

We have concluded (as have several meetings within the Partnership framework) that the website of the Partnership needs a more interactive profile. Continuous updating with news from the field, new articles, reports and an active discussion forum is a prerequisite for any website to be a success in terms of visits. The Coyote Magazine should be more Internet-based and serve also as the forum for discussion on Training in the European Youth field. The print copy of the magazine should however also be kept to ensure access for people with little access to the Internet and as a useful “leave behind” for the Partnership. A merger of part of the website with the SALTO website might be a way forward for exploring the synergies between the Partnership and the SALTO initiative. A common website for the three Covenants in the Youth field between the Institutions is another idea worth considering. A better link policy would also help. As it stands today, the links to the NAs (for example), are collected in a PDF document that requires scrolling of the whole document to find the NA one wishes to get to. It would be a better visualisation of the network of National Agencies and the Youth programme, if a direct link to each of them was created. A proper strategy for dissemination of the publications and results must cater for appropriate follow-up and monitoring of the dissemination. Today it is not possible, for example, to detect which specific items have been downloaded from the website, as the only monitoring regards downloads from the site in general. If the website is decided to be the appropriate tool for the Partnership, an expert on web-design and the Internet as a tool for marketing, dissemination and follow-up should be contracted to give advice on how to make full use of the tool.

5 . 4 . 5 . I N C L U S I O N O F N A T I O N A L A G E N C I E S

A stronger and more elaborate role of National Agencies and SALTOs in the Partnership has repeatedly been emphasised in reports from the TWP as well as from other meetings. The inclusion of NAs/SALTOs that have been put forward as a response to the requests from the side of the Partnership secretariat, concerns more one-way communication, by posting advertisements on the YOUTHNET that is accessible for all staff at NA/SALTOs, than a proper strategy for inclusion of NA/SALTOs in the implementation of the Partnership programme. A lack of interest from the National Agencies has been brought up in different reports as a potential obstacle to co-operation. It is possible that there has been a certain lack of interest amongst the NA/SALTOs, and our questionnaire directed towards the NA/SALTOs was only returned by approximately 50% of the NA/SALTOs. If this is the case it is even more important to emphasise inclusion and promote a sense of ownership amongst the NAs, since they form an important part of the structure for multiplication. To a certain extent the National Agency staff seem to have been regarded as information agents, rather than as resources who can add value in the implementation of training activities and production of publications. It is also important to make sure that NA representatives working on Partnership matters have the right profile for the task they are assigned to and that they are known and acknowledged as a representative of the NA network. The strategy for dissemination of information to the network of the National Agencies also needs to be looked into. Until now, as stated above, the YOUTHNET has been used as an arena for dissemination. This is one way of doing it, but the co-operation and involvement of the National Agencies would certainly benefit if more intensive ways of dissemination of the information were used. A more informal way of communication, with for example personal e-mail lists to the



appropriate staff of NAs/SALTOs, would ensure that the different actors feel more ownership of the Partnership.

5 . 4 . 6 . M U L T I P L I C A T I O N A N D T R A N S F E R - D I S S E M I N A T I O N O F R E S U L T S F R O M C E N T R E

T O P E R I P H E R Y

The fourth phase of the Covenant ought to be focusing inter alia on using the results of prior actions of the Partnership. Until now the work of the Partnership has nearly always meant production of tools and bringing people and/or knowledge from the “periphery” to the “centre”. For the next phase it is advisable to concentrate as well on using the results of prior phases and spreading the know-how on European training in the Youth field to a broader audience, incorporating the “not-already-convinced”. Ways for implementation of such a strategy could involve: - Involvement of previous participants in the TCs/ATTE in organising European Citizenship seminars in their local community/region or even nationally in co-operation with the National Agency concerned. A percentage of the budget of the Partnership could be earmarked for these kinds of activities. The initiative could come from former participants, NAs or be initiated by the secretariat. This is also discussed in relation to increasing Impact, as a way for creation of more co-operations between different actors and strengthening also the multiplier effects. - European trainers and/or authors of the T-kits could organise seminars on T-kits, in co-operation with NAs, in order to market them and establish a dialogue with the target groups. - Creation of TCs on regional/national level to bring together participants from various backgrounds forming together a multicultural/intercultural group. Participants could be recruited through applicant organisations to the Youth programme. By reduction of travel costs involved in the TC, those courses could be reasonably cheap in comparison to a TC abroad. Activities taking place in a local/regional community could also easily get publicity and thereby raise the general awareness of the Youth programme. Through these kinds of activities, exploring the synergies of the training tools of the Partnership, clear links can be created between the different partners and actors in the European youth field. Everybody will use their specific key-competence in the co-operation. The result of their co-operation will be an efficient combined promotion and dissemination activity, which would ensure that the objectives of the Partnership on a long-term basis are more efficiently fulfilled. Until now we have mainly focused our analysis of the Third Covenant Partnership on the Partnership as such, rather than on the individual roles of the funding institutions. It is clear that the role of the Council of Europe in the day-to-day work within the Partnership has been the stronger of the two. This is mainly due to the fact that the Partnership secretariat is situated within the premises of the Council of Europe and that the Partnership therefore has benefited from much more input in terms of staff-time by the Council of Europe than by the European Commission. Since the emphasis in work has been on production it was logical to keep a stronger involvement of the Council of Europe and benefit from the long experience and educational expertise accumulated within the Council. Moving in to the fourth phase, where more emphasis should be placed on multiplication and transfer, it is important that the European Commission, including the network of SALTO resource centres and National Agencies, becomes more involved, provides more input and takes on more responsibilities than they have during the past three years. As we have argued previously in the report, multiplication and dissemination are closely interrelated phenomena and for an efficient management of the two, the Partnership will have to rely much more on the networks of NAs/SALTOs. In relation to this it is very



important that all actors of the Partnership acknowledge the strength of the network of NAs in reaching out to regional/local level and that the European Youth Forum and the Council of Europe continue to use their internal channels for information in order to complement the dissemination of the NAs. The European Commission needs to state more clearly how the Partnership activities relate to the other training activities involving the National Agencies and make sure that the work of representatives of NAs in the Partnership activities and the TWP is understood and supported by the network of NAs/SALTOs.

5 . 4 . 7 . R E C O G N I T I O N

We have been asked especially to comment on the efforts made by the Partnership in relation to Recognition. Recognition and Validation are issues of great concern not only for the European Youth field. Within the Partnership the immense work of ensuring that experience from and work produced in the sector of European Youth work is recognised by neighbouring fields, mainly education, vocational training and employment, has only been briefly addressed. Recognition is very closely related to the development of quality criteria and standards. A group of experts, nominated within the Partnership, have during the last phase of the third Covenant tried to promote work in the European Youth field in relation to Quality standards, Evaluation and Validation. The results of the efforts of this expert group will hopefully take the work of the CQDG one step further and propose easily measurable standards that will help to create a common understanding within the field on what quality training in the youth field should look like. In order for the work and results produced within the European Youth training field to be recognised by the neighbouring fields, a common understanding of the concepts of quality within the field must be developed. The Partnership has the potential to create/produce this common understanding, provided the partners in the fourth phase really take the steps necessary to move into an area of work, where multiplication, dissemination and integration are leading concepts. A possible step in this direction would be to go ahead with the planning of short training modules in the area of European Citizenship. We have touched upon this already in the Chapter on Relevance. All major actors in the European Youth training field can by producing different parts of the Curricula on European Citizenship, based on the agreed quality standards. Participants or trainers wishing to get a “certificate”, “degree” or in another way be “recognised”, by the European Commission and the Council of Europe, as having the ability to train others in European Citizenship, would need to collect a full range of courses, including all elements/modules during a set time-frame. An agreement on course formats of this kind would demand co-operation amongst the actors/stakeholders of the Partnership. The fact that there is little if any other possibility to gain such a “certificate” elsewhere in the education or training fields, should mean that the certified course would certainly interest the other neighbouring fields, and that interest would be the first step towards getting recognition from them. It is important to identify the specificity of what is unique in the European Youth field and the non-formal education as practised in the European Youth field, rather than trying to monopolise the concept of non-formal education. Another aspect that is worth considering is to try to implement and promote an attitude of recognition of alternative ways of becoming involved in European Youth Work. Recognition from the outside would be facilitated by inclusion also of “outsiders” in the work, recognising their knowledge and competencies acquired outside the youth sector. Those “outsiders” might produce the shortcuts in the efforts produced on the way towards recognition!



5.5. Effectiveness

As a concluding evaluation criterion, we now address the effectiveness of the implementation of the programme, i.e. to what extent the outputs and results have been achieved with regard to the foreseen ones. A comparison between the descriptions of the annual activities (Annex D) in the three individual contracts between the European Commission and the Council of Europe, comprising the total timeframe of the 3rd Covenant, from 1 July 2000 to 30 June 2003 (with an extension of the contract until the end of February 2004) and the results and outputs produced (as described with an emphasis on quantitative aspects under “Quantitative descriptions of the training tools” and with an emphasis on qualitative aspects, under Results and Impact) provides an indication of the effectiveness. This comparison shows that the foreseen activities have to a very large extent been implemented. In the descriptions of planned activities, it is not until the 3rd phase of the 3rd Covenant that we have found any non-implemented activities. The first question-mark is in relation to the development and production of a training-kit (T-kit) on Evaluation. We have referred earlier to the issue of poor documentation in relation to certain work areas. If a decision to postpone or abandon the production of a T-kit on Evaluation has been made or not remains to us unclear. However, there are at this stage no references or indication in the material provided to us that the process of production has started. The production of issues number 8 and 9 of the Coyote Magazine and an additional supplement should have been implemented during this 3rd phase. We believe that at least an 8th issue is under production and eventually will be ready before the end of February 2004. A 9th issue seems impossible to produce within the framework of the 3rd Covenant. The planned activities according to Annex D also include the production of a political document outlining the added value of the Partnership programme. We can not identify a document answering that description. Common efforts have been made by the two Institutions to produce a joint document on non-formal education. A draft document was discussed in the last TWP meeting, but we have no further information on that matter. The global analysis of the effectiveness of the Partnership shows that the major bulk of activities planned have also largely been performed. The emphasis has been very much on the strict production of the agreed training tools, with some exceptions as stated above. We have also argued that the co-operation between the two institutions would benefit from a change of perspective towards more emphasis on multiplication, dissemination and use of the existing products and publications. The issues of follow-up of participants in training activities, monitoring of demand for publications including both print and electronic versions have to a certain extent been neglected. It is stated in Annex D of the three yearly presentations, that “Follow-up of the participants should form an integral part of the European co-operation in the field of youth training and related to non-formal education. This should include the establishment of a database of participants, as a resource for other organisations, and monitoring of the use that participants make of their experience after the training course. On completion of the course, participants should receive a document describing the content of the course they have followed (‘attestation’).” We have not been able to locate any references to such a database of participants in any of the documentation. It would certainly have been valuable for the institutions to be able to keep track of the progress made by participants in the field and their use of other training tools produced in the European Youth Training field. To our knowledge there exists no such database and from what we can see it has not even been discussed. This fact emphasis the importance of establishment of procedures for documentation and reporting in relation to the general development of the Partnership. It emphasises also the importance



of allocating enough financial resources for more than one full-time senior official employed by the Partnership. It is difficult at this stage, as discussed earlier in the chapter on Impact, to establish evidence on the use of Partnership tools in projects organised under the YOUTH programme. Since the effects of the Partnership on the quality of applications to the Youth programme, and on the implementation of activities with a European Citizenship theme, might in some cases only be possible to detect several years from now, it is important to stress the follow-up and monitoring of the Partnership’s activities. A way forward in establishing indicators of the usefulness of the Partnership activities would be to introduce a possibility to mention possible uses the projects’ organisers have made of the Partnership activities in the final report form. For the effectiveness of the Partnership it is also important to keep the activities in a project form with a clearly stated start and end dates, rather than to consider the Partnership as another “institution”. The overruling aim should be to “produce” added value to the institutions and other stakeholders, not to become another actor in the European Youth training field.



6. CONCLUSIONS AND RECOMMENDATIONS

We present our summary conclusions hereafter, followed by recommendations. They are identified with reference to the key evaluation criteria used in the study.

1. The Third Covenant Partnership has produced the results that were foreseen to a great extent. Three consecutive training courses on European Citizenship have been implemented, as have one long term training course for advanced trainers (the ATTE course), five new T-kits and 5 issues of the Coyote magazine. In addition to these training tools, the Partnership has launched and maintained a website, produced a promotional brochure, implemented two sessions of the Trainers’ Forum and initiated two separate expert groups in relation to Curriculum and Quality development and Quality, Evaluation and Validation. (Results/ Effectiveness) We conclude that the training tools have helped trainers and youth workers to acquire new knowledge, skills and competencies, in particular as far as integrating European Citizenship into their work. They can be further improved in terms of practical orientation in order for the participants to be able to use their newly acquired knowledge in their day-to-day work and activities. (Results/Relevance)

2. The four training tools seem generally appropriate, provided that coherence and synergies

between them is maximised. In achieving objectives regarding quality, the 9-day training course, with perhaps an over-extensive curriculum, can not realistically achieve the objectives of high quality. ATTE is much better placed to do this. This issue could be further exploited by reducing the programme within the 9-day course and offering a shorter format to appeal to more potential participants (Relevance/Effectiveness/Efficiency)

3. We can conclude that during the past three years the analysis of needs has not been in explicit

focus in the Third Partnership Covenant activities. It is however, of great importance that the needs of the target groups are kept as a central element within the Partnership, in order to ensure the relevance of the objectives, tools and methods. Our evaluation indicates that there is, for example, a need amongst the wider target group of the Youth programme and the Partnership activities for more practically-oriented training, ensuring that the European level training has an impact on youth activities at all levels, from European to local level. In this respect significant attention was not given to the Voluntary Service area. Another dimension of the needs issue is the point that the Youth programme’s National Agencies were not associated to the extent that could have been possible in the design and implementation of the Partnership. (Relevance)

4. Quality and related issues, although discussed, have not enjoyed the central place that would

have been expected. The Partnership offers a privileged place for the advancement of the (admittedly difficult) discussion on quality in European youth work training. (Results/Impact/Effectiveness)

5. The development of a database on the participants in the training activities was foreseen in the

Annex D, regulating the Partnership activities. A creation of a database on the trainers and participants in the training activities would have ensured better follow-up and monitoring of participants and their training activities and projects. It would have facilitated the monitoring of the Partnership and especially the follow-up of the training activities undertaken by participants in the Training courses on European Citizenship and the ATTE. A database of



this type would also ensure a more manageable dissemination of products, publications, calls for experts etc., provided it is based on a simple format that is easily updated and well maintained. (Results/Effectiveness)

6. It has been logical until now to have a strong involvement of the Council of Europe and

benefit from the long experience and educational expertise accumulated within the Council. While this should be continued, it is desirable that when moving in to the fourth phase, where more emphasis should be placed on multiplication and transfer, the European Commission, including the network of SALTO resource centres and National Agencies, should become more involved, provide more input and take on more responsibilities than was the case during the past three years. It is very important that the complementarity between all actors of the Partnership is acknowledged and that the strength of the network of NAs in reaching out to regional/local level is acknowledged. The European Youth Forum and the Council of Europe should continue to use their own channels in order to complement the dissemination through the NAs. (Relevance)

7. The optimal use of the publications could secure significantly greater impact for the

Partnership. An overarching dissemination strategy is not present. This would incorporate objectives, defined target groups, monitoring and tracking tools to follow outputs. As a result the multiplier effect of the partnership would be greatly enhanced, particularly in a context where individualism in popular culture may make it more difficult. (Impact/Effectiveness)

8. The potential of new technologies has not been fully explored. Examples of new technologies

are video conferencing, distance learning tools and interactivity as well as monitoring of visits to and on-line down-loading of publications from the website. Within the Partnership activities (both training activities and meetings) we can conclude that the possibilities that exist in the use of new technologies as a complementary tool for preparation, follow-up mechanisms and dissemination to more traditional training activities and meetings based on face-to-face contact have not yet been adequately deployed. (Impact/Efficiency)

9. Regarding the staffing within the Partnership, there have been limited resources available for

coordination, which has led to difficulties in keeping all actors, perspectives and objectives actively included in the process of implementation of the Covenant.(Efficiency)

10. In our evaluation we can conclude that in the selection of trainers and resource people for the

activities, the knowledge and experience of working with the Youth programme of the European Commission have not been in focus. Nevertheless the presence of representatives of National Agencies or other people with a good knowledge of the Youth programme has increased during the Third Covenant. (Efficiency)

11. The challenge of getting non formal education (as practised in the Youth field) recognised

outside the Youth field remains considerable. This would raise awareness and ultimately secure recognition of the importance of Youth work, Youth policy and the participation of young people in society and politics from the outside. (Efficiency)

The following summarises our recommendations, to be seen in relation to the above conclusions. The text of the report (especially Chapter 5) contains a much larger set of specific recommendations and suggestions.



1. For the Fourth Partnership Covenant we recommend that more emphasis should be placed on coordination between the Institutions and actors and on multiplication and transfer of experiences, publications and products. The Partnership activities should be clearly positioned in the broader EU and Council of Europe youth strategies.

2. We recommend that the Partnership incorporate more efficient ways of disseminating

information, publications and possibilities for participation in training activities. A database would serve as a means for more efficient dissemination, monitoring and follow-up in relation to the actors in and target groups of the European Youth training field.

3. We recommend that more focus is made on the analysis of needs of the wider target group of

the Youth programme and the Partnership activities in addition to the needs of those that are mainly active on European level. The analysis of needs is especially important to focus on for the institutional and management level.

4. We recommend that a more integrated approach be applied to the co-operation between the

actors in the Partnership. It is important to build a common sense of responsibility and ownership of the Partnership activities amongst National Agencies, SALTOs, the European Youth Forum and the two Institutions and that the contributions, financial or human resources are more equal.

5. We recommend that more emphasis be put on the practical use of the training activities and

that more focus is given to the development of easily measurable indicators of how the participants and/or target groups make practical use of the knowledge they acquire through the Partnership.

6. The work against racism and xenophobia should remain a priority. More efforts should be

made to help users and participants make the connection between work against racism and xenophobia and for instance work with European Citizenship.

7. We recommend that the Partnership properly underline the importance of participants sharing

their experiences with a wider audience. The sharing of experiences/multiplication should be considered a very important element in the dissemination strategy of the Partnership. The participants need to fully understand their expected role as active multipliers, prior to commencing the training activities. More resources should be devoted to the transfer of experience from the Partnership to the target groups and all the actors of the Partnership. New methods and instruments to explore the full potential of multiplication and transfer of information, knowledge and experiences in the European Youth training field should be assessed. Bringing existing websites together to create an integrated electronic forum for European youth activities should be explored as an option.

8. We recommend that new technologies are used in a more structured way in particular in

relation to the preparation of activities and products. Preparatory work partly assisted by, for example, on-line electronic conferences can reduce costs for travel and facilitate preparation of activities. Amongst the actors of the Partnership there is already considerable competence in the area of using new technologies and this should be exploited.

9. We recommend that the Institutions continue their co-operative work in promoting common

positions in the areas of setting quality standards, and increasing awareness of quality-related issues amongst the target groups of the Youth programme and the Partnership activities. It is also important to co-operate on promotion of the work implemented in the field, research on



and evaluations of the wider youth field as well as enhancing the political work in bringing European policy to national/regional and local level and bringing examples of good practice from the national/regional/local level to the European level.

10. We recommend that resources for co-ordination activities are prioritised for the Fourth

Partnership Covenant. The co-ordinator should only to a limited extent work directly on the activities, publications and products, which would be led by the trainers or experts contracted for the task. The coordinator position should rather be focused on managing the development and implementation of the partnership activities and development of a dissemination strategy (both internal and external). A structured and well-functioning management, including sufficient financial and human resources, with clear roles and responsibilities for all involved, is of utmost importance for the efficient management of the Fourth Covenant.

11. We recommend that the tendency to increase the involvement of National Agencies in the

implementation of the training activities continue and be included in a general strategy of enhancing the sense of ownership amongst the actors in the Partnership. This is especially important when it comes to National Agency and SALTO staff, in order for the potential of the Youth programme to be presented to the target groups in the Partnership activities and that participants understand how they can make use of the Youth programme or for that matter other funding possibilities.

12. We recommend that the Institutions continue to prioritise the promotion of Non-formal

education to the European Youth field as well as to neighbouring fields, for example that of vocational training.

13. We recommend that the Partnership contracts between the two Institutions should remain in a

project format with clear start and end dates and with a programme that is balanced in terms of aims and objectives versus the expected output and resources.



7. ANNEXES

Annex 1 - Questionnaire for participants in TC I, II and III and ATTE Annex 2 - Analysis of Questionnaire for TC I, II, III and ATTE Annex 3 - Questionnaire directed towards the National Agencies of the Youth programme,

Action 5 officers and to the SALTO resource centres Annex 4 - Analysis of Questionnaire to NAs/SALTOs Annex 5 – Comments from National agencies and SALTO resource centres Annex 6 - Overview of documentation reviewed Annex 7 – Terms of Reference



Annex 1 - Questionnaire for participants in TC I, II and III and ATTE Highlight your choice by putting it in BOLD. If you wish to comment your choice, please, make a short comment.

1. Have your participation in TC/ATTE provided by the Partnership, made you feel more confident in participating in/working with Youth training at European level? (Mark your choice by putting it in BOLD)

Yes definitely To some extent No If you can/wish, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

2. Have your participation in TC/ATTE, provided by the Partnership, supported your professional development and increased your competence to integrate European citizenship as a theme into your planned or implemented projects and/ or daily work with young people? (Mark your choice by putting it in BOLD)

Yes definitely To some extent No If you can/wish, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

3. Do you feel that you belong to a European network of trainers/youth workers? (Mark your choice by putting it in BOLD)

Yes, but I did already belong to it before the TC/ATTE Yes, with this experience I feel that I belong to a European network of trainers/ youth workers No, I still feel unfamiliar with European level Youth work and training and I would like to learn more in order to be able to feel that I belong to a European network of trainers and youth workers No, I am not interested in belonging to such a network If you can/wish, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

4. Have you actively used your newly acquired knowledge/professional development/information in your local/regional/national reality? (Mark your choice/choices by putting it in BOLD)

a) Yes, I have brought the issues of European citizenship into discussion



b) Yes, I have disseminated information on the existence of other training tools such as Coyote and T-kits, to my local/regional/national reality c) No, I have had difficulty in putting my new training into practice d) Yes, I have shared my experiences during the TC/ATTE with most of my colleagues and/or young people that I work with e) Yes, I have shared my experiences during the TC/ATTE with a few of my colleagues and/or young people that I work with f) No, my colleagues and/or young people that I work with were not interested g) Yes, I have published my experiences in a newsletter, magazine and/or newspaper h) Yes, I have implemented/plan to implement a European youth project within the framework of the Youth Programme of the European Commission i) Yes, I have implemented/plan to implement a European youth project within the framework of the Youth Programme, Action 5 “Support measures” of the European Commission j) No, not on local/regional /national level, but I have participated in other European level training activities If you can/wish, give a motivation to your choice/choices; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

5. In relation to which of the following issues have you gained an increased knowledge, due to your course? (Mark your choices by putting it in BOLD)

a) European Citizenship?

Not at all I know more about it Enough to train others b) Youth participation and inclusion of young people in society and politics?

Not at all I know more about it Enough to train others c) Intercultural learning?

Not at all I know more about it Enough to train others d) Concepts of quality in European Youth work?

Not at all I know more about it Enough to train others e) European youth policy and European funding programmes?

Not at all I know more about it Enough to train others f) Fight against racism and xenophobia Not at all I know more about it Enough to train others



If you can/wish, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

6. Do you consider the training activities you have participated in to equally promote and inform about different types of training activities and publications available in the European youth training field –such as publication and training possibilities within the Partnership, the European Commission, the National agencies/SALTO resource centres of the Youth programme, the Council of Europe or any other? (Mark your choice by putting it in BOLD)

Yes No If your answer is No, what possibilities do you know most/least about? ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

7. Mark in BOLD if you agree, don’t know or disagree with the following statements

a) The mix of methods of my TC/ATTE, corresponded well to the objectives of the activity I agree don’t know I disagree b) The mix of methods of my TC/ATTE corresponded well to the needs of participants in the activity. I agree don’t know I disagree c) The training should have focused more on personal development of participants I agree don’t know I disagree

d) The training should have included more input by external resource persons I agree don’t know I disagree e) I would have wanted more focus on how I can share/use my training experience back home I agree don’t know I disagree f) I would have wanted more information on funding programmes I agree don’t know I disagree g) The training should have included more intercultural learning and group dynamics I agree don’t know I disagree h) There was not enough input by the European Commission I agree don’t know I disagree i) There was not enough input by the Council of Europe I agree don’t know I disagree j) I had expected more discussion on the concept of European citizenship I agree don’t know I disagree



k) I would have wanted more guidance than I got in how to implement my training in a European youth project I agree don’t know I disagree l) There was not enough focus on development of concepts of quality I agree don’t know I disagree m) I have now a clear understanding of non-formal education I agree don’t know I disagree n) I had expected more working group sessions and project development exercises I agree don’t know I disagree o) I find it difficult to link my training to my day-to-day work I agree don’t know I disagree If you can/wish, give a comment to your choices; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

8. Which of the following T-kits have you; “Read in full”, “Read parts of” or “Not read at all”? (Mark your choices by putting it in BOLD)

T-kit 5 International Voluntary Sevice Read in full Read parts of Not read at all T-kit 6 Training Essentials Read in full Read parts of Not read at all T-kit 7 European Citizenship Read in full Read parts of Not read at all T-kit 8 Social Inclusion Read in full Read parts of Not read at all If you can/wish, give a comment to your choices; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

9. a) Do you consider the T-kits a good complementary tool to your participation in the TC/ATTE in order to improve your youth projects or training activities? (Mark your choice by putting it in BOLD)

Yes No



b) Have you recommended it to other people Yes No If you can/wish, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

10. Are you a subscriber to the Coyote Newsletter? (Mark your choice by putting it in BOLD)

Yes No

If you can/wish, give some details or reasons to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

11. Are you a frequent visitor of the partnership website www. training-youth.net? (Mark your choice by putting it in BOLD)

Yes, I visit it often, more than twice a week Yes, I visit it approximately once a month No, but I have visited the site 2-3 times No I have not visited it If you can/wish, give a brief comment to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Annex 2 - Analysis of Questionnaire for TC I, II, III and ATTE Analysis of Q TC I, II, III and ATTE

Reply-rate 25/30 83 % 9/60 15% 26/30 87% TOTAL %ATTE TC I, II TC III 60 (/120) 50% replyrate

1.Yes def 21 84% 7 78% 12 46% 40/60 67%T s Ex 4 16% 2 22% 14 54% 20/60 33%No 0 0 0%

2.Yes def 14 56% 3 33% 8 31% 25/60 42 %T s E 9 36% 5 56% 18 69% 32/60 53%No 2 8% 1 11% 0 3/60 5%

3.Belonged 11 44% 3 33,3% 4 15% 18/60 30 %with this 14 56% 3 33,3% 11 42% 28/60 47%No, unfami 3 33,3% 9 35% 12/60 20%not interested 1 4% 1 1.5%? 1 4% 1 1.5%

4.a) Brought to disc 11 44% 4 44% 4 15% 19/60 32%b) diss. Info 19 76% 7 78% 5 19% 31/60 52%c) diff put in prac 1 4% 0 1 4% 2/60 3%d) shared with all 9 36% 4 44% (Atte TcI+II 38%) 3 12% 16/60 27%e) shared with few 11 44% 5 55% (Atte TcI+II 47%) 4 16% 20/60 33%f) no interest 0 0 1 4% 1/60 1.5%g) published 2 8% 0 2 8% 4/60 7%h) impl youth project 14 56% 4 44% 4 16% 22/60 37% (ATTE TC 1+2 18/34 53%)i) impl youth 5 project 18 72% 2 22% 1 4% 21/60 35% (ATTE TC1+2 20/34 59%)j) no but further europ train 0 1 11% 0 1/60 1.5 %



5. 23 tota) Europ citizenshipnot 1 4% 0 1 4% 2/60 3%know more 15 65% 5 56% 14 56% 34/60 57%enough 7 31% 4 44% 10 40% 21/60 35%? 2 8% 1 4% 3/60 5%

b) youth participation and incl…not 2 9% 0 3 12% 5/60 8%know more 8 35% 7 78% 14 56% 29/60 48%enough 13 56% 2 22% 8 32% 23/60 38%? 2 8% 1 4% 3/60 5%

c) intercultural learningnot 1 4% 2 22% 6 24% 9/60 15%know more 5 22% 3 33% 13 52% 21/60 35%enough 17 74% 4 44% 6 24% 27/60 45%? 2 8% 1 4% 3/60 5%

d) concepts of qualitynot 0 1 11% 6 24% 7/60 12%know more 16 70% 6 67% 16 64% 38/60 63%enough 7 30% 2 22% 2 8% 11/60 18%? 2 8% 2 8% 4/60 7%

e) policy and fundingnot 5 22% 1 11% 5 20% 11/60 18%know more 7 30% 3 33% 18 72% 28/60 47%enough 11 48% 5 56% 1 4% 17/60 28%? 2 8% 2 8% 4/60 7%

f) fight against racism and xenoph.not 8 35% 5 56% 10 40% 23/60 38%know more 8 35% 4 44% 10 40% 22/60 37%enough 7 30% 5 20% 12/60 20%? 2 8% 1 4% 3/60 5%

6.Equal promotion and informationyes 19 76% 7 78% 19 73% 45/60 75%no 5 20% 2 22% 7 27% 14/60 23%? 1 4% 1/60 2%

7.a) corr objagree 22 88% 5 56% 16 62% 43/60 72%dn 3 12% 3 33% 2 8% 8/60 13%disagree 0 1 11% 8 30% 9/60 15%b) corr needagree 17 68% 5 56% 7 27% 29/60 48%dn 6 24% 4 44% 12 46% 22/60 37%disagree 2 8% 0 7 27% 9/60 15%c) per devagree 9 36% 5 56% 18 69% 32/60 53%dn 3 12% 1 11% 4 15% 8/60 13%disagree 12 48% 3 33% 4 15% 19/60 32%? 1 4%

d) ext inputagree 4 16% 1 11% 12 46% 17/60 28%dn 5 20% 4 44% 4 15% 13/60 22%disagree 15 60% 4 44% 10 38% 29/60 48%? 1 4% 1/60 2%

e) shar/use experienceagree 5 20% 8 89% 21 81% 34/60 57%dn 2 8% 1 11% 4 15% 7/60 11%disagree 17 68% 0 1 4% 18/60 30%? 1 4% 1/60 2%



f) info on fundingagree 4 16% 2 22% 17 65% 23/60 38%dn 2 8% 2 22% 6 23% 10/60 17%disagree 19 76% 5 56% 1 4% 25/60 42%? 2 8% 2/60 3%

g) ICL group dynamicsagree 6 24% 3 33% 9 35% 18/60 30%dn 0 3 33% 6 23% 9/60 15%disagree 19 76% 3 n 33% 9 35% 31/60 52%? 2 8% 2/60 3%

h) input comagree 8 32% 1 11% 9 35% 18/60 30%dn 3 12% 4 44% 10 38% 17/60 28%disagree 14 56% 4 44% 7 27% 25/60 42%

i) input coeagree 3 12% 0 3 12% 6/60 10%dn 1 4% 3 33% 12 46% 16/60 27%disagree 21 84% 6 67% 11 42% 38/60 63%

j) discussion on europ. citizenshipagree 8 32% 3 33% 12 46% 23/60 38%dn 3 12% 1 11% 3 12% 7/60 12%disagree 14 56% 5 56% 11 42% 30/60 50%

k) guidance in agree 3 12% 4 44% 13 50% 20/60 33%dn 4 16% 2 22% 6 23% 12/60 20%disagree 17 68% 3 33% 7 27% 27/60 45%? 1 4% 1/60 2% l) dev con quaagree 5 20% 4 44% 10 38% 19/60 32%dn 2 8% 4 44% 8 31% 14/60 23%disagree 17 68% 1 11% 8 31% 26/60 43%? 1 4% 1/60 2%

m) understanding of nfeagree 22 88% 5 56% 13 50% 40/60 67%dn 2 8% 1 11% 6 23% 9/60 15%disagree 0 2 22% 7 27% 9/60 15%? 1 4% 1 11% 2/60 3%

n) working groups and project devagree 2 8% 2 22% 10 38% 14/60 23%dn 5 20% 1 11% 7 27% 13/60 22% disagree 18 72% 6 67% 8 31% 32/60 53%? 1 4% 1/60 2%

o) diff link dtdagree 3 12% 3 33% 7 27% 13/60 22% dn 1 4% 4 44% 6 23% 11/60 18%disagree 20 80% 2 22% 13 50% 35/60 58%? 1 4% 1/60 2%

8.T-kit 5 int vol serfull 3 12% 1 11% 0 4/60 7%part 10 40% 2 22% 3 12% 15/60 25%not 12 48% 6 67% 22 88% 40 /60 67%? 1 4% 1/60 2%



T-kit 6 trainfull 8 32% 2 22% 2 8% 12/60 20%part 15 60% 3 33% 13 52% 31/60 52%not 2 8% 4 44% 10 40% 16/60 27%? 1 4% 1/60 2%

T-kit 7 eu citfull 6 24% 1 11% 5 20% 12/60 20%part 17 68% 6 67% 17 68% 40 /60 67%not 1 4% 2 22% 3 12% 6/60 10%? 1 4% 1 4% 2/60 3%

T-kit 8 soc inclfull 7 28% 1 11% 3 12% 11/60 18%part 6 24% 3 33% 11 44% 20/60 33%not 12 48% 5 56% 11 44% 28/60 47%

1 4% 1/60 2%

9.a) complementarityyes 21 84% 9 100% 26 100% 56/60 93%no 3 12% 3/60 5%? 1 4% 1/60 2%

b) recommendedyes 25 100% 9 100% 18 69% 52/60 87%no 0 8 31% 8/60 13%

10.Coyote subscriberyes 20 80% 2 22% 8 31% 30/60 50%no 5 20% 5 56% 18 69% 28/60 47%

2 22% 2/60 3%

11.Frequency web visitstwice a week 7 28% 3 33% 5 19 % 15/60 25%once a month 17 68% 6 67% 6 23% 29/60 48%2-3 times 1 4% 13 50% 14/60 23%not at all 0 2 8% 2/60 3%



Annex 3 - Questionnaire directed towards the National Agencies of the Youth programme Action 5 officers and to the SALTO resource centres

Questionnaire directed towards the National Agencies of the Youth programme, Action 5 officers and to the SALTO resource centres on the tools of the Partnership Covenant on Training-Youth Highlight your choice by putting it in BOLD. If you wish to comment your choice, please, make a short comment.

1. Have the 3 Training courses on European Citizenship (TCs) and the Advanced Training for Trainers in Europe (ATTE) provided by the Partnership, contributed to an increase in the quality of applications, submitted to the YOUTH programme in your country? (Mark your choice by putting it in BOLD)

Yes definitely To some extent No If you can, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

2. Have you been in contact with/contacted by someone that did participate in either of the TC:s or the ATTE? (Mark your choice by putting it in BOLD)

Yes No If yes, what was the result of this contact? ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

3. Do you consider it helpful in the implementation of the Youth programme and dealing with the target groups at your NA/SALTO, to be able to refer to people with the experience/knowledge of the TCs/ATTE or to use them as trainers in your training activities?

(Mark your choice by putting it in BOLD) Yes No If you can, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

4. Are you well informed about the Partnership Covenant? (Mark your choice by putting it in BOLD)

Yes, fully Yes, to a good extent No, I find it difficult to find information No and I am not interested



If you can, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

5. Have you noticed an increase of application including European Citizenship as a theme in their activities in 2002 and 2003? (Mark your choice/choices by putting it in BOLD)

Yes No If you can, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

6. Which of the following T-kits have you; Read in full, Read parts of or Not read at all? (Mark your choices by putting it in BOLD)

T-kit 5 International Voluntary Service Read in full Read parts of Not read at all T-kit 6 Training Essentials Read in full Read parts of Not read at all T-kit 7 European Citizenship Read in full Read parts of Not read at all T-kit 8 Social Inclusion Read in full Read parts of Not read at all If you can, give a comment to your choices; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

7. a) Are the T-kits, good complementary tools to the information and training materials/activities that you provide, in relation to the development of Youth projects or training activities within the Youth programme? (Mark your choice by putting it in BOLD)

Yes No b) Do you recommend it to the target groups? All the time Often Seldom Not at all



If you can, give a motivation to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

8. Do you read the Coyote magazine? (Mark your choice by putting it in BOLD)

Yes No

If you can, give a comment to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

9. Does Coyote, in your opinion, provide a forum for discussion of training related issues amongst Youth workers/trainers?

Yes No

If you can, give a brief comment to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

10. a) Do you provide a link to Partnership website “www.training-youth.net” on your NA/SALTO homepage? (Mark your choice by putting it in BOLD)

Yes No b) How many clicks does it take to reach it from your Start page? (Mark your choice by putting it in BOLD) one to two three to four five to seven eight or more c) Are the words “T-kit” and “Coyote” searchable on your homepage? ? (Mark your choice by putting it in BOLD) Yes No

If you can, give a comment to your choice; ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



11. Do the tools provided by the Partnership assist Youth workers and trainers to give a European dimension to their projects/training activities?

Very well to a good extent a little bit not at all 12. In your opinion, what are the main successes (or positive elements) and failures (or negative elements)

of the Partnership until now? Give a brief comment. ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Annex 4 - Analysis of Questionnaire to NAs/SALTOs NA SALTO Q Analysis 7. a) T-kit Good compl toolReply rate 20/38 53% Yes 100%1. Contribution to quality NoYes 0 b) Do you recommend itTsE 35% All the time 30%No 40% Often 70%? 25% Seldom

Not at all2. Contact with paxYes 60% 8. Do you read Coyote?No 40% Yes 95%

No 5%3. Acknowledgement of needYes 85% 9. Coyote -forum for discussion?No 10% Yes 75%? 5% No 15%

? 10%4. Level of info on PartnershipFully 20% 10.TGE 70% a) provide link to WebsiteDiff to find info 5% Yes 60%Not interested 5% No 35%

? 5%5. Increase of EU cit theme in appl b) how many clicks?Yes 10% 1 to 2 15%No 90% 3 to 4 25%

5 to 7 10%6. T-kits 8 or more 5%T-kit 5 Vol Service ? 5%Full 25%Parts 55% c) Searchability ofCoy and T-kitNot 15% Yes 25%? 5% No 65%T-kit 6 Training essentials ? 10%Full 25%Parts 50% 11. Not 20% Very well 15%? 5% TgE 50%T-kit 7 European Citizenship A little bit 30%Full 10% Not at all 5%Parts 60%Not 25%? 5%T-kit 8 Social inclusionFull 10%Parts 60%Not 25%? 5%



Annex 5 – Comments from National agencies and SALTO resource centres

Comments from National Agencies and SALTO:s in relation to question 12 of the questionnaire:

12. In your opinion, what are the main successes (or positive elements) and failures (or negative

elements) of the Partnership until now? Give a brief comment.

The Partnership is not promoted enough independently from us.The trainings are sometimes limited to people under 30, which doesn’t' always fit the needs of the youth workers. The Partnership enables a forum for the exchange of training essentials and thoughts. ------ + material!!! – more visibility of training contents needed!!! ------ Main successes are quality training courses and publications. Maybe even more activities could be organised. We feel that Partnership is not very known among Slovenian youth workers and could have more promotion. At the NA we would like to get more feedback on Slovenian youth workers that were involved in your activities. ------ I think it is very positive that many of training experiences, materials are gathered in T-Kits. Coyote magazines could be issued on more regular bases. European Citizenship concept is not very clear yet. ------ The relationships between some people didn’t facilitate the collective work; the selection of pax and trainers was not enough transparent and seemed to be more a COE affair than a partnership; but I think it was also the responsibility of the COMM side. ------ T –kits + Coyote: very positive Trainings: too far from daily YOUTH work in xxxxx -----



In order of priority:

1. Partnership website – an excellent tool for information, easily usable and very well built (all necessities included). Good thing to have the trainers info and comprehensive info on wide range of topics.

2. T-KITS – again very good tools for trainers and international projects implementers. The themes have been chosen in good coherence with the needs of the possible targets. Nicely explanatory and give the theoretical background and good tools for practice.

3. Coyote – justifies the expectations. Unfortunately because of the linguistic difficulties amongst grass root level youth work is not that widely spread and read within xxxxx.

4. Training Courses – a helpful way of training resource people and trainers at European level. As the access of xxxxx is really limited (so far) it is hard to evaluate the impact at this point.

5. Expert groups on various developmental issues – very important from European youth policy building aspects.

----- Positive: - joining forces, offer of innovating TC’s, supporting material for all actors involved in European youth work (it doesn’t matter in which framework), … Negative: unclear information and communication, most of the time “very far away” from our daily work, … ----- The youth centres in Strasbourg and Budapest and all the training they offer. ----- The main successes is that this Partnership exists. On the other hand in my everyday work I do not feel the dynamics and development of this structure at all. Should I? ----- In general its useful to acquire experience and knowledge especially in XXXX where there is no youth workers or experience in activities of the Youth programme. ----- + we set up new regional centres to eliminate below negative elements -national propagation strategy within YP and luck of newcomers ----- Positive elements: training activities, tools offered and service. negative elements: Some of the well established trainers are not very tolerant towards trainers with different background or lower experience and prefer to work in fixed teams. -----



An added value of the partnership could be close co-operation and herewith synergy effects between the DYS (CoE) and the YOUTH unit (EC). The YOUTH unit runs the YOUTH programme on decentralised level with NA´s and nowadays also SALTOs, so there is a need to involve those actors as well. Nowadays, the Partnership activities are not co-operation products between those actors, they are products decided and developed on centralised level between the CoE and EC staff only (top down process!). If products and human resources from the partnership activities should be "used" by NAs/SALTOs in a sufficient quantity and quality, all actors need to be involved actively (not only be informed by a short email or a questionnaire) in a much earlier stage than happening at the moment = ownership! My proposal for the next partnership would be to set up more small projects / issues for co-operation between CoE staff and NA/SALTO staff, to facilitate co-operation on “lower” level than the centralised one. This could e.g. be small work groups on priorities. The groups should be open for more NA staff to join, if there would be an interest, because this would increase synergy and multiplying aspects. All above would increase the multiplying effect of products from partnership. Synergy effects would allow actors to work on other important needs, which can not be covered yet due to financial or work load reasons. At the end of the days, youth worker and youth will benefit from co-ordinated and coherent activities from Partnership and from CoE and the YOUTH programme actors. ------ Not consistently visible enough. The resources, though of good quality, are not rooted in any national reality. It means they may not be sufficiently tailored to national needs which vary so much from one country to the next. This is the strength of working through and in partnership with national structures as we do as a national agency – we are able to have a close dialogue with the XXXX field. A comment has been that the approach tends to be too academic and theorethical, more of interest to people working in the field of higher education than hands-on youth workers wanting help in organising projects for and with young people.



Annex 6 – Overview of documentation T-kits

T-Kit 5 International Voluntary Service (2002) T-Kit 6 Training Essentials (2002) T-Kit 7 European Citizenship (2003) T-Kit 8 Social Inclusion (2003)

Coyote

Coyote issue nr 4,5,6,7 and two additional supplements Date Activity Type of

document 8-10/02/2000 Evaluation seminar 1st and 2nd Covenant Final report 23-24/05/2000 TWP 11 Report 16-17 September 2000

T-kit 5 Meeting nr 1 Report

19/09/2000 TWP 12 Report 5-6/12/2000 TWP 13 Report 31/1 – 3/2 2001 T-kit 6 Prep mtg 2 Report 19-21/03/2001 CQDG 3rd meeting, (Draft)

Final report 2-3/05/2001 TWP 14+15 Report 25-26/06/2001 TWP 16 Report 8-9/07/2001 Prep mtg 1 TC I Report 15-19/09/2001 Bridges for training Report 25-26/10/2001 TWP 17 Report 19-21/10/2001 Prep mtg 3 TC I Agenda 21/10/2001-01/11/2001

TC 1 Documentation & Evaluation report

07-11/11/2001 Intr. Seminar ATTE Reported together w. next

11-13/11/2001 Prep mtg ATTE Report 30/11 – 2/12 2001 T-kit 7 EU CIT 1st Prep Mtg Report 9-10/12/2001 Prep mtg ATTE Report 7-19/01/2002 First seminar ATTE Preliminary

documentation 17-18/01/2002 TWP 18 Report 8/03/2002 TWP 19 Report 17-20 March 2002 T-kit 7 Prep mtg 2 Report 21/03/2002 Prep mtg ATTE Report 25-27/03/2002 Evaluation mtg TC I Report 27-28/03/2002 Prep mtg 1TC II Report 3-4/04/ 2002

T-Kit on Social Inclusion 1st Experts’ Meeting EYC Strasbourg

Report

6-17/05/2002 Second seminar ATTE Preliminary documentation



11/06/2002 TWP 20 Report 16-27/06/2002 TC 2 Documentation &

Evaluation report 8-9/10/2002 TC Eval mtg Report 10-11/10/2002 Interim Evaluation Future Strategy (IEFS)

mtg TC I and II and ATTE Report and Meeting summary

17/10/2002 TWP 21 Report 14-16/11/2002 Trainers forum 2002 Report 31/01/2003 Evaluation & Prep mtg ATTE Report 10-12/03/2003 1st mtg Expert group on Quality standards,

Evaluation and Validation Prel. report

22/05/2003 TWP 22 Report 16-18/06/2003 2nd mtg Expert group on Quality standards Prel. report 11-12 /06/2003 Prep mtg 1 TC III Report 26-28/08/2003 Prep mtg 2 TC III Report 2-5/09/2003 Prep mtg ATTE Report



Annex 7 - Terms of Reference

Evaluation of the partnership ... - European Commission · training with the Council of Europe, in...

Documents

Transcript of Evaluation of the partnership ... - European Commission · training with the Council of Europe, in...