Evaluation: Building strategy and knowledge in education programmes

IEP - OSI Conference - January 31 - February 2, 2000

PURPOSE PAPER

The aim of the conference is to promote evaluation activity by National Foundations, which focuses on the systemic impact to be achieved through education strategies. The findings of these studies should intersect with the strategy development process of the Foundations and the Education Sub-Board, providing evidence for successful strategies, tactical options for unfolding strategies, or ideas of new strategies. Ideally then, the evaluation will extend the knowledge portfolio of the Board, and deepen the local knowledge of the strategic process. The conference would thus represent a first step towards putting in place a comprehensive approach for affecting decisions about education funding.

Through achieving its aim, the conference will therefore support the strategy development process. In order to follow up the implementation of the strategies, it will not be enough to assess the success of models, or to retreat in the role of "providing support to schools and to education institutions." Evaluation will focus on the sustainability and systemic change achieved by programmes. It will guide and inform the implementation of strategies and will support the building of capacity for the management of education reforms.

In order to achieve its aim, the conference should provide an understanding of the role and purpose of external evaluation of education programmes, together with the steps in planning and managing it. Internal evaluation and alternative approaches to evaluation will be subject of future conferences.

We will define each one of the topics intended to be addressed at this conference:

1. External evaluation

External evaluation is a formal research-oriented study of a social programme in terms of its purpose and design, or its implementation or its outcomes - or each of these together. A prime purpose of an external evaluation is to make judgements about the efficiency and effectiveness of the programme. External evaluations also usually present other substantiated findings and recommendations in terms of a predetermined evaluation brief, or that should be of interest to the stakeholders of the programme.

A further purpose of external evaluations is to inform financial decision and programme strategy. An external evaluation involves the contracting of an external expert whose approach, opinions and methodologies should reflect a commitment, as far as possible, to fair and objective process.

2. Evaluation management

Evaluation management, for the purpose of this discussion, involves the implementation of clear procedures for achieving both focus and clarity in terms of the purpose of an evaluation; the central evaluation questions; the selection of indicators and suitable methodologies; the interests of various stakeholders, and the expected evaluation outcomes.

In as far as it relates to external evaluation, evaluation management includes: establishing initial criteria for selecting evaluators, a selection procedure, and implementing a process for securing agreement on related financing, schedules and time frames and the allocation of responsibilities to various parties. While the evaluators should manage the evaluation itself, evaluation management, from the point of view of the contracting party and the evaluand, should enable efficiency and maximise learning from the study.

The outline of the main sessions of the conference, as resulting from the ideas listed above is presented below. Each session will consist in plenary presentations, followed by group work on the same topic, and plenary feed-back.

1. The strategic purpose of evaluation

Evaluation integrated in strategic planning: what is its role, and how is it planned. Sustainability, systemic impact of education reforms as strategic goals - evaluation as support in the pursuit of these goals, informing the implementation of strategies. Specific aspects of evaluation of change in education: stakeholder evaluation, descriptive evaluation for "unexpected goals". Need to combine external evaluation with self-evaluation.

2. International developments in evaluation

Types of evaluation used - outcomes of evaluation. Evolution of evaluation techniques. Evaluation of education reforms.

3. Evaluation management; evaluation management of education programmes

Elements of evaluation management. Main documents needed for evaluation management. How to select experts - how to define tasks - what to expect from an evaluation report. Ethics of evaluation.

4. Two international case-studies in evaluation management (Romania, Mongolia)

Detailed examples of evaluation.

5. Measurement and indicators for education programmes

Techniques of evaluation (theoretical and practical aspects). Detailed planning. Selecting, defining, interpreting and analysing indicators.

6. Evaluation planning

Comprehensive planning of evaluation: inclusion into strategy, management planning, techniques.

Conference participants:

Education Directors and National Foundation evaluation experts (and/or Foundation Board members) from across the Network

All staff on the Institute for Educational Policy

Representatives of Network Programmes