Part 5 - Completion and Evaluation
October 2005
Contents
2 Activity completion reporting
- 4.1 Purpose
- 4.2 Programming evaluations
- 4.3 Managing evaluations
- 4.4 Reporting
- 4.5 Capturing the learning
B Evaluation team briefing agendas
1. Introduction
AusGuide Part 5 Completion and Evaluation sets out the completion and evaluation aspects of an activity and follows on from two earlier parts of AusGuide:
AusGuide Part 3 Activity Design
AusGuide Part 4 Activity Implementation.
The activity completion and evaluation stage covers:
- activity completion reporting, which provides a balanced assessment of activity performance against design objectives at the time when Australian funded inputs cease
- activity hand-over, which formalises the transfer of Australian activity assets to the partner government (or counterpart agency) and
- evaluation, which provides a systematic and objective assessment of the relevance, efficiency, effectiveness, impact and sustainability of development activities.
2. Activity completion reporting
Delivery organisations are required to prepare activity completion reports (ACRs) before the end of their implementation contracts. The ACR should be brief and concentrate on key issues. It should serve the needs of AusAID, the partner government and counterpart agencies and beneficiaries.
The preparation of an ACR is typically specified in the design documentation and in the implementation contract for the delivery organisation. It should be scheduled in the last Annual Plan for the activity. ACR preparation will not normally require additional resources to be made available to the delivery organisation beyond those identified in the activity design documentation and provided for in the contract. The Activity Manager is responsible for monitoring the timely preparation of the report.
Some activities will also require independent completion reports (ICRs) to be produced by an independent consultant or consultants. ICRs are undertaken for activities which exceed the relevant financial threshold - for example, the threshold is $5 million for activities in PNG, Vietnam & Indonesia; $3 million for activities in Other Asian, African & Middle East countries and $2 million for activities in Pacific Island Countries (other than PNG).
The ICR will validate the performance data provided in the ACR and identify lessons learnt from a broader perspective. The target audience for the ICR is AusAID's management.
ACRs and ICRs enable AusAID to evaluate the success of an activity, and its wider influence on the aid program, at completion.
ACRs and ICRs contribute to AusAID's aid program by providing:
- accountability for expenditure of public funds
- a check on design quality and implementation performance
- a key input into country program effectiveness reviews and other reviews
- a source of lessons that can feed into program strategies and activity design
- information on the types of actions that may increase sustainability and
- a preliminary analysis for undertaking an ex-post evaluation.
Detailed guidelines on preparing ACRs and ICRs are provided in AusGuideline 5.1 Preparing completion reports. Guidance onaddressing sustainability issues in completion reports are set out in AusGuideline 6.4 Promoting practical sustainability.
The Office of Review and Evaluation (ORE) reviews ACRs and ICRs to identify activities that may warrant ex-post evaluation or which may yield useful lessons. ICRs are added to AusAID's Knowledge Warehouse (AKWa). This database is a source of information for program strategy and activity development.
3. Handover
3.1 Ownership of assets
Under AusAID's standard contracts for activity implementation, supplies (including assets) are purchased through the contract. The Australian Government reimburses the delivery organisation for the cost of such supplies, which are the property of the Australian Government. These supplies remain the property of the Australian Government until ownership is transferred to another party by agreement between the Australian Government and that other party. Usually, ownership of assets purchased by the Australian Government is transferred to the partner Government at a suitable point in activity implementation.
Many umbrella agreements for development cooperation with Australia's major partner governments contain standard provisions for transferring ownership of assets from the Australian Government to that partner.
One standard arrangement is for asset ownership to be transferred to the partner Government at the end of activity implementation. However, other umbrella agreements provide for ownership to be transferred at the point at which the asset enters the territory of the partner country.
In either case, there is usually an agreement that activity supplies are to be used for activity implementation during the life of the activity, and are only available for other uses after the activity is completed. This agreement may be made through the umbrella agreement, may be an informal arrangement or may be embodied in customary practice.
3.2 Handover note
A handover note is prepared at the completion of activity implementation. The handover note lists the activity records and activity assets (and other activity supplies) which are still in existence at the completion of the activity. The handover note records the agreed disposition of these records, assets, and any other supplies, subsequent to activity completion. This includes a record of the agreed ownership of activity records, assets and other activity supplies after completion.
The delivery organisation may be asked to prepare a draft of the handover note. The handover note should be prepared for signature by the Australian Government (usually AusAID at post) and by the partner organisation which takes post-activity ownership of the records and assets.
The delivery organisation is required to maintain an asset register over the life of the activity which records the status of all assets purchased with Australian Government funds. The list of assets in the handover note should be based on this asset register.
The Australian Government (usually AusAID) and the other signatory organisation should retain signed originals of the handover note, and a copy should be attached to the ACR for information. A sample hand-over note format is provided in Annex A.
3.3 Contract compliance check
Once the delivery organisation has completed the delivery of all the services under the contract (including all its inputs to the ACR and ICR), the Activity Manager checks that the scope of services set out in the contract has been fully executed. The compliance check should include a reconciliation of all payments against the contract sum. It must be completed prior to finalisation of payment to the delivery organisation.
When action is completed on all contracts, the Activity Manager should change the status of the activity from 'approved' to 'completed' on AidWorks (formerly the Activity Management System).
4. Evaluation
4.1 Purpose
Evaluation is the systematic and objective assessment of the relevance, efficiency, effectiveness, impact and sustainability of development activities. Ex-post evaluations are undertaken after the activity has been completed, typically around two to three years after activity completion.
While evaluation is generally considered as the last assessment process in the life of an activity, it does have implications for the design of future activities. No development activity should be planned or undertaken without first reviewing the lessons of past experience.
Evaluation aims to improve AusAID's policy analysis as well as the design and implementation of future AusAID activities by the identification and adoption of lessons learnt. It assesses:
- relevance: the extent to which the objectives of the development intervention were consistent with beneficiaries' requirements, country needs, global priorities and partners' and donors' policies
- efficiency: how economically resources/inputs were converted to results
- effectiveness: the extent to which the development intervention's objectives were achieved, taking into account their relative importance
- impact: the positive and negative, primary and secondary long term effects produced by a development intervention, directly or indirectly, intended or unintended and
- sustainability: the continuation of benefits from a development intervention after major development assistance has been completed.
It also draws out:
- lessons: generalisations based on evaluation experiences with activities, programs or policies that abstract from the specific circumstances to broader situations. Frequently lessons learnt highlight strengths or weaknesses in preparation, design and implementation that affect performance, outcome and impact and
- achievements in poverty reduction: the impact of the activity on poverty reduction.
Evaluations are also used to provide a measure of accountability in the aid program.
Evaluations are frequently undertaken for a cluster of activities in a country or sector, but may also be undertaken for single activities/programs.
4.2 Programming evaluations
Activities evaluated should be representative of the aid program as a whole. There should be a focus on those activities which can provide useful lessons on the factors affecting both success and failure, without undue emphasis on either successful or problem activities.
The Program Evaluation Section of ORE (EVAL), in consultation with program areas, develops an annual program of 'Branch-managed' evaluations. EVAL also develops and implements an annual program of higher level (thematic, program and policy) assessments which is approved by the Executive. The evaluations are normally highly responsive to AusAID's immediate and longer term needs.
Evaluations are generally carried out by independent consultants or consulting companies with no prior involvement in the design and implementation of the activity.
4.3 Managing evaluations
4.3.1 Preparation
Evaluations are undertaken by an Activity Manager in EVAL or in the geographic area tasked with implementing a 'branch-managed' evaluation. The Activity Manager:
- consults widely on the issues pertaining to the activities or theme to be assessed and prepares an Issues Paper which discusses the major issues and how they are to be addressed in the evaluation
- convenes a small Peer Group, comprising members from the relevant program areas and posts, ORE and sector advisers. This Group discusses the Issues Paper and recommends how the evaluation should proceed
- developsthe terms of reference (TOR) for the evaluation. These outline the objectives for the evaluation, its scope, the method to be employed, the composition of the Evaluation Team, the resources needed and the reports expected as the outcome of the evaluation. Annex A of AusGuideline 5.2 Undertaking evaluations provides guidelines on the format of TOR for evaluations and
- if appropriate, consults with other donors and/or the counterpart government to assess the possibility of joint evaluation or other forms of collaboration.
AusGuideline 6.2Planning and managing in-country missions provides advice on how to conduct in-country missions, including evaluation missions.
4.3.2 Selecting an evaluation team
Once the TOR for the evaluation have been approved, the evaluation Activity Manager selects the team or individual to undertake the study. The team may be selected by putting the evaluation out to public tender, or by selecting the team members from AusAID's period offers, following normal AusAID selection procedures. Contracts are drawn up with advice as necessary from Contract Services Group (CSG).
The Activity Manager also consults with the post in drawing up the scope of services for partner government representatives and in arranging for their availability. The Activity Manager should work closely with the evaluation study leader and consult also with the relevant desks and posts in drawing up a work plan and a list of agencies and/or people to interview. However, Evaluation Team leaders will need to exercise their own judgement in defining contacts.
4.3.3 Briefing the evaluation team
Briefing is in two parts:
- The Activity Manager provides relevant activity reports, file extracts and other relevant data to the Australia-based Evaluation Team members in Canberra prior to the field study. Relevant AusAID officers are asked to brief the team on specific aspects and telephone interviews with former activity managers may also be arranged. Normally the members of any activity advisory group (e.g. TAG) would be invited to participate in the briefing. Following this briefing, the Evaluation Team develops the Work Plan and interview guides for the field study, in consultation with the Activity Manager and the post and
- On the Evaluation Team's arrival in country the post briefs the full team (comprising both Australian and partner government personnel). The post provides the team with any relevant additional background information about the activity. This particularly includes their experience with the delivery organisation and the implementing agency during implementation, and any current institutional development within the implementing agency that is affecting the realisation of the outcomes of the activity. At this briefing, the post also advises the team of progress made or difficulties encountered in establishing the work program for the team in-country. Interviews or meetings arranged are outlined and a draft itinerary provided.
Following these initial briefings, the Evaluation Team commences other meetings, site inspections, data collection and analysis, and interviews with key informants and beneficiaries as set out in the Work plan.
4.4 Reporting
4.4.1 Formats
The key elements of the report cover the issues of relevance, efficiency, effectiveness, impact and sustainability, with a final chapter on conclusions and lessons learnt. Details of the evaluation report format are set out in AusGuideline 5.2 Undertaking activity evaluations.
The first draft, and all subsequent, reports produced for circulation and comment must include a disclaimer that the views expressed are those of the Evaluation Team and do not necessarily reflect the views of AusAID or the Australian Government.
4.4.2 Reviewing, revising and finalising documents
On the conclusion of field work, the Evaluation Team prepares an aide memoire which includes a summary of the findings of the evaluation. The aide memoire is used as the basis for a debriefing session in country involving the line agency, aid coordinating agency (or agencies) and the post.
Following the return of the Evaluation Team to Australia:
- the Team prepares and forwards the draft report to the Activity Manager for review. Comments are sought from the Peer Group established for the evaluation and the team holds a debriefing seminar on the evaluation findings
- the draft report is revised following consideration by the Evaluation Team of the comments provided by AusAID
- the revised draft report is submitted to the partner government, former AMCs and other interested parties for their comment
- the draft final report is prepared by the Evaluation Team after consideration of the comments from the AMCs and counterparts
- the report is submitted to the relevant Branch Head for 'approval', and may be submitted to the Executive to 'note' if it is of sufficient importance
- in selected cases, the evaluation Activity Manager will draft a submission referring the report to the Minister for information and
- the report is normally published and posted on the AusAID internet, intranet and AKWa.
Further information on the approval process for evaluation reports is included in AusGuideline 5.2 Undertaking activity evaluations.
4.5 Capturing the learning
Feeding back lessons learned is the most important stage of the ex-post evaluation process. Lessons learned must be fed into policy and ongoing country programming processes to assist in making future activity design and management activities more efficient and effective.
To assist this process, the Activity Manager should develop a distribution strategy for the report, to ensure that it reaches key individuals and agencies in Australia and partner countries.
A. Sample hand-over note
Addressee: (in the counterpart agency or coordinating agency)
Activity title:
Advise that the [name of activity] activity has now been completed in accordance with the agreement expressed in the memorandum of understanding dated [include date]. Include reference to the status of the completed activity and provisions for its sustainability (e.g., training and ongoing maintenance provisions). Reference arrangements in place or proposed for on-going partner government (agency) responsibilities in this regard.
Briefly refer to assets purchased during the life of the activity and available for transfer to the partner government (agency) as mutually agreed. Refer to relevant paragraph in the MOU indicating that these assets are now the property of the government of [insert country name]. Attach detailed list of assets.
Request signature of appropriate government representative, signifying acceptance of the assets, and request one of the signed originals be returned to the post.
Name and signature of authorised AusAID officer
Date
Name and signature of authorised officer of partner government agency
Date
B. Evaluation Team briefing agendas
Evaluation Team briefing agenda in Australia
- The aims and objectives of the evaluation study
- History of the activity
- Relevant documents are given to team members. These should include, where available, the feasibility study/activity design documentation, appraisal, review reports and activity completion report. Other documents such as consultants' technical reports and training course evaluations should also be made available.
- Australian government policies and the relevance of the activity in the country program.
- Australian and partner government policies that were in effect during the life of the activity, such as poverty reduction, the environment, gender and population.
- Specific information on the country and sector.
- The activity as it was implemented. The Evaluation Manager can use the Logical Framework in presenting a summary of the activity. The briefing should consider:
- activity objectives
- logic of the activity structure and activity components
- issues of sustainability
- database, data sources and specific AusAID data requirements and
- areas of strength and weakness in the activity. The findings of previous review activities, if any, should be used.
- Terms of Reference (TOR) for the evaluation
- approach
- work plan, tasks and list of persons identified for interview.
- Reporting requirements, as specified in the TOR.
Joint Evaluation Team briefing agenda in country
- Brief history of the activity.
- Partner government and Australian perspectives, policies and related activities.
- Consideration of the activity from the partner government point of view.
- Evaluation TOR:
- Discuss the program and tasks for individual team members, particularly to the extent that local team members may take on specific responsibilities under the evaluation.
- Evaluation methodology:
- Discuss with the partner government team members the approach developed during the briefing in Australia and modify as appropriate.
- Itineraries and support services:
- agree on individual itineraries and means of arranging transport and accommodation;
- identify agencies which can support individual team members; and
- finalise arrangements for other support services (including report preparation).
- Report preparation and production. Describe and discuss responsibilities and means of preparing working papers and the draft evaluation report.
Associated AusGuidelines
- AusGuideline 5.1 Preparing completion reports [PDF - 232KB]
- AusGuideline 5.2 Undertaking activity evaluations [PDF - 158KB]
About this website | Disclaimer | Privacy | Feedback
