Consultant Midterm Evaluation

1. Background

 

In July/August 2010 the most devastating floods in the history of Pakistan covered and ravaged one third of the country. Almost 2.000 people were killed and 3.000 injured; at large, more than 20 million people in 78 out of a total of 141 districts from the Himalayas to the Arabian Sea were affected, with millions losing their property, houses and livelihood. The humanitarian needs are exacerbated by political instabilities, including a volatile security context, unpredictable international relations and shrinking humanitarian space.

Although present since 2002, it was from the 2005 Kashmir earthquake that DKH’s involvement in the country intensified, and since the onset of the devastating floods in the summer of 2010, DKH became active in Khyber Paktunkhwa (KPK) and Punjab Provinces. In cooperation with local partners, DKH has supported Emergency Aid, Rehabilitation, and Disaster Risk Reduction (DRR) projects.

In response to the flood 2010, Diakonie Katastrophenhilfe has to date allocated approximately €14.000.000,- from own funds and different donors, and as of May 2012 there are 19 ongoing projects addressing different needs such as livelihood recovery, WASH, DRR, and the rehabilitation of housing and basic infrastructure. Diakonie Katastrophenhilfe thereby aims to mainstream DRR and gender sensitivity in all of its operations. Even though all projects are thus far carried out with local partner NGOs, Diakonie Katastrophenhilfe currently employs approximately 75 staff working in three project offices in Pakistan to contribute to the organization, administration, and expertise of the projects.

 

2. Objectives of the evaluation

 

This evaluation is carried out two years after the start of the flood response, with two years remaining in the mission strategy[1], and is therefore considered a mid-term evaluation.

 

1)    This evaluation shall assess the DKH response to the 2010 Pakistan floods with respect to the evaluation questions below. It shall thereby identify the main strengths, weaknesses, and gaps in the DKH response to the flood 2010 in Pakistan.

2)    The main objective of this evaluation is to ascertain lessons learnt and recommendations for improvement of the planning and implementation for the remaining two years of the mission and of future emergency response missions more generally. Identified constraints shall be expressed in conjunction with concrete recommendations for points of action. A particular focus shall be to analyze DKH in the way it works with its partners, the adequacy of its monitoring & evaluation system and the efforts which are made to strengthen local capacities. The evaluation shall also serve to provide transparency and accountability to beneficiaries, donors, and stakeholders.

 

Audience and use of the Evaluation

 

The evaluation of the Pakistan post-flood intervention is carried out to learn from the first phase of the response and to adjust the ongoing and future program. Therefore, field staff and implementing partners, as well as headquarter personnel will be the main audience of this evaluation. They will use the evaluation results to modify and plan the next phases of the ongoing program accordingly. The evaluation shall also be made available to the public for accountability and transparency.

 

3. Evaluation criteria & questions

After providing an overview of the context and program, this evaluation shall assess the DKH Pakistan mission with respect to the questions below.

 

a)   Relevance and appropriateness:

Were the needs indentified through DKH assessments amongst the priority needs of the communities? Did and do the defined objectives of the implemented projects address those needs adequately? Were the beneficiaries satisfied with the assistance provided? What real difference has the operation made to the beneficiaries? The DKH country strategy Pakistan has been formulated in the course of the 2^nd year of implementation, and shall guide the remaining mission. To what extent are its objectives valid? How does the mission fare in its goal to adhere to the wider principles such as Code of Conduct and Sphere standards? What are the factors adversely affecting this goal, and how can this be addressed?

 

b)    Effectiveness & timeliness

How effective are the projects in the different program sectors in furthering their main goals, i.e.:

• Reducing food insecurity and poverty (Livelihood)?
• Reducing mortality thanks to better hygiene conditions (WASH)?
• Reducing the vulnerability of the target population through access to housing and basic infrastructure such as Basic Health Units (Habitat)?
• Reducing the negative impact of disasters (DRR)?

 

What are the short or medium term, intended and unintended, expected and unexpected outcomes of the examined projects? To what extent is the selected target group reached? What are the major factors influencing the achievement or non-achievement of the objectives, i.e. what would have to be changed in order to achieve them? Was the implementation of the response timely?

 

c)            Efficiency

Are the financial resources and other inputs efficiently used to achieve results? How does the specific structure of the DKH Pakistan office, for example the special circumstance that procurement for projects is done by DKH Pakistan rather than implementing partners, enhance or weaken the efficiency of the mission? Which changes are advised in order to make the mission more efficient towards achieving its goals? To what extent is the complex Pakistani context influencing the efficiency of the program, and is this adequately dealt with?

 

d)            Coverage

Was the identification of beneficiaries a fair process, i.e. were the selected beneficiaries amongst the hardest hit by the flood and most vulnerable? What efforts were made to ensure that the most vulnerable groups and areas were not overlooked? What efforts were made to address the DKH mainstreaming goal of gender issues?

 

e)            Coherence

To what extent is it possible to align the multitude of projects, sectors, and implementing partners into a coherent set of strategies and principles as detailed in the DKH mandate and strategy? To what extent has this been achieved so far, and what needs to be redressed?

 

f)             Impact

Are there any indicators pointing to probable longer term impacts of the operation in relation to the situation of the beneficiaries, including positive, negative, impacts produced by the operation, directly or indirectly, intended or unintended? What wider effects of the operation on individuals, different gender- and age groups, communities and institutions might be expected? Are DKH-supported projects expected to have an impact with regard to the central mainstreaming goal of sustainable reduction of disaster risk?

 

g)            Connectedness & Sustainability

What are the major factors which influence the achievement or non-achievement of sustainability of the program? What efforts are made to build on local capacities? What efforts are made to strengthen local implementing partners? Is there an adequate balance between the partners’ and DKH’s input to project design and implementation? Do partners feel adequately trained and supported by DKH? What efforts are made to include the local stakeholders, such as government, population and beneficiaries, in the planning and implementation of the projects? Are these adequate and successful? What efforts are made to increase awareness of the importance of DRR with relevant stakeholders in Pakistan? To what extent will the positive effects or changes of the program last? What efforts have to be made to ensure continuation of the benefits from the operation after its completion? How adequate and successful is the current M&E mechanism to verify the above, and how can it be improved? Does an exit strategy exist (including time plan and how to hand over responsibilities to other institutions)?

 

h)   Accountability

Are the intentions of DKH and implementing partners sufficiently presented and explained to the beneficiary communities? Is enough space provided for the input by beneficiary communities, including adequate complaint mechanisms?

 

4. Evaluation Methodology

 

The evaluation methodology will include a review of relevant documents (proposals, reports, policies), interviews of key informants, DKH staff, and implementing partners; as well as focus group discussions with beneficiaries and other stakeholders involved, such as the responsible government agencies and departments.

It will be impossible to consider all projects of the mission in detail; however a representative number of projects shall be visited and assessed by the evaluators. The projects to be visited shall proportionately represent all sectors of intervention, and closed as well as ongoing projects, and they shall be selected together with DKH staff. The evaluation team will use participatory and gender sensitive approaches to find out the views of the beneficiaries, staff, local partners and other stakeholders.

A more detailed description of the methodology needed to best complete the tasks of this evaluation shall be established by the evaluators and included in the inception report. 

 

5. Elements of the Evaluation

 

Evaluation Team

 

The evaluation team shall consist of one expat evaluator and two Pakistanis – of which one man and one woman. One of the three shall be the team leader. Additionally, one DKH Pakistan staff shall be the focal person for Pakistan and provide them with the information and support necessary to successfully carry out the assessment.

 

The evaluation team is answerable to the Head of Mission (HoM) and the responsible project officer at headquarter. The field projects and visits will be decided by the evaluation team in consultation with the HoM and headquarter. The logistics and security department in Islamabad will monitor the security situation at the time of evaluation, and shall take appropriate steps when deemed necessary.

 

Reporting

 

1)    Inception Report: Before the evaluation takes place, an inception report shall be produced. Given the wide range of evaluation questions, it shall condense the scope of the evaluation to what can realistically be expected from the final report. The inception report shall also further specify the most appropriate evaluation methodology.

2)    A formal interim report is not expected. However, the HoM and project officer at HQ shall be informed of any important issues regarding the evaluation or the implementation of the projects.

 

3)    Draft/Final Report: The final report shall be structured as specified below. Recommendations shall be a crucial part of this evaluation. The evaluation team should identify and state which DKH practices serve as good and innovative practice examples, which have an added value and contribute towards reaching the strategic goals. Likewise the evaluation team should identify which general lessons can be drawn from the hitherto practice evaluation and which points should be included in the prospective (re-) framing and planning of the program. This shall help DKH staff to plan concrete points of action. A time plan will be established within which the points of action have to be realized, and it will be the HQ's and HoM’s role to oversee this after the evaluation.

 

The structure of the final report shall be as follows:

 

1. Table of contents

II.         Executive summary (max. 2 pages)

1. Introduction: Context & Program Description

III.1  Political and socio-economical context in Pakistan

III.2  Description of the DKH Pakistan mission 2010-14; its main objectives, program sectors, projects, implementing partners, and target groups

IV.        Evaluation objectives, design, and methodology

V.        Key findings & Lessons learnt according to the evaluation criteria and questions posed.

VI.       Recommendations for points of action

VIII.     Annexes: (ToR, Timetable, Persons interviewed, References, …)

 

Geographical area of Evaluation

Subject to the security situation at the time, it will be required that evaluators travel to the DKH Pakistan project offices and some project sites of the DKH Pakistan-funded operations, i.e. Islamabad, Khyber Pakhtunkhwa Province (KPK).[2]

The Language of the evaluation is English.

Timetable

The field visit to carry out the main part of the evaluation shall take place after Ramadan 2012 (ends 20.08.2012), which means at the end of August and in September.

 

No	Duties and Responsibilities	Days /Person	Tentative Time-frame
1	Preparation: Kick-off, analysis of relevant documents, development of evaluation design and methodology.	4	Mid/End July
2	Preparation and handing over of inception report to Diakonie Katastrophenhilfe. Acceptation of inception report by DKH within 10 days	Team leader 2	By 10.08.2012
3	Implementation of assessment in Pakistan	14	Between 21.08.2012-12.09.2012
4	Work-shop in Islamabad: Feed-back of the preliminary findings.	1
5	Preparation of draft report.	5/ 7 for team leader
6	Hand over draft report to DKH		30.09.2012
7	Feed-back on the draft report by DKH and other stakeholders		20.10.2012
8	Hand over of final report to Diakonie Katastrophenhilfe	1	31.10.2012

 

: