Baseline and Endline Evaluation Consultant

Job DescriptionJob Description:

Abbreviated Scope of Work: Combined Baseline and Endline Assessments for Three USAID/BHA Awards in Ethiopia Reports To: Maya Rao, MERL Advisor Coordinating with: Needa Malik, Disaster Risk Manager Estimated Level of Effort: 55 days Period of Performance: May 27 – Aug 9, 2024 Country of Assignment: Remote BACKGROUND Plan International USA is a girls’ rights organization. Powered by supporters, we partner with adolescent girls, children and young women to overcome oppression and gender inequality. We provide the resources that are unique to their needs and the needs of their communities, ensuring they achieve their full potential with dignity, opportunity and safety. Plan is currently implementing two USAID/Bureau for Humanitarian (BHA) projects: one in Borena Zone of Oromia Region and one in Northern Ethiopia across the three regions or Amhara, Afar and Tigray. The project in Borena provides child protection assistance to drought-affected communities through a combination of in-kind goods and services and voucher assistance. The project in Northern Ethiopia provides multisectoral assistance to internally displaced persons (IDPs), people who have returned from displacement (returnees), and households hosting displaced and returnees (host communities). Sectors of assistance include child protection, nutrition, health, and water, sanitation and hygiene (WASH), delivered through in-kind goods and services and voucher assistance. Both the project in Borena and Northern Ethiopia are closing out in June and September 2024 respectively, with a new award starting up in May 2024. The new award will combine the Borena and Northern Ethiopia responses under one project; however the scope will be reduced, covering a subset of woredas from the prior two awards. Child protection activities will continue only in one woreda in Borena zone while multisectoral interventions will continue under a reduced set of target woredas in Amhara and Tigray Regions (Afar is removed). With the closeout of the current Borena and Northern Ethiopia projects, BHA M&E guidelines require a final evaluation of the Northern Ethiopia project, while the Borena project requires only an endline assessment, with indicators to be submitted in the indicator tracking table (ITT) as part of the final performance report. As the new award is starting up, Plan is also required to collect and report baseline data to BHA within the first 90 days. Given the overlap in implementation sites and most indicators maintained from the prior to new award, Plan aims to combine the prior award endline data collection and new award baseline collection into one assessment managed by one consultant in collaboration with Plan Ethiopia project teams on the ground. SPECIFIC TASKS Background document review (ITT, BHA M&E guidelines, project award documents, baseline reports and terms of reference, detailed Statement of Work Annexes (see below) Develop work plan and sampling strategyClean and analyze baseline and endline data for Borena endline, Northern Ethiopia final evaluation, Review and monitor the database throughout data collection and share updates and issues with field staffDraft and finalize final evaluation and baseline study report for Northern Ethiopia current and new awards DELIVERABLESTerms of Reference with detailed sampling plan, budget and workplan for Borena endline, Northern Ethiopia final evaluation and new award baseline. Due by June 3Borena project data analysis and results write-up – As a final evaluation is not required for this project, a draft write-up of methodology and key findings is sufficient for internal documentation. Due by June 28.[1]Northern Ethiopia Final Evaluation Report – Due in early August[2] New Award Baseline Assessment Report – Due in early August2 Cleaned datasets and syntax files for Borena endline, Northern Ethiopia final evaluation, New Award baseline. Due in early August.2 Roles and responsibilities of the consultant The consultant will lead the technical aspects of the evaluation, includingDocument reviewRefinement of data collection tools and sampling strategyDevelop/update guidance for data collection and data quality considerationsProvide technical support in facilitation of enumerator trainingReview pilot test data and support any further adjustments needed for toolsReview incoming data during collection and provide progress reports and data quality issues to the field teamsData cleaning and analysisReport writingPresentation of results in an online/hybrid validation meeting with core project team (time permitting) Roles and responsibilities of Plan International Plan Ethiopia will provide the following support for the final evaluation: Review and propose adjustments to data collection tools Update digital tools according to any revisions neededDevelop handout for enumerators to support local translation of key termsHire, train and manage enumeratorsLogistics for enumerator training, pilot testing and data collectionProviding relevant and clear information to the consultant for background, planning, regular updates during data collection and debriefing for data cleaning and interpretation of resultsPrepare local government stakeholders and community representatives for facilitating access and mobilizing communities and providing guidance to the team for implementing the evaluation activities on the groundReview report and provide feedback to consultant for any revisions neededArrange logistics and invitations for participants to results validation and dissemination meetings. Key qualifications for the consultant The evaluation consultant should have the following skills and experience:A minimum of 5 years of evaluation experience, including substantive involvement in reviews at program and/or outcome levels in related fields with international organizations.Familiarity with the policies, programs and operating styles of the local and national governments that are participating in the projectUnderstanding and sensitivity to the diversity of cultures, traditions and languages;Excellent proficiency in spoken and written English. Advanced knowledge and experience on gender and safeguarding issuesExperience working with communities using participatory approachesSolid methodological and research skills HOW TO APPLY: Please submit your resume/CV along with a technical offer describing your methodological approach, workplan and team composition, as well as a financial offer. Annex1: Abbreviated Terms of Reference: “Integrated Multi-sectoral Response to Critical Needs of IDPs and Returnees in Northern Ethiopia” (current award) and “Strengthening community resilience through addressing the critical multi-sectoral needs of returnees, IDPs and host communities in Ethiopia” (new award) Evaluation Purpose The purpose of this evaluation is to assess the performance and effectiveness of the project activities undertaken throughout the current award from USAID’s Bureau for Humanitarian Assistance (BHA) to achieve project outcomes and to establish baseline measurements for the new award. Endline measurement of outcome-level indicators will be integrated as part of the performance assessment. Endline results in project sites to be continued under the new award will also be considered as baseline measurements. The evaluation plan will also include data collection in new project sites under the new award to collect baseline measures in the new implementation areas. The evaluation team will also examine unintended consequences of the project interventions to inform future emergency assistance in Northern Ethiopia. The overall objectives of this evaluation areTo evaluate the project in terms of relevance and effectiveness, with a strong focus on comparing achievements against project targets,To collect baseline measurements under the new award,To identify key successes, gaps, and constraints to be addressed in future humanitarian programming andTo document good practices within the current programs and to generate evidence-based lessons learned that can inform and strengthen the strategies of both ongoing and future programs.Evaluation Type This is a mixed-methods, performance evaluation, primarily focused on assessing the extent to which outcomes were achieved and collecting baseline measurements for the new award. The evaluation will also examine the project’s implementation processes in terms of their gender responsiveness, inclusiveness and relevance to the local context.Evaluation Questions Performance: To what extent have the activity’s interventions adhered to planned implementation (e.g., schedules, participant targeting, service delivery, and outputs) and achieved intended outcomes?Effectiveness: To what extent has the intervention appropriately assisted women, men, boys, girls, people with disabilities and other vulnerable groups affected by the conflict in Northern Ethiopia? How did the project prioritize and engage these groups in design and implementation?Challenges and action taken: What were the key barriers or challenges to program implementation? How were they addressed?Baseline status: What are the baseline levels of key performance indicators in participating communities under the new project award? The evaluation team will aim to highlight best practices and lessons learned under questions 2 and 3, in terms of engagement strategies that worked well and innovative solutions to barriers that could be replicated and scaled for future interventions. Evaluation Methods This evaluation will use mixed methods, including project document review, participant-based surveys, direct observations, key informant interviews (KII) and discussions with male and female focus groups. Quantitative Data Collection The quantitative data will be collected using a household questionnaire for both child protection, WASH, health and nutrition indicators. The tool will be administered using an electronic mobile phone platform (Kobo Tool Box). The survey will include demographic questions to determine if a household has children 0-5 months, 6-23 months. If the household has a child in this age group, the enumerator will request to interview the mother of these children, as she would be best placed to provide accurate responses to questions related to indicators N08 and N09. The demographic questions will also screen for women of reproductive age. These women would then respond to the sub-questionnaire on dietary diversity to measure indicator N10. All other questions related to WASH and health indicators may be answered by the head of household or any representative of the household above the age of 18. The respondents for the child protection household questioners will be children within different age categories of 5-9, 10-14, and 15-18 years old with the questions and interview length adjusted for younger age groups. An overview of the indicators to be measured, corresponding tools and respondent type are presented in the table below. Note that one WASH indicator related to sanitation practices has been replaced under the new award with a different indicator (W13 replaced with W12). Sampling strategy for child protection indicator CO2a: Two objectives influenced the planned sample size: achieving an acceptable margin of error (MOE) at project level but having no less than 20 observations in each stratum to make the data useful at regional level. A total sample size of 558 was thus established at the 95% confidence level, distributed over the three regions proportionally based on the size of each region’s client base. Each regional sample is equally distributed over six age-sex strata, namely boys aged 5-9 years of age, girls aged 5-9 years, boys aged 10-14 years, girls aged 10-14 years, boys aged 15-18 years, and girls in this age bracket. See table below for the planned sample distribution. Borena zone of Oromia, which was previously under a separate BHA award will be integrated under the new award, with child protection activities continuing in one Woreda only. The sample size for this integrated zone for child protection baseline data collection will be finalized in discussion with the consultant. Qualitative Data Collection Qualitative data will be collected through focus group discussions (FGD) and key informant interviews (KII). The FGDs will be held with children and adolescents participating in child-friendly space (CFS) activities. The focus groups will be designed to measure indicator CO2.b (percent of children and adolescents participating in child-friendly space (CFS) activities who report an improved sense of safety and well-being at the end of the program). Selected HF staff, Government Bureau representatives and community leaders will serve as the key informants and be interviewed using a KII guide. The guide for each group of respondents focused on the overall performance of the project. The questions will include a critical reflection that allows respondents to mediate on both project success and challenges and to capture new knowledge and actions for the future. FGDs and KIIS will focus first on what worked well within the project and participants’ suggestions for improvements. These elements will be translated into a Strengths and Opportunities, Weaknesses and Threats (SWOT) Analysis which will inform and enhance the sustainability and development of future phases of the project. For example, strengths and opportunities could be leveraged and scaled in future iterations of the project, while weaknesses and threats could be addressed through adaptations to future project designs to mitigate those challenges. Qualitative sampling: The sampling for the FGD will be a purposive selection of children in the age groups 5-9, 10-14 and 15-18 years old who have been enrolled in CFS and attended at least 8 CFS sessions to ensure diverse perspectives from children who have actively participated in the programs. The FGDs will have 8-10 participants, with separate focus groups for girls and for boys for each of the age groups. The discussion length will be shorter for the younger groups. The KIIs will be purposefully selected participants from health facilities, project staff and local stakeholders—such as community leaders, focal points, volunteers supporting implementation—based on their involvement in child protection, WASH, health and nutrition project activities. For feasibility, the FGDs and KIIs will take place in the same kebeles sampled for the quantitative surveys. Analysis Requirements Quantitative data will be analyzed with MS Excel and SPSS. The analysis will include descriptive statistics and comparison of results from baseline and endline. Qualitative data will be analyzed using qualitative software, such as NVivo or Dedoose. The analysis will use codes to identify and extract key themes, and illustrative (deidentified) quotes will be integrated into the presentation of res

---