TERMS OF REFERENCE EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME INTERVENTION DISTRICTS IN MALAWI - UNGM
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
TERMS OF REFERENCE EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI 1. BACKGROUND Malawi is a landlocked country with a population of nearly 17 million people – which is estimated to surpass 20 million in the next five years.1 Malawi has 28 administrative districts, which are further divided into traditional authorities (TA) and villages, the smallest administrative unit. Malawi’s economy has expanded over the past 30 years, with real GDP growth estimated at 2.9% in 2016. It remains predominantly an agricultural country, with agriculture, forestry, and fishing contributing 28% of GDP. Currently, GDP per capita is approximately $380, and given that inflation and population growth currently outpace economic growth, average living standards are falling. In 2010-11, 29% of households lived under the international poverty line of $2 per day. Poverty remains particularly prevalent in rural areas, where more than 80% of the population live. Despite recent achievements, Malawi has not yet achieved optimal health outcomes. Life expectancy remains low at 61 years. Over half of the country’s total disability-adjusted life years (DALYS) are a result of the top four leading causes– HIV/AIDS, lower respiratory infections, malaria, and diarrheal diseases. Malawi has reduced its child mortality rate, leading to achievement of Millennium Development Goal (MDG) 4. However, other indicators remain stagnant or even face declines. The leading cause of morbidity in under five children are Malaria (29%), diarrhoea (22%) and Acute Respiratory Infections (ARI) (5%) (MDHS, 2015). Nationally, 52% of total disease burden can be attributed to sanitation related factors. Estimates indicate that Malawi loses US$57 Million annually due to poor sanitation (HMIS, 2015). Hence, maintaining sanitation is central to increasing quality of life and years of healthy life of Malawians (HMIS, 2015). The Malawi health sector operates under a decentralised system guided by the Local Government Act (1998). The Act delegates authority and funding from central government ministries to district assemblies, who guide health sector planning, budgeting, procurement, and service delivery at district and community levels. At central level, the Ministry of Health (MoH) sets strategic direction and formulates sector-wide governing policies. 29 district health offices oversee services provided in and outside of the district hospital. Five Zonal Health Support Offices (ZHSOs) provide technical support to districts in planning, delivery, supervision, and monitoring of health services. In total, the percent of government expenditure going toward public health averaged 10.4% from 2012-13 to 2014-15, well below the Abuja target of 15%. Therefore, donors have contributed the majority of resources for the health sector in recent years. 2. EVALUATION OBJECT The Government of Malawi’s Ministry of Health with support from Korea International Cooperation Agency (KOICA) through UNICEF implemented a quality improvement of maternal and newborn health services in high priority districts. In 2015 the 5 target districts had an estimated total population of 4.2million people with about 51.6% being women, while children below the age of 5 years, adolescents between the age 10 to 19 years and women of child-bearing age (15 to 49 years) account for about 0.7, 0.9 and 1 million of the population, respectively. The estimated annual number of births in 2015 was 210,000. The project considered gender as a key cross-cutting issue and thus planned to develop forums where communities’ voices could be raised in a constructive way, and promote engagement and leadership from women and youth, including supporting activities and accountability mechanisms featuring discussion of gender issues. 1 All information in this section comes from the HSSP II, the NCHS Situation Assessment, and the Malawi Demographic Health Survey; please refer to these documents for sources. 1|Page
TERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
Table 1: Project summary
Project Name Improving Maternal, Newborn and Child Survival in High
Priority Districts in Malawi
Project Location Malawi, districts of Nkhata Bay, Dedza, Mangochi, Thyolo
and Blantyre
Project reference number SC170390
Project budget KOICA: USD4,499,608.00
UNICEF: USD448,661.00
Total: USD4,948,269.00
Donor(s)/ funding sources KOICA and UNICEF.
Project duration 25 July 2017 to 31 December 2019
Implementing partners 1) Paediatrics and Child Health Association (PACHA)
2) District Councils in the 5 focused districts
3) Maikhanda Trust
Key roles of implementing partners are summarised in table 2 below.
Table 2: Roles of key partners in the project
Partner Roles in the RMNCH project
Name
1 Paediatrics - Improve coordination of neonatal care activities at the National and district level
and Child and across the continuum of care. PACHA supports districts to conduct
health hospital/district based review meetings, peer review meetings for all implementing
Association hospitals once in a quarter for the hospitals/districts and also facilitated the sharing
(PACHA) of experiences and progress made and together identify solutions to challenges.
- Improve capacity for health workers in the management of new-borns during
delivery and illness and management of data systems in SNCUs in selected
hospitals.
- have improved quality of new-born care and strengthened accountability
mechanisms. Training and mentorship of health workers in the management and
use of essential new-born care equipment.
- adaption and implementation of new-born care standards through training and
mentorship.
- conduct neonatal and perinatal death and systems audits to identify gaps and
address them as quality of care will be a major issue in this project.
2 District -Coordination of implementation of project activities
Council -Supportive supervision and mentorship
3 Maikhanda -Mobilize communities through interpersonal and group communications.
Trust • Engage HSAs on collection and collation of community based monthly data
on number of pregnancies, new births, antenatal care attendance, and place
of delivery, pregnancy outcome and newborn care to be compiled and report
ed quarterly.
• Supportive monitoring to VHC members on their role on identification of
pregnant women and enhancing timely referral of sick pregnant women and
newborns from community level to the nearest facility
-Produce and air radio programs capturing pertinent RMNCH issues affecting
access and utilization of services.
• Work with relevant stakeholders to produce radio programs addressing
identified gaps on RMNCH care.
• Engage relevant community radios for distribution of key messages
addressing key barriers associated with RMNCH. The messages are
expected to be aired for about 150 minutes spread over the assignment
2|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
period.
• Monitor dissemination and follow possible impact of messages aired.
-Produce biannual community score cards and utilize community engagement
meetings
• Training relevant community structures (and HC staff) on the Community
score card process and guidelines.
• Synthesizing district data and produce community score cards for use in
advocacy and planning discussions.
• Facilitating community score card meetings between VHCs, Health facility
staff and District health authorities in relevant forums to monitor progress,
and address barriers related to access of RMNCH services.
-Build capacity of service providers for the delivery of age appropriate adolescent
reproductive health services
• Continued monitoring support provided to the facilities equipped for
adolescent health services
• Provision of mentorship, supervision, data collection and analysis and
coordinate review meeting
• Conduct quarterly supportive supervision to the trained providers, involving
national level program leads.
- Support HSAs for CBMNC
• Provide HSAs with CBMNC supplies as per national standards (counseling
cards, laminated mother-baby cards, timers, salter scales, thermometers,
back-packs).
• Ensure that all HSAs are supervised monthly by their immediate supervisors
• Conduct district level CBMNC supervision visits, involving MOH, on quarterly
basis
• Provide on the job training for facility level health workers to mentor HSAs for
CBMNC and provide respectful maternity care
• Conduct Program based program supervision from National level
4 Ministry of -Enabling environment through various policies and guidelines. provide
Health guidance in the implementation and monitoring.
Headquarter
Project outcomes and outputs
The overall objective of the project is to improve quality of Reproductive, Neonatal and Child health
services in selected districts in Malawi. The project expected outcomes are in threefold:
1. Improved availability and readiness of quality (Reproductive) Maternal, Neonatal and Child
Health Services including emergency obstetric and newborn care in project district(s).
2. Increased utilization of (Reproductive) Maternal, Newborn and Child Health services in project
district(s).
3. Improved community awareness and demand for quality (Reproductive) Maternal, Newborn and
Child Health services in project district(s).
The project expected outputs:
1. Community Health Workers in project district(s) are capacitated to deliver quality maternal and
Newborn Health services
2. Health staff at the targeted health facilities have improved knowledge and skills to provide
essential (R)MNCH services
3. Targeted health facilities are adequately refurbished, equipped, and provided with necessary
commodities (including innovative technology) to provide essential (R)MNCH and Emergency
Obstetric and Newborn Care (EmONC) services
4. Quality assurance mechanism in place for integrated (R)MNCH services including Improved
management of essential RMNCH medicines and commodities
3|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
5. Reduced barriers of the access for pregnant and post-partum women's and their newborns'
visits to health facilities for MNH services including EmONC
6. Reduced barriers of access for RMNCH services
7. Organized (R)MNCH awareness program at the community level in place
Project beneficiaries
The project aimed to reach approximately 500,000 women of child-bearing age (15-49), 150,000
Adolescents, 100,000 newborns with various RMNCH services. The indirect beneficiaries from this
project are expected to be about 2.6 million people in the target districts.
Human rights and gender
For women in Malawi, poor health outcomes are compounded by gender inequalities that subordinate
women, limiting their access to education, livelihoods opportunities, health care and other social
services. Women in Malawi generally have a lower socio-economic status and lower educational
attainment than men, and are “oriented from childhood to be submissive to males. This results in an
expectation that women should impress their spouses/partners at all costs, even at the expense of their
health.”
Sexual and gender-based violence (SGBV) is also a significant concern that impacts women’s health,
increasing the risk for unwanted pregnancies and sexually-transmitted diseases such as HIV. Of women
15-49 years old, 41% have been the victims of violence and 28% of women in Malawi report that they
have experienced sexual violence. Harmful cultural practices such as sexual initiation ceremonies and
traditional beliefs which negatively impact care-seeking or child care behaviors also affect the health
and survival of women and children. And young women in Malawi face particular challenges which put
them more at risk. The country’s child marriage and adolescent birth rates are among the highest in the
world. Over 10% of girls are married by the age of 15 and a full 50% by the age of 18, and 35%of girls
have given birth by the age of 18. These trends are even more pronounced in rural areas, with early
marriage and adolescent fertility occurring at much higher rates, and highly correlated with low levels
of education and wealth.
The promotion of gender equality and the empowerment of women and girls is central to the mandate
of UNICEF and its focus on equity. In order to achieve the results for children that UNICEF sets forth
in pursuing its mandate and to realize the rights of every child, especially the disadvantaged, it is
essential to address one of the most fundamental inequalities that exist in all societies –gender
inequality. The Gender Action Plan 2018-2021 for UNICEF outlines two programming gender results:
• Integrating gender in all UNICEF programme result areas – health, nutrition, HIV/AIDS,
education, water, hygiene and sanitation, child protection and social policy
• Targeted priorities for the empowerment and well-being of adolescent girls primarily:
- Promoting adolescent girls’ nutrition, pregnancy care, and prevention of HIV/AIDS
and human papilloma virus (HPV)
- Advancing adolescent girls’ secondary education, learning and skills, including
Science, Technology, Engineering and Mathematics (STEM)
- Preventing and responding to child marriage and early unions;
- Preventing and responding to GBV in emergencies
- Facilitating accessible and dignified menstrual hygiene management (MHM).
In line with that, the RMNCH project developed forums where communities’ voices are raised in a
constructive way, and promote engagement and leadership from women and youth, including in
supporting activities and accountability mechanisms featuring discussion of gender issues. Activities
on Adolescent Friendly Health Services also aimed at promoting gender equality and coherence and
target vulnerable groups like adolescent girls who are at risk of teen age pregnancy, SGBV and other
related harmful practices.
4|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
Evaluation target audience
The targeted audience for the evaluation will include women, children, community members (adults),
Ministry of Health, District Councils, Ministry of Local for policy guidance. UNICEF’s implementing
partners. Other specific targets include: UNICEF Malawi to judge if the overall programme achieved
results in line with the ToC; and to identify operational improvements to improve efficiency and
effectiveness and to reduce risks and improve the positive outcomes, and KOICA for programme
accountability as the main donor.
3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE
3.1 PURPOSE OF THE EVALUATION
The purpose of the evaluation is to establish and document the effectiveness of project interventions to
render accountability to donors and government. The evaluation is expected to provide data on the
performance and sustainability of project interventions. The findings and recommendations will
contribute to a learning process which will enable UNICEF and the Ministry of Health to draw lessons
from its experience in order to improve the quality and coverage of RMNCH service to Malawian
children, women and communities, assessing the extent to which the outcomes of the project have
been achieved, determine relevance, efficiency, effectiveness and sustainability of the interventions
and document new knowledge. The evaluation will also help to assess the effectiveness of beneficiary
accountability systems and mechanisms that were used during the project implementation period.
This evaluation is summative as it takes stock of the implementation of the maternal, newborn and child
survival programme over the last two years. It is also forward looking, providing the evidence to inform
the re-design of the next generation of the maternal, newborn and child survival interventions at
national, district and community level – thus in this sense formative. The overall evaluation approach
will be based on the RMNCH conceptual framework/strategy and logic models and results framework
outlined in the project proposal. The evaluation will further be guided by the evaluative baseline
assessment.
3.2 EVALUATION OBJECTIVES
The evaluation will examine, as systematically and objectively as possible, the relevance, effectiveness,
efficiency and sustainability of the maternal, newborn and child survival programme in districts of
Dedza, Mangochi, Blantyre, Thyolo and Nkhata Bay supported by UNICEF with funding from KOICA.
a) Effectiveness: To articulate and test or reconstruct the Theory of Change (ToC) of the
maternal, newborn and child survival and assess the reasons behind the achievement or not of
the objectives.
b) Coverage: To determine who was supported by the RMNCH project (children, women), and
why.
c) Efficiency: To review the set of financial and managerial factors that supported or hindered the
attainment of maximum efficiency/value-for-money.
d) Sustainability: To analyze the extent to which the programme had set up systems to improve
the long-term adherence to the new processes and behaviors.
e) Relevance/Appropriateness: To explore the appropriateness and relevance of the current
delivery strategies to the context and also the role and value addition of UNICEF support to the
RMNCH programme in the 5 districts in comparison to non-UNICEF funded districts.
3.3 EVALUATION SCOPE
This evaluation will review RMNCH work done by UNICEF with funding from KOICA and its partners
since 2017 and will focus on the 5 intervention districts of Dedza, Mangochi, Blantyre, Thyolo and
5|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
Nkhata Bay in comparison to non-intervention districts of Mzimba North, Ntcheu and Chiradzulu. The
evaluation will take place from 2 January 2020 to 20 April 2020.
The evaluation will examine, as systematically and objectively as possible, the relevance, effectiveness,
efficiency, sustainability and gender and human rights of the efforts in this maternal, newborn and child
survival programme. The evaluation will not cover the impact criteria since the evaluation is being
carried out soon after closure of the project activities and it is anticipated that impact may not be
demonstrated. The evaluation process, gender concerns should be addressed. All data should be
disaggregated by sex and age. Different needs of women, newborn and children targeted by the
intervention should be considered throughout the evaluation process. The aspect of human rights will
be assessed to give a measure of how the design and delivery of the project incorporated rights issues,
given that Malawi has ratified several related international treaties.
4. EVALUATION CONTEXT
In Malawi, nearly 20 percent of all deaths among women of reproductive age are as a result of
pregnancy or pregnancy-related causes. Most of all maternal deaths in Malawi occur due to direct
obstetric causes. The five main causes remain to be post-partum haemorrhage (23 percent), infection
(19 percent), pre- eclampsia/eclampsia (16 percent) and ruptured uterus (13 percent) and unsafe
abortion (7 percent). The main indirect causes include malaria, anaemia, HIV/AIDS and tuberculosis.
Despite a high institutional delivery rate of 90 percent, indicators of quality and equity remain
unsatisfactory. Antenatal care is important to identifying and mitigating risk factors in pregnancy such
as poor nutrition or HIV. Although 95 percent of pregnant women do receive ANC care from a health
care worker, only one-fifth of women start antenatal care during the first trimester and half of all pregnant
women do not receive the recommended four antenatal visits during pregnancy, important gaps in
quality of ANC services. Newborn mortality has increased as a share of overall child mortality, reaching
40 percent currently, compared to 23 percent in 2000. In Malawi, 89 percent of all neonatal deaths
results from just 3 conditions: prematurity, infection and asphyxia. Prematurity also results in disability
and poor health, even later in life. Every year in Malawi 120,000 babies are born premature and 28,300
newborns die or are stillborn. Malawi has the highest preterm birth rate in the world: 18 percent of all
babies are premature, and 12.9 percent have low birth weight of less than 2,500 grams. Approximately
80 percent of neonatal deaths occur during the first week of life. Data also suggest that 30% of new
HIV infections in infants occur during the breastfeeding period.
5. EVALUATION CRITERIA
The evaluation will employ the five specific evaluation criteria from the Organization for Economic
Cooperation and Development – Development Assistance Committee (OECD-DAC) namely
Relevance, Effectiveness, Efficiency, Sustainability and Gender and Human Rights.
Table 4: Selected OECD-DAC evaluation criteria
Relevance The extent to which the programme design and implementation was relevant to
national/provincial priorities and needs of men and women beneficiaries in terms
of achieving expected results across all socio-cultural groups including the
vulnerable in the targeted communities
Effectiveness The extent in which the implementation strategies were effective and successful
in achieving the planned outcomes/results women and children across all socio-
cultural groups including the vulnerable in the targeted communities
6|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
Efficiency The extent to which the outputs of the intervention has been achieved (in terms
of quality and quantity) with the allocated resources/inputs (such as funds, time,
and procedures)
Sustainability The extent to which various stakeholders including civil society and households
are likely to sustain the behaviour change related to the goals of the programme
after it has ended
Gender and The extent to which the intervention contributed to observance of human rights
Human Rights and gender equality in the focused five districts
The extent to which stakeholders regardless of sex, geographical location,
disabilities, age have benefited from the intervention
8. EVALUATION FRAMEWORK AND EVALUATION QUESTIONS
In line to the evaluation criteria, the evaluation will address the following key questions, which will be
further developed by the evaluation team during the inception phase. Collectively, the questions should
aim at highlighting the key lessons and performance of the UNICEF funded RMNCH programme in 5
districts, which could inform future strategic and operational decisions at community, health center,
district zonal and national levels. Specifically, the evaluation seeks to answer, but not be limited to, the
following questions:
a. Relevance
• Were the objectives of the project appropriate in the overall problem context, and did they meet
common needs and priorities?
• Were the RMNCH interventions formulated according to international norms and agreements
on human rights and gender (e.g. Convention on the Elimination of all Forms of Discrimination
Against Women (CEDAW), Universal Declaration of Human Rights (UDHR), Convention on the
Rights of Persons with Disabilities (CRPD), and to national and local strategies to advance
human right and gender?
• Does the program logic allow to achieve the project’s objectives (Quality of ToC)?
• Were the targets set realistically?
• Did the project team as well as partners have the institutional capacity in terms of staffing, local
knowledge and experience to implement the project’s targets?
• To what extent did the community and stakeholders participate in planning and implementation
of project interventions?
• Were the activities and outputs of the project consistent with the overall goal and the attainment
of its objectives?
• Did the RMNCH activities address the underlying causes of inequality and discrimination?
• Are the intervention results contributing to the realization of international human rights and
gender norms and agreements (e.g. CEDAW, UDHR and CRPD), as well as to national and
local strategies to advance human rights and gender?
b. Efficiency
• To what extent are human rights and gender a priority in the overall intervention budget?
• To what extent are human rights and gender a priority in the overall intervention budget?
• To what extent have the various activities transformed the available resources into the intended
outputs in terms of quantity, quality and timeliness?
• How efficient was the collaboration, coordination at national, zonal, district, community level
• Were there any constraints (e.g. political, practical, bureaucratic) to addressing human rights
and gender efficiently during implementation? What level of effort was made to overcome these
challenges?
• Was the use of intervention resources to address human rights and gender in line with the
corresponding results achieved?
7|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
c. Effectiveness
• To what extent has the availability and readiness of quality RMNCH services improved, including
emergency obstetric and newborn care in project districts?
• Are human rights and gender outcome and outputs clearly stated in the results framework,
including short, medium and long‐term objectives?
• To what extent has the community awareness improved and demand for quality RMNCH
services increased or decreased from baseline in project districts?
• To what extent has the project reduced the barriers to the utilization of reproductive, maternal
and new-born health services?
• To what extent has the project strengthened the health system for improved maternal and
neonatal health outcomes?
• To what extent has the project enhanced the quality of maternal and neonatal health services
in selected districts in Malawi?
• To what extent has the utilization of RMCNH services such as postnatal care within 72 hours
after birth, early ANC attendance within 12 weeks, BEmONC increased or decreased in project
districts compared to the baseline?
• What objective indications are there showing improvements in reproductive, maternal, neonatal
and child health?
• What difference has the project made in reproductive, maternal, neonatal and child health of the
intended beneficiaries as compare to the baseline?
• Did the intervention implementation maximize efforts to build the capacity of rights holders and
duty bearers?
• What are the unintended positive and negative consequences?
• Was monitoring data collected and disaggregated according to relevant criteria (gender, age,
geo-location etc.)?
• What were the main results achieved by the intervention towards the realization of human rights
and gender?
d. Sustainability
• To what extent has the intervention strengthen the implementation capacities of national and
local partners?
• Did the intervention design include an appropriate sustainability and exit strategy (including
promoting national/local ownership, use of local capacity, etc.) to support positive changes in
human rights and gender after the end of the intervention? To what extent were stakeholders
involved in the preparation of the strategy?
• Did the project plan and implement an adequate transition and exit strategy that ensures longer-
term positive effects and reduces risk of dependency?
• Will the beneficiaries continue to access RMNCH services on a long- term basis?
• Will the Government of Malawi maintain the project benefits?
• Assess capacity of key actors to contribute to sustaining the positive changes according to the
developed ToC
• What were/are the major factors which influenced the achievement or non-achievement of
sustainability of the project?
• Did the intervention activities aim at promoting sustainable changes in attitudes, behaviors and
power relations between the different stakeholder groups?
6. EVALUATION METHODOLOGY AND APPROACH
The evaluation will be transparent, follow a collaborative and participatory mixed methods approach
that draws on both existing and new quantitative and qualitative secondary and primary data
triangulated from different methods and sources to enhance the reliability of findings. The evaluation
will follow a theory-based approach and will combine evaluation tools based on international standards
8|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
and guidelines which are OECD DAC Quality Standards. The main sources of secondary information
will be from various RMNCH registers, baseline report, District Health Management Information System
(DHIS2), partner and donor reports. The evaluation team will:
a. assess data availability and reliability as part of the inception phase expanding on the information
provided. This assessment will inform the data collection
b. systematically check accuracy, consistency and validity of the data in the registers, reports and
acknowledge any limitations/caveats in drawing conclusions using the data.
Data will be disaggregated by relevant criteria (wherever possible): age, gender, marginalized and
vulnerable groups.
Design
The evaluation will use the same study design which was used during the evaluative baseline
assessment. The approach includes 3 comparisons for the counterfactual. The evaluative baseline
methodology was guided by the RMNCH conceptual framework, the logic models and results
frameworks. The approach used allowed a reflexive pre-project and post-project comparative analysis
to estimate the project attribution in order to assess the extent of achievement of the key project
indicators
The comparison districts are based on falling within the same health or geographical zone of the priority
districts of Mzimba North, (North Health Zone), Ntcheu (Central East Zone) and Chiradzulu (South East
West). Selecting comparison districts within the same geographical and heath zone as the programme
districts increases the chances of obtaining a good comparison group. In this regard, it is assumed that
the priority districts and comparison districts and their populations are defined in a similar manner and
there are no reasons to think that these two groups are substantially different as regards to socio,
economic, demographic and that utilization of RMNCH services in the groups has come due to
implementation of UNICEF funded RMNCH project in the intervention districts.
The study design recognizes that multiple programmes and partners are implementing similar
RMNCH activities and therefore UNICEF -RMNCH project activities in the districts only strengthens
and intensity implementation rather than complete attribution.
Evaluative baseline assessment
An evaluative baseline for the project was conducted in 2018 primarily to document the pre-intervention
situation of maternal and newborn health in target districts and come up with appropriate baseline
values for key project indicators, details are in Annexe 1. The baseline focused on ANC coverage,
labour and delivery, postnatal care, nutrition, immunisation, family planning, availability of emergency
obstetric care services signal functions, WASH at health facility and household levels, maternal and
neonatal births and deaths.
Desk Review
A desk review should be conducted by the consultancy firm to inform the methodology and development
of the tools. The desk review process should serve as guiding for the evaluation team to continue
gathering resources that would enable the team to carry out development of tools and hence conduct
the evaluation. In addition, the conducted desk review should cover the following documents but not
limited to:
I. Project proposal,
II. Logical framework with objectives, targets and progress indicators and literature related to the
RMNCH
III. Evaluative baseline report
IV. Partner project reports
9|PageTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
V. Annual donor reports
VI. Annual Review reports for the health sector for 2014/2015, 2015/2017 and 2018/2019
VII. Malawi Health Sector Strategic Planning II 2017 to 2022
VIII. Malawi health Sector Strategic Plan I 2011 to 2016
IX. UNICEF Malawi Country Programme 2011 to 2018
X. UNICEF Malawi Country Programme 2019 to 2023
XI. Malawi Demographic Health Survey (MDHS) 2015 -2016
XII. Millennium Development Goals Endline Survey 2014
XIII. Data collection tools and existing data collection tools related to ANC, postnatal care, child
health, immunisation used by the primary health facilities, sampling methodologies,
XIV. 2018 Malawi Housing Census report
XV. 2008 Malawi Housing Census report
XVI. 2018-2021 UNICEF Gender Plan
XVII. Integrating Human Rights and Gender Equality in Evaluation ‐Towards UNEG Guidance
XVIII. Convention on the Elimination of all forms of Discrimination Against Women (CEDAW)
XIX. Universal Declaration of Human Rights (UDHR)
Qualitative component
The qualitative part of the evaluation will involve data collection from communities, service providers
and health authorities: in-depth interviews (IDIs) with pregnant, breastfeeding mothers and caregivers;
key informant interviews (KIIs) with community members, Health Surveillance Assistants (HSAs),
District Health Officers (DHO); and focus group discussions (FGDs) in each district. Insights from these
qualitative interviews and discussions with community members provide complementary data to that
obtained through the quantitative survey and will allow examination of issues at depth. Based on the
above generic guideline, the prospective firm will propose a detailed design and methodology for the
evaluation. The final methodology and questions will be agreed with UNICEF at the inception phase.
The evaluation team is required to review the baseline report which includes the indicators for
answering each evaluation question.
Quantitative component
The primary quantitative evaluation will be based on a robust and adequate evaluation design
comparable to the evaluative baseline.
Sample size calculation and sampling
The methodology for sample size calculation should be the same as evaluative baseline method
with an exception of using the 2018 Malawi Population and Housing Census instead of the 2008
census as the basis the basis for sampling clusters for the evaluation with EAs as primary
sampling units. An adequate representative sample of eligible women of child bearing age who
gave birth within the last 2 years prior to the evaluation should draw randomly from an
appropriate sampling frame of Enumeration Areas (EAs). The study areas should be randomly
selected in both intervention districts (5) and comparison districts (3) to provide district level
estimates of core indicators. This enabled the study team to compute population weights that
were used in the analysis. In each district, the study team stratified and included rural and urban
enumeration areas.
The sample size for interviews should be calculated using the percent of women attending at
least one ANC visit during last pregnancy at 5 percent level of significance and a statistical
power of 80 percent.
The evaluation team will provide a detailed sampling plan that will include at minimum: Power
calculations and sample size determination at the facility, community and household level to
ensure robust measures of change at outcome level;
10 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
• Clearly defined probabilities of selecting the sampling units;
• Sampling weights used in the data analysis. The approach and methodology proposed
in the inception phase should include, but not limited to, the following:
a. An appropriate survey design with relevant approaches that will document status to
enable measure change from baseline, contributions and cost effectiveness while
meeting the quality criteria.
b. A quality assurance plan for all data collection processes, analysis and training of
field staff, as relevant.
c. A data triangulation plan taking stock of all relevant data sources such as facility
registers and reports available as identified during the evaluation.
d. A data analysis plan, in which the procedures related to the data to be analysed
under the evaluation design and sampling plan will be described and detailed. The
data analysis plan is integral part of study plan.
Statistical Analysis
The data analysis methods should be in line with the design of the evaluation to provide fair,
unbiased judgement of the programme. Statistical tests (Chi-Squared tests where observed
frequencies are more than 5 and Fishers Exact test where observed frequencies are less than
5) should be used to test if the data collected have confirmed the lack of significant differences
between the intervention and comparison group on the RMNCH key indicators in table 2 at
baseline and endline.
Ethical consideration
The evaluation will follow UNICEF procedure for ethical standards in Research, evaluation, data
collection and analysis. All participants in the evaluation will be fully informed about the nature and
purpose of the evaluation and their involvement. Only participants who have given their written or verbal
consent will be included in the evaluation.
The prospective consultant is expected to provide a detailed plan on how the following principles will
be ensured throughout the evaluation:
1) Respect for dignity and diversity
2) Fair representation;
3) Compliance with codes for vulnerable groups (e.g., ethics of research involving young
children or vulnerable groups);
4) Redress;
5) Confidentiality; and
6) Do no harm.
Specific safeguards must be put in place to protect the safety (both physical and psychological) of both
respondents and those collecting the data. These should include:
• A plan is in place to protect the rights of the respondent, including privacy and confidentiality
• The interviewer or data collector is trained in collecting sensitive information
• Data collection tools are designed in a way that are culturally appropriate and do not create
distress for respondents
• Data collection visits are organized at the appropriate time and place so as to minimize risk to
respondents
• The interviewer or data collector is able to provide information on how individuals in situations
of risk can seek support
The contracted institution is expected to seek approval from the Malawi National Health Science
Research Committee (NHRC) before commencement of the evaluation.
11 | P a g eTERMS OF REFERENCE EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI 12 | P a g e
TERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI
7. EVALUATION WORK PLAN AND DELIVERABLES
The evaluation process has been divided in three phases: 1) planning phase: from 1 August 2019 to December 2) Implementation phase: from
1 January 2020 to 20 April 2020 and 3) Follow up phase: from 1 June to 31 December 2020. Table 5 details plan of activities, deliverables and
estimated timelines.
Table 5: Evaluation workplan and key deliverables
Estimated Responsible
number of Person
days and Estimated
No Activities Deliverable suggested Completion
technical Date
resources
required
Implementing the evaluation
Briefing of the Evaluation Team: ERG will give/provide access to all ERG
relevant documents (including UNEG Norms and Standards, UNICEF ET briefed on the
procedure for ethical standards in research, evaluation, data collection evaluation and key 15th January
1 1
and analysis, Standards in conducting, programme documents (including documents 2020
proposal and reports), evaluative baseline report for the project, list of key provided
stakeholders and etc) to the Evaluation Team (ET).
Develop evaluation plan in consultation with ERG outlining specific dates Evaluation Team
Evaluation plan 15th January
2 for key deliverables 1
developed 2020
Inception Report and ethical approval by NHSRC. The ET Evaluation Team
to clarify in writing and through presentations the understanding and
Approved
expectations of how the evaluation will be undertaken, the ET will prepare
evaluation of the
and submit to the ERG. The inception report should further refine the 31st January
3 project by NHSRC 30
overall evaluation scope, approach, design and timeframe, provide a 2020
detailed outline of the evaluation methodology including data collection
Inception report
tools
Data collection, and analysis. The ET collects and analyses data using Report on data Evaluation Team
31th March
4 various methods agreed in the inception report collection (field 45
2020
work) and analysis
13 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI
Preliminary Findings: The ET presents preliminary finding to the ERG Draft evaluation Evaluation Team
and conduct validation workshop with UNICEF, MoH – District and report and
5 15 15th April 2020
National and implementing partners validation workshop
conducted
Evaluation report: The ET prepares the report in accordance with the Final evaluation Evaluation Team
UNICEF- Adapted UNEG Evaluation report standards and the report report incorporating th
6 5 20 April 2020
should be logically structured, containing evidence-based findings, comments from the
conclusions, lessons and recommendations. validation workshop
14 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
8. GENDER AND HUMAN RIGHTS, INCLUDING CHILD RIGHTS
Gender Equality and empowerment of women should be mainstreamed throughout the evaluation.
Further details of the criteria can be sourced from: http://www.unevaluation.org/document/detail/22. The
evaluation process, gender concerns should be addressed. All data should be disaggregated by sex
and age. Different needs of women, newborn and children targeted by the intervention should be
considered throughout the evaluation process. The aspect of human rights will be assessed to give a
measure of how the design and delivery incorporated in the programme as a rights issue, given that
Malawi has ratified several related international treaties. This evaluation should document whether the
RMCH project was implemented in a gender sensitive and socially acceptable manner.
9. MANAGEMENT, OVERSIGHT AND CONDUCT OF THE EVALUATION TEAM
The evaluation team will conduct the evaluation under the direction of its team leader and in close
communication with the Chief of REKM. The evaluation team will not have been involved in the design
or implementation of the subject of evaluation or have any other conflicts of interest. Further, they will
have to act impartially and respect the code of conduct of the evaluation profession as stipulated by
UNEG.
10. EVALUATION QUALITY ASSUARANCES
To enhance the quality and credibility of this evaluation, an Evaluation Reference Group (ERG) is
established. The ERG is team will provide a review of the draft Terms of Reference, evaluation of the
evaluation proposals from the bidding companies, provide systematic feedback from an evaluation
perspective, on the quality of the draft inception and evaluation report and recommendations on how to
improve the quality of the final inception/evaluation report. The inception report and final report will also
be reviewed by the Eastern and Southern Africa UNICEF regional office and the final report will further
be reviewed by the Global Evaluation Oversight System (GEROS).
11. EVALUATION REPORT
The Inception Report should include but not be limited to the following:
- evaluability assessment
- constructed RMNCH theory of change
- evaluation design detailing the sampling method the team intends to follow
- data collection tools
- any interview guides
- evaluation forms or other data collection instruments to be used
The Evaluation Report which shall be compliant with the UNICEF – Adapted UNEG standards should
include - but is not limited to - the following components:
• Executive Summary.
• Background (including adequate description of the RMNCH programme)
• Evaluation Objective, Purpose and Scope
• Methodology (including limitations and ethical considerations)
• Findings
• Conclusions and lessons learned
• Recommendations
• Appendices, including ToRs, tools, and people interviewed, evaluation matrix, results
framework in comparison to the baseline.
15 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
A tightly-drafted, executive summary is an essential component. It should be short and not more than
five pages. It should focus mainly on the key purpose or issues of the evaluation, outline the main
analytical points, and clearly indicate the main conclusions, lessons learned and specific
recommendations. The executive summary shall include the performance rating of the main 5
evaluation criteria. Cross-references should be made to the corresponding page or paragraph numbers
in the main text that follows.
The recommendations should be the subject of a separate final chapter. Wherever possible, for each
key conclusion there should be a corresponding recommendation. The key points of the conclusions
will vary in nature but will often cover aspects of the key evaluation criteria (including performance
ratings).
The final evaluation report will strictly follow “UNICEF Evaluation Report Standards” and UNICEF
Evaluation Technical Notes and be aligned with UNEG Standards and Norms. A self-assessment of
the draft report against the GEROS UNICEF tool will also be required.
12. PROPOSED PAYMENT SCHEDULE
Payments to be based on outputs to deliverables, after approval b UNICEF
Payment #1: Deliverables 3 30%
Payment #2: Deliverable 4 10%
Payment #3: Deliverable 5 30%
Payment #4: Deliverables 6 30%
13. QUALIFICATION REQUIREMENTS
Qualified and experienced vendor to undertake the evaluation must have access to a network of experts
with experience in undertaking evaluations for health programmes, particularly Reproductive, Maternal,
Newborn and Child health. The consultants should additionally have:
• commitment to gender equality, and knowledge and experience in evaluating gender equality
interventions
• commitment to human rights, and knowledge and experience in evaluating human rights
interventions
• an understanding and application of United Nations mandates on human rights and gender
Company experience:
• The institution must possess at least 10 years of institutional experience in the evaluation of
development programmes, especially health programmes and health research with an
emphasis on mixed method data collection and analysis.
• Proven recognized expertise on evaluation of high impact maternal and new-born health
interventions.
• Establish working relationship with the Ministry of Health of Malawi.
Proposed Team experience:
The Organisation should propose a team that fully responds to the requirements of this TOR, therefore
it is the Organisations responsibility to propose a strong team, with well-defined roles and
responsibilities. The following qualifications and experience should be incorporated into the team
proposed:
16 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
• Demonstrated field experience of team members on other assignments related to this activity.
The Lead quantitative researcher must have an advanced University degree (Masters or PhD)
in one of the following disciplines: Public Health, Epidemiology, Biostatistics, Health Economics,
or other relevant social science.
• The team will essentially have an experienced quantitative data analyst and evaluation
specialist.
14. APPLICATION AND EVALUATION PROCESS
Each proposal will be assessed first on its technical merits and subsequently on its price. In making
the final decision, UNICEF considers both technical and financial aspects. The proposal reviewers will
first review the technical aspects of the offer, followed by review of the financial offers of the technically
compliant vendors. The proposal obtaining the highest overall score after adding the scores for the
technical and financial proposals together, that offers the best value for money will be recommended
for award of the contract.
The Technical Proposal should include but not be limited to the following:
- Company Profile (5 points)
Ensure to include information related to the experience of the company as required and outlined
in item 13 of this document.
- Methodology (35 points)
Detailed Methodology / approach to requirement detailing how to meet or exceed UNICEF
requirements for this assignment
- Work Plan (5 points)
Proposed work plan showing detailed sequence and timeline for each activity and man days of
each proposed team member
- References (5 points)
Details of similar assignments undertaken in last three years including the following information:
o Title of Project
o Year and duration of project
o Scope of Project
o Outcome of Project
o Reference / Contact persons
- CV’s (15 points)
CV of each team member (including qualifications and experience)
Ensure to include information related to the qualifications and experience of each proposed team
member as required and outlined in item 13 of this document.
- Any project dependencies or assumptions
The Financial Proposal should include but not be limited to the following:
Bidders are expected to submit a lump sum financial proposal to complete the entire assignment
based on the terms of reference. The lump sum should be broken down to show the detail for the
following:
- Resource costs
Daily rate multiplied by number of days
17 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH
INTERVENTION DISTRICTS IN MALAWI
- Conference or workshop costs (if any)
Indicate nature and breakdown if possible
- Travel Costs
All travel costs should be included as a lump sum fixed cost.
For all travel costs, UNICEF will pay as per the lump sum fixed costs provided in the proposal.
A breakdown of the lump sum travel costs should be provided in the financial proposal.
- Any other costs (if any)
Indicate nature and breakdown
- Copy of the company registration
- Recent Financial Audit Report
Report should have been carried out in the past 2 years and be certified by a reputable audit
organization.
Bidders are required to estimate travel costs in the Financial Proposal. Please note that i) travel costs
shall be calculated based on economy class fare regardless of the length of travel and ii) costs for
accommodation, meals and incidentals shall not exceed the applicable daily subsistence allowance
(DSA) rates, as propagated by the International Civil Service Commission (ICSC). Details can be
found at http://icsc.un.org
15. EVALUATION WEIGHTING CRITERIA
Cumulative Analysis will be used to evaluate and award proposals. The evaluation criteria associated
with this TOR is split between technical and financial as follows:
70 % Technical
30 % Financial
100 % Total
A submission must obtain a minimum of 49 points (70%) to pass the technical evaluation. Financial
proposals will only be opened where the technical proposal has reached the required pass mark.
Financial proposals will be opened and points assigned. The maximum score of 30 points will be
assigned to the financial proposal that provides the lowest overall cost. Allocations to the activities as
well as to program management will also be considered. All other financial proposals will receive scores
in inverse proportion according to the following formula:
Score for price proposal A = (Maximum score for price proposal (e.g. 30) * Price of lowest priced
proposal)/Price of proposal A.
The technical and financial scoring will then be combined to provide an overall score for each technical
compliant proposal. Award should then be made to the proposal that gains the highest score following
combining the technical and financial scores.
18 | P a g eTERMS OF REFERENCE EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI 19 | P a g e
TERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI
Annexes
Annex 1. Baseline, Planned results (Outcomes and Outputs) and achievements as at 31 December 2018
Outcome 1 Indicator 1.1 Baseline Target Dec 2018
Improved availability and readiness % of health facilities
of quality (reproductive) maternal providing all 7 Basic
and Neonatal Health services, Emergency Obstetric and 42% 60% 72.7%
including emergency obstetrics and Newborn Care (BEmONC)
newborn care in project districts signal functions
Output 1.1 Activity 1.1.1 Indicator 1.1.1 Baseline Target
Community Train/refresh HSAs Number (and proportion) of 1000 693 (69.3%)
Health Workers on core CBMNC Community Health Workers
in project competencies who have completed the
district(s) are Community based Maternal
capacitated to and Newborn Care
deliver (CBMNC) training during the
standard current reporting period
(R)MNCH Activity 1.1.2 Indicator 1.1.2 Baseline Target
services Complete and Number of Community 800 493
analyze data on Health Workers who have
pre- and post- test obtained the minimum
of all Health required score after the
Surveillance training during the current
Assistants (HSAs) reporting period
trained
Activity 1.1.3 Indicator 1.1.3 Baseline Target
Senior HSA visits Number of HSAs received at 1000 1684
all HSAs working least one supervision visit
in Village Clinics in and submitted activity report
catchment area during the current reporting
quarterly period
Output 1.2 Activity 1.2.1 Indicator 1.2.1 Baseline Target
20 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI
Health Staff at Train health staff in Number of health staffs who 320 320
the targeted facilities on have completed the training
health facilities BEmONC on BEmONC services and
have improved Active Management of the
knowledge and Third Stage of Labor
skills to provide (AMTSL) for vaginal delivery
essential during the current reporting
Reproductive, period
Maternal, Activity 1.2.1 Indicator 1.2.2 Baseline Target
Newborn and Complete and Number of health staffs who 240
Child Health analyze data on have obtained the minimum
(RMNCH) pre- and post- test required score after the
services of all health facility training provided during the
staff trained current reporting period on
BEmONC services and
AMTSL for vaginal delivery
Activity 1.2.3 Indicator 1.2.3 Baseline Target
Follow health Number of health facilities 43
facilities for post- with health staff who have
training obtained the minimum
assignment of staff required score after the
to relevant units training provided during the
current reporting period on
BEmONC services AMTSL
for vaginal delivery
Output 1.3 Activity 1.3.1 Indicator 1.3.1 Baseline Target
Targeted Based upon Number of health facilities 5
health facilities results of surveys, refurbishment completed for
are adequately health facilities MNCH services during the
refurbished, provided with current reporting period
equipped, and Indicator 1.3.2
21 | P a g eTERMS OF REFERENCE
EVALUATION OF MATERNAL, NEWBORN AND CHILD SURVIVAL PROGRAMME IN HIGH INTERVENTION DISTRICTS IN MALAWI
provided with equipment and Number of health facilities - 43
necessary supplies identified provided with equipment and
commodities supplies during the project
(including period
innovative
technology) to
provide
essential
(R)MNCH and
Emergency
Obstetric and
Newborn Care
(EmONC)
services
Output 1.4 Activity 1.4.1 Indicator 1.4.1 Baseline Target
Community Conduct Number of health facilities 43 38
Health Workers monitoring and received at least one
in project supervision visits supervision visit on the
district(s) are to labor, delivery quality of (R)MNCH services
capacitated to and neonatal care including AMTSL for vaginal
deliver units regularly delivery,
standard with the checklist fully
(R)MNCH completed during the
services reporting period
Activity 1.4.2 Indicator 1.4.2 Baseline Target
Establish/support % of health facilities with 75% 100%
Quality functional QI teams
Improvement (QI)
teams in health
facilities
Activity 1.4.3 Indicator 1.4.3 Baseline Target
MPDR held % of health facilities and 100%
including relevant communities conducting 100%
staff MPDR meetings
22 | P a g eYou can also read
