Measurement, Monitoring, Evaluation, and Action-Oriented Learning

Monitoring and Evaluation:

• Created improved periodic and routine RMNCH metrics, tools, measurement guidelines, and processes: MCSP contributed to the development of improved RMNCH metrics and guidance through its engagement with global monitoring and evaluation working groups. For example, MCSP collaborated with the Every Newborn Action Plan/Ending Preventable Maternal Mortality core metrics group to help design a multi-country study of new maternal newborn health (MNH) indicators for national management health information systems (HMIS) and develop a core set of maternal health indicators for global tracking. In addition, MCSP’s participation in the Global Immunization Data Quality and Use Working Group with WHO and UNICEF resulted in the provision of support to countries with tools and processes for improving immunization data quality. MCSP also provided feedback to this working group on findings from cross-country learning. As part of the Health Data Collaborative community working group’s work plan, MCSP supported the Africa Regional Workshop on Improving Routine Data for Child Health in National Health Information Systems. Almost 100 participants from ministries of health (MOHs), nongovernmental organizations, WHO, and UNICEF discussed emerging best practices, gaps and challenges, and developed an action plan to address them. MCSP also contributed to activities of the community and interoperability working groups of the Health Data Collaborative, which seek to increase country capacity to measure progress toward national targets for the Sustainable Development Goals. MCSP also led the Child Health Task Force and co-chaired the M&E sub-group.
• Validated routine intrapartum care indicator: MCSP conducted a study in Tanzania to test the validity of a new facility-based indicator on perinatal mortality. The study yielded compelling results, documenting a high level of sensitivity and specificity between information on newborn outcomes recorded in the health facility maternity register and the gold-standard perinatal death audit for the outcomes of fresh stillbirth, macerated stillbirth, and newborn deaths. These results are notable as this indicator allows for measurement of potentially preventable perinatal mortality using HMIS data and could serve as a sentinel measure of the quality of intrapartum care at health facilities. A manuscript summarizing the indicator validation findings was published in PLOS One; an implementation research brief is also available.
• Improved household survey tools for measurement of RMNCH service utilization, intervention coverage, quality, and gender equity: MCSP updated the knowledge, practices, and coverage (KPC) household survey toolkit and made it available as a global resource. The following survey modules were revised: sick child; malaria in pregnancy; pregnancy spacing; nutrition; water, sanitation and hygiene; maternal, newborn and child health (MNCH); and immunization. In addition, MCSP created a new gender module with questions that can be integrated into other KPC survey modules. The gender module seeks information about women’s and men’s roles in household activities, household decision-making, decision-making for women’s and children’s health care, and attitudes around gender norms. Questions aim to provide program implementers with a better understanding of how gender-based attitudes, norms, roles and behaviors may affect health-seeking behaviors and health outcomes, including RMNCH. Understanding this context is crucial for integrating gender in program design as well as better enabling families and communities to practice healthy behaviors and to seek and access health services. This KPC module differs from others in that it includes a questionnaire for both women and men. MCSP also developed mobile versions of the KPC household survey modules using an open-source mobile software application, CommCare, building on work conducted by MCSP in Tanzania. MCSP created mobile survey modules in generic versions of XML forms. These mobile applications, which enable built-in data quality control checks and real-time tracking of data collection, allow for streamlined data collection and cleaning, paving the way for improved data quality and faster data analysis.
• Reviewed the RMNCH content of national HMIS in 24 countries: MCSP completed a review of the MNH and child nutrition content of national HMIS across 24 USAID priority countries to provide country and global stakeholders with a better understanding of the data elements available for calculating current global standard indicators and other indicators on content of care and health outcomes. The global review informed revisions to facility registers and patient records, and supported advocacy at the national level for incorporating recommended high-priority MNCH data elements and indicators into national HMIS. Results were disseminated to key stakeholders at international meetings and a webinar series that also included results of a family planning HMIS review conducted by MCSP. The HMIS review findings informed discussions on and plans to address gaps in the content of the national HMIS in Rwanda and Nigeria during country-led revisions to these systems.
• Strengthened visualization and use of routine RMNCH data in supported countries: MCSP provided resources, tools and support for improved data collection, quality and use at the national and sub-national levels in multiple countries. In several sub-Saharan African countries — Rwanda, Nigeria, Madagascar, Malawi, Uganda and Tanzania — MCSP increased the use of routine RMNCH data for decision-making through capacity building on data use and application of data dashboards and scorecards at community, facility, district and/or regional/state levels. MCSP developed a comprehensive Routine RMNCH Data Visualization and Use Resource Package for health facility providers and district-level supervisors. MCSP also revised two components of this package — Visualizing and Using Routine Reproductive, Maternal, Neonatal, and Child Health Data at Health Facilities: A Resource Package for Health Providers and District Managers — based on experience applying them with our country programs, including the laminated wall chart template and supportive supervision module. Components of this resource package were adapted and used in the English speaking Caribbean countries of Barbados, Guyana and St. Lucia, where MCSP supported Zika-related work as part of a technical updates and quality improvement training for health workers on postnatal care and family planning.
• Assessed the feasibility, acceptability and usefulness of routine RMNCH-related indicators in two countries: The aim of this work was to test the utility, feasibility and acceptability of RMNCH indicators that may potentially be introduced into existing national HMIS in Nigeria and Madagascar. The results from this activity were helpful for determining: 1) whether these indicators will provide key stakeholders with meaningful, quality data that can be used for decision-making and progress monitoring; and 2) if so, whether these indicators should be prioritized for introduction into health systems as the content of these systems are revised. Analysis and documentation of findings were undertaken in both countries. Results were used in Nigeria to advocate with the federal MOH for inclusion of new indicators that were found to be useful, acceptable and feasible to collect into the national HMIS as it is currently being revised.
• Compiled evidence on and applied geographic information system analyses to RMNCH in low-resource settings: MCSP led the preparation of an RMNCH geographic information systems journal supplement that was published in BMC Global Health. In Rwanda, MCSP used innovative geospatial techniques to assess the relationship among the use of antenatal care services, client’s sociodemographic background, and the closest health facility’s readiness to provide antenatal care, with findings presented at the 11th Geospatial Health Symposium and at the MEASURE Evaluation Annual Geographic Information System Working Group meeting. In Nigeria, MCSP used geospatial techniques to assess the relationship between various health facilities’ emergency obstetric and newborn care readiness scores (using data from a comprehensive health facility readiness assessment conducted by MCSP in Cross River, Kogi and Ebonyi states) and sociodemographic characteristics of the catchment population, such as literacy and access to improved water sources, as proxies for wealth. The study did not demonstrate significant differences in the sociodemographic distribution. MCSP’s exploratory work was presented at the Global Digital Health Forum and has continued to inform discussion on how to better address underlying health inequities and continue to target underserved populations. The innovative geospatial techniques of linking population survey data with health facility data was identified as of interest for use by future programs.
• Assessed use of process indicators for routine immunization system strengthening: To advance global learning on the effective use of process indicators for strengthening routine immunization (RI), MCSP drew from its country-level technical assistance in three countries (Malawi, Nigeria, and Uganda) to achieve two primary objectives: 1) explore health facility and district staff understanding and utilization of a selected set of process indicators; and 2) identify mechanisms that promote the use of process indicators for decision-making by district and health facility staff. Quantitative data extracted from the monitoring system demonstrated major improvement: more than 80% of health facilities in Malawi and Nigeria and 75% of health facilities in Uganda had complete microplans for RI in January-March 2018 compared to less than 20% at the baseline when MCSP started supporting RI in these countries. Having microplans available resulted in better planning of sessions. In Nigeria, the number of fixed sessions conducted increased from 9,930 at baseline in July-September 2014 to 17,339 in January-March 2018. In Malawi, the proportion of planned outreach sessions conducted increased from 55% at baseline to 98% over the same time period. Interpretation of process indicators also considers context. For example, in Nigeria, the significant increase in number of sessions within a short time period is a likely a result of partners being involved in a memorandum of understanding to strengthen the RI system. Health workers reported that the indicators are useful for gaining a holistic snapshot of the status of the RI system. Some health workers, however, expressed a lack of understanding of the indicators and their use. The results were shared and adapted in the remaining regions/districts in the three assessment countries. In addition, the results informed adaptation of the Reaching Every District (RED) guide by 17 countries in the Eastern and Southern Africa region during the revised RED guide adaptation. Results were also shared with the WHO Strategic Advisory Group of Experts, which used the findings and key learning to inform their global guidance.

Learning:

• MCSP contributed lessons from its global learning efforts and those of other organizations to develop new action-oriented approaches that help countries successfully implement and scale-up health programming. MCSP’s learning agenda included more than 80 traditional studies across all country programs and technical area, and more systematic approaches to implementation of activities to enable more focused and high-quality information to be generated and used to drive program improvements. Illustrative key learning activities in individual countries include:
  o MCSP conducted implementation research in Tanzania that found that the use of a microplanning tool significantly improved the quality and comprehensiveness of reported funding needs in district health planning, which prompted the MOH to use the tool in other areas of the country.
  o MCSP continued to work with the Mozambique MOH to strengthen its referral systems for maternal and child health from communities to facilities and between facilities. By tracking referral and counter referral completion rates and through lesson sharing in regular learning meetings, target districts adapted and improved their practice.

• Knowledge Management
• Learning

• Monitoring & Evaluation