MONITORING PROGRESS AND ENSURING ACCOUNTABILITY OF REPRODUCTIVE
HEALTH SERVICES PROVISION - A ROAD MAP
Dr. A. T. P. L. Abeykoon

Introduction

The consensus reached by the International Conference on Population and Development (ICPD), substantially enlarges the scope of population programmes and places family planning (FP) within the broader context of reproductive health (RH).  Monitoring progress and ensuring accountability of RH programmes and services will remain a challenge for programme managers as greater emphasis is placed on the need to measure progress in achieving population and RH goals.  Following recent international conferences, countries endorsed a number of goals and targets in the broad area of sexual reproductive health.  Many of these goals and targets have been formulated with explicit, quantifiable and time-bound objectives and adopted by countries as part of their national health related policies, programmes and services.

In order to assess achievement of goals, it is necessary to establish systems for monitoring and evaluation.  This generally involves the definition of essential indicators and guidelines and how to use them (Box 1).  With the expansion and evolution of services addressing reproductive health, many agencies have been working on developing indicators.  Consequently, there has been a proliferation of indicators, in addition to existing national indicators.

The aim of this chapter is to present a guide to a manageable and practical set of quantitative and qualitative indicators for RH services and programmes.  The indicators here are presented in a conceptual framework which can be used as a "road map" to assist programme managers, planners and decision-makers to 1) measure achievement of targets and objectives; 2) assess changes/trends in RH status or compare the level of achievements between programme areas etc.; and 3) identify possible problem areas and contribute to decision-making to improve management and accountability.

Conceptual Framework

An important objective of a conceptual framework for programme monitoring and evaluation is to depict clearly the desired programme and population outcomes targetted by interventions and the main paths of influence that connect the pertinent actions (see Figure 1).  A conceptual framework for reproductive health helps those involved in programme design, management, implementation and evaluation to monitor and evaluate whether and how their interventions have helped achieve RH objectives for instance, reducing the incidence of sexually transmitted diseases (STDs), unwanted pregnancy, unsafe abortions, pregnancy related nutritional deficiencies, gynaecologic and obstetric morbidities and maternal and pre-natal mortality.

Organisational Resources and Policies

In a RH programme, specific interventions directed at achieving the desired outcomes need to be supported by a conducive environment, where policies and organisational resources are in place.  The inputs needed to meet the desired results are resources and the policy environment.  Resources include manpower, material and financial resources.  Policies and administrative procedures include national policies and legislation with regard to reproductive health to create an enabling environment for the effective implementation of activities.  RH indicators directed at policies and administrative issues are designed to show whether the enabling national policy conditions and guidelines are in place to support appropriate RH interventions.  All these policy indicators require qualitative information on the existence, or not, of policy statements or legislation in support of RH goals.  Their status as indicators need further research in the development of effective indicators for measuring the enabling environment.

Implementation

Implementation of RH activities is the process through which the desired interventions are carried out to achieve the expected outcomes.  The process indicators of reproductive health address operational issues and questions that can be answered with programme level data and measures.  The indicators may enable policy-makers and programme managers to assess and improve RH services so that clients can achieve their reproductive intentions.

Service Outputs and Utilisation

If the activities of a RH programme are implemented as desired, then the results of such activities should meet with programme objectives.  The output indicators of a RH programme are knowledge of RH, utilisation of RH services and prevalence of contraception, etc.

Reproductive Health Impact

The effect of RH interventions introduced through programme activities have to be eventually measured at the population level.  The outcome indicators that measure changes at the population level are fertility, mortality and morbidity rates.

Monitoring Progress of Reproductive Health Services Provision

Monitoring

The purpose of monitoring is to achieve efficient and effective programme performance by providing feedback to programme management at all levels.  This enables the management to improve operational plans and to take timely corrective action in case of shortfalls and constraints.  Monitoring is thus a part of the management information system and is an internal activity.  As an integral component of the management function, and hence an essential part of good management practice, monitoring needs to be conducted by those responsible for programme implementation at every level of the management hierarchy.

Administrative reports such as progress report on physical and financial inputs and outputs, supplemented by investigative studies or in-depth analyses of persistent problem areas, provide the information base for monitoring. Health indicators are used to summarise data that have been collected in order to answer questions relevant to the planning and management of RH programmes.  These indicators are used in a number of ways.  For example:

Accountability

The attainment of reproductive health by the population requires the participation of all concerned - the government, the private sector, non-government organisations (NGOs) and the people themselves who are seeking reproductive health.  These parties in turn are held accountable to successfully implement programmes by:

As new programme areas develop and existing ones evolve, accountability also expands.  Regular review of performances at programme implementation, sectoral and central levels is essential so that matters relating to achievements and targets that require the attention of, and action by the programme managers or authorities at the higher level, are brought to the attention of those who make decisions.

Review Report

For effective management, it is crucial that there is proper reporting of clinic performance.  This is to ensure that clinics perform according to the work plan; that organisational goals are being met; that quality of care is provided; that costs are within budget and the time frame is kept.  Other considerations are the data collection mechanism - adequate, sufficient and well kept to allow thorough evaluation of clinic services' impact on the community.

The indicators most useful to the health worker in maternal and child health/family planning (MCH/FP) are the process and output indicators rather than the outcome/impact indicators. Data can be collected through the health information system, i.e. registers, records and reports.  The health worker can assist in the collection and analysis of the data.  More importantly, the health worker should be able to use information on the selected indicators for his/her work planning and self-assessment.

Self-Assessment1

The health worker can undertake a continuous assessment of his or her performance based on records and reports, work plans, and client feedback.  By using household registers, maternal and child care registers, FP registers, etc. and the periodic reports that the health worker submits to the next higher level, he or she can discern the coverage of target groups, identify weak programme areas and potential problems and opportunities.  The health worker can share this self-assessment with the community groups or the Village Health Committee, if one exists, and with his/her supervisor.  Together, the health worker, supervisor and the community representatives, can explore alternative solutions to improve programme performance.

Work plans are an excellent instrument for monitoring performance.   By comparing what was planned and achieved, the health worker can identify the time, resource or other constraints which impede better performance.  Often, certain activities may be completed well ahead of time or better coverage of certain target groups may be achieved.  By identifying factors which facilitated implementation, such as increased community acceptance or special campaigns undertaken in the area, the health worker can revise his/her work plans to focus on areas of under achievement.

Work plans are also shared with the community representatives.  The community members can, therefore, monitor the programme performance based on what was planned and what was achieved.  The health worker should encourage such an assessment by the community as it not only generates a sense of ownership but it can also be a source for future support to achieve programme objectives and services targets.

Client's feedback is an important measure of the quality of services and the acceptance level of the RH services.  The health worker, during his/her routine interaction with the community, can learn a lot about the satisfaction level of the clients.  During household visits, group meetings, information, education and communication (IEC) sessions, clinic visit, etc. the health worker should encourage the clients to provide feedback on the range of services, the quality of services, accessibility and affordability of services, etc.

Monitoring Other Health Service Providers2

The health worker is also responsible for the performance of the other health care providers in her/his work area, such as trained traditional birth attendants (TBAs) and community health workers (CHWs), individuals or groups involved in community-based distribution and social marketing.  When mutually agreed specific responsibilities are assigned to these individuals, usually some measures to assess performance are included.  For example, identification of pregnant women may be a specific responsibility of the TBA/CHW.  The number or percentage of pregnant women identified in the first trimester can be taken as one measure of assessing performance.  The number of eligible couples, with two or less number of children, who were provided with contraceptives can be a measure of performance for individuals assigned the responsibility of community-based contraceptive distribution.

Improving Quality of Care

Service data and clinic performance may reveal dissatisfaction with facilities and supplies among both clients and service providers. Therefore, various structural input, supplies and training following clinic performance assessment should be provided to create enabling conditions for workers to perform their tasks well.

Evaluation

Evaluation is a process for systematically and objectively determining the relevance, efficiency, effectiveness and impact of activities in the light of their objectives.  It is an organisational process for improving activities still in progress and for aiding management in future planning, programming and decision-making.

Evaluation in the context of meeting the RH needs of all people, requiring services throughout their life span, is concerned with the assessment of effects Ð benefits or ill effects (intermediate objectives) and impact (long-range objectives) Ð on the beneficiaries, who are preferably classified into income and gender groups.  Its concerns are who or which group has benefited (or has been adversely affected), by how much (compared to the situation before the activity), in what manner (directly or indirectly), and why (establishing causal relationships between activities and results to the extent possible).

Evaluation here is therefore viewed as a learning process in the implementation of the service programme.  The very process of carrying out an evaluation can be just as important as the conclusions drawn, since involvement in the process itself often induces a better understanding of the activities under evaluation, and a more constructive approach to their implementation and to any future action required.  Limitations of evaluations often include the varying degree of sensitivity of the indicators selected; the availability of necessary information and its quality; and the different capabilities of those who carry out the observations of the implementation of the service programme.

Selecting Indicators for Monitoring of Reproductive Health

Most health indicators focus on problems that need action rather than function as tools for diagnosis.  For example, an indicator shows a drop in new acceptors for intra-uterine device (IUD) without any apparent reason.  However, the drop may prompt a check on the stock of IUDs in the clinic.  Thus, indicators often stimulate other activities to be carried out.  It is important to identify specific uses for RH indicators by asking which service areas should be assessed, which aspects need to be monitored and which effects can be evaluated.  Perhaps, above all, one should be clear on what purpose a given indicator is to serve.  Once this is established, efforts should be made to ensure the indicator is well defined, measurable, responsive to intervention inputs, and estimated at regular intervals.

Most indicators are expressed in terms of absolute numbers, rates, proportion, averages or categorical variables (i.e. presence or absence).  In the case of rates, ratios and proportions, it is necessary to clearly specify the numerator and demoninator.  It is also important to have a clear understanding of data collection methods and their limitations.

Over the last 25 to 30 years, the prevalence of contraceptive use has been the indicator of individual reproductive behaviour change and also that of societal changes.  These changes constitute a profound transformation of child-bearing patterns, with smaller family size and lower fertility rates.

Criteria for Indicator Selection

Indicator selection raises technical questions about the implication of data collection as well as other operational and financial issues.  For some programmatic issues, the basic statistics required to construct indicators already exist and the major task is to ensure consistent use and proper interpretation.  However, for others, considerable innovative thinking is required to develop appropriate indicators that are sensitive to the complex change involved.  A good indicator has a number of important attributes and those recommended by the World Health Organisation (WHO)3 are outlined below.

To be ethical, an indicator requires data which are ethical to collect, process and present in terms of the rights of the individual to confidentiality, freedom of choice in supplying data, and informed consent regarding the nature and implications of the data required.

To be useful, at the national and international level, an indicator must be able to act as a marker of progress towards improved RH status, either as a direct or proxy measure of impact or as a measure of progress towards specified process goals.  Since computation of national level indicators usually requires aggregation of data collected at a local level, the data should also be useful locally, i.e. follow-on action should be immediately apparent.

To be scientifically robust, an indicator should be a valid, specific, sensitive and reliable reflection of that which it purports to measure.  A valid indicator actually measures the issue or factor it is supposed to measure.  A specific indicator only reflects changes in the issue or factor under consideration.  The sensitivity of an indicator depends on its ability to reveal important changes in the factor of interest.  A reliable indicator is one which would give the same value if its measurement was repeated in the same way on the same population and at almost the same time.

To be representative, an indicator must adequately  encompass all the issues or population groups it is expected to cover; for national and global level indicators the group of interest is the population as a whole including minority groups and adolescents.

To be understandable, an indicator must be simple to define and its value must be easy to interpret in terms of RH status.

An accessible indicator is one for which the data required are already available or relatively easy to acquire by feasible methods that have been validated in field trials.

Indicators for Reproductive Health

Indicators relating directly to ICPD RH programme evaluation have been developed by WHO, United Nations Population Fund (UNFPA) and United States Agency for International Development (USAID).  The list has expanded to cover gender equality and equity, empowerment of women and global/country population stabilisation.  WHO recommends that some of these indicators be selected and adopted, depending on the service needs, readiness and the availability of expertise.  However, the selected indicators need to be integrated into the existing system of data collection and monitoring in the current MCH/FP and PHC service programmes.

WHO, together with other relevant agencies developed 15 sets of RH indicators (Box 2).   UNFPA came up with 120 indicators (summarised in Appendix 1).  The Reproductive Health Indicators Working Group4 and USAID have also developed RH indicators which serve to supplement those already developed by other international agencies.

The indicators have been arranged in a framework that makes their use largely self-explanatory.  The core set of indicators have been organised in sections centering on the following RH components: policy and administrative procedures; FP; maternal health; RTIs/STDs; abortion and post-abortion care; infertility; harmful practices; advocacy/IEC; media promotion; community involvement and outreach; capacity-building of personnel; knowledge, attitude and practice of RH/FP; population education; and other advocacy and IEC.  The indicators used in the RH indicators tables (Appendix I ) are developed by the Technical and Evaluation Division of UNFPA and ICOMP.  The Quality of Care indicators in RH Services for Women and Access of Adolescents to Information and Services - context and process indicators, are by the Latin American and Caribbean Women's Health Network (LACWHN, 1998).

Priority Indicators

From the overall list of indicators present in the proceeding section, the UNFPA has specified a priority sub-set of indicators for use in RH programmes.  The basic set of 20 priority indicators the UNFPA has selected are presented below.  It is believed that these indicators will frequently be required to assess the performance of RH programmes.

Input Indications

Data for measuring RH indicators can be obtained from a variety of sources, such as population and housing census; civil registrations systems; administrative records; service statistics; facilities and service-provider and clients follow-up surveys; health information including hospital records.  The main data source for compiling a given indicator will vary from country to country.  In situations where complete and reliable data systems are lacking, it would be necessary to estimate the indicators through indirect procedures.

Quality of Care Indicators

The quality of care for RH services has may dimensions, and although all these dimensions require attention, it may be useful, when instituting a process of quality improvement, to begin by emphasising, follow-up, and inter-personal aspects of services.

 ¥ Counselling
  Average time spent with a client
  Percentage of contraceptive users who know about side effects
  Percentage of couples who received RH counselling

 ¥ Follow-up
  Percentage of clients who know when they should return for service
  Percentage of sterilisation client visited at home by the health worker within one week of the procedure

 ¥ Technical
  Percentage of providers who know about proper practices
  Percentage of procedures that meet specified standards
  Percentage of procedures in which aseptic conditions were maintained

The data for these indicators come from observations of service provision and interactions between the service provider and clients, and from client feedback and provider interviews.  Many dimensions of quality cannot be measured quantitatively and require qualitative measures.  For instance, measuring the inter-personal dimension of quality requires judgement by observers.  The extent to which a programme meets the needs of women as well as men can be assessed in part by the involvement of clients in programme planning, implementation, and evaluation.  How well the organisation of service delivery addresses men's and women's concerns and responds to their social situation is another indicator of met or unmet need.

Conclusion

Over the last few years there has been a huge amount of work on the development of RH indicators, particularly since the ICPD in 1994.  Many of these initiatives have generated compilations of indicators without any clear description of what selection criteria were used and how.  Demand for indicators has generally outstripped the supply of necessary data and few developing countries have the data generation capabilities required to report on many of the indicators currently defined for monitoring RH status and progress.

At the local level, it has been recommended that countries should select indicators most appropriate to their needs and capacity for data collection (Graham and Macfarland, 1997). In a context defined by a general shortage of health information, particularly at community level, the proliferation of RH indicators is a matter of concern to the extent that it tends to impose unwelcome reporting burdens on national data collection systems.

The move towards constructing common set of indicators to monitor progress of RH services for each country has many advantages.  It enables the aggregation of data at various levels Ñ country, region and the world.  It also enables countries to achieve uniform technical standards in the form of standard concepts, definitions, procedures and methodologies.  Indicator selection is a dynamic and complex process.  In this chapter, an attempt has been made to classify indicators according to the needs of programme management.  Development of a meaningful set of indicators requires testing in the field and subsequent adaptation.  This difficult and invaluable exercise can only be carried out through systematic efforts by governments and donor agencies in a cost-effective manner.

Process Indicators
Percentage of population within one hour walk from FP service delivery point
Percentage of FP SDPs with provision of RTI/STD services
Percentage of post-partum women (six weeks after delivery) offered FP

Output Indicators
Percentage of married women of reproductive age who want to postpone or stop child-bearing and who are not currently using any contraceptive method
Percentage of clients asked about their:
   i.   Reproductive intentions
   ii.  Concerns about contraceptive methods
Adolescent (<age 20) fertility rate *

Output Indicators
Percentage of delivering women who developed obstetric complications and received emergency obstetric care
Percentage of deliveries  that are C-section
Percentage of pregnant women attending ante-natal services who received
   i.   Iron/folate (100 tablets)
   ii.  Tetanus immunisation (two doses)
Percentage of pregnant women receiving maternal services expressing satisfaction with:
   i.    Pre-natal care
   ii.   Delivery services
   iii.  Post-natal care

Output Indicators
Prevalence of RTIs/STDs among women attending gynaecological clinics
Estimated prevalence of HIV among adolescents, men and women
Prevalence of urethral discharge among men aged 15-49
Percentage of clients expressing satisfaction with RTI services

Output Indicators
Annual number of:
   i.   Legal abortions
   ii.  Estimated illegal abortions
Percentage of obstetric and gynaecological admittances due to abortion complications

Output Indicators
Percentage of service providers trained in counselling techniques/interpersonal skills

Output Indicators

Output Indicators

Output Indicators
Number of organisation/membership of coalitions formed to achieve advocacy objectives

Bertrand, Jane and Tsui, Army (1995). Indicators for Reproductive Health Programme Evaluation.  Chapel Hill: University of North Carolina.

ESCAP (1998).  Implementation Challenges of Reproductive Health, including Family Planning and Sexual Health: Monitoring, Evaluation and Impact Assessment:  Development and Utilisation of Indicators, background paper at the High-level Meeting to review the ICPD Programme of Action, March, Bangkok.

Family Planning Management Development (1999).  "Human Resource Development Assessment Tool," a supplement to The Manager, vol. 8, no. 1, Spring 1999.

LACWHN (1998).  The Cairo Consensus: Women Exercising Citizenship through Monitoring - The Cairo+5 Process.

Satia, Jay and Subramanian, Sangeeta Sokhi (1999).  "Developing an Alternative System of Monitoring Indicators for the Family Welfare Programme."  In Improving Quality of Care in India Family Welfare Programme, Koenig, Michael A. and Khan, M.E. (eds.).  New York: The Population Council.

UNFPA (1997).  Indicators for Population and Reproductive Health Programmes.  New York: Technical and Evaluation Division.