Evaluating implementation of the World Health Organization’s Strategic Approach to strengthening sexual and reproductive health policies and programs to address unintended pregnancy and unsafe abortion
Reproductive Health volume 14, Article number: 153 (2017)
We conducted a process evaluation to assess how the World Health Organization’s (WHO) Strategic Approach to strengthening sexual and reproductive health policies and programs (“the SA”) was used in 15 countries that requested WHO’s technical support in addressing unintended pregnancy and unsafe abortion. The SA is a three-stage planning, policy, and program implementation process. We used the social ecological model (SEM) to analyze the contextual factors that influenced SA implementation.
We used a two-phased sequential approach to data collection and analysis. In Phase A, we conducted a document and literature review and synthesized data thematically. In Phase B, we conducted interviews with stakeholders who used the SA in the countries of interest. We used a qualitative method triangulation technique to analyze and combine data from both phases to understand how the SA was implemented in each country.
Data from 145 documents and 19 interviews described the SA process and activities in each country. All 15 countries completed Stage 1 activities. The activities of Stage 1 determined activities in subsequent stages and varied across countries. Following Stage 1, some countries focused on reforming policies to improve access to sexual and reproductive health (SRH) services whereas others focused on improving provider-level capacity to enhance SRH service quality and improving community-level SRH education. We identified factors across SEM levels that affected SA implementation, including individual- and community-level perceptions of using the SA and the recommendations that emerged from its use, organizational capacity to conduct SA activities, and how well these activities aligned with the existing political climate. Stakeholders perceived SA implementation to be country-driven and systematic in bringing attention to important SRH issues in their countries.
We identified key success factors for influencing the individual, organization, and system change required for implementing the SA. These include sustaining stakeholder engagement for all SA stages, monitoring and reporting on activities, and leveraging activities and outputs from each SA stage to obtain technical and financial support for subsequent stages. Results may be used to optimize ongoing implementation efforts to improve access to and the quality of SRH services.
Plain English summary
We evaluated how the World Health Organization’s planning, policy, and program implementation process, (called the Strategic Approach to strengthening sexual and reproductive health policies and programs or “the SA”), was used in 15 countries to address unintended pregnancy and unsafe abortion. SA implementation was influenced by individual-, community-, organizational-, and policy-level factors. Stakeholders perceived SA implementation to be country-driven and systematic in bringing attention to important sexual and reproductive health issues in their countries. Key success factors for implementing the SA include ensuring stakeholder engagement during all stages of SA activities, monitoring activities, and using outputs from each SA stage to gain support for subsequent stages. Findings from this study can guide efforts in countries that are currently using the SA or conducting similar activities to improve sexual and reproductive health services.
Of the 210 million pregnancies that occur globally each year, it is estimated that 80 million are unintended . In low- and middle-income countries, 24% of pregnancies ended in abortion in 2010 to 2014 . Women are more likely to resort to an unsafe abortion when provisions for safe abortions are restricted, unavailable, or inaccessible . Recent studies estimate that 8–18% of maternal deaths worldwide are due to unsafe abortion, and the estimated number of abortion-related maternal deaths in 2014 ranged from 22,500 to 44,000 [4,5,6]. Almost all abortion-related deaths occur in low- and middle-income countries, and estimates for 2012 indicate that 6.9 million women in the developing world were treated for complications from unsafe abortion .
Reducing the incidence and consequences of unsafe abortion are essential for achieving United Nations Sustainable Development Goal 3 — to ensure universal access to sexual and reproductive health (SRH) care and reduce global maternal death rates. Over the past 15 years, SRH decision makers in several countries have worked in collaboration with non-governmental organizations (NGOs) and international agencies to adopt the World Health Organization’s (WHO) Strategic Approach to strengthening SRH policies and programs (herein referred to as “the SA”). The SA is a planning, policy, and program implementation process that countries can use to assess their SRH needs and strengthen policies and programs to address these needs. Since being field tested in 1993 as a systematic approach to contraceptive introduction, the SA has been used to address a broad range of SRH issues (particularly reproductive tract infections, HIV/AIDs, maternal health, and family planning). From 1997 to 2014, 15 countries (i.e., Vietnam, Romania, Bangladesh, Mongolia, Ghana, Moldova, Macedonia, Ukraine, Zambia, Guinea, Malawi, Russian Federation, Senegal, Kyrgyzstan, and Sierra Leone) requested technical support from the WHO to use the SA to address unintended pregnancy and unsafe abortion. The current study is a process evaluation of how the SA was applied in these 15 countries. This evaluation was not intended to measure the impact of the SA on reducing unsafe abortion in these countries.
The SA is guided by public health, social and management sciences, and the International Conference on Population and Development (ICPD) principles; it is a three-staged implementation process that uses a systems framework, a participatory process emphasizing country ownership, and a reproductive health philosophy focusing on improving equitable access to and quality of care . The SA is composed of the following stages (Fig. 1):
Stage 1: Conducting a field-based strategic assessment to identify and prioritize SRH needs and generate consensus recommendations for addressing these needs;
Stage 2: Developing and pilot testing innovations (e.g., policy, programs, and services) on a limited scale at different levels of the health system and evaluating these interventions to determine if implementation is feasible, acceptable, effective, and sustainable in the particular context; and
Stage 3: Scaling up to expand access beyond pilot sites and strengthening health system capacity to sustain the provision of programs and services.
Evaluating the “process” of implementing the SA
There is a gap in research on understanding the processes and mechanisms by which complex interventions are implemented in practice [9,10,11]. Many SRH programs report on program impacts without describing how, when, where, and under what conditions programs were developed and implemented . As a result, it is difficult to replicate programs that are successful in achieving intended outcomes. For unsuccessful programs, it is unclear whether the program is ineffective, whether it was limited by contextual factors, or whether it was not implemented well . Process evaluations can help to unpack the “black box” of implementation by providing insight into how program components interact with factors in the implementation context that help or hinder implementation processes [11, 12].
We conducted a process evaluation guided by the following objectives:
To understand how the SA was applied in each of the 15 countries and analyze the processes and outputs of each stage of the SA;
To identify and analyze contextual factors that influenced implementation of the SA using the Social Ecological Model (SEM; Additional file 1) as an analytical framework to examine health system dynamics across countries and time; and
To identify key success factors for implementing the SA to inform current and future implementation efforts.
We used a two-phased qualitative sequential approach (Fig. 2).
Phase A: document and literature review
The primary document review phase (conducted between April and December 2015) was designed to collect and analyze the strategic assessment reports for all 15 countries. We then identified and analyzed secondary documents (e.g., country background papers; historical documents; national standards, policies, and guidelines for abortion care; donor reports; and information educational materials).
A librarian conducted a literature search of databases (e.g., MEDLINE and EMBASE) and grey literature in July 2015 to identify English-language articles (published between 1946 and 2015) on the SA and reproductive health issues in the countries of interest. Search terms consisted of keywords, such as “World Health Organization”, “strategic approach”, “abortion”, “birth control”, “contraception”, “fertility control”, “family planning”, “unwanted pregnancy”, and “unplanned pregnancy”, (Additional file 2). Two reviewers independently screened the titles and abstracts of records to select articles for full-text review. Articles were included for full-text review if they explicitly mentioned the WHO’s SA recommendations and/or described SA initiatives being implemented at the national or local level in any of the 15 countries of interest. Articles were excluded if they did not directly mention the WHO’s SA or described initiatives outside of the countries of interest. Inter-rater reliability was calculated to assess the degree to which the two reviewers consistently agreed on which articles were eligible for full-text review. The calculated Kappa coefficient, k = 0.82, indicated substantial inter-rater agreement ; discrepancies were reconciled through discussion. Full-text versions of the selected articles were then retrieved and reviewed by two independent reviewers to determine eligibility.
Primary and secondary documents were organized by country or document type. Data were abstracted from the documents using a standardized template (Additional file 3). The data abstraction template was piloted by two reviewers, who independently used the template to review a primary document from one country selected at random. The reviewers then met to deliberate and refine the template. Subsequently, 20% of the primary documents were randomly selected, double coded, and analyzed for reliability across two independent reviewers (k = 0.66). Reviewers reconciled discrepancies through discussion and further revised the data abstraction template to create a final version. Reviewers then used the final template to independently abstract data from the remaining documents.
Two analysts reviewed abstracted data from the documents. Following familiarization with the data, analysts generated initial codes, identified themes among the list of codes, and developed a thematic framework of analysis . Analysts then reviewed and synthesized abstracted data according to the major themes.
Phase B: semi-structured interviews
Interviews were conducted between September 2015 and January 2016 to explore perceptions of two stakeholder groups: (1) frontline implementers who used the SA for policy and program strengthening in their countries and (2) technical staff who had a role in development, operations, or technical support of the SA activities in these countries (Fig. 2).
Stakeholders were identified and contacted by the WHO and introduced to the research staff. The research staff asked country stakeholders to provide additional documents and names of potential stakeholders who could be recruited for interviews through purposive and snowball sampling. Interviews were 45–60 min in length and conducted in English by experienced interviewers via telephone or in person using semi-structured interview guides (Additional file 4). Interview guide development was informed by research on factors that affect the implementation process . Interviews were audio recorded and transcribed verbatim and de-identified prior to analysis.
The SEM, a theory-based framework [16–18], was used to understand the interplay of factors and conditions under which the SA has been used and the processes involved with implementing the SA over time (1997–present) and in different contexts (15 countries). An overview of the SEM and other relevant frameworks is provided in Additional file 1.
Two qualitative analysts independently familiarized themselves with the data by reviewing interview field notes and transcripts to develop an initial list of codes and themes. Analysts then compared their initial list of codes and developed a thematic framework of analysis (i.e., the codebook), which was subsequently applied to the data (Additional file 5). This framework was piloted with two transcripts and further refined to fit the data. The framework was adapted throughout the coding process to capture emergent themes . A modified audit consensus coding approach  was used whereby interview transcripts were divided into six groups and coded in sequential rounds. Analysts independently applied the coding framework to interview transcripts and at the end of each round, inter-rater reliability between analysts was determined using the Kappa coefficient. Analysts discussed discrepancies and made changes to the coding file until a Kappa coefficient ≥ 0.6  was achieved for each individual code/transcript combination. The number of discrepancies between coders decreased after the second round of coding. For the remaining transcripts, the primary analyst coded all transcripts and the secondary analyst audited one randomly selected transcript per round. Once all data were coded, both analysts met to discuss the emergent themes and to chart data according to theme and country using the framework matrices tool in NVivo 10.
A multi-source (i.e., document and literature review and interviews) qualitative method triangulation technique  was applied to analyze the data. Data collected from all sources were combined through a matrix where data were mapped according to country, SEM, and stage of the SA. This method allowed analysts to consider data from the document and literature review and interviews collectively to better understand how the SA was implemented in each country.
Ethics, consent, and permissions
Ethics approval was obtained from the University of Toronto Health Sciences Research Ethics Board (REB #31181), and verbal informed consent was obtained from all participants. Data were de-identified prior to analysis.
The number of data sources (i.e., document and literature review from Phase A and semi-structured interviews from Phase B) per country varied (Additional file 6), with the highest proportion of data coming from Malawi (n = 21) and Ghana (n = 18), followed by Macedonia (n = 16) and Zambia (n = 16). Romania (n = 4) had the lowest number of available data sources. A breakdown of the results from Phase A and Phase B is provided below.
Phase A: document and literature review
For the primary document review, we analyzed 15 strategic assessment reports. For the secondary document review, we analyzed 61 documents from country stakeholders and 69 documents from the literature search. The literature search (Fig. 3) yielded 169 records after duplicates were removed. After screening for title and abstract (Stage 1) and conducting full-text screening of articles (Stage 2), 69 documents were included in the final analysis. Additional file 7 provides the complete list of citations.
The 69 documents selected from the literature search described a wide range of SRH policies, programs, and practices. Of the 44 peer-reviewed articles, 13 articles examined abortion methods and practices, 11 articles evaluated programs to improve access to and quality of SRH services, eight articles reported the incidence of abortion and/or abortion-related deaths, four articles were related to reproductive rights and policy reform, three articles described social factors and attitudes towards contraceptives and abortions, three articles examined specific strategies and tools to improve the quality of SRH services, and two articles offered guidance on providing high-quality SRH services. The 25 grey literature documents included progress reports, highlights reports, as well as biennial and annual technical reports from WHO’s Department of Reproductive Health and Research.
Phase B: semi-structured interviews
We completed 19 interviews with a response rate of 83%: 11 with technical staff and eight with frontline implementers.
Overview of SA implementation and factors affecting progression through the SA stages
Figure 4 presents the timeframe during which the SA was implemented in each country. All 15 countries completed Stage 1 activities. All except one country (Guinea) initiated Stage 2 activities based on Stage 1 outputs/recommendations. Seven countries (i.e., Ghana, Moldova, Mongolia, Romania, Russian Federation, Ukraine, and Vietnam) were perceived to have simultaneously engaged in Stage 2 activities (i.e., piloting certain innovations) and Stage 3 activities (i.e., scaling up other innovations from Stage 2). Interview participants could not confirm the status of activities (i.e., whether these activities were ongoing or completed) in the countries that had initiated Stage 3. Examples of common country-specific activities by stage are provided in Table 1.
A number of factors were perceived to have influenced the implementation of the SA overall and progression through the SA stages.
Individual perceptions of using the SA to address SRH issues
Some senior-level stakeholders believed that they understood the scope of unintended pregnancy and unsafe abortion in their respective countries and felt that they did not need to collect additional data through Stage 1 assessment activities. Others believed that qualitative assessment activities (i.e., interviews and discussions) were not appropriate in their context because no one would openly discuss a stigmatized topic, such as abortion, and quantitative methods were preferred to understand the magnitude of the problem. Participation in global SRH conferences was perceived to have stimulated stakeholders’ (e.g., from Ghana, Malawi, and Zambia) interests in using the SA. Similarly, Ministry of Health (MOH) and WHO endorsement of the SA provided credibility to the process and influenced country stakeholders’ decisions to use the SA.
In-country technical capacity to develop, pilot test, and evaluate interventions
In some countries (e.g., Bangladesh and Ghana), strong local NGO support facilitated SA implementation and was used to leverage more reluctant stakeholders; changes in local NGO interest or leadership were perceived to have affected implementation efforts. Technical support from international organizations, such as the WHO, was perceived to facilitate implementation. In particular, support for the in-country scientific and ethical review processes that preceded the assessment fieldwork and engagement of SA implementers from other countries to share their experiences with stakeholders in countries planning to use the SA were found to be especially helpful. Establishing a central governmental organization to work in collaboration with both local and international organizations to coordinate SA activities was also perceived to have facilitated implementation.
Countries’ ability to secure funding for all stages of SA implementation
Although all 15 countries secured funding to initiate the SA and conduct Stage 1 activities (i.e., strategic assessment fieldwork), countries varied in their opportunities to mobilize resources for the activities in subsequent stages. Knowledge transfer and exchange of information between countries on the implementation of Stage 2 interventions was perceived to have helped mobilize resources for Stage 3. However, many countries did not consistently monitor, evaluate, or report on the implementation of Stage 2 interventions. As a result, there was limited data on whether Stage 2 interventions were effective; this was identified as a potential barrier to scaling up these interventions.
Changes in the political climate during SA implementation
Changes in the political climate affected progression through SA stages. In Macedonia, for example, the political party that came to power following Stage 1 activities promoted aggressive, anti-abortion messages and changed the abortion law to make it more restrictive, thereby making it difficult to implement subsequent SA activities. An interview participant who provided technical assistance to a number of different countries mentioned how staff turnover resulting from political changes also affected SA implementation:
‘I have been in countries where over a course of five years they had six different Ministers of Health, so that kind of change at that level often has knock down effects at lower levels in the ministries in changing priorities’. – Interview Participant # 118.
Implementation of SA Stage 1: strategic assessment to identify needs and generate consensus recommendations to address needs
All 15 countries completed the Stage 1 strategic assessment activities in a similar manner with small logistical variations (e.g., size and composition of assessment team and coverage of assessment areas). Representatives from ministries, NGOs, professional associations, and national and international health care organizations participated in the following:
Preparation of a background paper on existing socio-demographic, cultural, political, economic, and public health issues and available research on unintended pregnancy and abortion in the country;
An assessment planning workshop using evidence from the background paper to develop strategic questions for guiding assessment activities, selecting and training assessment team members, and identifying assessment sites;
Fieldwork involving iterative data generation through in-depth interviews and group discussions with a broad range of key informants to explore knowledge and perspectives of unintended pregnancy, contraception, abortion rights, and strategies for addressing SRH issues in the country. Data were analyzed and used to draft recommendations for specific follow-up actions; and
A national dissemination meeting to present assessment findings to multiple stakeholders, such as policymakers, program managers, health care providers (HCPs), NGOs, UN agencies, SRH advocates, and local human rights organizations) who worked to generate consensus on the follow-up recommendations. See Additional file 8 for examples of Stage 1 recommendations commonly identified across countries.
Table 2 outlines the activities of Stage 1 and factors influencing Stage 1 implementation organized by SEM level (i.e., individual, community, organizational, and national/policy).
Implementation of SA Stage 2: developing and testing innovations based on Stage 1 recommendations
The focus of Stage 2 activities varied among countries based on the legal status of abortion at the time of SA implementation; specifically whether they had restrictive abortion laws (e.g., Senegal, Sierra Leone and Malawi) or less restrictive abortion laws (e.g., Ghana, Kyrgyzstan, Moldova, Ukraine and Zambia). Table 3 outlines the Stage 2 activities and factors influencing Stage 2 implementation.
Implementation of SA Stage 3: scaling up successful interventions
Stage 3 activities differed for each country depending on the progress made toward implementing Stage 2 activities and continued stakeholder buy-in. Although interview participants indicated that scale-up activities had been initiated in seven countries (i.e., Ghana, Moldova, Mongolia, Romania, Russian Federation, Ukraine, and Vietnam), there was limited information about the specific nature of these activities. Ukraine, Ghana, Moldova, and Romania initiated scale-up activities to expand access to family planning and safe abortion services beyond pilot sites. For example, Ukraine conducted a workshop to develop a scaling-up strategy for the Comprehensive Care for Unwanted Pregnancies project; the strategy was approved by the government and distributed to all regions for implementation. Table 4 outlines Stage 3 activities and factors influencing Stage 3 implementation.
Perceptions of SA implementation
Interview participants’ descriptions of how the SA was implemented in their settings and perceptions of the key strengths of the SA directly aligned with the essential features or guiding principles of the approach itself: country-driven, participatory, and staged process using a systems framework.
Interview participants shared that the decision to use the SA was always country initiated and activities were largely country-driven. Stakeholders selected their area of focus (e.g., contraceptive introduction, assessment of SRH policies and programs, or assessment of abortion service delivery), and developed strategic questions based on their existing systems and structures. They led all stages of the fieldwork, from identifying the pool of participants to selecting sites to reflect regional, cultural, socio-economic, and programmatic differences. The opportunity to adapt the SA to the local context facilitated ownership and acceptance of the assessment process, findings, and recommendations by country stakeholders. Stage 2 activities were also locally driven, often by local NGOs. Although the WHO and other international partners provided technical support on various topics (e.g., best practices for guideline development), country stakeholders led the process of developing local solutions based on their expertise in local challenges.
The participatory, multidisciplinary, and multisectoral approach of the Stage 1 strategic assessment process was thought to have provided an opportunity to integrate perspectives of often disparate groups of stakeholders to develop a shared understanding of key issues, as described by the quote below:
‘I think that’s very important… the sort of the multidisciplinary nature of it is really unusual and really valuable aspect of the process, because very often, people don’t talk to each other. And also, many of them never really have had the opportunity previously to go deeply into communities to look at these issues and to talk to communities and to really see what the issues are on the ground. So the multidisciplinary, multi-stakeholder, really participatory nature of the Stage 1 of the strategic assessment are really critical I think.’ – Interview Participant # 116.
Participation in assessment activities helped establish networks for decision making on how to deal with an issue like unsafe abortion that is often affected by social, political, and economic considerations. Participation in Stage 1 also increased team members’ commitment to implementing recommendations in Stage 2. For example, in Romania, the participation of representatives from the MOH, National Health Insurance, and National College of Physicians as part of the Stage 1 assessment team led to piloting a free contraceptives intervention for vulnerable groups in three districts (Stage 2) and scaling it up to 42 districts (Stage 3) in a relatively short period of time. To date, the Romanian government continues to allocate funds to this program every year.
Brings attention to important issues
Interview participants perceived the SA to have played an important role in creating dialogue and drawing attention to a sensitive and often taboo topic, as described in the quote below:
‘Even for the program managers at the Ministry of Health who are involved in the strategic assessment, when they go out and talk to people they start understanding, aha, this is happening. I thought it was only a problem of young university students, you know, other’s problem, there are a lot of stereotypes about abortion. But when they go to the street and talk to people, then hear it is just like from the horse’s mouth. They go to a rural area and people in the rural area tell them that yes, during the three months 2 women died because of you know, botched abortion and things like that, an eye opener’.- Interview Participant # 101.
Stage 1 helped to dispel common misconceptions about abortion and allowed individuals and groups with decision-making authority to examine whether their country’s existing policies and services were responsive to community needs. In some cases, the assessment helped highlight important social issues; for example, in Zambia, the assessment brought attention to the important issue of violence against women, and the country has since developed guidelines and programs to address some of these issues.
Staged and systematic approach
A number of interview participants, particularly frontline implementers of the SA, perceived the approach to serve as a structured and systematic process for identifying important issues and driving key activities. Outputs of Stage 1 (e.g., consensus recommendations) helped country stakeholders prioritize what to do next and introduce change in discrete, manageable units. The staged, incremental process allowed countries to move cautiously towards adopting recommended activities. For example, the SA was used to implement the national health strategy in Moldova, where linking pilot testing with ongoing monitoring and evaluation activities provided an opportunity to see the CAC model in action and demonstrate how to follow standards for medical abortion and manual vacuum aspiration for training HCPs.
The SA is designed as a participatory approach within a systems framework to identify and mobilize resources and relationships existing within the health system, which affect the feasibility, prioritization, acceptability, effectiveness, and sustainability of SRH interventions. This evaluation yielded important insights into the processes and mechanisms by which the SA was used to address issues related to unintended pregnancy and unsafe abortion in 15 countries. It helped unpack the “black box” of implementation by providing insight into how the activities within each stage of the SA were carried out and the factors in the implementation context that helped and hindered the implementation process.
Gaining perspectives from individuals who were involved with implementing the SA in these countries helped identify the strengths of the SA (i.e., how it is a country-driven, participatory, and systematic process for identifying and addressing key SRH issues) as well as the limitations of the SA. Implementing the SA is a complex and time consuming process; it requires countries to critically assess their existing SRH policies, programs, and services; consider the contextual factors and stakeholder relationships that influence each of these policies, programs and services; and identify and reach consensus on priority areas to focus SRH improvement efforts. To progress through SA stages, countries need to obtain and sustain stakeholder commitment and secure financial support and technical assistance for navigating challenges related to specific activities in each stage. Although Stage 1 outlines specific strategic assessment activities for all countries to engage in, country stakeholders have less direction on how to proceed through subsequent stages (e.g., specific activities related to piloting and scaling up) due to the variation in the scope and range of Stage 2 activities by country.
Key success factors and opportunities to enhance SA implementation
This evaluation identified key success factors for implementing the SA and opportunities to leverage these factors to enhance ongoing and future implementation efforts in varied settings. It also identified country stakeholders’ key role in developing, piloting, and scaling up policies and programs to address SRH issues in their respective countries. First, countries that made the most progress towards implementing the SA were successful in engaging and maintaining country stakeholder support beyond Stage 1 activities. Opportunities to sustain engagement of country stakeholders beyond Stage 1 include sharing resources on SRH guidelines and interventions, inviting them to share their insights on how to plan for sustainability as part of Stage 1 activities and building capacity on how to scale up interventions, by sharing tools such as EXPANDNET capability building resources . Second, clarifying that Stage 1 of the SA is an opportunity to better understand unintended pregnancy and unsafe abortion rather than to promote abortion services enhanced country stakeholders’ buy-in to launch the SA. Third, in some countries, following up Stage 1 assessment activities (i.e., qualitative interviews and focus group discussions) with quantitative data collection offered a complementary method for providing evidence to strengthen stakeholder support for reforming policies and programs. Fourth, establishing a central coordinating body (e.g., the strategic assessment team) to coordinate SA activities was perceived to facilitate SA implementation. When establishing a central coordinating body, countries can consider assessing leadership buy-in  or using coordinated leadership and organizational development strategies  to improve leadership and organizational supports for implementing and sustaining SA interventions. Countries can also employ a contingency plan for personnel turnover through an ongoing process of training and feedback to establish a stable team . Coordination efforts can be further strengthened by using Communication for Development (C4D) strategies  to comprehensively target each level of the SEM for achieving the behavioral and social change needed to implement the SA. Fifth, leveraging Stage 1 outputs to obtain support and funding for Stage 2 activities facilitated implementation. For example, coordinating and integrating efforts of local NGOs and research organizations interested in reproductive health issues helped maximize in-country technical capacity to implement and evaluate Stage 2 interventions. Securing technical assistance from the WHO and other international organizations early on helped countries avoid losing the momentum or enthusiasm generated during Stage 1. Sixth, countries that monitored and reported on SA interventions were able to use these data to show progress and secure additional funding. Incorporating a formal evaluation plan (including process and outcome metrics) and using SRH reporting tools, such as Programme Reporting Standards for SRH programmes , can help countries ensure consistent monitoring of process and outcomes.
There are limitations to evaluating the process of implementing the SA. First, interviews were conducted with a relatively small group of individuals. The SA is a multi-year process, and in some cases, retrospectively evaluating the process meant that the stakeholders who were initially involved with SA activities were no longer involved with ongoing efforts. This may have limited the number of country stakeholders available to participate in the evaluation. Second, the study design and methods were influenced by considerations of feasibility in terms of available resources, number of country sites, and availability of information. For example, face-to-face engagement with country stakeholders may have encouraged more responsiveness than email contact and telephone interviews; however, the former was not feasible due to resource limitations. Language and communication barriers in cases where English was not a first language may have also affected stakeholder involvement with the evaluation. Third, there was considerable variation in the quantity of data available for each country, which may affect the generalizability of findings. In addition, information from documents was not always linked directly to the SA. Most country documents had limited descriptions of Stage 2 and 3 activities, and these were often not explicitly linked to Stage 1 recommendations. Fourth, the document review phase included English-language articles only, potentially limiting the review of related documents in other languages. The document review process also required interpretation by the study team, which may have led to an information bias, misclassification, or errors in linking activities to output variables. We note, however, that the data calibration exercises were done with good agreement, and the document review phase was followed by interviews, which enabled the study team to supplement findings by collecting richer insights on factors related to implementation process and context . Finally, because bias may have been introduced due to self-reporting, findings may not be applicable to other implementers and their experiences. The study team designed interview questions to encourage participants’ retrospective reflections on the implementation process, and data from multiple informants was compared and synthesized, which likely helped reduce bias from memory. Conclusions were based on the consistency of data from multiple sources and triangulation of data aimed at increasing the rigour of the data collection and analysis process.
This study focused on how the SA was applied in 15 countries to address issues of unintended pregnancy and unsafe abortion. Although we could not directly assess the quality with which each country applied the SA, the alignment between the perceived strengths of the SA and the key elements of the SA (e.g., country-driven, participatory, and systematic) suggest that the SA has been implemented as intended in most countries. The study identified gaps in the implementation process that should be addressed as well as opportunities that could be leveraged by others who are considering using the SA to address SRH issues in their countries.
Future studies may consider conducting an outcome or impact evaluation using quantitative outcomes data along with process data to identify changes in health systems and outcomes as a result of applying the SA.
The SA is a country-driven, participatory, and comprehensive process model for identifying and addressing SRH issues. This process evaluation of 15 countries that applied the SA to address unintended pregnancy and unsafe abortion demonstrated how implementation is influenced by interactions between individuals, evidence, and context. The evaluation identified contextual factors that affected implementation at multiple levels in each country using the SEM as an analytical framework. This afforded a deeper understanding of the key success factors that are required for planning and executing SA activities. These factors include continued stakeholder engagement, coordination, monitoring documentation, and technical and financial support for implementing the SA. Lessons learned from this evaluation can be used to optimize the continued use of the SA and to inform future applications of the SA. Findings can also be used to guide similar activities, programs, and policies to improve access to and the quality of SRH services.
Comprehensive abortion care
Dilation and curettage
Health care providers
International Conference on Population and Development
Ministry of Health
Strategic Approach to strengthening sexual and reproductive health policies and programs
Social ecological model
Sexual and reproductive health
World Health Organization
WHO. Unsafe abortion: global and regional estimates of the incidence of unsafe abortion and associated mortality in 2008. Geneva: WHO; 2011.
Sedgh G, Bearak J, Singh S, Bankole A, Popinchalk A, Ganatra B, et al. Abortion incidence between 1990 and 2014: global, regional, and subregional levels and trends. Lancet. 2016;388(10041):258–67.
WHO. World Health Organization information sheet in unsafe abortion incidence and mortality global and regional levels in 2008 and trends during 1990–2008. Geneva: WHO; 2012.
Kassebaum NJ, Bertozzi-Villa A, Coggeshall MS, Shackelford KA, Steiner C, Heuton KR, et al. Global, regional, and national levels and causes of maternal mortality during 1990-2013: a systematic analysis for the global burden of disease study 2013. Lancet. 2014;384(9947):980–1004.
Say L, Chou D, Gemmill A, Tunçalp Ö, Moller A-B, Daniels J, et al. Global causes of maternal death: a WHO systematic analysis. Lancet Glob Health. 2014;2(6):e323–e33.
Singh S, Darroch J, Ashford LS. Adding it up: the costs and benefits of investing in sexual and reproductive health 2014. New York: Guttmacher Institute; 2014.
Singh S, Maddow-Zimet I. Facility-based treatment for medical complications resulting from unsafe pregnancy termination in the developing world, 2012: a review of evidence from 26 countries. BJOG. 2015;123:1489–98.
WHO. The WHO strategic approach to strengthening sexual and reproductive health policies and programmes. Geneva: WHO; 2007.
Pearson M, Brand SL, Quinn C, Shaw J, Maguire M, Michie S, et al. Using realist review to inform intervention development: methodological illustration and conceptual platform for collaborative care in offender mental health. Implement Sci. 2015;10(1):1–12.
Brewster AL, Curry LA, Cherlin EJ, Talbert-Slagle K, Horwitz LI, Bradley EH. Integrating new practices: a qualitative study of how hospital innovations become routine. Implement Sci. 2015;10:168.
Salter KL, Kothari A. Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci. 2014;9:115.
Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci. 2009;4:40.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.
Lewis RJS OCW. Carrying out qualitative analysis. In: Ritchie J LJ, editor. Qualitative research practice: a guide for social science students and researchers. Thousand Oaks, California: SAGE; 2008;219–62.
Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41(3–4):327–50.
Bronfenbrenner U. Lewinian space and ecological substance. J Soc Issues. 1977;33(4):199–212.
Stokols D. Translating social ecological theory into guidelines for community health promotion. Am J Health Promot. 1996;10(4):282–98.
McLeroy KR, Bibeau D, Steckler A, Glanz K. An ecological perspective on health promotion programs. Health Educ Q. 1988;15(4):351–77.
Schreier M. Qualitative content analysis in practice. In: Flick U, editor. The SAGE handbook of qualitative data analysis. Thousand Oaks: Sage Publications; 2012. p. 170–83.
Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–3.
Miles MB, Huberman AM, Saldana J. Qualitative data analysis: a methods sourcebook. Thousand Oaks: SAGE; 2014.
EXPANDNET. In: Simmons R, Fajans P, Ghiron L, editors. Scaling up health service delivery: from pilot innovations to policies and programmes. Geneva: WHO; 2007.
Aarons GA, Ehrhart MG, Farahnak LR. The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9:45.
Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10(1):1–12.
Edmondson AC, Bohmer RM, Pisano GP. Disrupted routines: team learning and new technology implementation in hospitals. Adm Sci Q. 2001;46(4):685–716.
York UN. Position paper. Communication for development (C4D) realizing strategic shifts and accelerating results for children. New York: UNICEF; 2009.
Kågesten A, Tunçalp Z, Ali M, Chandra-Mouli V, Tran N, Gülmezoglu AM. A systematic review of reporting tools applicable to sexual and reproductive health Programmes: step 1 in developing programme reporting standards. PLoS One. 2015;10(9):e0138647.
Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38:44–53.
This work was supported by the Department of Reproductive Health and Research, World Health Organization, Geneva, Switzerland.
We would like to thank the many people who contributed to this evaluation, particularly the country and organizational partners who shared their insights on the SA process. We would also like to thank Brooke Ronald Johnson, Yolanda Scoleri, Kristen Dufresne, David Flaherty, and Nadia Bashir.
The work was funded by the World Health Organization.
Availability of data and materials
All data supporting the findings in this manuscript are present in the manuscript. Supplemental information can be found in the additional files. For any additional data, please contact the lead author (SR).
Ethics approval and consent to participate
Ethics approval was obtained from the University of Toronto Health Sciences Research Ethics Board (REB #31181), and verbal informed consent was obtained from all participants. Data were de-identified prior to analysis.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Overview of frameworks. (DOCX 164 kb)
Literature search strategy. (DOCX 17 kb)
Data abstraction template. (DOCX 21 kb)
Interview guides. (DOCX 28 kb)
Coding criteria for analyzing interview data. (DOCX 26 kb)
Overview of data sources by country. (DOCX 18 kb)
Document review data sources. (DOCX 37 kb)
Examples of common SA Stage 1 recommendations. (DOCX 16 kb)
About this article
Cite this article
Rashid, S., Moore, J.E., Timmings, C. et al. Evaluating implementation of the World Health Organization’s Strategic Approach to strengthening sexual and reproductive health policies and programs to address unintended pregnancy and unsafe abortion. Reprod Health 14, 153 (2017). https://doi.org/10.1186/s12978-017-0405-3