An outcome evaluation of a perinatal education programme

How to cite this article: Rundare, A., & Goodman, S. (2012). An outcome evaluation of a perinatal education programme. SA Journal of Human Resource Management/SA Tydskrif vir Menslikehulpbronbestuur, 10(3), Art. #424, 8 pages. http://dx.doi.org/10.4102/ sajhrm.v10i3.424 Orientation: The World Health Organisation (WHO) (2005) reported that at least 260 mothers and babies die every day in South Africa. Most of these deaths are attributed to low standards of maternal and infant care. Research has shown that one way of reducing the maternal and infant mortality rates in hospitals is to improve the standard of care.


Introduction
In the light of the high child mortality rates in South Africa (SA), this evaluation assessed whether or not a Perinatal Education Programme (PEP) delivered its intended short-term outcomes, namely improved knowledge and skills of midwives, in a Western Cape maternity hospital.

Background to the study
Worldwide, approximately 130 million babies are born every year and four million of them do not survive beyond the first seven days after delivery (United Nations, 2001).Developing countries contributed the greatest proportion of these mortalities.The World Health Organisation (WHO) (2005) reported that at least 260 mothers and babies die every day in SA.Most of these deaths are attributed to low standards of maternal and infant care.
One of the Millennium Development Goals (MDGs) of the WHO initiative is to reduce the mortality rate of children under five years of age from 95 per 1000 to 31 per 1000 by 2015 (United Nations, 2001).The United Nation's Children's Fund (UNICEF) reported improved mortality rates of 60 per 1000 for 2010 but warned that this reduction was not sufficient to comply with the MDG standard for child mortality (UNICEF, 2011).To attain this MDG, developing countries like SA need to take drastic measures to reduce both child and maternal deaths by improving the standard of care of mothers and their babies (Osorno, Campos, Cook, Vela & Dávila, 2006;Pattinson, Woods, Greenfield & Velaphi, 2005).

Trends from the research literature
Research has shown that one means of reducing the maternal and infant mortality rates in hospitals is to improve the standard of care (Pattinson et al., 2005;Theron, 1999aTheron, , 1999b)).In order to provide improved care to mothers and their newborn babies, there is a growing need for the continuous professional education and training of midwives (Harris, Yates & Crosby, 1995;Kattwinkel, Nowacek, Cook, Hurt & Short, 1984;Le Roux, Pattinson, Tsaku & Makin, 1998;Osorno et al., 2006).To meet this need, the Perinatal Continuing Education Programme (PCEP) was developed at the University of Virginia, in the USA, between 1975and 1978(Osorno et al., 2006).The use of the PCEP has not only spread across 43 states in the USA, but it has been translated, adapted and implemented in other countries such as China, Poland, Indonesia and Mexico (Cook, 1992;Hesketh, Zhu & Zheng, 1994;Kattwinkel et al., 1984;Katwinkel et al., 1997;Osorno et al., 2006).Osorno et al. (2006), evaluated the effect of the PCEP in Mexico.The population for this evaluation consisted of 1421 doctors, nurses and nursing assistants who were working with pregnant mothers and newborns.The sample consisted of 924 who were enrolled in the programme.Sixty-five percent of these programme participants completed the programme.Improvement was observed in 14 of 23 evaluated neonatal practices.The participants rated the learning material as clear and useful in practice.
In Brazil, an evaluation was undertaken which compared the effectiveness of a conventional five-day training programme with a self-directed, five-week learning programme similar to the PCEP (Suely Arruda Vidal et al., 2001).A pre-test post-test design was used to measure knowledge acquisition and ward practice.No significant differences were found between the learning outcomes of these two training methods.However, self-directed learning was found to be more cost-effective and the better managed of the two.
The PCEP has been adapted for South African conditions by a group of experts comprising obstetricians, neonatologists and midwives.It is called the Perinatal Education Programme (PEP) (Woods, 1999).When adapting the PCEP for SA, Woods evaluated whether or not midwives understood the content of the training manuals.A group of midwives volunteered to complete different learning units within the training manual.Woods found that this group's knowledge of maternal and newborn care improved by approximately 20%.The feedback provided by the volunteers also enabled the manual designers to make specific improvements to the learning material.Theron (1999b) conducted a randomised control trial in three South African towns in the Eastern Cape, in order to assess whether or not the maternal care manual of the PEP was effective in improving the knowledge of midwives.After completion of the PEP intervention (i.e. after 12 months), Theron found a significant improvement in knowledge acquisition in the experimental group.
In their study of three obstetric units in Mpumalanga, SA, Le Roux et al. (1998) showed that PEP programme recipients, in two obstetric units, showed a significant improvement in obstetric practice when compared to the control group.
From the few evaluations that exist of the PCEP and the PEP, it seems that these training programmes were effective at improving the knowledge of midwives.The PEP has been introduced in the Western Cape in SA, but no evaluation exists of its effectiveness in this specific province.The current evaluation, conducted at a maternity hospital in Cape Town, aimed to provide information regarding the effectiveness of perinatal training in this context.

Context and description of the perinatal education programme
The health care system in SA is divided into three levels: primary, secondary and tertiary.Most of the training of health personnel, including midwives, occurs at tertiarylevel facilities which are located in cities.As is the case in other developing countries, SA is not sufficiently resourced to conduct midwifery training at both primary and secondary-level facilities (Theron, 1999a(Theron, , 1999b;;Woods, 1999).In addition, the midwives stationed at primary health facilities, located in rural areas, undergo the least training of all because they are the furthest away from training hospitals (Davies, Hall, Harpin & Pullan, 2005;Theron, 1999b;Woods, 1999).
The PEP was designed to meet some of these training challenges (Theron, 1999a;Woods, 1999).It is a self-help initiative, which does not require a tutor.Students are responsible for their own learning (Theron, 1999a) at their own pace within their places of work (Woods, 1999).This delivery mode makes the PEP cost-effective and easy to manage.
The PEP consists of the following eight learning manuals: 1. Maternal care 2. Newborn care 3. Perinatal HIV and AIDS 4. Primary new-born care 5. Mother and baby-friendly care 6. Saving mothers and babies 7. Birth defects 8.Primary new-born care.
Each of the eight manuals contains different learning units with specific objectives.Before midwives commence with a learning unit, they complete 20 multiple-choice questions (MCQs) in order to assess the current state of their knowledge and skills.Midwives mark their own MCQs with the help of an answer sheet.Thereafter, they study the unit on their own.A study group, consisting of 5-10 midwives, meets every three weeks to discuss a specific learning unit, to share information and working experiences relating to the unit and to learn from each other.At the end of the group meeting, midwives complete and mark another set of 20 MCQs.This assessment is used to identify any existing deficiencies in knowledge and any sections of the learning material, which need more attention.A coordinator, usually a senior nurse, arranges the group meetings, oversees discussion, assessments, practical demonstrations and administers the final examination.After a whole manual has been completed, a formal examination is administered.A mark of 80% is required to pass the examination.A certificate is awarded to students who successfully complete all the examinations.At the maternity hospital where this evaluation was conducted, the PEP was introduced in 2006.At the time of the evaluation, the PEP intervention consisted of the maternal care and newborn care manuals.To date, 42 midwives have completed the maternal care manual of the PEP.The programme is managed by one of the chief professional nurses.Two senior midwives fulfil the role of tutor.They convene discussion groups on a weekly basis and oversee pre-and post-tests.The meetings are scheduled during normal working hours and attendance is compulsory.Kattwinkel et al. (2004) described four components of an evaluation plan that are essential when conducting evaluations of perinatal outreach education programmes:

Evaluation questions
• The first component outlines a clear statement of the purpose of the programme.• The second component identifies the stakeholders.
• The third component measures the process of evaluation in terms of programme structure and information delivery, programme activities and participant characteristics.• The fourth component measures the outcomes of the education programme as exhibited by the target population.
As regards the first requirement of an evaluation plan, namely a clear statement of purpose, the evaluators have utilised the notion of a programme theory.A programme theory describes '… cause-and-effect sequences in which certain program activities are the instigating causes and social benefits are the effects they eventually produce' (Rossi, Lipsey & Freeman, 2004, p. 168).For the PEP, it is assumed that when midwives are exposed to the learning material and process, their knowledge and skills would improve and this would improve the standard of care, which would reduce infant and maternal mortality (see Figure 1).
From the programme description provided earlier, it is clear that the stakeholders are the support staff who assist in programme implementation (a chief professional nurse and two senior midwives) and the programme designers (the group of specialists who adapted the PCEP).The primary recipients are the midwives and the secondary recipients are the mothers and new-born babies.
The following evaluation questions, based on the programme theory and presented in Box 1, will be the focus of this evaluation.

The potential value-add of the study
This evaluation will provide evidence for hospital authorities to make informed decisions on whether or not to increase programme coverage and continue the PEP.

Research approach
A quasi-experimental design (single group pre-test post-test type) was used to determine knowledge and skills acquisition of midwives.

Research participants
The PEP was introduced in the maternity hospital in Cape Town in 2006.By 2008, 42 midwives had completed the maternal care manual.Of these 42 recipients, three were chief professional nurses, two were senior nurses and 35 were professional nurses.These midwives came from different departments of the hospital (e.g.education, admissions, antenatal, post-birth and neonatal wards or units).At the time of the study, all the midwives had sufficient records of their pre-test and final examinations.

Question 1: Coverage
The coverage assessment provides information about the target population and the extent to which this population has participated in the PEP (Babbie & Mouton, 2003).The evaluation questions for coverage were: a.How many midwives practised at this maternity hospital during the evaluation period?b.What was the proportion of midwives who have participated in the PEP at this maternity hospital?c.What was the attendance rate for the midwives who were enrolled in the PEP? d.What proportion of midwives completed the PEP successfully per year?
Question 2: Service delivery The evaluation questions relating to service delivery pertained to the suitability of the programme activities and content for the target population: a. Was the instructional design of the maternal care manual appropriate for midwives and for the purposes of improving their knowledge?b.Were the specific content areas contained in the maternal care manual appropriate for the midwives' work context (maternity wards)?c.Were the theoretical and practical aspects of maternal care adequately covered in the manual and facilitated during group sessions?

Question 3: Knowledge and skill acquisition outcomes
The knowledge and skill acquisition outcomes were the main focus of this evaluation.The following evaluation questions sought to assess these proximal outcomes for midwives: a. Has the PEP improved the perinatal knowledge of the midwives?b.Has the PEP improved the perinatal skills of the midwives?c.In general, have midwives benefited from the PEP and would they recommend it to others?d.Were there any unintended outcomes which could be attributed to the PEP? e.What other perinatal learning initiatives had been in place for midwives prior to their enrolment in the PEP?
Question 4: Organisational support In order to reinforce newly acquired skills and knowledge, trainees are required to practise these in an environment, which is conducive to experimentation (Noe, 2003).The following evaluation question assessed whether or not such an environment was present in the maternity hospital: a. Was the climate in the maternity wards conducive to applying newly acquired knowledge and skills at work? PEP, perinatal education programme.
Apart from programme records, questionnaires were sent to all recipients (42 midwives) and their supervisors (eight supervisors).Thirty-seven (88%) midwives responded; the rest had left the employ of the hospital.Three midwives were male and 34 female, all between the ages of 30 and 35.All midwives had at least one year's experience of midwifery.Of the eight supervisors, five responded.

Measuring instruments
A paper-based questionnaire consisting of four parts was distributed to collect participant and programme data.A paper-based questionnaire containing three 5-point Likerttype items was developed for supervisors.Supervisors had to indicate whether or not they thought that the PEP had improved the knowledge and skills of midwives, whether or not the manual was suitable for midwives and whether or not the programme schedule was user-friendly.An openended item inviting suggestions for the improvement of the module concluded the questionnaire.

Research procedure
The evaluators obtained ethical clearance from the Research Ethics Committee in the Commerce Faculty at the University of Cape Town to conduct the evaluation.
The first evaluator then collated the relevant data for each respondent from the programme records.Thereafter the questionnaire was distributed to the midwives.

Statistical analyses
Descriptive statistics were utilised to analyse the primary data from the questionnaire.Inferential repeated measures statistics were used to analyse the secondary data from the programme records.

Results
The main objective of this evaluation was to assess whether or not the PEP delivered its intended short-term outcomes, namely improved knowledge and skills of midwives in a Western Cape maternity hospital.Such an evaluation will contribute evidence to assist hospital managers to make informed decisions on whether to continue with the PEP or not.
The evaluation results are presented in terms of the evaluation questions formulated earlier.Question 2: Service delivery a. Was the instructional design of the maternal care manual appropriate for midwives and for purposes of improving their knowledge?
All participants reported that the manual was appropriate for them.In general, participants rated the instructional design of the programme positively, except for the time allocated to the different modules and the scheduling of the group meetings (see Table 1).
Four out of the five supervisors indicated that, in their opinion, the scheduling of group meetings suited participants and they strongly recommended that midwives working in their wards attend the programme.

b. Were the specific content areas contained in the maternal care manual appropriate for the participants' work context (maternity wards)?
Most of the participants (89.5%) strongly agreed or agreed that the content of the maternal care manual was applicable to their daily work in the wards.In addition, three of the five supervisors strongly agreed that the maternal care manual was appropriate for the midwives working in the wards.However, one supervisor indicated that the content on asphyxia was too basic.
c. Were the theoretical and practical aspects of the maternal care manual adequately covered in the manual and facilitated during group sessions?
Most participants indicated that the theoretical aspects of the PEP were adequately described and facilitated.However, a minority indicated that assessment of foetal growth and medical problems during pregnancy could be expanded.This is reflected in Table 2.
The responses of the midwives suggested that the practical skills of the maternal care manual were adequately covered (see Table 3).However, some concerns were expressed about coverage and facilitation of skills such as determining foetal condition or gestational age.

Question 3: Knowledge and skill acquisition outcomes
a. Has the PEP improved the perinatal knowledge of the midwives?
The knowledge acquisition of recipients was measured using the pre-test and examination results.
The mean score obtained by the midwives in the pre-test (M = 62.5, SD = 8.61) differed significantly from the mean score in the examination (M = 89.0,SD = 8.15).The t-test for paired means showed a significant difference between the pre-test and post-test means (t [42] = -10.6,p < 0.05).This result demonstrated that there was evidence of improved knowledge.
In addition to the comparison of pre-and post-test results, participants were also asked whether or not they had gained new knowledge.Most participants (94%) strongly agreed or agreed that they had experienced improved knowledge, whilst 3% indicated that they were not sure and 3% reported no improvement.

b. Has the PEP improved the perinatal skills of the midwives?
The majority of participants (97.1%) indicated that they strongly agreed or agreed with the statement regarding increased skills.
Four of the five supervisors indicated that their observations led them to believe that recipients of the PEP intervention were able to apply their new knowledge and skills in the workplace.

c. In general, have midwives benefited from the PEP and would they recommend it to others?
Most participants (96%) indicated that they benefited from the maternal care course.
All participants strongly agreed or agreed that they would recommend the PEP to other midwives working in maternity wards.
d. Were there unintended outcomes which could be attributed to the PEP?
Besides the learning outcomes such as the new knowledge and skills, another outcome of the PEP included increased team work and cooperation.Ninety-five percent of the participants strongly agreed or agreed that participation in the PEP has brought about this change.
e. What other perinatal learning initiatives had been in place for midwives prior to their enrolment in the PEP?
In order to establish if interventions other than the programme might have led to knowledge acquisition, midwives were asked whether or not they had attended maternal care courses other than PEP.Ten percent of the participants attended other maternal care courses in the previous 12 months.These courses included lactation, medication, a perinatal update  Sixty-two percent of the participants believed that there were organisational factors, which reinforced their newly acquired maternal care knowledge and skills.A thematic analysis of this open-ended question indicated that most of these participants cited their impressions, that the PEP was well integrated in the wards as many of the ward procedures were closely linked to the content of the PEP.

Ethical considerations
The Ethics in Research Committee of the Faculty of Commerce, University of Cape Town, approved the evaluation.

Potential benefits and hazards
There were no benefits or hazards for participants who took part in the evaluation.

Recruitment procedures
The chief professional nurse provided the primary evaluator access to the programme records of the midwives and arranged for the distribution of paper-based questionnaires to the midwives and their supervisors.

Informed consent
On the first page of the questionnaire, the purpose, confidentiality of the data and the time it would take to complete were described in detail.

Data protection
The primary evaluator was the only person who had access to the data in the programme documents and the paperbased questionnaires.No participant was identified by name as number codes were used for the data.

Trustworthiness Reliability
Specific evaluation questions listed at the end of the Introduction section were used to collect data.

Validity
The evaluation focused on a specific programme and did not seek to generalise the findings to other, similar programmes.

Discussion
The main objective of this evaluation was to assess whether or not the PEP delivered its intended short-term outcomes, namely the improved knowledge and skills of midwives.The evaluation makes an important contribution to midwifery, as it provides evidence for hospital authorities to make informed decisions on whether or not to increase programme coverage and continue the PEP.
In summary, the results indicated that the 42 midwives who participated in the PEP had a good attendance rate and all completed the module successfully.With regard to service delivery, the participants were of the opinion that the PEP manual was appropriate as far as theoretical and practical content was concerned.In terms of outcomes, most of the participants noted that the PEP improved their perinatal knowledge and skills, and most of them agreed that they would recommend it to other midwives.Finally, the PEP had sufficient organisational support and was perceived as a well-organised module, which was closely aligned with the everyday work of the participants.
The following discussion, like the results, is organised according to the evaluation questions.Over the three years during which the PEP had been running, 42 midwives out of 200 (21%) completed the maternal care module of the PEP successfully.This could be interpreted as coverage of 7% per year, which is quite limited.Given the good attendance and success rate of the module, it is recommended that more programme modules should be rolled out to more midwives.The PEP could even be implemented in a staggered fashion during the year to utilise instructional resources optimally.
Question 2: Service delivery a. Was the instructional design of the maternal care manual the most appropriate for midwives and for purposes of improving their knowledge?
In general, participants were of the opinion that the programme was designed in such a way that it was appropriate for them.Most of the participants expressed positive reactions towards the user-friendly, easy-to-understand learning material, and the effectiveness of the small group discussions and revision sessions.These findings were similar to those of Woods (1999) who concluded that the PEP manual was pitched at the appropriate level of understanding.It should be noted that the PEP was intended to be a self-study programme.However, in the maternity hospital where the evaluation was undertaken, self-study was supplemented with group sessions.From the results it was clear that this aspect of instructional design contributed to the effectiveness of the programme.All of these results, relating to instructional design, confirmed that the PEP module was implemented optimally as a learning intervention.
However, approximately a third of the participants indicated that they would have liked more time to absorb and integrate the learning material.It is suggested that the facilitators check understanding at the end of each session.If participants indicate that they are uncertain about some content areas, they could be requested to review the material for the next session.
Approximately 14% of midwives indicated that the scheduling of the facilitated session did not suit them.A staggered approach, where participants could choose convenient times to attend sessions would solve this problem.
b. Were the specific content areas contained in the maternal care manual appropriate for the participants' work context (maternity wards)?
Most of the participants agreed that the content of the PEP was applicable to their daily work in the maternity wards.In terms of adult learning principles, this is a positive finding as adults in general learn better when they perceive that their new knowledge and skills can be applied to their jobs.
c. Were the theoretical and practical aspects of the maternal care manual adequately covered in the manual and facilitated during group sessions?
In general, participants indicated that theoretical and practical skills were adequately covered during sessions of the PEP.A small number of participants said that the theoretical aspects of determining foetal condition and gestational age, and the practical aspects of foetal growth and medical problems during pregnancy could be covered in more depth.It is suggested that facilitators design a timetable for the discussion groups to ensure that all theoretical and practical aspects are covered sufficiently.
Question 3: Knowledge and skill acquisition outcomes a. Has the PEP improved the perinatal knowledge of the midwives?
Based on comparisons of the mean pre-and post-test results, it is clear that the PEP significantly improved the maternal care knowledge of midwives.Subjective self-reports also confirm that midwives were of the opinion that their knowledge had improved.This knowledge increase has been supported by other studies involving the PEP (Theron, 1999a).However, it should be kept in mind that the pre-and post-tests used different measures (results based on MCQs and a formal examination).It is unknown whether or not the formal examination was easier than the MCQs and whether or not the knowledge increase could be attributed to this.

b. Has the PEP improved the perinatal skills of the nurses?
A high percentage (97%) of participants indicated that their maternal care skills had improved because of the PEP.Four of the five supervisors who completed a separate questionnaire also confirmed this and reported that they had observed increased skills in PEP participants on the wards.This PEPrelated skill increase was also confirmed in other evaluations (Hesketh et al., 1994;Le Roux et al., 1998;Theron, 1999b).
In the self-report data of this evaluation there is evidence of improved skills and the application thereof.The evaluators suggest that in future evaluations of the PEP, skill acquisition and the application that follows should be measured by means of third-party observations and check lists.This would strengthen evidence of positive skills gain and supplement the results based on the subjective data.
c.In general, have midwives benefited from the PEP and would they recommend it to others?
Ninety-six percent of the participants reported that they had benefited from the PEP and would recommend the programme to others.This is indeed a strong endorsement for the PEP and could be used to market the programme effectively to potential participants.
d. Are there unintended outcomes which could be attributed to the PEP?
Most participants (95%) mentioned that increased teamwork and cooperation were spin-offs of the PEP.It could be speculated that the group sessions may have caused this unintended benefit.
e. What other perinatal learning initiatives had been in place for midwives prior to their enrolment in the PEP?
Three participants indicated that they had attended training programmes in lactation, medication, perinatal updates and BCG.Even if the results of these three participants were removed from the data set, it would not affect the overall positive effect of the programme significantly.

Organisational support
a. Was the climate in the maternity wards conducive to applying newly acquired knowledge and skills at work?
Approximately two-thirds of the participants reported a perceived alignment of ward practice and procedures with the PEP content.In addition, four of the five supervisors indicated their support for the PEP and stated that they had recommended it to midwives who reported to them.This supportive climate in the wards is ideal and can only strengthen programme uptake.

Conclusion
In summary, this evaluation showed that the maternal care module of the PEP was successfully implemented and endorsed by participants and supervisors alike.The addition of group facilitation to the self-study mode of the programme has produced additional increases in knowledge, skills and group work.The evaluation has provided a positive, sound basis for programme managers to increase programme coverage and continue the good work already evident from the results.

FIGURE 1 :
FIGURE 1: Programme theory of the perinatal education programme.
was the proportion of midwives who have participated in the PEP at this maternity hospital?c.What was the attendance rate for the midwives who are enrolled in the PEP? d.What was the proportion of midwives who successfully completed the PEP per year?
Part one contained 5-point Likert-type questions on coverage, service delivery, knowledge and skills acquisition, and organisational support.Part two consisted of two lists of items with yes or no tick boxes.These items focused on adequate theoretical content areas and practical skills.Part three had open-ended items dealing with unintended outcomes and other interventions, which could have influenced programme outcomes.The last part of the questionnaire elicited the participants' demographic details.The questionnaire is available from the evaluators.Programme records, kept by the chief professional nurse, were used to obtain information regarding pre-test results, post-discussion results and examination results of midwives.Pre-test results consisted of each recipient's pre-test score for the multiple-choice test before studying the maternal care module.Post-test results were the formal examination marks obtained at the end of the programme.The pass mark for the examination was set at 80%.The time interval between the pre-test and the post-test was 12 months.

TABLE 1 :
Participant perception of the programme's educational activities.
Values provided in percentages.

TABLE 2 :
Adequate coverage and facilitation of theoretical aspects.

TABLE 3 :
Adequate coverage and facilitation of practical aspects.