A process evaluation of a supervisory development programme

How to cite this article: Buys, J., & Louw, J. (2012). A process evaluation of a supervisory development programme. SA Journal of Human Resource Management/SA Tydskrif vir Menslikehulpbronbestuur, 10(3), Art. #423, 13 pages. http://dx.doi.org/10.4102/ sajhrm.v10i3.423 Orientation: An important evaluation function is to assess the adequacy of the programme process, including the extent to which the appropriate people participate in its activities.


Introduction Key focus of the study
In this article we examine the attributes of a management development programme in the hospitality industry.Faced with the dual challenges of high staff turnover and expensive recruiting practices, a major hotel group in the Western Cape introduced a supervisory development programme to develop a cohort of young, trained employees to become potential supervisors, who could be ready at short notice to fill vacancies as they arose.This report describes the extent to which the implemented programme has achieved this and other objectives.

Trends from the research literature
Management development is generally defined as the process through which individuals acquire the necessary capabilities to perform professional management tasks (Mabey & Finch-Lees, 2008;Wexley & Baldwin, 1986).There is extensive debate about whether or not these management capabilities include the development of leadership capabilities (Amos, Ristow, Ristow & Pearse, 2008;Kotter, 1990;Paauwe & Williams, 2001;Zaleznik, 1977).The development programme that formed the basis of this evaluation aims to instil both managerial and leadership skills, as well as knowledge and attributes, related to these skills, in participants.It assumes that these capabilities include leadership, and therefore we pay no further attention to this particular debate.A further point of debate was whether or not there is a difference between training, development and education of managers.According to Mabey and Finch-Lees (2008), training refers to the acquisition of concrete and job-specific skills, whilst development is a longer-term learning process, which cuts across jobs.For these authors, management education is typically provided by formal academic institutions like universities and business schools, and is often more theoretical by nature (e.g. a Master's degree in business administration) than training or development.Mabey and Finch-Lees (2008) tried to clarify the debate by suggesting a classification of the three processes of training, development, and education under the super-ordinate label 'management learning'.The programme under evaluation contains elements of both management education and training and can be categorised under management learning.

Background to the study
The programme under evaluation is a supervisory development programme (SDP) established by a large luxury hotel group based in Cape Town, South Africa.The programme has been running since 2006.The following factors led to the programme's implementation: • the high turn-over in the hospitality industry, and at this particular hotel • the high cost of replacing exiting staff • lengthy delays in the appointment of new staff as a result of stringent labour legislation • the higher workload of mid-level managers who often have to perform tasks assigned to incompetent supervisors (according to the programme manager, personal communication, June 2009).
According to the programme manager the main aim of the SDP is to create a pool of competent supervisors who will address the company's current and future staff needs for these critical positions.There are specific objectives for the participants and two main stakeholder groups of the SDP.
For the participants these are: • gain knowledge about the company as a system • develop a strategic outlook on the organisation • learn the skills required for supervisory positions within their business units.
The objective for the managers of participants is improved customer service by supervisors who can exhibit the full range of activities required by their positions.
The objectives for the company are: • more skilled employees to fill vacant supervisory positions • improved customer service ratings • compliance with recruitment and selection standards.
The target population for the SDP was current supervisors within the organisation (inclusive of those recently appointed from outside) and employees not yet in supervisory positions, but identified by their line managers as potential supervisors.
Employees from all business units are eligible for selection into the programme, as the same management structure is implemented across the organisation.

Final group presentation
Apart from the hotel group, an international hotel school and a private consultancy (represented by a lecturer from a local business school) participated in presenting the SDP.The duration of the SDP is three months and it is presented bi-annually.The programme activities are illustrated in Figure 1, with arrows indicating a direct relationship between activities and shaded activities indicating major assessments.
Figure 1 shows that the SDP begins with a two-hour training session for senior managers and human resource (HR) staff who are responsible for facilitating and supporting the various programme activities.In this session the external consultant introduces the principles and practice of action learning.For participants, the SDP starts with two half-day modules presented by the consultancy.On the first day, a morning module introduces the principles of action learning.
During the afternoon session, a business acumen module is presented which introduces business concepts, terminology and knowledge associated with fulfilling a managerial role within a luxury hotel.At the start of the programme the participants are divided into groups (consisting of 4 to 7 employees each) for the purposes of group assignments and an action learning project.The action learning project aims to integrate the different programme activities.At the end of the programme this summative project is presented to a senior management panel who act as assessors.
The next phase of the programme consists of a series of half-day modules, presented to the participants, spaced two weeks apart.Modules include: • HR Principles Activities are assessed on an ongoing basis.Students assess their own work, as well as that of their peers.In addition, assessment is conducted by internal and external staff who facilitate the programme activities, the line managers of participants, as well as clients dealing with participants.These assessments are moderated by an international hotel school, which co-sponsors certification of the programme's successful participants.
According to the programme manager, the SDP is based on action learning principles.A basic definition of action learning emphasises that a group of colleagues who form a mutual support structure examine actual, work-related problems and then take action to solve these problems (McGill & Beaty, 1995).
In general, programme staff hold assumptions about how their programme will change the programme's recipients.
To elicit these assumptions, the evaluators used a model proposed by Rossi, Lipsey and Freeman (2004) which delineates programme theory into three main parts: • programme impact theory (a causal theory which describes the changes expected in the target population as a result of programme activities)

Short-term outcomes
Medium-term outcomes Long-term outcomes • a service utilisation plan (how the target population will interact with the programme services) • an organisational plan (whether or not the programme has sufficient resources for effective service delivery).
Although the SDP did not have a formalised programme theory in place, the programme manager could articulate specific programme goals.The evaluators elicited these goals by means of a series of semi-structured and structured interviews with the programme staff and an analysis of programme documents.From the interviews and documents it became clear that the hotel's senior management responded to the problems of high turnover and skills scarcity, at the supervisory level, with a management training and education programme.Figure 2 reflects the assumptions of senior management regarding the change that the SDP will bring about in the participants.
The programme's impact theory, as set out in a causal chain in Figure 2, shows how the SDP will lead to changing the state of affairs identified as organisational problems by senior management (i.e., reduced recruitment costs and increased client satisfaction).
In order to elicit the programme's service utilisation plan, the evaluators attempted to analyse the programme activities.
The programme documentation did not contain a detailed description of these activities, apart from a diagram showing the sequence of activities and their duration.In this regard, the designers of the SDP assume that attendance of seven half-day modules on specific topics, followed by individual and group action learning projects and finally coaching sessions and reflective journal keeping, would bring about the intended changes in the participants.
In terms of the organisational plan, the following implicit assumptions of the programme manager and programme sponsors were elicited: • in-house senior management and a consultant would constitute sufficient resources to deliver the SDP • the length of time scheduled for the SDP would be appropriate for such a programme.

Evaluation questions
Carefully formulated evaluation questions serve to focus and structure an evaluation in order to provide useful answers for programme stakeholders (Donaldson & Lipsey, 2006;Rossi et al., 2004).The evaluators, in cooperation with the stakeholders, formulated the following evaluation questions for the formative evaluation questions presented in Box 1, of the SDP.

The potential value-add of the study
Formative evaluations are conducted to provide programmes, and their managing staff, with information that allows them to improve the programme's activities, delivery, performance and outcomes.Thus the contribution of this study is to: • assist programme staff to understand the assumptions of the programme better • provide programme staff with information they can use to reflect on the programme's current performance and future direction.
The study also provided an example of how programme theory can be developed for this type of intervention.Furthermore, the study aimed to find out whether or not such an evaluation can be used within the hospitality industry to examine management education programmes.

Research approach
The study employed a mixture of quantitative and qualitative methods to answer the above-mentioned evaluation questions.Data for the coverage evaluation questions were obtained from the SDP's attendance register and assessment documentation.
The following data sources were selected for the service delivery evaluation questions: • a literature review with the search parameter, 'action learning' The data for questions resorting under organisational resources included financial records made available by the programme manager.Additionally, an interview with the programme BOX 1: Questions the evaluators formulated for the formative evaluation of the supervisory development programme.
Question 1: Coverage a. How many recipients have successfully completed the SDP? b.How many recipients have started but not completed the SDP and why? c.From which business units were recipients selected and were all business units equally represented?
Question 2: Service delivery a.Is the SDP a typical action learning programme and is action learning an appropriate medium for management development?b.Are the actual programme activities aligned with the intended ones?manager provided data relating to time expenditure and the expertise of programme staff.
Finally, the data for the evaluation questions probing mediumterm outcomes were obtained from programme records of assessment outcomes and the hotel's personnel records.

Research participants
The programme manager was an important research participant, in terms of giving approval to conduct the study and providing access to documents, as well as being a key interviewee.In addition to the programme manager, the programme administrator was interviewed a number of times to collect data on the programme.

Measuring instruments
The interviews were used to ask about the programme's activities and participants, the rationale behind the programme, whether or not clear needs were identified by the programme sponsors and personnel, and which questions the evaluation would focus on.

Research procedure
The first evaluator was introduced to the programme manager by a mutual business acquaintance.An e-mail and a telephone conversation outlining the proposed evaluation led to an invitation to meet the programme manager.The evaluation was conducted at no cost to the organisation and, as such, a verbal agreement between the programme manager and the evaluator served to initiate the evaluation.Formal documents that addressed the scope of the evaluation, ethics and the use of evaluation results were later drawn up and signed by the programme manager.Initial interviews with the programme manager elicited the goals of the programme.These were used to develop the programme theory discussed earlier.The programme manager agreed that the programme theory represented the causal logic of the SDP.
After signing an agreement on the scope of the evaluation and preliminary evaluation questions, the programme manager introduced the evaluator to the programme administrator.Monthly meetings for the duration of the evaluation were scheduled with the programme administrator, and each lasted between 30 minutes and 90 minutes.When face-to-face meetings were not possible, the evaluator communicated with the administrator via e-mail.This was found to be less effective, as numerous e-mail messages were needed to deliver the same results as a short interview.
Meetings with the administrator were focused on developing an understanding of the programme and its activities.Through these meetings, the evaluator also gained access to relevant programme documentation and other information needed to answer the evaluation questions.Four such meetings took place before the evaluator received all the available programme documents.Thereafter, ad hoc meetings and e-mail messages, with probing or clarifying questions, were used successfully to elicit further information.This iterative process was followed until all available programme documentation and information needed to answer the evaluation questions were obtained.The frequency of communication decreased towards the end of the evaluation.
As the evaluation involved human participants, the evaluators obtained ethical clearance for the evaluation from the Research Ethics Committee of the Commerce Faculty at the University of Cape Town.

Statistical analyses
The evaluators incorporated the information for each participant into a Microsoft © Excel 2008 for Mac spreadsheet.The data included which year and SDP group the participant attended, an identification number, basic demographic information, employment information, attendance of SDP activities and outcomes from SDP assessments.These data were imported into PASW Statistics 18.0 for further analysis.Descriptive statistics, which were mainly frequency counts and percentages, were used to answer the evaluation questions.

Results
The results of the evaluation will be presented in terms of the specific evaluation questions.In order to assess completion rates, the available information on attendance, and marks for the 69 delegates who completed the SDP, were compared to the criteria set by the programme staff.The duly performed criteria and academic standards could be obtained from the marking schedules of the SDPs, and included the following: • attending all sessions • absence only with the HR Manager's permission or a doctor's note • handing in all assignments and reflective journals • handing in all assignments and reflective journals on time • a minimum average for the course of 50% or more • no more than one assignment scored between 40% and 49% • any assignment receiving a mark below 40% would have to be resubmitted in order to obtain a minimum mark of 40% which would allow the candidate to proceed with the course.
If a participant failed to meet either the duly performed criteria or minimum academic standards, he or she would be asked to withdraw from the programme, subject to review by the SDP Academic Board.
As is evident in Table 2, the documentation of programme activities for SDP 2006(1), 2007(1) and 2007(2) was incomplete.This made it difficult to comment on the degree to which participants, who completed the SDP, adhered to the performance criteria stipulated by the programme staff for these specific programmes.In order to answer the evaluation question, an analysis of the available programme documents (attendance registers, marks, and group projects) for the 69 participants, who had completed the SDP, was undertaken.
The results are presented in Table 2.
Although there were some missing data regarding marks, all recorded marks met the academic standards set by the programme staff.Significantly, this held true for the two 2008 cohorts for which between 80% and 90% of programme activities were documented.With regards to duly performed criteria: • five participants did not submit all reflective learning journals during the 2006 SDP • three participants did not submit all reflective learning journals during the first 2007 SDP • three participants did not submit all reflective learning journals during the second 2007 SDP • one participant was absent from one half-day coaching session during the first 2008 SDP • one participant was absent from one half-day coaching session and one participant did not complete the final written assessment during the second 2008 SDP.
These findings should be interpreted cautiously for two main reasons.Firstly, the data for the 2006 and 2007 SDPs were incomplete, but results for the two 2008 SDPs could be interpreted with a higher degree of confidence because of more complete data sets.Secondly, no records were available to indicate whether or not the failures to perform duly were for reasons that the programme staff would accept as legitimate (operational requirements or illness).It is therefore possible that an even higher percentage of participants performed duly than is indicated in Table 2.
In summary, the available data indicated that the majority of participants, completing the SDP between 2006 and 2008, did so successfully as described by the programme standards.The reasons for dropout were not documented.The evaluator was also not able to interview the exiting individuals as they had both resigned from the organisation at the time of the evaluation.From the programme documentation it was clear that both participants exited the programme at an early stage.
There are several possible reasons for the low dropout rate recorded for the SDP between 2006 and 2008.Firstly, upon selection into the programme, performance criteria for the SDP are made clear to the participants.Secondly, participants sign a compulsory bonding agreement to repay the organisation R3500 should they leave the SDP before their successful completion of this programme or should they leave the organisation within six months after their successful completion of this programme.Thirdly, the SDP provides participants with an opportunity for promotion to a supervisory position.Lastly, most of the programme's activities are scheduled during working hours and thus do not require participants to spend time after hours on developmental activities.
c. From which business units were recipients selected and were all business units equally represented?
The main business units within the hotel are: • The majority of participants were selected from the organisation's two biggest business units, the Rooms Division and Food and Beverage Division.
Question 2: Service delivery a.Is the SDP a typical action learning programme and is action learning an appropriate medium for management development?
In order to answer this evaluation question, a brief literature review was conducted, focusing on the prerequisite activities of an action learning mode of delivery.
A basic definition of action learning describes a group or set of colleagues who form a mutual support structure, who work through actual work-related problems faced by the individuals within the group (McGill & Beaty, 1995).
Learning focuses on taking action, as the name of this approach suggests, through the support of peers.Dotlich and Noel (1998, p. 15) describe action learning generically as 'an alternating series of workshops and field experiences' and as a context for rapid learning.According to this definition, action learning provides both the tools that participants need to solve business-related problems, as well as the context to engage in problem solving.
Many action learning approaches exist (Smith & O'Neil, 2003).Based on their review of action learning literature published between 1994 and 2000, these researchers identified the following generic components of action learning programmes: • participants grapple with actual business problems as they occur • participants work in small and stable learning groups, called sets • sets meet regularly • there is a fixed programme duration • the business problems participants deal with are relevant to their own working environment • participants take action between meetings to solve the identified business problems they face • questioning and reflection play a major role in the learning process • a supportive and collaborative working environment is created in the set.
In Table 3, the generic components described by Smith and O'Neil (2003) are compared with the SDP activities to ascertain the level of congruence between them.
Based on this comparison it became clear that the SDP contained the majority of the action learning components described in the literature review by Smith and O'Neil (2003).
The most notable exception was that participants did not deal with actual business problems they were facing at work, but simulated problems related to the various business units.
Based on this comparison of action learning literature and SDP programme activities, it can be concluded that the SDP is an action learning programme that includes the majority of activities described in the reviewed action learning literature.
However, is action learning the appropriate approach to address the problems faced by the hotel management?Is action learning a mode of delivery that is commonly used to grow a pool of supervisors, who need to think holistically about their organisation, and have a pro-active management style?Again, the available literature seems to support the choice of action learning in the context described above.Fitzsimmons and Fitzsimmons (1994) highlight the key roles of the service manager as those of independent thought and taking appropriate action.Mumford (1997) describes both planned learning exercises and seizing learning opportunities, as they emerge from the reality of managerial life, as key strategies in management development.McGill and Beaty (1995) describe action learning as a learning and reflective process that takes place with the support of a group of colleagues working with real problems, with the intent of achieving tangible results.Action learning also typically produces unintended competencies such as: gaining increased self-confidence, selfawareness, other-awareness, readiness to take responsibility and initiative, learning a more disciplined way of working, learning how to formulate more informed actions, learning to relate to and communicate effectively with others, and learning to learn (Johnson, 1998).
Action learning theorists paint a picture of action learning that appropriately matches the approach followed by the SDP.This congruence provides some evidence that action learning may be an appropriate approach for supervisory development.
b. Are the actual programme activities aligned with the intended ones?
The answer to evaluation question 4 indicated that the programme activities of the SDP, as intended, were aligned with an action learning mode of delivery.However, the evaluators also examined whether or not these intended activities were the actual activities that participants received whilst on the SDP.Based on this partial evidence, it would be safe to say that the majority of programme activities between 2006 and 2007 were planned, scheduled and paid for.Unfortunately, the available programme documents did not allow a definitive statement as to the actual implementation of these activities.
The current programme administrator has been involved with the SDP since the beginning of 2008 and has been able to produce more complete programme documentation.This is reflected in Figure 3, which shows that the number of documented activities increased substantially in 2008.
The planned programme activities remained relatively consistent during the three years covered by the evaluation.
Of the 18 to 20 programme activities originally planned (Figure 3) only three were changed between 2006 and 2008.
In summary, it can be concluded that the planned programme activities took place in 2008.The programme showed a high degree of stability over the three years and five programme instances evaluated, with only one new activity added and two frequency adjustments made during this time.
Question 3: Organisational resources a. Are there sufficient resources (e.g.finances, time and expertise) to implement the SDP effectively?
The programme utilised considerable financial, time and human resources.In terms of financial resources, hotel management has spent approximately R350 000 between 2006 and 2008, as Table 4 indicates.
Most of the funds were spent on fees charged by the private consultancy.Accreditation and co-sponsorship of the SDP by the international hotel school, mentioned earlier, required the second largest investment of financial resources.Although total expenditure for the SDP from 2006 to 2008 was considerable, it equated to an average of R5072.46 per participant per three-month programme.It could be argued that the SDP is a cost-effective programme.
Two factors enabled the hotel to keep programme costs relatively low.Firstly, the hotel used its own training venues.
Secondly, an agreement between the external consultancy and hotel management existed to train hotel managers to facilitate programme activities and phase out the need for a consultant.
Expenditure of time is another important resource consideration, as time spent by senior managers facilitating the SDP, and participants attending the programme, converts to time being spent on non-core activities, which  subsequently results in a decline in productivity.During the three months that the SDP ran, at least fourteen full workdays were spent on training, and this process involved between 12 and 15 participants, twice a year.If not properly planned for, this could add strain to the workload of senior managers facilitating the course, to colleagues of participants who may have to perform extra duties whilst participants are undergoing training, and to the participants themselves.This indirect programme cost should be accounted for in addition to the fees paid to external service providers to achieve a realistic sense of programme cost.
Two staff members were closely and continually involved in the SDP.The programme manager for the SDP is a senior HR executive within the hotel group, and the programme administrator is an HR manager within the hotel who oversaw the daily planning, running and administration of the programme's activities.
Question 4: Medium-term outcomes a.Does a pool of skilled supervisors exist after five SDPs have been run, and if so, how many recipients form part of this pool?
The successful completion of all programme activities, according to the standards set by the programme staff, was chosen as an indicator of a skilled supervisor.The data are presented in Table 5.
Of the 69 participants who completed the SDP between 2006 and 2008, 45 were still employed by the organisation at the end of August 2009.If it is assumed that all those who completed the SDP have developed the required supervisory skills, then the simple answer to the evaluation question is that a pool of 45 skilled supervisors is still employed by the organisation.
However, if evidence of actual skill upon programme completion is required, as reflected in the number of trainees who have complied with all the standards and criteria of success, a different answer emerges.There was documented evidence that 55 participants (or 79.7%) adhered to all the standards set by programme staff.Of these 55 skilled participants, 35 (or 63.6%) were still employed by the hotel in 2009.When the six successful SDP graduates who had since been appointed (and, thus, do not form part of the pool of the skilled supervisors that were available for appointment when a vacancy arose) were subtracted from this number, 29 participants remained in the pool.
Within the limitations of available programme records, and the evaluation method chosen, the most accurate answer to this evaluation question is that a pool of skilled supervisors did exist and that it consisted of 29 employees who had successfully completed the SDP.The aforementioned 29 beneficiaries are still employed by the organisation and have not been appointed into vacant junior management or supervisory positions.The organisation can report back to stakeholders that their investment and efforts have resulted in the creation of a pool of 29 employees who have been trained to be competent supervisors and who are ready to be appointed to fill vacancies when they arise.Significantly, the SDP has delivered at least 35 skilled junior managers or supervisors within a period of three years.This result indicates that, to a large degree, the programme is being implemented as planned, up to this point, and has started to deliver its intended short-term outcomes.
b. Are recipients who successfully completed the SDP being appointed into vacant supervisory positions?
According to the programme records, a total of 69 SDP participants completed the programme between 2006 and 2008, of whom 35 could be labelled as having acquired the required skills, in accordance with the programme's standards.The programme manager also indicated that in some cases newly appointed supervisors, from outside the organisation, were also enrolled in the programme.Table 6 reflects the number of participants appointed prior to and after the SDP.
Of the 55 participants who successfully completed the SDP between 2006 and 2008, 22 (40%) were already employed as supervisors whilst 6 (10.9%) were appointed after completion.Thus, the majority of these appointments took place before SDP enrolment and it could be concluded that only six participants were developed by the SDP and appointed permanently when a vacancy arose, as the programme theory stipulates.The 22 incumbents who were appointed did not fit the current understanding of how the programme was supposed to work, which is not necessarily a negative or an unintended outcome for the programme, but it should be incorporated into the formal programme theory.Furthermore, two internal appointments of non-SDP graduates and two external appointments were made during the evaluation period.These four appointments were not aligned with the programme theory at all.The continued appointment of non-SDP and external candidates could indicate a serious flaw in the implementation of the SDP, with potential repercussions for how it is perceived by the intended target group.

c. How long does a typical recipient have to wait between completion of the SDP and selection into a vacant supervisory position?
This question is only relevant to the six SDP graduates who were appointed after graduation -the 22 who were first appointed and then enrolled on the SDP, of course, experienced no waiting period.We could only find data on four appointments, and they waited on average for a rather lengthy period of 17 months, after completing the course to be appointed.
Twenty-nine successful SDP graduates were still employed by the organisation but had not yet been appointed into supervisor positions.The waiting time by the end of the evaluation period, for these beneficiaries, is indicated below: Although programme staff make it clear to participants that graduation from the SDP does not guarantee promotion, these findings indicated that the relatively small number of SDP graduates who had been appointed, and who were not incumbent supervisors, were appointed after approximately a year and a half.The majority of SDP graduates who had not yet been appointed into supervisory positions had been waiting for promotion for an average of 21 months.The majority of supervisory vacancies (32) were filled with employees who were at that time either current or future SDP graduates.Four vacancies were filled with individuals from within the organisation, who were not and did not become SDP graduates, whilst two external applicants were employed as supervisors.
Based on the information gathered, there are two main reasons to believe that the SDP is developing a surplus of skilled junior managers.Firstly, only 50.9% of successful graduates had been appointed into vacancies by August 2009 when this evaluation was concluded.For the 49.1% of delegates not yet appointed, the average waiting time was calculated as 21 months, with certain graduates already awaiting promotion for 37 months.It seems that there were twice as many gra duates as vacancies during the evaluation period.Secondly, interviews with the programme administrator indicated that programme staff were planning to increase the number of SDP delegates during 2009.The uptake for the first 2009 SDP was reportedly 24 participants, with a similarly sized group planned for the second 2009 SDP.This could potentially increase the number of skilled junior managers by almost 48 individuals within one year.If the number of vacancies does not increase accordingly, even more SDP graduates will form part of the pool of skilled supervisors awaiting promotion.
In summary, it could be asserted that the programme theory, as initially formulated by the programme staff, was well implemented up to the creation of a pool of skilled junior managers.Only 10.9% of SDP graduates were appointed from this pool of skilled individuals into vacant supervisory positions.The majority (40%) of supervisors were appointed into vacant supervisory positions first and then enrolled into the programme.Therefore it seems that the SDP has  5.
This revised programme theory indicates that there are two groups of SDP participants, namely potential supervisors and incumbent supervisors.As indicated in Figure 5, both groups already participated in the SDP.After graduation, only the potential supervisors proceeded to the pool of skilled supervisors.These participants were ready to be appointed into supervisory vacancies and were thus able to attain both the medium and long-term outcomes specified in the original programme theory (Figure 1).After graduating from the SDP, the participants who were incumbent supervisors upon entering the programme (indicated by the grey block and perforated lines in Figure 5) did not form part of the pool of skilled participants who were available to be appointed into vacancies.These participants had already been appointed into supervisory vacancies and, thus, could not attain all medium and long-term outcomes originally specified.Although these participants also function more efficiently, independently and productively, they did not necessarily help the organisation to reduce the cost of filling vacancies.Participants in this second group might also have been appointed from outside the organisation, thus negating the outcomes of reduced recruitment costs and reduced induction periods.
implementation and short-term outcomes within a limited time frame and limited resources (Donaldson & Lipsey, 2006).Relevant information regarding the programme's functioning was collected, interpreted and reported in a user-friendly manner.Based on the main findings of the evaluation, aspects of the programme that required attention or improvement have been highlighted for programme staff.Furthermore, the programme stakeholders have been provided with a revised programme theory.This evaluation has also summarised the available information for the five SDPs running between 2006 and 2008.By explicating the basic programme theory and studying programme implementation, the evaluation also served as a starting point for future evaluations of the SDP.
The main results of the evaluation indicated that the programme was not implemented according to the programme plan.There was a risk that this could lead to unintended consequences, such as the participants' perception that the organisation had violated the psychological contract (Smithson & Lewis, 2000) of development and promotion.
Between 2006 and 2008, the SDP functioned mainly as a training programme for incumbent junior managers.If it was mostly used as a training programme, rather than as a management development programme, then the programme activities and theory should be re-evaluated.There may be more efficient ways than the current SDP to train incumbent managers within the organisation.At present, the evaluation results indicated that the SDPs, running between 2006 and 2008, might not have had the intended outcome of reducing recruitment costs to fill junior management and supervisory positions within the organisation.
Following Mabey and Finch-Lees (2008), this programme was treated as a management learning exercise.Our results indicated that learning indeed took place, as a pool of skilled managers was created.The theory failure (Rossi et al., 2004) that we identified had to do with what had happened further down the outcome chain, in terms of how these newly trained managers were treated.

Conclusions
It is clear that the SDP was not implemented as intended and that the way in which the participants were treated after completing the programme might undermine the positive effect of the programme.It is conceivable that the SDP could develop a surplus of skilled SDP graduates that will negate the need to run the programme for a certain period of time.

Recommendations
It is recommended that the period in which SDP graduates awaited appointment should be taken into account when SDP graduates are considered for a supervisory vacancy.This would reduce the waiting periods for those who have completed the programme successfully.It would be virtually impossible to predict future vacancies precisely or to appoint all graduates immediately after graduation, but despite this it is important to ensure that programme uptake and output are aligned more closely, and that waiting periods are reduced.Continually training a surplus of SDP graduates would counteract the initial programme goal of reduced recruitment and selection expenditure.It would also increase the waiting period between graduation and appointment, thus, increasing the risk of dissatisfaction of successful participants.

Limitations
All evaluations operate in situations where there is a tension between what 'the best' methods to use are, and what is possible or feasible.In the present case, the evaluators had to rely on rather limited data sources, and even with these they experienced great difficulties obtaining and extracting the information.It would have been more informative if, for example, the evaluator had attended one of the SDPs to see the training in action, and also had observed how the programme proceeded.

Suggestions for future research
Ideally, one would like to study more programmes like this, to expand the knowledge base about management learning in the local context.For this particular programme, it would have been useful to investigate the perceptions of the trainees who had been waiting to be placed in supervisory positions, given the risks that the study has identified concerning them.Finally, a cost-benefit analysis, in terms of the resources allocated to the programme, would yield important information on how to go forward with it.
hard copies of the conceptual flowcharts of SDP programme activities • a final flow chart of SDP programme activities in electronic format • hard copies of internal e-mail messages sent between programme staff during the planning phase of the first SDP in 2006 • hard copies of e-mail messages sent between the programme manager and the external consultant during the planning phase of the first SDP in 2006 • hard copies of assessed group projects collected between 2006 and 2008.

Question 3 :
Organisational resources a. Are there sufficient resources (e.g.finances, time and expertise) to implement the SDP effectively?Question 4: Medium-term outcomes a.Does a pool of skilled supervisors exist after five SDPs have been run, and if so, how many recipients form part of this pool?b.Are recipients who successfully completed the SDP being appointed into vacant supervisory positions?c.How long does a typical recipient have to wait between completion of the SDP and selection into a vacant supervisory position?d.Are a sufficient number of supervisors being developed by the SDP? SDP, supervisory development programme.
Rooms Division (typically comprising of Front office, Reservation, Housekeeping, Uniformed services, Telephone services and concierge) • Food and Beverage Division (including specialty restaurants, bars, coffee shops, banqueting, lounges, catering and kitchens) • Sales and Marketing Department • Finance Department • HR Department • Maintenance Department • the hotel's Spa.
FIGURE 3: Number of planned and documented programme activities from 2006 to 2008.

d.
Are sufficient supervisors being developed by the SDP?This evaluation question was answered by comparing the number of skilled supervisors developed by the SDP between 2006 and 2008 with the vacancies reported by the organisation between 2006 and August 2009, when the evaluation was concluded.This information is reflected in Figure 4.

TABLE 1 :
Participants completing supervisory development programmes between 2006 and 2008.
Both attendance and performance, in relation to programme standards, were taken into consideration to calculate the number of successful participants.If the standards set by the programme were adhered to strictly, a total of 55 participants successfully completed the SDP.
b. How many recipients have started but not completed the SDP and why?All participants who enrolled in 2006 and 2007 completed the programme.Only two participants dropped out of the programme during the evaluation period, one from the 2008(1) SDP and one from the 2008(2) SDP.This constitutes a dropout rate of 2.8% (total n = 71) between 2006 and 2008.

TABLE 2 :
Adherence to stated performance standards for participants completing the supervisory development programme from 2006 to 2008.
SDP, supervisory development programme; N, used as means of number.

TABLE 3 :
Comparison of supervisory development programme activities and action learning literature.
We compared the scheduled programme activities for the SDPs running between 2006 and 2008 and the actual activities recorded in available electronic and paper-based programme documents for the same period.It became clear that available documentation for the three SDPs running between 2006 and 2007 was limited.The reason for this was the resignation of the first programme administrator and the subsequent lack of access to her electronic programme documents.During further interviews with the current administrator, it was established that the missing programme activities did in fact take place each year.

TABLE 4 :
Approximate supervisory development programme cost and number of qualifiers from 2006 to 2008.

TABLE 5 :
Comparison of participants completing and successfully completing the supervisory development programme and still employed within the organisation from 2006 to 2008.SDP, supervisory development programme; N, used as means of number.†,Total number of participants who have completed the programme according to programme staff, regardless of whether or not all stated criteria were met.‡,Number of participants who have completed the programme and complied with all stated standards and criteria.

TABLE 6 :
Supervisory development programme participant appointments prior to or after supervisory development programme.Temporary appointments whilst supervisors were on leave.Not added to totals.
SDP, supervisory development programme.FIGURE 4: Supervisory development programme graduate appointments into vacant junior management and supervisory positions from 2006 to 2009.