A formative evaluation of a pay-for-performance system

How to cite this article: Joseph, L., Emmett, K., & Louw-Potgieter, J. (2012). A formative evaluation of a pay-forperformance system. SA Journal of Human Resource Management/SA Tydskrif vir Menslikehulpbronbestuur, 10(3), Art. #426, 12 pages. http://dx.doi.org/10.4102/ sajhrm.v10i3.426 Orientation: Pay-for-performance (PFP) systems emerged during the 1980s as performance improvement tools. However, research findings have shown contradictory evidence as to whether these systems motivate employees to improve their performance.


Introduction Key focus of the evaluation
The main aim of this evaluation was to assess whether the pay-for-performance (PFP) that at a South African university system introduced for administrative employees improved their performance.A secondary aim was to examine whether the university implemented the system as it intended to.

Background to the evaluation
Pay-for-performance systems emerged during the 1980s as the way to motivate employees and improve organisational performance (Cannell & Wood, 1992;Dowling & Richardson, 1997).However, research shows contradictory evidence as to whether PFP systems are effective motivational and performance improvement tools (Armstrong, 2006;Dowling & Richardson, 1997).
It seems that organisational contexts influence the effectiveness of these systems (Armstrong, 2003;Belcher, 1996;Gomez-Mejia, Wiseman & Dykes, 2005).According to Bruns (1996), PFP systems are most effective when implemented in organisations which foster trust between management and employees, do not have strong trade unions which oppose PFP systems and have an entrepreneurial or performance culture.Research also shows that PFP systems are less effective in improving team performance because PFP measurement tends to be individualised (Armstrong, 2003;Belcher, 1996;Bruns, 1996;Reilly, 2003;Wright, 2004).

Elements of pay-for-performance systems
According to Ulrich and Brockbank (2005), two elements are the basis of PFP systems: measurement and pay.
Measurement consists of formal assessment and the ratings of managers.The main functional requirements for the measurement element of a PFP system are to distinguish good from bad performance and to separate past from future performance.
Records of past appraisals determine past performance whereas employees' development plans define future performance (Armstrong, 2003).Furthermore, the people the system affects need to perceive the measurement element of PFP systems as fair if it is to fulfil its motivational function (Armstrong, 2003;Edwards, Scott & Raju, 2003).Doran (2008) reinforced this idea and suggested that the employee ratings managers give should undergo independent moderation so that employees will see rating decisions as less subjective and discriminatory.
The pay element of PFP systems shows what and who organisations value (Holbeche, 2004).In addition, this element should reinforce positive employee behaviour so that employees can achieve their goals and improve their performance (Henderson, 2006).This element of PFP systems is most effective when it links visibly to performance, when it gives employees feedback soon after they achieve outstanding performance and when it deducts pay if the recipients do not continue to perform (Eichinger, Lombardo & Ulrich, 2004;Rosenthal, Landon, Howitt, Song & Epstein, 2007).
The pay element of PFP systems should provide an adequate pay incentive (Doran, 2008;Rosenthal et al., 2007).According to Henderson, reward-related pay made in lump sums is more effective than percentage increases because lump sums have a more positive effect on employees' motivation and, therefore, on improved performance.In addition, Armstrong (2003) states that percentage increases do have a positive effect on employees' motivation and performance as long as the increase is between 10% and 15%.
Human resources (HR) publications rarely explore the pay element of PFP systems.One may attribute this to the fact that HR does not exclusively control the pay function of PFP systems.Quite often, finance departments are responsible for the pay element.In these cases, HR has focused on the measurement element.This could be problematic because it could separate the pay and measurement elements in PFP systems.Organisations need to align both of these elements to produce effective PFP programmes.

Standard requirements for implementing pay-forperformance systems
The essential requirements for implementing PFP systems successfully are: • the commitment of managers to the system • a focus on the implementation process rather than on designing the systems • support from key stakeholders • clear communication about the systems (Armstrong, 2003;Rosenthal et al., 2007).Streib and Nigro (1993) emphasised the importance of process training for the effective implementation of PFP systems because it guides employees how to set clear and realistic performance objectives.It also helps employees and their managers to go about interacting and collaborating in the process.In addition, process training shows employees how systems link pay to performance.This understanding is important because linking pay to performance is the main motivating mechanism for employees.

Pay-for-performance systems and non-management employees
Organisations use PFP systems mainly for employees at management level (Armstrong, 2003;Belcher, 1996;Pennings, 1993;Reilly, 2003;Schwab & Olson, 1990;Wright, 2004).They seldom target administrative and support employees for this type of pay system because they do not produce a specific product or service.Their work is usually task-related and they often work in small work groups instead of on their own.
Research shows that trade unions could hinder the implementation of PFP systems for their members (Reilly, 2003;White & Druker, 2000;Wright, 2004).PFP systems reward employees individually, whilst trade unions prefer to increase pay equally for all employees (Doran, 2008;Reilly, 2003;White & Druker, 2000;Wright, 2004).Trade unions often resort to collective bargaining to compress the range of pay for performance.Compressed pay ranges lead to decreases in the discretionary amounts available for distribution to employees, increased perceptions of the unfairness of systems and decreased motivation (Doran, 2008;Wright, 2004).
Trade unions also influence the level of rewards, the structure of rewards, the reward systems organisations choose and all procedures concerned with managing rewards (White & Druker, 2000).Because of the wide-ranging influence trade unions have on PFP systems, these systems may not suit unionised environments because they reduce the functionality of the systems (Reilly, 2003;White & Druker, 2000;Wright, 2004).

Evaluation objectives
The main aim of this evaluation was to assess whether a PFP system, which a South African university introduced for administrative employees, improved their performance.
A secondary aim was to examine whether the university implemented the system as it intended to.

Introduction of a pay-for-performance system at a South African university
In 2001, a South African university introduced a PFP system for administrative staff.Most of the staff belonged to unions.According to the Performance Development Resource Guide (University of Cape Town, 2006), which the university provided, and from interviews with key stakeholders, the new PFP system aimed to improve administrative employees' performance and help them to clarify their roles.It also intended to increase their skills, knowledge and abilities for growth and future performance.
The system intended to achieve these goals through a two-fold process.The first process, performance management, intended to assess employees' performance against agreed outcomes and then reward employees for excellent performance.The second process, performance development, aimed to identify development opportunities so that the university could develop employees' skills for future performance.
According to the university's Salary Assessment Resource Guide (University of Cape Town, 2006), the university deliberately separated the performance management and development discussions.This was to ensure that the performance management discussion, with its concerns about assessment and pay, did not cloud the development discussion.Although the university dealt with them separately, both processes used a single set of performance objectives that employees would develop.The objectives would link to the university's strategic objectives.These objectives were the link between the performance management and development processes.
The link the objectives provided between the performance management and development processes also created a performance development cycle.This cycle contains six steps: • agree on objectives The university introduced its PFP system in 2004.It affected all administrative employees in the bargaining unit.A halfday workshop, presented over a three-month period, gave administrative staff and their line managers the necessary knowledge to use the system.Attendance at these workshops was voluntary.Human resource advisers offered the workshops to each department and provided a Performance Development Resource Guide to all employees who attended the workshops.
Administrative employees and line managers attended different workshops.The administrative employees' workshop focused on how the PFP system was going to affect them and how their performance levels would determine their pay.It also trained them to set objectives.The line managers' workshop trained line managers in the performance development process, to review their employees' performance, to provide constructive feedback and to rate employees' performance.
Based on discussions with the programme managers, the short-term outcomes of an effective PFP system would be cooperation between line managers and administrative staff to produce performance objectives and a development plan.See Figure 1.
At this stage of the evaluation, the evaluators did not know which variables would mediate between the programme and its short-term outcomes.

Evaluation questions
According to Scriven (1991), a formative evaluation is one that guides programme improvement.The evaluators formulated the evaluation questions in Box 1 to improve the PFP system.

The potential value-add of the evaluation
The evaluation seeks to contribute to the social science literature on the effectiveness of PFP systems by examining whether these systems work.If they do not, the evaluators seek to find out why.Question 4: Longer-term outcomes a.After two implementation cycles, how did line managers and administrative staff rate the effectiveness of the system?b.Did the PFP system at the university work for some administrative employees and not for others?c.Are there interventions or variables that can predict for whom the system would work?

Research procedure
The evaluators distributed the questionnaire electronically to participants.The questionnaire was available on-line for a period of two months and the evaluators sent two reminders to participants during that period.

Statistical analysis
The evaluators used descriptive statistics to analyse the data for evaluation questions 1-9.For questions 10 and 11, the evaluators performed a reliability analysis, a factor analysis, a linear regression and a stepwise discriminant analysis.

Results
The evaluators reported their results in terms of the evaluation questions.Seventy-two per cent of line managers and 60.7% of administrative staff indicated that they had received training since the introduction of the system.
The main reasons for non-attendance were: • unaware that training had been scheduled (15% of line managers and 26% of administrative staff reported this) • lack of time to attend training (7.8% of line managers and 10% of administrative staff reported this) • perceptions that the training would not be useful (7% of line managers and 8.4% of administrative staff had these perceptions).
Question 2: Resources a. Were there sufficient human and physical resources for implementing the new performance management system?
There was no significant difference between the answers of line managers and those of administrative staff about the helpfulness of HR advisers during the implementation process.Thirty per cent of both groups indicated that the HR resource was very helpful.The line managers also rated other resources in terms of their helpfulness.They indicated that they found their senior managers (33%), colleagues (28%), training manuals (23%) and training workshops (20%) very helpful.

Question 3: Short-term training outcomes
With regard to the short-term training outcomes, the evaluators focused mainly on how well the two groups of trainees understood various aspects of the system and how skilled they thought they were in some aspects of the measurement part of performance management.

Understanding:
a. Did the line managers and administrative staff understand the main components of the system after the training?
Table 2 shows that the line managers understood these aspects better than the administrative staff did.

Skills:
b. Did line managers and administrative staff have the necessary participative skills to implement parts of the system after the training?
Line managers and administrative staff should cooperate when setting performance objectives for administrative staff.The evaluators compared the two groups with regard to their perceived skill for this task.Line managers rated themselves as more skilled in setting objectives than administrative staff did (line managers M = 4.09, administrative staff M = 3.87; t = 2.40, p < .017).When the evaluators compared the managers' ratings with the selfratings of administrative staff on setting skills objectively for administrators, there was a significant difference (line managers' ratings of administrative staff M = 3.33, administrative staff members' self-ratings M = 3.87; t = -5.137,p < .000).

c. Did line managers have the necessary skills to implement specific aspects of the system?
The evaluators also assessed the line managers' perceptions of their own performance management skills for implementing specific aspects of the system.Table 3 shows that line managers regarded themselves as generally skilled in these tasks.Table 4 reports the line managers' perceived longer-term outcomes of the performance management system.

Plans
Table 5 reflects the perceptions of the administrative staff after two implementation cycles.
The evaluators added an open-ended question to the questionnaire.It required all respondents to comment on   the potential advantages and disadvantages of the PFP programme.
Two major themes emerged from the comments.These were the lack of meaningful rewards (mentioned by 45% of line managers and 15.6% of administrative staff) and nonprocedural evaluation by line managers (mentioned by 36.4% of line managers and 35% of administrative staff).
b. Did the PFP system at the university work for some administrative staff and not for others?c.Are there interventions or variables that can predict for whom the system would work?
In order to answer these two questions, the evaluators conducted three analyses.Firstly, they subjected all items that relate to the administrative staff's perceptions of the PFP system to a reliability analysis.Secondly, they reduced these items to three factors using factor analysis.Thirdly, they conducted a linear regression and a step-wise discriminant analysis to ascertain for which administrative staff the PFP system worked.They discuss each of these steps below.
The evaluators used Cronbach's alpha to identify the internal consistency of items 6 to 22 in the administrative employees' questionnaire.Cronbach's alpha = .91,which suggested a high internal consistency reliability of the 17 item scale (Pallant, 2007).They also used item total statistics to show the correlation of each item with the overall interval scale.There were no items with an item-total correlation below .3.
Therefore, items 6 to 22 all measured the same construct (employees' perceptions of the PFP system), and the evaluators did not need to delete any of the items (Pallant, 2007).
In order to reduce the data of the interval scale items into separate independent variables, the evaluators conducted an exploratory factor analysis (EFA) on items 6 to 22 (they excluded item 12, the dependent variable) of the questionnaire for the administrative staff.The Kaiser-Meyer-Olkin value was .86.This exceeds the recommended value of .6. Bartlett's test of sphericity was statistically significant as its p value was 0.00 (< 0.05) (Pallant, 2007).Therefore, the evaluators confirmed that it is appropriate to conduct an EFA with the interval items from the administrators' questionnaire.
A principal component analysis revealed the presence of three factors with Eigenvalues that exceed 1.This explained 62% of the cumulative variance.In addition to this, the pattern matrix confirmed that all the items had loaded onto the three factors.However, items 10, 11 and 15 loaded onto more than one factor.The difference between these crossloading items was smaller than .25 (Pallant, 2007) and the evaluators removed them.These items show that there is a robust support system for administrative employees on performance management.The factor contributed 42% variance of the scale.
The evaluators labelled the second factor 'employees' understanding' (of the PFP system).It consisted of items 6 ('please indicate the extent to which you are skilled in setting performance objectives'), 7 ('I have a clear understanding of what the primary goals of the PFP system are'), 8 ('I understand how the PFP system works') and 9 ('I understand how the performance assessment links to the PFP system').This factor contributed 12% of the total variance of the scale.
The evaluators called the last factor 'employees' motivation to work harder'.It included items 19 ('I consciously work harder because of the PFP system'), 20 ('the financial rewards I receive reflect my individual contributions'), 21 ('the PFP system contributes significantly to my receiving the credit and recognition I deserve') and 22 ('the financial incentive of the PFP system increases my determination to achieve my objectives').This factor contributed 10% of the total variance of the scale.The cumulative variance of these three factors was 64%.
To measure the relationship between the three independent variables the evaluators identified in the EFA and the dependent variable (item 12: 'I participate with my manager in setting my performance objectives' -that they labelled 'cooperative objective setting'), they used a linear regression.
The evaluators first used collinearity diagnostics to assess whether there was a possibility of multi-collinearity.The variance inflation factor (VIF) and tolerance posed no problem of multi-collinearity because the VIF for all the scores was smaller than 10 and the tolerance scores were larger than .10(Pallant, 2007).Table 7 shows this.
The evaluators entered all three of the independent variables (guidance, support and training; employees' understanding; and employees' motivation to work harder) into the regression model.Upon evaluating the regression model, they found that the model explained 49.3% of the variance of cooperative objective setting.They also found that this model reached statistical significance because the analysis of variance (ANOVA) p value was 0.00 (p < .005).See Pallant (2007).
Guidance, support and training made the strongest unique contribution to explaining cooperative objective setting when the evaluators controlled for all the other variables in the model.This variable had the highest beta value.It also made a statistically significant contribution to predicting cooperative objective setting.When the evaluators scanned the part correlation coefficients, they found that guidance, support and training explained 32.8% of the variance in the dependent variable.Employees' understanding and employees' motivation had p values larger than 0.05.Therefore, they did not make a statistically significant contribution to explaining cooperative objective setting (see Table 7).
The evaluators conducted a step-wise discriminant analysis to investigate whether the three independent variables (guidance, support and training; employees' understanding; and employees' motivation to work harder) could be predictors of the dependent variable: the existing development plan (item 3: 'I had a performance development plan for 2006').Of the 262 administrative employees, 213 had a performance development plan.
Table 8 shows the uni-variate ANOVAs of all three independent variables.There was a statically significant difference between all the grouping variable means because the p values for all the variables were smaller than 0.05.
The evaluators identified only a single step for this analysis.The only variable they entered was guidance, support and training.This variable had the lowest Wilks' Lambda value of 0.93 and the highest F value of 19.43.The tolerance value of this variable was 1.00.This suggests that this variable contributed strongly to the analysis (Pallant, 2007).The evaluators found that step 1 accounted for 100% of the variance of whether employees had an existing performance development plan in 2006.In addition, this discriminant function was statistically significant because the p value was 0.00 (p < 0.05).See Pallant (2007).The evaluators also found a canonical correlation of 0.26 and an Eigenvalue of 0.75.
Table 9 shows which criterion groupings have the most accurate classifications when the evaluators used the discriminant functions they developed in the analysis.The analysis was more likely to classify accurately predicted group membership of employees who did not have an existing development plan compared to the employees who did have one.

Ethical considerations
The Ethics in Research Committee of the Faculty of Commerce, University of Cape Town, approved the evaluation.

Potential benefits and hazards
There were no benefits or hazards for participants who took part in the evaluation.

Recruitment procedures
The evaluators distributed the questionnaire electronically to all participants.

Informed consent
On the first page of the questionnaire the purpose, confidentiality of the data and the time it would take to complete were described in detail.

Data protection
The webmaster stripped the data of all identifiers and downloaded the data onto a spreadsheet.The first two evaluators were the only people who had access to the data.

Trustworthiness Reliability
The reliability of the measuring instrument is described on page 6.

Validity
The evaluation focused on a specific programme and did not seek to generalise the findings to other, similar programme.

Discussion
The main aim of the evaluation was to assess whether a pay-for-performance (PFP) system, which a South African university introduced for administrative employees, improved their performance.A secondary aim was to examine whether the university implemented the system as it intended to.This evaluation will contribute to the social science literature on whether PFP systems work, and if they do not, why.
In general, the results showed that there was a discrepancy between the perceptions of the line managers and those of the administrative staff about the PFP system.The line managers tended to be more positive about the system outcomes than the administrative staff were.The evaluators structured the discussion according to the evaluation questions they set earlier.

Question 1: Training coverage
At first glance, the number of staff who received training seems adequate.However, one has to question whether it is acceptable that approximately 40% of administrative staff, who had to complete the complex task of developing measurable objectives, had not had any training.The evaluators recommend that the university presents more training workshops in order to increase the skill levels of administrative staff.

Question 2: Resources
It seems that there were adequate resources for implementing the system.For line managers, senior managers emerged as the resource that was most often helpful.These managers could be involved in the training the evaluators suggest.

Question 3: Short-term outcomes
Understanding: a. Did the line managers and the administrative staff understand the main components of the system?
From the results, it was clear that line managers understood the system's goals, operation, measurement and the link between pay and performance better than the administrative staff did.This is quite problematic, as the administrative staff members were the recipients of the system and experienced its consequences.When the university presents follow-up training courses, it should clarify these aspects of the system to the administrative staff.

Skills:
b. Did the line managers and administrative staff have the necessary participative skills to implement parts of the system after training?
Again, line managers rated themselves as more skilled than the administrative staff did in setting performance objectives.Line managers also questioned whether the administrative staff had adequate skills for setting objectives.
In the evaluators' experience, it is fatal simply to prescribe Specific, Measurable, Attainable, Relevant, Timely (SMART) objectives in a manual (the Performance Development Resource Guide, University of Cape Town, 2006) without practising how to set these objectives.The evaluators recommend that any additional training should include the clear guidelines of Kettner, Moroney and Martin's (1999) for setting measurable objectives.

c. Did line managers have the necessary skills to implement specific aspects of the system?
Line managers' self-ratings of their skills showed that they thought they had mastered how to develop a performance management plan and how to give constructive feedback.Although more than 70% indicated that they were skilled or highly skilled in rating performance and in differentiating between different levels of performance, additional training in these two aspects might provide better coverage for all skills.

Plans: d. Did administrative staff apply the training to create development plans?
At face value, the answer to this question is that they did.In 2005, 74% of administrative staff had plans and this percentage increased to 81% in 2006.However, HR performed a quality check of these plans in 2005 and found that in only eight of 21 departments were the plans satisfactory or good.There was no intervention to improve the quality of the development plans.
The evaluators noted earlier that, in the training manuals, the performance objectives of administrative staff did not link to the strategic objectives of the university or the job descriptions of the staff members.One can easily group administrative jobs into job categories (like departmental secretary or finance officer) where there is considerable overlap in performance objectives.One can strengthen the perception that measurement is consistent by clarifying the link between objectives and job descriptions and by emphasising the similarities between objectives in a job category.
Question 4: Longer-term outcomes a.After two implementation cycles, how did line managers and administrative staff rate the effectiveness of the system?
Too many line managers agreed or strongly agreed that performance management is an administrative task with little pay-off (34.1%) and that it was a lengthy, time-consuming task (54.3%).The clear link between performance objectives and job description or category could change this perception significantly.This link would reduce the number of irrelevant objectives that administrative staff members often create because there are clear parameters for setting objectives.In any event, the evaluators suggest that administrative staff do not set more than between seven and nine objectives.These objectives should focus on the key performance areas of their jobs.Essential objectives rather than irrelevant detail should drive performance measurement.
Of even more concern is the perception of inconsistency in implementing the system, endorsed by 83% of line managers.This poses a threat to the credibility of the system.Again, job categories with comparable objectives might reduce the notion that each person's objectives and their measurement are unique or arbitrary.
On the positive side, the results showed that line managers reported high commitment to staff development and the required skills to implement pay-for-performance in their departments.However, when the evaluators compared these opinions, about the availability of training for performance improvement, to those of administrative staff members (see Table 5) a different picture emerges.Only 34.7% of administrative staff agreed or strongly agreed with the statement that there are adequate training opportunities for improving performance.
Administrative staff members indicated that, in their opinion, the PFP system had little value.
Research on the standards required for effective PFP systems showed that the recipients should see the system as fair (Barnes-Farrell & Lynch, 2003), perceive a link between performance and pay (Holbeche, 2004;Ulrich & Brockbank, 2005) and perceive differential rewards for poor and excellent performance (Ulrich & Brockbank, 2005).
The results of the evaluation showed that if one took 50% of administrative staff as an arbitrary cut-off point, one could conclude that more than half of the administrative staff: • did not see the system as fair • did not work harder because of the system • felt that the rewards did not reflect their performance • felt that they did not get credit and recognition for their performance • felt that financial incentives did not increase their motivation to achieve their objectives.
Where staff had more control over the outcomes of the system, their perceptions were more positive.More than half of them strongly agreed or agreed that their performance objectives were clear enough to show what they were supposed to achieve (58.7%) and that they had agreed on realistic deadlines with their line managers to achieve their objectives (51.6%).
These results showed that the recipients of the system were not well disposed towards it.This could jeopardise the whole PFP endeavour.The evaluators recommend an intervention to change these perceptions.They also recommend that the university creates a clear link between objectives and job descriptions or categories in order to introduce the notion of consistency and reinforce perceptions of fairness.Thereafter, line managers should receive follow-up training in performance-rating skills.Clear performance categories and criteria (standards) are available in current manuals and additional training will strengthen line managers' confidence in their performance judgements.
Addressing the link between pay and performance seems to be relatively easy.To date, the university has not implemented the PFP as it intended to.In 2005 (a year after the evaluation), the university committed itself to a PFP system but set the lowest end of the range at a 5% increase.In other words, the poorest performers still received a 5% increase.The upper limits of the range depended on the job class, with the highest job class in the bargaining unit getting the highest possible increase, namely 13.4%.In 2006, the university introduced a major pay correction for administrative staff based on salaries in the national job market.The PFP portion of the total increase all but disappeared with this adjustment.In 2007, the first time the university implemented the PFP performance system as it intended to, it fixed the ranges at between 6.2% and 7% (please note that the evaluation did not cover this period).
This range is problematic, as there is too little differentiation between poor and excellent performers.Therefore, it undermined one of the main goals of the system.The university managers need to revisit the uneasy alliance between collective bargaining and PFP, taking into account the tendency for unions to compress pay structures in these systems (White & Druker, 2000).
According to Holbeche (2004), the pay element of PFP systems shows what and who organisations value.In the open-ended questions, too many line managers and administrative staff questioned whether the system delivered meaningful rewards and whether there were fair performance decisions.These results do not hold great promise for one to regard the system as effective: recipients and appraisers alike questioned whether pay showed which employees and behaviours the university valued.They also questioned the fairness of the system.These doubts go to the heart of any performance management system.The university needs to address them to improve the system.b.Did the PFP system at the university work for some administrative employees and not for others?c.Are there interventions or variables that can predict for whom the system would work?
Figure 1 presented the outcome variables of an effective PFP system.These are cooperative objective setting and existing development plans.Using the results they obtained from the evaluation, the evaluators completed the diagram.
Figure 2 shows that the system worked for administrative employees who had guidance, support and training.This variable consisted of four items.The evaluators discuss each one below.

I receive guidance from my line manager in setting my performance objectives
Guidance from line managers in setting objectives is important and should aim at defining objectives, in task and behavioural terms, developing observable objectives and setting clear deadlines for achieving these objectives (Eichinger et al., 2004;Harper, 2003).To ensure that line managers are fully competent when they guide their employees to set objectives, they should receive training in how to set objectives and how to guide their employees effectively.In addition to this, the evaluators recommend that this training focuses on guiding employees to set objectives that satisfy the current and future needs of the university (Eichinger et al., 2004).Therefore, the evaluators propose that HR educates line managers in how to turn the university's strategic goals into practical and measurable objectives for administrative employees (Lawler, 1990).

I receive guidance from my human resources adviser on how to set the performance objectives that are relevant to my job
There is little research on the role of HR in supporting or guiding employees in PFP processes.The lack of this research is because of the notion that some HR professionals believe that their role is strategic and not participative when it comes to implementing PFP systems (Brown & Purcell, 2007).Furthermore, there is often an understanding that HR is responsible for designing programmes, whilst line managers are accountable for implementing programmes.
Because of this split in responsibilities, line managers often find themselves in situations where they have insufficient time or financial resources to implement the PFP system effectively (Brown & Purcell, 2007).Therefore, support and guidance from HR is vital for implementing reward systems (Brown & Purcell, 2007).This is consistent with the results of this evaluation because the evaluators found that the PFP system worked for administrative employees who received guidance from their HR advisers on how to set objectives that are relevant to their jobs.
According to Streib and Nigro (1993), this type of process training guides employees to set clear and realistic performance objectives.It also defines the roles and responsibilities of managers and employees when they set performance objectives (Purcell & Hutchinson, 2007).Process training can also provide clear guidelines on how employees and their managers should be contributing to, and interacting in, the process of setting objectives.This will improve the implementation of the system (Streib & Nigro, 1993).

I receive regular informal advice from my line manager about how to improve my job performance
Receiving regular informal advice from line managers on how to improve performance was another of the items that contributed to the successful functioning of the PFP system at the university.According to Armstrong (2003), regular performance feedback is one of the most important aspects of effective PFP systems.This feedback reinforces perceptions about the fairness of the system (Armstrong, 2003).
The evaluators recommend that this informal feedback process extends to training for line managers.It could include how to distinguish between good and bad performance, how to give appropriate feedback according to performance and how frequently to give informal feedback (Armstrong, 2003).

Human resource's additional role in a pay-forperformance system
It is clear that the four items the evaluators have discussed are associated with line managers' support and guidance.However, enabling and eliciting this support from managers is not simple.Line managers often point out that it is difficult to provide the necessary support because of conflicting priorities, heavy workloads as well as lack of training and support (Brown & Purcell, 2007).
Therefore, in order to ensure that managers support their employees properly, the evaluators suggest that managers themselves receive support from their HR department.HR's support should involve training and regular communication between the two parties, shared accountability for implementing PFP processes as well as giving feedback to line managers about any on-going monitoring processes (Brown & Purcell, 2007).

Conclusions
In conclusion, the administrative staff whose working lives the PFP system affected had a low opinion of its effectiveness.With regard to the goals of the system, which are to improve the performance of employees and to reward excellence, one must conclude that the university has not achieved them.Administrative staff members stated unambiguously that they did not work harder because of the system.Line managers and administrative staff members felt that the reward part of the system did not distinguish between poor and excellent performance.
However, there are positive results that the university could use to improve the system.Line managers seemed to be confident about their understanding and skills to implement the system.A clear link between job description, or category, and performance objectives as well as reducing the number of performance objectives would address line managers' current perceptions of the amount of time the exercise of setting objectives takes.
After training, line managers could also clarify the range of performance standards and the link between these standards and payment to administrative staff.
Three factors made implementing the PFP system difficult at this university.
Firstly, after decades of incremental increases, administrative staff found themselves in the unfamiliar territory of performance scrutiny.Secondly, a major salary adjustment in 2006 masked the effects of the PFP system.Thirdly, the university introduced the system in a unionised environment where collective bargaining sets pay parameters.
It will take more time to stabilise the system and attend to the unfavourable perceptions of the recipients.With a few more interventions to improve the system, the university could achieve its goals of improving the performance of employees and of rewarding excellent work.
The literature and the findings of this evaluation confirm that guidance, support and training are vital for the PFP system to work at the university.This factor relates directly to programme implementation and its importance.It seems that the PFP system at the university was ineffective because of poor implementation rather than poor design.
The evaluators mentioned earlier that line managers and HR often disagree and deny their responsibilities when implementing the PFP system.As a result, implementing the system usually suffers.Implementers often ignore the human dynamics of implementation.Therefore, most of the recommendations this evaluation makes relate to issues of interaction and communication between administrative employees, line managers and HR.These recommendations are practical and will clarify roles as well as improve implementation.Better implementation seems to be the key to an effective PFP system at the university.

Suggestions for future research
The university used the recommendations in this evaluation to improve the implementation of its PFP system in 2009.The evaluators suggest that the university conducts a systematic process and outcome evaluation to assess whether the system is now achieving its intended outcomes.contractors.K.E.(University of Cape Town) completed the second part of the evaluation as a requirement for her master's degree in programme evaluation.J.L.-P.(University of Cape Town) supervised the evaluation.

FIGURE 1 :
FIGURE 1: Diagram of an effective pay-for-performance programme.
: d.Did line managers have the necessary skills to implement specific aspects of the system?e.How many administrative staff had these plans?Administrative staff did create development plans after the training.In 2005, 74% of the administrative staff had plans.This increased to 81% in 2006.Question 4: Longer-term outcomes a.After two implementation cycles, how did line managers and administrative staff rate the outcomes of the system?

FIGURE 2 :
FIGURE 2: Programme theory for an effective pay-for-performance system for administrative employees.

TABLE 1 :
Questionnaire for collecting data from administrative employees.
ItemStatements or questions 1 Please indicate in which department or faculty you are located 2 I had a performance development plan for 2005 3 I had a performance development plan for 2006 4 I received training on how the performance development system works 5 I was aware of the training workshops the human resources department offered but did not attend because 6 Please indicate the extent to which you are skilled in setting performance objectives 7 I have a clear understanding of what the primary goals of the PFP system are 8 I understand how the PFP system works 9 I understand how the performance assessment links to the PFP system 10 I understand how my performance is measured 11 I believe the way in which my performance is measured is fair 12 I participate with my line manager in setting my performance objectives 13 I receive guidance from my line manager in setting my performance objectives 14 I receive guidance from my HR adviser on how to set the performance objectives that are relevant to my job 15 The objectives in my performance development plan are clear enough for me to understand what I am supposed to achieve 16 I agree on clear deadlines with my line manager for achieving my objectives 17 I receive regular informal advice from my line manager about how

TABLE 2 :
Understanding the aspects of the pay-for-performance system.

TABLE 3 :
Self-rating of performance management skills (line managers).
Note that totals are in percentages.

TABLE 4 :
Longer-term perceived outcomes of the performance management system (line managers).
Note that totals are in percentages.

TABLE 5 :
Longer-term perceived outcomes of the performance management system (administrative staff).
Note that totals are in percentages.

Table 6
manager in setting my performance objectives'), 14 ('I receive guidance from my HR adviser on how to set the performance objectives relevant to my job'), 16 ('I agree on clear deadlines with my line manager for achieving my objectives'), 17 ('I receive regular informal advice from my line manager about how to improve my job performance') and 18 ('There are adequate training opportunities in my department to enable me to achieve my objectives').
presents the three distinctive and coherent factors.The evaluators labelled the first factor 'guidance, support and training'.It consisted of items 13 ('I receive guidance from my line

TABLE 6 :
Pattern matrix and communalities of the three-factor solution of administrative employee perception scale items.
Note that the major loadings for each item are in bold.

TABLE 7 :
Linear regression analysis for variables that predict cooperative objective setting.

TABLE 8 :
Discriminant analysis for variables that predict the existing development plan.

TABLE 9 :
Classification of results to determine the accuracy of predicted group membership.

There are adequate training opportunities in my department to enable me to achieve my objectives
Appropriate training, access to this training and opportunities to practise, as well as to apply, the new skills or knowledge in the workplace could change the perception that there are insufficient opportunities to develop.